jostheim/loom

A streaming cross-cat inference engine

Loom

Loom is a streaming inference and query engine for the Cross-Categorization model mansinghka2009cross, shafto2011probabilistic.

Data Types

Loom learns models of sparse heterogeneous tabular data, with hundreds of features and millions of rows. Loom currently supports the following feature types and models:

• boolean fields as Beta-Bernoulli
• categorical fields with up to 256 values as Dirichlet-Discrete
• unbounded categorical fields as Dirichlet-Process-Discrete
• count fields as Gamma-Poisson
• real fields as Normal-Inverse-Chi-Squared-Normal
• sparse real fields as mixture of degenerate and dense real
• text and keyword fields as booleans for word absence/presence
• date fields as a combination of absolute, relative, and cyclic parts
• optional fields as a boolean plus one of the above feature models

See input format docs for details.

Data Scale

Loom targets tabular datasets of sizes 100-1000 columns 10^3-10^9 rows. To handle large datasets, loom implements subsample annealing obermeyer2014scaling with an accelerating annealing schedule and adaptively turns off ineffective inference strategies. Loom's annealing schedule is tuned to learn 10^8 cell datasets in under an hour and 10^10 cell datasets in under a day (depending on feature type and sparsity).

   Full Inference:     Partial Inference:  Greedy Inference:
   structure
   hyperparameters     hyperparameters
   mixtures            mixtures            mixtures
 |-------------------> ------------------> ------------------>
 1   many-passes   ~10^4   accelerate   10^9   single-pass  10^4
row                 rows                rows               row/sec

Documentation

• Installing
• Quick Start
• Using Loom
• Adapting Loom
• Examples

Authors

• Fritz Obermeyer https://github.com/fritzo
• Jonathan Glidden https://twitter.com/jhglidden

Loom is a streaming rewrite of the TARDIS engine developed by Eric Jonas https://twitter.com/stochastician at Prior Knowledge, Inc.

Loom relies heavily on the distributions library.

License

Copyright (c) 2014 Salesforce.com, Inc. All rights reserved. Copyright (c) 2015, Google, Inc.

Licensed under the Revised BSD License. See LICENSE.txt for details.

The PreQL query interface is covered by US patents pending:

• Application No. 14/014,204
• Application No. 14/014,221
• Application No. 14/014,225
• Application No. 14/014,236
• Application No. 14/014,241
• Application No. 14/014,250
• Application No. 14/014,258

Dependencies

• metis - Apache 2.0
• pymetis - MIT
• numpy - BSD
• scipy - BSD
• pandas - BSD
• matplotlib - PSF
• scikit-learn - BSD
• simplejson - MIT
• google protobuf - Apache 2.0
• google perftools - New BSD
• parsable - MIT
• distributions - Revised BSD
• goftests - Revised BSD
• nose - LGPL
• mock - New BSD