lixiangnlp / pysrilm Goto Github PK
View Code? Open in Web Editor NEWThis project forked from njsmith/pysrilm
An extremely simple Python wrapper for the SRI Language Modeling toolkit
License: BSD 2-Clause "Simplified" License
This project forked from njsmith/pysrilm
An extremely simple Python wrapper for the SRI Language Modeling toolkit
License: BSD 2-Clause "Simplified" License
This is an extremely simple Python wrapper for SRILM: http://www.speech.sri.com/projects/srilm/ Basically it lets you load a SRILM-format ngram model into memory, and then query it directly from Python. Right now this is extremely bare-bones, just enough to do what I needed, no fancy infrastructure at all. Feel free to send patches though if you extend it! Requirements: - SRILM - Cython Installation: - Edit setup.py so that it can find your SRILM build files. - To install in your Python environment, use: python setup.py install To just build the interface module: python setup.py build_ext --inplace which will produce srilm.so, which can be placed on your PYTHONPATH and accessed as 'import srilm'. Usage: from srilm import LM # Use lower=True if you passed -lower to ngram-count. lower=False is # default. lm = LM("path/to/model/from/ngram-count", lower=True) # Compute log10(P(brown | the quick)) # # Note that the context tokens are in *reverse* order, as per SRILM's # internal convention. I can't decide if this is a bug or not. If you # have a model of order N, and you pass more than (N-1) words, then # the first (N-1) entries in the list will be used. (I.e., the most # recent (N-1) context words.) lm.logprob_strings("brown", ["quick", "the"]) # We can also compute the probability of a sentence (this is just # a convenience wrapper): # log10 P(The | <s>) # + log10 P(quick | <s> The) # + log10 P(brown | <s> The quick) lm.total_logprob_strings(["The", "quick", "brown"]) # Internally, SRILM interns tokens to integers. You can convert back # and forth using the .vocab attribute on an LM object: idx = lm.vocab.intern("brown") print idx assert lm.vocab.extern(idx) == "brown" # .extern() returns None if an idx is unused for some reason. # There's a variant of .logprob_strings that takes these directly, # which is probably not really any faster, but sometimes is more # convenient if you're working with interned tokens anyway: lm.logprob(lm.vocab.intern("brown"), [lm.vocab.intern("quick"), lm.vocab.intern("the"), ]) # There are detect "magic" tokens that don't actually represent anything # in the input stream, like <s> and <unk>. You can detect them like assert lm.vocab.is_non_word(lm.intern("<s>")) assert not lm.vocab.is_non_word(lm.intern("brown")) # Sometimes it's handy to have two models use the same indices for the # same words, i.e., share a vocab table. This can be done like: lm2 = LM("other/model", vocab=lm.vocab) # This gives the index of the highest vocabulary word, useful for # iterating over the whole vocabulary. Unlike the Python convention # for describing ranges, this is the *inclusive* maximum: lm.vocab.max_interned() # And finally, let's put it together with an example of how to find # the max-probability continuation: # argmax_w P(w | the quick) # by querying each word in the vocabulary in turn: context = [lm.vocab.intern(w) for w in ["quick", "the"]] best_idx = None best_logprob = -1e100 # Don't forget the +1, because Python and SRILM disagree about how # ranges should work... for i in xrange(lm.vocab.max_interned() + 1): logprob = lm.logprob(i, context) if logprob > best_logprob: best_idx = i best_logprob = logprob best_word = lm.vocab.extern(best_idx) print "Max prob continuation: %s (%s)" % (best_word, best_logprob)
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.