liulhdarks/darks-learning

Darks learning is the machine learning algorithm library. It contains Word2vec,DBN, RBM, MLP, LSA, PLSA, SDA, Maxent, regression, etc.

Darks Learning

Darks learning is the machine learning algorithm library. It contains Word2vec,DBN, RBM, MLP, LSA, PLSA, SDA, Maxent, regression, etc.

Load Corpus

The corpus type are divided into Corpus, Documents and ModelSet.

Corpus

It can be used for Word2vec,LSA,pLSA,etc. which are used to documents and words related to non classification algorithm.

CorpusLoader loader = new CorpusLoader();
loader.addFilter(...);
loader.addStopwords(...);
Corpus corpus = loader.loadFromFile(new File(...));

Documents

It can be used for Maxent, bayes, etc. which are used to documents and words related to classification algorithm.

File input = new File(...);
File labels = new File(...);
Documents docs = Documents.loadFromFile(input, labels, "UTF-8");
File corpusFile = new File(...); 
//The corpusFile both contains labels and features, which's labels and features of each line must be separated by Tab(\t).
Documents docs = Documents.loadFromFile(corpusFile, "UTF-8");

ModelSet

It can be used for regression, MLP, DBN, RBM, SDA, SOFTMAX, etc. which are used to classification based on double matrix.

ModelSet modelSet = ModelLoader.loadFromFile(...);
ModelSet modelSet = ModelLoader.loadFromStream(...);

Naive Bayes

Navie bayes contains the BINAMIAL and BERNOULLI modes.

• BINAMIAL is fine-grained to words by default.
• BERNOULLI is coarse-grained to documents.

Example

How to train

Documents docs = Documents.loadFromFile(corpusFile, "UTF-8");
NaiveBayes bayes = new NaiveBayes();
bayes.config.setLogLikelihood(true)
			.setModelType(NaiveBayes.BINAMIAL);
bayes.train(docs);

How to classify

String sentence = ...; //Sentence string must be separated by spaces.
String classify = bayes.predict(sentence);
String classify = bayes.predict(new String[]{...});

Maxent

Maxent only supports GIS algorithm.

Example

How to train

Documents docs = Documents.loadFromFile(corpusFile, "UTF-8");
Maxent maxent = new GISMaxent();
MaxentModel model = maxent.train(docs, 1000); //Train model with 1000 iterations.
model.saveModel(...);

How to load

GISModel model = GISModel.readModel(...);
GISMaxent maxent = new GISMaxent(model);
...

How to classify

String[] terms = ...;
int labelIndex = maxent.predict(terms);
String classify = maxent.getLabel(labelIndex);

LSA (Latent Semantic Analysis)

Example

How to train

CorpusLoader loader = new CorpusLoader(Corpus.TYPE_TF_IDF);
File file = ...;
Corpus corpus = loader.loadFromFile(file, "UTF-8");
LatentSemanticAnalysis lsa = new LatentSemanticAnalysis();
lsa.train(corpus);
lsa.saveModel(...);
lsa.loadModel(...);

How to predict

String result = lsa.predict(new String[]{...});
int index = lsa.predictIndex(new String[]{...});

darks-learning's Issues

mvn repo down

The maven repositories no longer appear to work.

http://mvnrepo.alibaba-inc.com
ERR_CONNECTION_TIMED_OUT

Missing test data files

Please provide the 'corpus' directory so users can run the tests.

Tests in error:
  testDynamicalCluster(darks.learning.test.cluster.TextClusterTest): Cannot find corpus corpus\train_data.txt
  testPLSA(darks.learning.test.lsa.PlsaTest): Cannot find corpus corpus\corpus_ali_small.txt
  testMatrixSubAdd(darks.learning.test.math.MathTest): For input string: "9223372036854775807"
  testMaxentGIS(darks.learning.test.maxent.MaxentTest)
  testTrain(darks.learning.test.word2vec.Word2VecTest): Cannot find corpus corpus\text8_limit.txt