microsoft/distributed_word_embedding

Distributed word embedding

This repo was deprecated, please check the new version.

Distributed Word Embedding

The Distributed Word Embedding tool is a parallelization of the Word2Vec algorithm on top of our DMTK parameter server. It provides an efficient "scaling to industry size" solution for word embedding.

For more details, please view our website http://www.dmtk.io.

Build

Linux (Tested on Ubuntu 12.04)

Run $./build.sh to build the project.

Windows

For windows users, please refer to README in windows folder.

Microsoft Open Source Code of Conduct

This project has adopted the Microsoft Open Source Code of Conduct. For more information see the Code of Conduct FAQ or contact [email protected] with any additional questions or comments.

distributed_word_embedding's Issues

Fetch enwiki train and vocab files

The "run.bat" script has the options text, read_vocab and train_file.

• Which is the input format expected for these files (e.g. the a dump https://dumps.wikimedia.org/enwiki/latest/enwiki-latest-pages-articles.xml.bz2 is it ok?
• Is text a specific file in the train_file folder?

How to generate/retrieve a enwiki vocabulary file?

Is it ok to process the dump and generating a vocabulary like the one made by the script mkvocab.pl

THE     84503449
AND     33700692
WAS     12911542
FOR     10342919
THAT    8318795

What's the "clock" actually means in distributed_word_embedding?

In multiverso document, it says:
BSP (max_delay=0): All worker processes are barriered forcibly at the end of a clock.

What's the "clock" actually means in distributed_word_embedding? Some mini-batches or a period of time? If it means mini-batch, what's the size of a mini-batch?

no limitation for "%s" while calling fscanf()

distributed_word_embedding/src/huffman_encoder.cpp

Line 65 in 3fc473d

fscanf(fid, "%s", sz_label, kMaxWordSize);