to-shimo/chinese-word2vec

word2vec/glove/swivel binary file on chinese corpus

chinese-word2vec

word2vec/glove/swivel binary file on chinese corpus

word2vec: https://code.google.com/p/word2vec/

glove: http://nlp.stanford.edu/projects/glove/

swivel: https://github.com/tensorflow/models/tree/master/swivel

        http://arxiv.org/abs/1602.02215

训练语料：百科网页+多个分类新闻语料(全半角转换后)

binary file: （Word2vec的二进制读取可以参考其代码）

    word2vec CBOW: utf8  2.18G

        http://pan.baidu.com/s/1qX334vE

    word2vec SKIP: utf8  2.17G

        http://pan.baidu.com/s/1bogTzfp

    word2vec CBOW: gb18030（精简语料）  2.31G

        http://pan.baidu.com/s/1jHumqjW

    word2vec SKIP: gb18030（精简语料）  1.47G

        http://pan.baidu.com/s/1ntVJBYD

    swivel: utf8  

        待上传

    glove: utf8   9.19G

        http://pan.baidu.com/s/1i3XowWP

其他：

    使用1.25亿英国twitter训练的词向量：

        参考地址：https://figshare.com/articles/UK_Twitter_word_embeddings/4052331

        下载：https://ndownloader.figshare.com/articles/4052331/versions/1

chinese-word2vec's Issues

请问glove的bin文件如何打开？

用gensim的load_word2vec_format 会提示UnicodeDecodeError

glove-utf8无法使用

请问glove-utf8的语料库能提供所训练的词吗，词向量中貌似只有权重。。。

能详细介绍下训练过程吗？

这个训练之前有对文本进行分词吗？用的是什么分词工具？
关于训练参数，训练数据之类的详细说明一下？

请问 Glove 的格式是什么？

请问如何读取 glove 的二进制格式？

请问能上传处理好（分词后等）的训练预料吗？

unpickling stack underflow

Hi, recently I'm using your model for an NLP project. But it occurs some bugs.

In [2]: model = models.Word2Vec.load('cn.cbow.bin')
---------------------------------------------------------------------------
UnpicklingError                           Traceback (most recent call last)
<ipython-input-2-01c34d25b411> in <module>()
----> 1 model = models.Word2Vec.load('cn.cbow.bin')

/usr/local/lib/python2.7/dist-packages/gensim/models/word2vec.pyc in load(cls, *args, **kwargs)
   1760     @classmethod
   1761     def load(cls, *args, **kwargs):
-> 1762         model = super(Word2Vec, cls).load(*args, **kwargs)
   1763         # update older models
   1764         if hasattr(model, 'table'):

/usr/local/lib/python2.7/dist-packages/gensim/utils.pyc in load(cls, fname, mmap)
    246         compress, subname = SaveLoad._adapt_by_suffix(fname)
    247
--> 248         obj = unpickle(fname)
    249         obj._load_specials(fname, mmap, compress, subname)
    250         logger.info("loaded %s", fname)

/usr/local/lib/python2.7/dist-packages/gensim/utils.pyc in unpickle(fname)
    910     with smart_open(fname) as f:
    911         # Because of loading from S3 load can't be used (missing readline in smart_open)
--> 912         return _pickle.loads(f.read())
    913
    914

UnpicklingError: unpickling stack underflow

Platform:
Ubuntu 15.10/ Python2.7.9

good job! what's the dimension and total words of glove model? I want to use gensim package: Word2Vec.load_word2vec_format() function to load your glove model, so i have to manually add the total words and dimension of word-embedding.

读取cn.skipgram.bin.tar.gz错误

model = gensim.models.KeyedVectors.load_word2vec_format(fdir + 'cn.skipgram.bin.tar.gz', binary=True)

ValueError: invalid literal for int() with base 10: 'cn.skipgram.bin\x00\x00。。。。。。

你好，读取cn.skipgram.bin.tar.gz文件出现这个错误，查了很久都不知道原因

very useful model, How to fine tune?

thanks so much for the model, it's been very useful.
I'd like to apply your model in my application, but I don't know how to fine tune it, Can you help me out?
Thks a lot

语料链接打不开了

RT

词向量维度问题

你好，请问一下这个项目里面的词向量都是多少维的呢？