marsan-ma-zz/chat_corpus

chat corpus collection from various open sources

chat_corpus's Issues

How to create a corpuss like cornell movie format?

how can I create a corpus like cornell movie`s corpus?
I have a bunch of questions&answers in MySQL format and I want use them n this code

保险行业问答语料

Take a look at https://github.com/Samurais/insuranceqa-corpus-zh
It is a corpus for train FAQ models.

lots of duplicate lines in movie subtitles

I did a simple dedupe

cat movie_subtitles_en.txt | uniq > movie_subtitles_en-uniq.txt

and get a big reduction in size

-rw-r--r-- 1 dc staff 16M Jul 3 12:20 movie_subtitles_en-uniq.txt
-rw-r--r--@ 1 dc staff 24M Jul 3 12:18 movie_subtitles_en.txt

any reasons for this?

歌词文件行重复

twitter big corpus cat error

After catting the files with: cat twitter_en_big.txt.gz.part* > twitter_en_big.txt.gz
when I open the resulting file I get the error: "The archive is either in unknown format or damaged"