累计:3.2136亿条语料(处理后)
清洗后的语料数:24138.4891 万条 24292.0234
清洗后的语料数:3258.2909 万条 3280.0360
清洗后的语料数:4739.2483万条 4761.7525
nohup python3 src/new_zh/nwes_zh_filter.py > logs/news.log 2>&1 &
nohup python3 src/baike_qa/baike_qa_filter.py >logs/baike.log 2>&1 &
nohup python3 ./filter_wiki.sh > logs/wiki.log 2>&1 &