yongzhuo/nlg-yongzhuo

中文文本生成（NLG）之文本摘要（text summarization）工具包, 语料数据(corpus data), 抽取式摘要 Extractive text summary of Lead3、keyword、textrank、text teaser、word significance、LDA、LSI、NMF。（graph，feature，topic model，summarize tool or tookit）

nlg-yongzhuo's Issues

feature_base/text_teaser.py的问题

计算title为什么使用
(len_mix_word + 1.0) / (len_mix_word + 2.0) / len_title_word
论文原文里应该是
len_mix_word/len_title_word

使用pip安装过程中出现下列错误，请问如何解决？

error: subprocess-exited-with-error

python setup.py egg_info did not run successfully.
exit code: 1

[15 lines of output]
C:\Users\hs6\AppData\Local\Temp\pip-install-mw7z_iai\pandas_fd94fc07bd02449c82bdc4a3c67de894\setup.py:12: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html
import pkg_resources
C:\Users\hs6\anaconda3\Lib\site-packages\setuptools_init_.py:84: _DeprecatedInstaller: setuptools.installer and fetch_build_eggs are deprecated.
!!

      ********************************************************************************
      Requirements should be satisfied by a PEP 517 installer.
      If you are using pip, you can try `pip install --use-pep517`.
      ********************************************************************************

!!
dist.fetch_build_eggs(dist.setup_requires)
error in pandas setup command: 'install_requires' must be a string or list of strings containing valid project/version requirement specifiers; Expected end or semicolon (after version specifier)
pytz >= 2011k
~~~~~~~^
[end of output]

note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

Encountered error while generating package metadata.

See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.

ValueError: max_df corresponds to < documents than min_df

Hello, why does the program report ValueError: max_df corresponds to < documents than min_df when I call model nmr, lda, lsi or nmf several times?

意见反馈

能够提供一个完整的东西，给一篇文章，直接输出摘要。

怎么保留句子的原始顺序呢？

怎么按句子的原始顺序，而不是按分数呢？

这个适配python的哪个版本啊，安装老是报错

关于text pprocess问题

你好,感谢你的代码开源!
有两个问题请教:

1. LDA模型如果想用在移动段效率怎么样
2. c/c++中有可用的文本前处理工具包吗,用于分词词干提取之类的

Lead-3 疑问

对于Lead-3算法，我有两点疑问？

1. Lead3Sum类中的summarize中，num_min的计算是否必要？
2. 在计算得分的公式有什么原理吗？而且计算后的结果是降序到最后一个，最后一个又突增，在我看来和mix没有区别。

Question about sklearn's lda result

topic_lda.py
...
res_lda_v = lda.components_
...
"lda.componets" represent words distribution about the topic.
However,
'''
else:
### 方案二, 获取最大主题概率的句子, 不分主题
res_combine = {}
for i in range(len_sentences_cut):
res_row_i = res_lda_v[:, i]
res_row_i_argmax = np.argmax(res_row_i)
res_combine[self.sentences[i]] = res_row_i[res_row_i_argmax]
'''
所以用res_lda_v[:, i]取出来的是单词的概率, 但最后计算的是句子的概率,所以这个地方我有些疑问,句子是怎么抽取的?是不是在前面用lda算的时候就是用的句子呢

语句排序的一些问题

第一次跑还好好地，下午回来突然就G了

pip 安装的问题

thank you for your contribution
ubuntu18.04试过python3.7 3.8,安装会出现问题,同#13
python3.6 bug如下:

File "/home/xxx/Software/yes/envs/nlg/lib/python3.6/site-packages/smart_open/s3.py", line 9
from future import annotations
^
SyntaxError: future feature annotations is not defined

在import时出现问题：cannot import name '_get_n_jobs'

ImportError                               Traceback (most recent call last)
<ipython-input-11-bb77d769d63d> in <module>
     15 from bs4 import BeautifulSoup
     16 from nltk.stem import WordNetLemmatizer
---> 17 from nlg_yongzhuo import *

~/anaconda3/envs/python36/lib/python3.6/site-packages/nlg_yongzhuo/__init__.py in <module>
     15 from nlg_yongzhuo.text_summarization.extractive_sum.nous_base.lead_3.lead_3 import Lead3Sum
     16 
---> 17 from nlg_yongzhuo.text_summarization.extractive_sum.topic_base.topic_lda import LDASum
     18 from nlg_yongzhuo.text_summarization.extractive_sum.topic_base.topic_lsi import LSISum
     19 from nlg_yongzhuo.text_summarization.extractive_sum.topic_base.topic_nmf import NMFSum

~/anaconda3/envs/python36/lib/python3.6/site-packages/nlg_yongzhuo/text_summarization/extractive_sum/topic_base/topic_lda.py in <module>
     13 # sklearn
     14 from sklearn.feature_extraction.text import CountVectorizer
---> 15 from sklearn.decomposition import LatentDirichletAllocation
     16 import numpy as np
     17 

~/anaconda3/envs/python36/lib/python3.6/site-packages/sklearn/decomposition/__init__.py in <module>
      9 from .incremental_pca import IncrementalPCA
     10 from .kernel_pca import KernelPCA
---> 11 from .sparse_pca import SparsePCA, MiniBatchSparsePCA
     12 from .truncated_svd import TruncatedSVD
     13 from .fastica_ import FastICA, fastica

~/anaconda3/envs/python36/lib/python3.6/site-packages/sklearn/decomposition/sparse_pca.py in <module>
     11 from ..linear_model import ridge_regression
     12 from ..base import BaseEstimator, TransformerMixin
---> 13 from .dict_learning import dict_learning, dict_learning_online
     14 
     15 

~/anaconda3/envs/python36/lib/python3.6/site-packages/sklearn/decomposition/dict_learning.py in <module>
     18 from ..externals.joblib import Parallel, delayed, cpu_count
     19 from ..externals.six.moves import zip
---> 20 from ..utils import (check_array, check_random_state, gen_even_slices,
     21                      gen_batches, _get_n_jobs)
     22 from ..utils.extmath import randomized_svd, row_norms

ImportError: cannot import name '_get_n_jobs'

数据问题

你好，你的摘要数据用的是百度云里哪一个数据啊？

topic_lda.py中的n_topics

高版本的skl(>0.19)中应将其改为n_components

关于模型加速的问题

请问这个text_summary_merge能不能使用GPU加速呢，因为要大批量的处理数据，如果能使用GPU加速的话就好很多，直接跑的速度确实有点太久了