bheinzerling/pyrouge

A Python wrapper for the ROUGE summarization evaluation package

pyrouge's Issues

syntax error near unexpected token `newline'To fix `script not found` :

          To fix `script not found` :

pip uninstall pyrouge
git clone https://github.com/bheinzerling/pyrouge
cd pyrouge
pip install -e .
pyrouge_set_rouge_path </absolute/path/to/ROUGE-1.5.5/directory>
python -m pyrouge.test

Originally posted by @astariul in #25 (comment)

$ pyrouge_set_rouge_path </absolute/path/to/ROUGE-1.5.5/directory>
-bash: syntax error near unexpected token `newline'

How to set the path to store the intermediate files and results?

I am running this code in a docker container, which has not enough space left. Seems that it would store all the files under the path /tmp/. Could you please tell me how to set this path myself? Thanks in advance!

ImportError: cannot import name 'Rouge155' from 'pyrouge' (unknown location)

I have installed pyrouge using this solution:
https://stackoverflow.com/questions/45894212/installing-pyrouge-gets-error-in-ubuntu

When executing the command :
python -m pyrouge.test

I am getting ok.

But when i import pyrouge like:
from pyrouge import Rouge155

I am getting the following error:
ImportError: cannot import name 'Rouge155' from 'pyrouge' (unknown location)

What is it that I am doing wrong?

Plus at readme file your wrote this script:
from pyrouge import Rouge155

r = Rouge155()
r.system_dir = 'path/to/system_summaries'
r.model_dir = 'path/to/model_summaries'
r.system_filename_pattern = 'some_name.(\d+).txt'
r.model_filename_pattern = 'some_name.[A-Z].#ID#.txt'

output = r.convert_and_evaluate()
print(output)
output_dict = r.output_to_dict(output)

Where am i suppose to add this code snippet in my code?

Is this result normal?

Recently I was doing a text summarization related research and trained a simple model. I want to use ROUGE to check the validity of the model, and I got the following results.

1 ROUGE-1 Average_R: 0.41775
1 ROUGE-1 Average_P: 0.39336
1 ROUGE-1 Average_F: 0.39289

1 ROUGE-2 Average_R: 0.18253
1 ROUGE-2 Average_P: 0.17314
1 ROUGE-2 Average_F: 0.17203

1 ROUGE-3 Average_R: 0.10546
1 ROUGE-3 Average_P: 0.10178
1 ROUGE-3 Average_F: 0.10011

1 ROUGE-4 Average_R: 0.07039
1 ROUGE-4 Average_P: 0.06904
1 ROUGE-4 Average_F: 0.06724
...

And it shows that ROUGE_F score is smaller than ROUGE_P and ROUGE_R? Does anyone know why?
Is this result normal?

Exception: Cannot set data directory because the path /absolute/path/to/ROUGE-1.5.5/directory/data does not exist.

Hi,

Thanks for the package. How do you handle this? : https://stackoverflow.com/questions/45894212/installing-pyrouge-gets-error-in-ubuntu

UnicodeDecodeError

i'v been trying to run my code
although i run this correctly :python3 -m pyrouge.test
and it is answer me correct
when i run this :
from pyrouge import Rouge155

r = Rouge155()
r.system_dir = 'sys-summs'
r.model_dir = 'ref-summs'
r.system_filename_pattern = 'summ.(\d+).txt'
r.model_filename_pattern = 'summ.[A-Z].#ID#.txt'

output = r.convert_and_evaluate()
print(output)
output_dict = r.output_to_dict(output)

i get this errorUnicodeDecodeError: 'utf-8' codec can't decode byte 0x80 in position 3131: invalid start byte

my txt format is utf8 i converts them to utf8 but again i get this error

get error in testing pyrouge!

I run these commands, successfully:

git clone https://github.com/bheinzerling/pyrouge
cd pyrouge
python setup.py install
pyrouge_set_rouge_path /home/mahsa/ROUGE-1.5.5/

But when I write this command, I receive an error:

python -m pyrouge.test

Here is this error:

2017-12-14 01:53:07,990 [MainThread  ] [INFO ]  Written ROUGE configuration to /tmp/tmpy6pce_k0/rouge_conf.xml
.2017-12-14 01:53:08,505 [MainThread  ] [INFO ]  Processing files in data/SL2003_models_plain_text.
2017-12-14 01:53:08,506 [MainThread  ] [INFO ]  Processing SL.P.10.R.A.SL062003-10.html.
2017-12-14 01:53:08,506 [MainThread  ] [INFO ]  Processing SL.P.10.R.A.SL062003-04.html.
2017-12-14 01:53:08,506 [MainThread  ] [INFO ]  Processing SL.P.10.R.A.SL062003-11.html.
2017-12-14 01:53:08,506 [MainThread  ] [INFO ]  Processing SL.P.10.R.A.SL062003-25.html.
2017-12-14 01:53:08,507 [MainThread  ] [INFO ]  Processing SL.P.10.R.A.SL062003-14.html.
2017-12-14 01:53:08,507 [MainThread  ] [INFO ]  Processing SL.P.10.R.A.SL062003-13.html.
2017-12-14 01:53:08,507 [MainThread  ] [INFO ]  Processing SL.P.10.R.A.SL062003-01.html.
2017-12-14 01:53:08,507 [MainThread  ] [INFO ]  Processing SL.P.10.R.A.SL062003-02.html.
2017-12-14 01:53:08,508 [MainThread  ] [INFO ]  Processing SL.P.10.R.A.SL062003-05.html.
2017-12-14 01:53:08,508 [MainThread  ] [INFO ]  Processing SL.P.10.R.A.SL062003-23.html.
2017-12-14 01:53:08,508 [MainThread  ] [INFO ]  Processing SL.P.10.R.A.SL062003-09.html.
2017-12-14 01:53:08,508 [MainThread  ] [INFO ]  Processing SL.P.10.R.A.SL062003-24.html.
2017-12-14 01:53:08,509 [MainThread  ] [INFO ]  Processing SL.P.10.R.A.SL062003-15.html.
2017-12-14 01:53:08,509 [MainThread  ] [INFO ]  Processing SL.P.10.R.A.SL062003-19.html.
2017-12-14 01:53:08,519 [MainThread  ] [INFO ]  Processing SL.P.10.R.A.SL062003-03.html.
2017-12-14 01:53:08,519 [MainThread  ] [INFO ]  Processing SL.P.10.R.A.SL062003-12.html.
2017-12-14 01:53:08,520 [MainThread  ] [INFO ]  Processing SL.P.10.R.A.SL062003-22.html.
2017-12-14 01:53:08,521 [MainThread  ] [INFO ]  Processing SL.P.10.R.A.SL062003-17.html.
2017-12-14 01:53:08,521 [MainThread  ] [INFO ]  Processing SL.P.10.R.A.SL062003-08.html.
2017-12-14 01:53:08,522 [MainThread  ] [INFO ]  Processing SL.P.10.R.A.SL062003-21.html.
2017-12-14 01:53:08,522 [MainThread  ] [INFO ]  Processing SL.P.10.R.A.SL062003-20.html.
2017-12-14 01:53:08,523 [MainThread  ] [INFO ]  Processing SL.P.10.R.A.SL062003-18.html.
2017-12-14 01:53:08,523 [MainThread  ] [INFO ]  Processing SL.P.10.R.A.SL062003-16.html.
2017-12-14 01:53:08,524 [MainThread  ] [INFO ]  Processing SL.P.10.R.A.SL062003-07.html.
2017-12-14 01:53:08,524 [MainThread  ] [INFO ]  Processing SL.P.10.R.A.SL062003-06.html.
2017-12-14 01:53:08,525 [MainThread  ] [INFO ]  Saved processed files to /tmp/tmpai9wsudt.
.2017-12-14 01:53:08,569 [MainThread  ] [INFO ]  Written ROUGE configuration to /tmp/tmpvsc7_s_k/rouge_conf.xml
2017-12-14 01:53:08,569 [MainThread  ] [INFO ]  Running ROUGE with command /home/mahsa/ROUGE-1.5.5/ROUGE-1.5.5.pl -e /home/mahsa/ROUGE-1.5.5/data -c 95 -2 -1 -U -r 1000 -n 4 -w 1.2 -a -m /tmp/tmpvsc7_s_k/rouge_conf.xml
Cannot open exception db file for reading: /home/mahsa/ROUGE-1.5.5/data/WordNet-2.0.exc.db
E2017-12-14 01:53:09,874 [MainThread  ] [INFO ]  Processing files in data/SL2003_models_rouge_format.
2017-12-14 01:53:09,874 [MainThread  ] [INFO ]  Processing SL.P.10.R.A.SL062003-10.html.
/home/mahsa/anaconda3/lib/python3.6/site-packages/bs4/__init__.py:181: UserWarning: No parser was explicitly specified, so I'm using the best available HTML parser for this system ("lxml"). This usually isn't a problem, but if you run this code on another system, or in a different virtual environment, it may use a different parser and behave differently.

The code that caused this warning is on line 4 of the file /home/mahsa/anaconda3/bin/pyrouge_convert_rouge_format_to_plain_text. To get rid of this warning, change code that looks like this:

 BeautifulSoup(YOUR_MARKUP})

to this:

 BeautifulSoup(YOUR_MARKUP, "lxml")

  markup_type=markup_type))
2017-12-14 01:53:09,879 [MainThread  ] [INFO ]  Processing SL.P.10.R.A.SL062003-04.html.
2017-12-14 01:53:09,881 [MainThread  ] [INFO ]  Processing SL.P.10.R.A.SL062003-11.html.
2017-12-14 01:53:09,883 [MainThread  ] [INFO ]  Processing SL.P.10.R.A.SL062003-25.html.
2017-12-14 01:53:09,885 [MainThread  ] [INFO ]  Processing SL.P.10.R.A.SL062003-14.html.
2017-12-14 01:53:09,887 [MainThread  ] [INFO ]  Processing SL.P.10.R.A.SL062003-13.html.
2017-12-14 01:53:09,889 [MainThread  ] [INFO ]  Processing SL.P.10.R.A.SL062003-01.html.
2017-12-14 01:53:09,890 [MainThread  ] [INFO ]  Processing SL.P.10.R.A.SL062003-02.html.
2017-12-14 01:53:09,892 [MainThread  ] [INFO ]  Processing SL.P.10.R.A.SL062003-05.html.
2017-12-14 01:53:09,893 [MainThread  ] [INFO ]  Processing SL.P.10.R.A.SL062003-23.html.
2017-12-14 01:53:09,895 [MainThread  ] [INFO ]  Processing SL.P.10.R.A.SL062003-09.html.
2017-12-14 01:53:09,896 [MainThread  ] [INFO ]  Processing SL.P.10.R.A.SL062003-24.html.
2017-12-14 01:53:09,897 [MainThread  ] [INFO ]  Processing SL.P.10.R.A.SL062003-15.html.
2017-12-14 01:53:09,898 [MainThread  ] [INFO ]  Processing SL.P.10.R.A.SL062003-19.html.
2017-12-14 01:53:09,899 [MainThread  ] [INFO ]  Processing SL.P.10.R.A.SL062003-03.html.
2017-12-14 01:53:09,901 [MainThread  ] [INFO ]  Processing SL.P.10.R.A.SL062003-12.html.
2017-12-14 01:53:09,902 [MainThread  ] [INFO ]  Processing SL.P.10.R.A.SL062003-22.html.
2017-12-14 01:53:09,903 [MainThread  ] [INFO ]  Processing SL.P.10.R.A.SL062003-17.html.
2017-12-14 01:53:09,904 [MainThread  ] [INFO ]  Processing SL.P.10.R.A.SL062003-08.html.
2017-12-14 01:53:09,906 [MainThread  ] [INFO ]  Processing SL.P.10.R.A.SL062003-21.html.
2017-12-14 01:53:09,907 [MainThread  ] [INFO ]  Processing SL.P.10.R.A.SL062003-20.html.
2017-12-14 01:53:09,909 [MainThread  ] [INFO ]  Processing SL.P.10.R.A.SL062003-18.html.
2017-12-14 01:53:09,910 [MainThread  ] [INFO ]  Processing SL.P.10.R.A.SL062003-16.html.
2017-12-14 01:53:09,912 [MainThread  ] [INFO ]  Processing SL.P.10.R.A.SL062003-07.html.
2017-12-14 01:53:09,913 [MainThread  ] [INFO ]  Processing SL.P.10.R.A.SL062003-06.html.
2017-12-14 01:53:09,914 [MainThread  ] [INFO ]  Saved processed files to /tmp/tmpab6id8d1.
.2017-12-14 01:53:10,475 [MainThread  ] [INFO ]  Writing summaries.
2017-12-14 01:53:10,477 [MainThread  ] [INFO ]  Processing summaries. Saving system files to /tmp/tmph0s8_f0z/system and model files to /tmp/tmph0s8_f0z/model.
2017-12-14 01:53:10,477 [MainThread  ] [INFO ]  Processing files in data/systems_plain.
2017-12-14 01:53:10,477 [MainThread  ] [INFO ]  Processing D30002.M.100.T.A.
2017-12-14 01:53:10,478 [MainThread  ] [INFO ]  Processing D30003.M.100.T.A.
2017-12-14 01:53:10,478 [MainThread  ] [INFO ]  Processing D30001.M.100.T.A.
2017-12-14 01:53:10,479 [MainThread  ] [INFO ]  Processing D30005.M.100.T.A.
2017-12-14 01:53:10,479 [MainThread  ] [INFO ]  Saved processed files to /tmp/tmph0s8_f0z/system.
2017-12-14 01:53:10,479 [MainThread  ] [INFO ]  Processing files in data/models_plain.
2017-12-14 01:53:10,479 [MainThread  ] [INFO ]  Processing D30005.M.100.T.C.
2017-12-14 01:53:10,480 [MainThread  ] [INFO ]  Processing D30002.M.100.T.E.
2017-12-14 01:53:10,480 [MainThread  ] [INFO ]  Processing D30001.M.100.T.C.
2017-12-14 01:53:10,481 [MainThread  ] [INFO ]  Processing D30001.M.100.T.B.
2017-12-14 01:53:10,481 [MainThread  ] [INFO ]  Processing D30003.M.100.T.F.
2017-12-14 01:53:10,482 [MainThread  ] [INFO ]  Processing D30003.M.100.T.C.
2017-12-14 01:53:10,482 [MainThread  ] [INFO ]  Processing D30001.M.100.T.D.
2017-12-14 01:53:10,483 [MainThread  ] [INFO ]  Processing D30005.M.100.T.G.
2017-12-14 01:53:10,483 [MainThread  ] [INFO ]  Processing D30002.M.100.T.C.
2017-12-14 01:53:10,483 [MainThread  ] [INFO ]  Processing D30002.M.100.T.B.
2017-12-14 01:53:10,484 [MainThread  ] [INFO ]  Processing D30005.M.100.T.B.
2017-12-14 01:53:10,484 [MainThread  ] [INFO ]  Processing D30003.M.100.T.B.
2017-12-14 01:53:10,484 [MainThread  ] [INFO ]  Saved processed files to /tmp/tmph0s8_f0z/model.
2017-12-14 01:53:10,486 [MainThread  ] [INFO ]  Written ROUGE configuration to /tmp/tmp6a8qeb0b/rouge_conf.xml
2017-12-14 01:53:10,486 [MainThread  ] [INFO ]  Running ROUGE with command /home/mahsa/ROUGE-1.5.5/ROUGE-1.5.5.pl -e /home/mahsa/ROUGE-1.5.5/data -c 90 -2 -1 -U -r 1000 -n 2 -w 1.2 -a -m data/config_test2.xml -m /tmp/tmp6a8qeb0b/rouge_conf.xml
Cannot open exception db file for reading: /home/mahsa/ROUGE-1.5.5/data/WordNet-2.0.exc.db
Traceback (most recent call last):
  File "/home/mahsa/anaconda3/bin/pyrouge_evaluate_plain_text_files", line 4, in <module>
    __import__('pkg_resources').run_script('pyrouge==0.1.3', 'pyrouge_evaluate_plain_text_files')
  File "/home/mahsa/anaconda3/lib/python3.6/site-packages/setuptools-27.2.0-py3.6.egg/pkg_resources/__init__.py", line 744, in run_script
  File "/home/mahsa/anaconda3/lib/python3.6/site-packages/setuptools-27.2.0-py3.6.egg/pkg_resources/__init__.py", line 1499, in run_script
  File "/home/mahsa/anaconda3/lib/python3.6/site-packages/pyrouge-0.1.3-py3.6.egg/EGG-INFO/scripts/pyrouge_evaluate_plain_text_files", line 25, in <module>
    main()
  File "/home/mahsa/anaconda3/lib/python3.6/site-packages/pyrouge-0.1.3-py3.6.egg/EGG-INFO/scripts/pyrouge_evaluate_plain_text_files", line 21, in main
    output = rouge.convert_and_evaluate(args.system_id, args.split_sents)
  File "/home/mahsa/anaconda3/lib/python3.6/site-packages/pyrouge-0.1.3-py3.6.egg/pyrouge/Rouge155.py", line 364, in convert_and_evaluate
    rouge_output = self.evaluate(system_id, rouge_args)
  File "/home/mahsa/anaconda3/lib/python3.6/site-packages/pyrouge-0.1.3-py3.6.egg/pyrouge/Rouge155.py", line 339, in evaluate
    rouge_output = check_output(command).decode("UTF-8")
  File "/home/mahsa/anaconda3/lib/python3.6/subprocess.py", line 336, in check_output
    **kwargs).stdout
  File "/home/mahsa/anaconda3/lib/python3.6/subprocess.py", line 418, in run
    output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['/home/mahsa/ROUGE-1.5.5/ROUGE-1.5.5.pl', '-e', '/home/mahsa/ROUGE-1.5.5/data', '-c', '90', '-2', '-1', '-U', '-r', '1000', '-n', '2', '-w', '1.2', '-a', '-m', 'data/config_test2.xml', '-m', '/tmp/tmp6a8qeb0b/rouge_conf.xml']' returned non-zero exit status 255.
E.2017-12-14 01:53:11,148 [MainThread  ] [INFO ]  Writing summaries.
2017-12-14 01:53:11,151 [MainThread  ] [INFO ]  Processing summaries. Saving system files to /tmp/tmpub0odyyt/system and model files to /tmp/tmpub0odyyt/model.
2017-12-14 01:53:11,151 [MainThread  ] [INFO ]  Processing files in data/systems_plain.
2017-12-14 01:53:11,151 [MainThread  ] [INFO ]  Processing D30002.M.100.T.A.
2017-12-14 01:53:11,152 [MainThread  ] [INFO ]  Processing D30003.M.100.T.A.
2017-12-14 01:53:11,152 [MainThread  ] [INFO ]  Processing D30001.M.100.T.A.
2017-12-14 01:53:11,153 [MainThread  ] [INFO ]  Processing D30005.M.100.T.A.
2017-12-14 01:53:11,154 [MainThread  ] [INFO ]  Saved processed files to /tmp/tmpub0odyyt/system.
2017-12-14 01:53:11,154 [MainThread  ] [INFO ]  Processing files in data/models_plain.
2017-12-14 01:53:11,154 [MainThread  ] [INFO ]  Processing D30005.M.100.T.C.
2017-12-14 01:53:11,155 [MainThread  ] [INFO ]  Processing D30002.M.100.T.E.
2017-12-14 01:53:11,155 [MainThread  ] [INFO ]  Processing D30001.M.100.T.C.
2017-12-14 01:53:11,156 [MainThread  ] [INFO ]  Processing D30001.M.100.T.B.
2017-12-14 01:53:11,156 [MainThread  ] [INFO ]  Processing D30003.M.100.T.F.
2017-12-14 01:53:11,157 [MainThread  ] [INFO ]  Processing D30003.M.100.T.C.
2017-12-14 01:53:11,157 [MainThread  ] [INFO ]  Processing D30001.M.100.T.D.
2017-12-14 01:53:11,158 [MainThread  ] [INFO ]  Processing D30005.M.100.T.G.
2017-12-14 01:53:11,158 [MainThread  ] [INFO ]  Processing D30002.M.100.T.C.
2017-12-14 01:53:11,159 [MainThread  ] [INFO ]  Processing D30002.M.100.T.B.
2017-12-14 01:53:11,159 [MainThread  ] [INFO ]  Processing D30005.M.100.T.B.
2017-12-14 01:53:11,160 [MainThread  ] [INFO ]  Processing D30003.M.100.T.B.
2017-12-14 01:53:11,160 [MainThread  ] [INFO ]  Saved processed files to /tmp/tmpub0odyyt/model.
2017-12-14 01:53:11,162 [MainThread  ] [INFO ]  Written ROUGE configuration to /tmp/tmprdoxel4d/rouge_conf.xml
2017-12-14 01:53:11,162 [MainThread  ] [INFO ]  Running ROUGE with command /home/mahsa/ROUGE-1.5.5/ROUGE-1.5.5.pl -e /home/mahsa/ROUGE-1.5.5/data -c 95 -2 -1 -U -r 1000 -n 4 -w 1.2 -a -m /tmp/tmprdoxel4d/rouge_conf.xml
Cannot open exception db file for reading: /home/mahsa/ROUGE-1.5.5/data/WordNet-2.0.exc.db
Traceback (most recent call last):
  File "/home/mahsa/anaconda3/bin/pyrouge_evaluate_plain_text_files", line 4, in <module>
    __import__('pkg_resources').run_script('pyrouge==0.1.3', 'pyrouge_evaluate_plain_text_files')
  File "/home/mahsa/anaconda3/lib/python3.6/site-packages/setuptools-27.2.0-py3.6.egg/pkg_resources/__init__.py", line 744, in run_script
  File "/home/mahsa/anaconda3/lib/python3.6/site-packages/setuptools-27.2.0-py3.6.egg/pkg_resources/__init__.py", line 1499, in run_script
  File "/home/mahsa/anaconda3/lib/python3.6/site-packages/pyrouge-0.1.3-py3.6.egg/EGG-INFO/scripts/pyrouge_evaluate_plain_text_files", line 25, in <module>
    main()
  File "/home/mahsa/anaconda3/lib/python3.6/site-packages/pyrouge-0.1.3-py3.6.egg/EGG-INFO/scripts/pyrouge_evaluate_plain_text_files", line 21, in main
    output = rouge.convert_and_evaluate(args.system_id, args.split_sents)
  File "/home/mahsa/anaconda3/lib/python3.6/site-packages/pyrouge-0.1.3-py3.6.egg/pyrouge/Rouge155.py", line 364, in convert_and_evaluate
    rouge_output = self.evaluate(system_id, rouge_args)
  File "/home/mahsa/anaconda3/lib/python3.6/site-packages/pyrouge-0.1.3-py3.6.egg/pyrouge/Rouge155.py", line 339, in evaluate
    rouge_output = check_output(command).decode("UTF-8")
  File "/home/mahsa/anaconda3/lib/python3.6/subprocess.py", line 336, in check_output
    **kwargs).stdout
  File "/home/mahsa/anaconda3/lib/python3.6/subprocess.py", line 418, in run
    output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['/home/mahsa/ROUGE-1.5.5/ROUGE-1.5.5.pl', '-e', '/home/mahsa/ROUGE-1.5.5/data', '-c', '95', '-2', '-1', '-U', '-r', '1000', '-n', '4', '-w', '1.2', '-a', '-m', '/tmp/tmprdoxel4d/rouge_conf.xml']' returned non-zero exit status 255.
E./tmp/tmpeswzphxi/config_test.xml data/config_test.xml
...
======================================================================
ERROR: test_evaluation (pyrouge.tests.Rouge155_test.PyrougeTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/mahsa/pyrouge/pyrouge/tests/Rouge155_test.py", line 156, in test_evaluation
    pyrouge_output = rouge.evaluate(system_id=11).strip()
  File "/home/mahsa/pyrouge/pyrouge/Rouge155.py", line 339, in evaluate
    rouge_output = check_output(command).decode("UTF-8")
  File "/home/mahsa/anaconda3/lib/python3.6/subprocess.py", line 336, in check_output
    **kwargs).stdout
  File "/home/mahsa/anaconda3/lib/python3.6/subprocess.py", line 418, in run
    output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['/home/mahsa/ROUGE-1.5.5/ROUGE-1.5.5.pl', '-e', '/home/mahsa/ROUGE-1.5.5/data', '-c', '95', '-2', '-1', '-U', '-r', '1000', '-n', '4', '-w', '1.2', '-a', '-m', '/tmp/tmpvsc7_s_k/rouge_conf.xml']' returned non-zero exit status 255.

======================================================================
ERROR: test_options (pyrouge.tests.Rouge155_test.PyrougeTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/mahsa/pyrouge/pyrouge/tests/Rouge155_test.py", line 218, in test_options
    pyrouge_output = check_output_clean(pyrouge_command)
  File "/home/mahsa/pyrouge/pyrouge/tests/Rouge155_test.py", line 17, in <lambda>
    check_output_clean = lambda c: check_output(c).decode("UTF-8").strip()
  File "/home/mahsa/anaconda3/lib/python3.6/subprocess.py", line 336, in check_output
    **kwargs).stdout
  File "/home/mahsa/anaconda3/lib/python3.6/subprocess.py", line 418, in run
    output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['pyrouge_evaluate_plain_text_files', '-m', 'data/models_plain', '-s', 'data/systems_plain', '-sfp', 'D(\\d+).M.100.T.A', '-mfp', 'D#ID#.M.100.T.[A-Z]', '-id', '1', '-rargs', '"-e /home/mahsa/ROUGE-1.5.5/data -c 90 -2 -1 -U -r 1000 -n 2 -w 1.2 -a -m data/config_test2.xml"']' returned non-zero exit status 1.

======================================================================
ERROR: test_rouge_for_plain_text (pyrouge.tests.Rouge155_test.PyrougeTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/mahsa/pyrouge/pyrouge/tests/Rouge155_test.py", line 173, in test_rouge_for_plain_text
    pyrouge_output = check_output_clean(pyrouge_command.split())
  File "/home/mahsa/pyrouge/pyrouge/tests/Rouge155_test.py", line 17, in <lambda>
    check_output_clean = lambda c: check_output(c).decode("UTF-8").strip()
  File "/home/mahsa/anaconda3/lib/python3.6/subprocess.py", line 336, in check_output
    **kwargs).stdout
  File "/home/mahsa/anaconda3/lib/python3.6/subprocess.py", line 418, in run
    output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['pyrouge_evaluate_plain_text_files', '-m', 'data/models_plain', '-s', 'data/systems_plain', '-sfp', 'D(\\d+).M.100.T.A', '-mfp', 'D#ID#.M.100.T.[A-Z]', '-id', '1']' returned non-zero exit status 1.

----------------------------------------------------------------------
Ran 11 tests in 3.937s

FAILED (errors=3)

Please help me. I dont know what to do?

returned non-zero exit status 255.

I am getting this error. Help me please.

Benchmarking

Hi,

Would you be interested in benchmarking Python ROUGE wrappers?

I worked on my own solution for ROUGE evaluation over files (using two files f1, f2, reading each line l1_i in f1, l2_i in f2 and scoring rouge(l1_i, l2_i).

My system uses multithreading to speed up the process (good gain in practice as the wrapping require I/O).

Do you have any information about processing speed using your solution?
We could run experiments using a common process & data to compare.
Note that I don't want my solution "win", I just want the fastest python ROUGE scorer.

Regards

WindowsError: [Error 193] %1 is not a valid Win32 application

WindowsError                              Traceback (most recent call last)
<ipython-input-4-b6efe7b2e6d3> in <module>()
----> 1 rouge_eval('../Documents/rslt/cp/reference/', '../Documents/rslt/pg/decoded/')

<ipython-input-2-17db6565b0ff> in rouge_eval(ref_dir, dec_dir)
      7     r.system_dir = dec_dir
      8     logging.getLogger('global').setLevel(logging.WARNING) # silence pyrouge logging
----> 9     rouge_results = r.convert_and_evaluate()
     10     return r.output_to_dict(rouge_results)

C:\Anaconda\lib\site-packages\pyrouge-0.1.3-py2.7.egg\pyrouge\Rouge155.pyc in convert_and_evaluate(self, system_id, split_sentences, rouge_args)
    362             self.split_sentences()
    363         self.__write_summaries()
--> 364         rouge_output = self.evaluate(system_id, rouge_args)
    365         return rouge_output
    366 

C:\Anaconda\lib\site-packages\pyrouge-0.1.3-py2.7.egg\pyrouge\Rouge155.pyc in evaluate(self, system_id, rouge_args)
    337         self.log.info(
    338             "Running ROUGE with command {}".format(" ".join(command)))
--> 339         rouge_output = check_output(command).decode("UTF-8")
    340         return rouge_output
    341 

C:\Anaconda\lib\subprocess.pyc in check_output(*popenargs, **kwargs)
    565     if 'stdout' in kwargs:
    566         raise ValueError('stdout argument not allowed, it will be overridden.')
--> 567     process = Popen(stdout=PIPE, *popenargs, **kwargs)
    568     output, unused_err = process.communicate()
    569     retcode = process.poll()

C:\Anaconda\lib\subprocess.pyc in __init__(self, args, bufsize, executable, stdin, stdout, stderr, preexec_fn, close_fds, shell, cwd, env, universal_newlines, startupinfo, creationflags)
    709                                 p2cread, p2cwrite,
    710                                 c2pread, c2pwrite,
--> 711                                 errread, errwrite)
    712         except Exception:
    713             # Preserve original exception in case os.close raises.

C:\Anaconda\lib\subprocess.pyc in _execute_child(self, args, executable, preexec_fn, close_fds, cwd, env, universal_newlines, startupinfo, creationflags, shell, to_close, p2cread, p2cwrite, c2pread, c2pwrite, errread, errwrite)
    957                                          env,
    958                                          cwd,
--> 959                                          startupinfo)
    960             except pywintypes.error, e:
    961                 # Translate pywintypes.error to WindowsError, which is

WindowsError: [Error 193] %1 is not a valid Win32 application

how to generate ROUGE scores for each sentence in the text?

I want to do a supervised text summarization task on DUC2007. There are not any labels for this corpus and so I must generate ROUGE scores for all of the sentences in the corpus. how can I do this? should I put every sentence in a file? it is hard.

I use your pyrouge. but for example for pyrouge written in https://github.com/andersjo/pyrouge, there is a simple way for generating rouge score for any sentence. May you guide how can I produce ROUGE score for any sentence using your interface?

AssertionError when installing pyrouge

how to deal with this? please help.
https://stackoverflow.com/questions/43289915/assertionerror-when-installing-pyrouge

Unittests fail

Hi, I've installed pyrouge on my Mac running it within a virtual environment under Python 2.7.13. After activating the virtual environment and calling pyrouge_set_rouge_path I run the unittests with python -m pyrouge.test. However some of them fail saying:

======================================================================
FAIL: test_config_file (pyrouge.tests.Rouge155_test.PyrougeTest)

Traceback (most recent call last):
File "/Users/frank/Projects/ve.pyrouge/lib/python2.7/site-packages/pyrouge/tests/Rouge155_test.py", line 154, in test_config_file
add_data_path("ROUGE-test_11.xml")))
AssertionError: False is not true

======================================================================
FAIL: test_evaluation (pyrouge.tests.Rouge155_test.PyrougeTest)

Traceback (most recent call last):
File "/Users/frank/Projects/ve.pyrouge/lib/python2.7/site-packages/pyrouge/tests/Rouge155_test.py", line 172, in test_evaluation
self.assertEqual(pyrouge_output, orig_rouge_output)
AssertionError: u'---------------------------------------------\n11 ROUGE-1 Average_R: 0.22651 ( [truncated]... != u'---------------------------------------------\n11 ROUGE-1 Average_R: 0.77650 ( [truncated]...
Diff is 5949 characters long. Set self.maxDiff to None to see it.

======================================================================
FAIL: test_rouge_for_plain_text (pyrouge.tests.Rouge155_test.PyrougeTest)

Traceback (most recent call last):
File "/Users/frank/Projects/ve.pyrouge/lib/python2.7/site-packages/pyrouge/tests/Rouge155_test.py", line 192, in test_rouge_for_plain_text
self.assertEqual(pyrouge_output, orig_rouge_output)
AssertionError: u'---------------------------------------------\n1 ROUGE-1 Average_R: 0.40054 (9 [truncated]... != u'---------------------------------------------\n1 ROUGE-1 Average_R: 0.61336 (9 [truncated]...
Diff is 6253 characters long. Set self.maxDiff to None to see it.

======================================================================
FAIL: test_write_config (pyrouge.tests.Rouge155_test.PyrougeTest)

Traceback (most recent call last):
File "/Users/frank/Projects/ve.pyrouge/lib/python2.7/site-packages/pyrouge/tests/Rouge155_test.py", line 211, in test_write_config
self.assertTrue(xml_equal(config_file, target_xml))
AssertionError: False is not true

Until now I wasn't able to identify the reason for this issue. Any help is highly appreciated.

Summary length restriction n_bytes or n_words

How to restrict summary length in pyrouge ?

The original ROUGE supports this feature [-b n-bytes|-l n-words]

Usage: ROUGE-1.5.4.pl
[-a (evaluate all systems)]
[-c cf]
[-d (print per evaluation scores)]
[-e ROUGE_EVAL_HOME]
[-h (usage)]
[-b n-bytes|-l n-words]
[-m (use Porter stemmer)]
[-n max-ngram]
[-s (remove stopwords)]
[-r number-of-samples (for resampling)]
[-2 max-gap-length (if < 0 then no gap length limit)]
[-3 <H|HM|HMR|HM1|HMR1|HMR2>]
[-u (include unigram in skip-bigram) default no)]
[-U (same as -u but also compute regular skip-bigram)]
[-w weight (weighting factor for WLCS)]
[-v (verbose)]
[-x (do not calculate ROUGE-L)]
[-f A|B (scoring formula)]
[-p alpha (0 <= alpha <=1)]
[-t 0|1|2 (count by token instead of sentence)]
[-z <SEE|SPL|ISI|SIMPLE>]
[]

-b: Only use the first n bytes in the system/peer summary for the evaluation.
-l: Only use the first n words in the system/peer summary for the evaluation.

Subprocess error

While running some example code similar to the one found in your documentation, I get the following error (running on Windows 7 x64, Python 3.4):

**2017-02-02 09:53:30,910 [MainThread ] [INFO ] Running ROUGE with command ..\RELEASE-1.5.5\ROUGE-1.5.5.pl -e ...\RELEASE-1.5.5\data -c 95 -2 -1 -U -r 1000 -n 4 -w 1.2 -a -m C:\Users...\AppData\Local\Temp\tmpwbiw8z7b\rouge_conf.xml
Traceback (most recent call last):
File "rouge_eval.py", line 14, in
output = r.convert_and_evaluate()
File "C:\Python34\lib\site-packages\pyrouge\Rouge155.py", line 361, in convert_and_evaluate
rouge_output = self.evaluate(system_id, rouge_args)
File "C:\Python34\lib\site-packages\pyrouge\Rouge155.py", line 336, in evaluate
rouge_output = check_output(command).decode("UTF-8")
File "C:\Python34\lib\subprocess.py", line 607, in check_output
with Popen(*popenargs, stdout=PIPE, kwargs) as process:
File "C:\Python34\lib\subprocess.py", line 858, in init
restore_signals, start_new_session)
File "C:\Python34\lib\subprocess.py", line 1111, in _execute_child
startupinfo)
OSError: [WinError 193] %1 is not a valid Win32 application
Press any key to continue . . .

Any assistance would be highly appreciated

IndexError: tuple index out of range

Getting the following error. Any suggestion to solve it?

2018-06-04 02:49:35,663 [MainThread ] [INFO ] Writing summaries.
2018-06-04 02:49:35,664 [MainThread ] [INFO ] Processing summaries. Saving system files to /tmp/tmp2x4ur084/system and model files to /tmp/tmp2x4ur084/model.
2018-06-04 02:49:35,664 [MainThread ] [INFO ] Processing files in ./data/system_summaries.
2018-06-04 02:49:35,664 [MainThread ] [INFO ] Processing test.txt.tgt.tagged.
2018-06-04 02:49:35,717 [MainThread ] [INFO ] Saved processed files to /tmp/tmp2x4ur084/system.
2018-06-04 02:49:35,718 [MainThread ] [INFO ] Processing files in ./data/model_summaries.
2018-06-04 02:49:35,718 [MainThread ] [INFO ] Processing predicted_summary.txt.
2018-06-04 02:49:35,777 [MainThread ] [INFO ] Saved processed files to /tmp/tmp2x4ur084/model.
Traceback (most recent call last):
File "seq_ats/test_pyrouge.py", line 9, in
output = r.convert_and_evaluate()
File "/home/wasiahmad/software/anaconda3/lib/python3.6/site-packages/pyrouge-0.1.3-py3.6.egg/pyrouge/Rouge155.py", line 367, in convert_and_evaluate
rouge_output = self.evaluate(system_id, rouge_args)
File "/home/wasiahmad/software/anaconda3/lib/python3.6/site-packages/pyrouge-0.1.3-py3.6.egg/pyrouge/Rouge155.py", line 334, in evaluate
self.write_config(system_id=system_id)
File "/home/wasiahmad/software/anaconda3/lib/python3.6/site-packages/pyrouge-0.1.3-py3.6.egg/pyrouge/Rouge155.py", line 318, in write_config
self._config_file, system_id)
File "/home/wasiahmad/software/anaconda3/lib/python3.6/site-packages/pyrouge-0.1.3-py3.6.egg/pyrouge/Rouge155.py", line 271, in write_config_static
id = match.groups(0)[0]
IndexError: tuple index out of range

Error for python -m pyrouge.test

Cannot open exception db file for reading: /usr/local/ROUGE/RELEASE-1.5.5/data/WordNet-2.0.exc.db
Traceback (most recent call last):
File "/home/lab214/.local/bin/pyrouge_evaluate_plain_text_files", line 25, in
main()
File "/home/lab214/.local/bin/pyrouge_evaluate_plain_text_files", line 21, in main
output = rouge.convert_and_evaluate(args.system_id, args.split_sents)
File "/home/lab214/.local/lib/python3.5/site-packages/pyrouge/Rouge155.py", line 361, in convert_and_evaluate
rouge_output = self.evaluate(system_id, rouge_args)
File "/home/lab214/.local/lib/python3.5/site-packages/pyrouge/Rouge155.py", line 336, in evaluate
rouge_output = check_output(command).decode("UTF-8")
File "/usr/lib/python3.5/subprocess.py", line 626, in check_output
**kwargs).stdout
File "/usr/lib/python3.5/subprocess.py", line 708, in run
output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['/usr/local/ROUGE/RELEASE-1.5.5/ROUGE-1.5.5.pl', '-e', '/usr/local/ROUGE/RELEASE-1.5.5/data', '-c', '95', '-2', '-1', '-U', '-r', '1000', '-n', '4', '-w', '1.2', '-a', '-m', '/tmp/tmp8bewjckd/rouge_conf.xml']' returned non-zero exit status 13
E.E..
======================================================================
ERROR: test_evaluation (pyrouge.tests.Rouge155_test.PyrougeTest)

---

Traceback (most recent call last):
File "/home/lab214/anaconda3/envs/torch15tf220/lib/python3.7/site-packages/pyrouge/tests/Rouge155_test.py", line 156, in test_evaluation
pyrouge_output = rouge.evaluate(system_id=11).strip()
File "/home/lab214/anaconda3/envs/torch15tf220/lib/python3.7/site-packages/pyrouge/Rouge155.py", line 336, in evaluate
rouge_output = check_output(command).decode("UTF-8")
File "/home/lab214/anaconda3/envs/torch15tf220/lib/python3.7/subprocess.py", line 411, in check_output
**kwargs).stdout
File "/home/lab214/anaconda3/envs/torch15tf220/lib/python3.7/subprocess.py", line 512, in run
output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['/usr/local/ROUGE/RELEASE-1.5.5/ROUGE-1.5.5.pl', '-e', '/usr/local/ROUGE/RELEASE-1.5.5/data', '-c', '95', '-2', '-1', '-U', '-r', '1000', '-n', '4', '-w', '1.2', '-a', '-m', '/tmp/tmpt78ojqv5/rouge_conf.xml']' returned non-zero exit status 13.

======================================================================
ERROR: test_options (pyrouge.tests.Rouge155_test.PyrougeTest)

Traceback (most recent call last):
File "/home/lab214/anaconda3/envs/torch15tf220/lib/python3.7/site-packages/pyrouge/tests/Rouge155_test.py", line 218, in test_options
pyrouge_output = check_output_clean(pyrouge_command)
File "/home/lab214/anaconda3/envs/torch15tf220/lib/python3.7/site-packages/pyrouge/tests/Rouge155_test.py", line 17, in
check_output_clean = lambda c: check_output(c).decode("UTF-8").strip()
File "/home/lab214/anaconda3/envs/torch15tf220/lib/python3.7/subprocess.py", line 411, in check_output
**kwargs).stdout
File "/home/lab214/anaconda3/envs/torch15tf220/lib/python3.7/subprocess.py", line 488, in run
with Popen(*popenargs, **kwargs) as process:
File "/home/lab214/anaconda3/envs/torch15tf220/lib/python3.7/subprocess.py", line 800, in init
restore_signals, start_new_session)
File "/home/lab214/anaconda3/envs/torch15tf220/lib/python3.7/subprocess.py", line 1551, in _execute_child
raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: 'pyrouge_evaluate_plain_text_files.py': 'pyrouge_evaluate_plain_text_files.py'

======================================================================
ERROR: test_rouge_for_plain_text (pyrouge.tests.Rouge155_test.PyrougeTest)

Traceback (most recent call last):
File "/home/lab214/anaconda3/envs/torch15tf220/lib/python3.7/site-packages/pyrouge/tests/Rouge155_test.py", line 173, in test_rouge_for_plain_text
pyrouge_output = check_output_clean(pyrouge_command.split())
File "/home/lab214/anaconda3/envs/torch15tf220/lib/python3.7/site-packages/pyrouge/tests/Rouge155_test.py", line 17, in
check_output_clean = lambda c: check_output(c).decode("UTF-8").strip()
File "/home/lab214/anaconda3/envs/torch15tf220/lib/python3.7/subprocess.py", line 411, in check_output
**kwargs).stdout
File "/home/lab214/anaconda3/envs/torch15tf220/lib/python3.7/subprocess.py", line 512, in run
output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['pyrouge_evaluate_plain_text_files', '-m', 'data/models_plain', '-s', 'data/systems_plain', '-sfp', 'D(\d+).M.100.T.A', '-mfp', 'D#ID#.M.100.T.[A-Z]', '-id', '1']' returned non-zero exit status 1.

======================================================================
ERROR: test_write_config (pyrouge.tests.Rouge155_test.PyrougeTest)

Traceback (most recent call last):
File "/home/lab214/anaconda3/envs/torch15tf220/lib/python3.7/site-packages/pyrouge/tests/Rouge155_test.py", line 197, in test_write_config
check_output(command.split())
File "/home/lab214/anaconda3/envs/torch15tf220/lib/python3.7/subprocess.py", line 411, in check_output
**kwargs).stdout
File "/home/lab214/anaconda3/envs/torch15tf220/lib/python3.7/subprocess.py", line 488, in run
with Popen(*popenargs, **kwargs) as process:
File "/home/lab214/anaconda3/envs/torch15tf220/lib/python3.7/subprocess.py", line 800, in init
restore_signals, start_new_session)
File "/home/lab214/anaconda3/envs/torch15tf220/lib/python3.7/subprocess.py", line 1551, in _execute_child
raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: 'pyrouge_write_config_file.py': 'pyrouge_write_config_file.py'

======================================================================
FAIL: test_config_file (pyrouge.tests.Rouge155_test.PyrougeTest)

Traceback (most recent call last):
File "/home/lab214/anaconda3/envs/torch15tf220/lib/python3.7/site-packages/pyrouge/tests/Rouge155_test.py", line 147, in test_config_file
add_data_path("ROUGE-test_11.xml")))
AssertionError: False is not true

---

Ran 10 tests in 0.367s

FAILED (failures=1, errors=4)

it token almost 4h to solve these problems, but it does not work, How to solve this?thanks a lot.

Rouge-L result is very low

I am working on a model of a text summary. For the output of the model, the test is performed using pyrouge. The test results show that the rouge-1 and rouge2 indicators are normal and consistent with the effect of the paper. However, the results of rouge-L differ greatly from the results of the paper. What is doubtful is how the rouge-L is calculated. What could be the cause of this?

Getting 3 errors out of 11 while testing pyrouge

Hi everyone!
While testing if everything is installed correctly, running:
python -m pyrouge.test
I've got 3 errors out if 11.

As I understood the error occurs when execution perl script ROUGE-1.5.5.pl

OSError: [Errno 8] Exec format error: '/home/olha/Desktop/GNLP/Project/fast_abs_rl/ROUGE-1.5.5/ROUGE-1.5.5.pl'
E./tmp/tmpf95nxjbp/config_test.xml data/config_test.xml
...

ERROR: test_evaluation (pyrouge.tests.Rouge155_test.PyrougeTest)

Traceback (most recent call last):
File "/olhasempienv/lib/python3.6/site-packages/pyrouge/tests/Rouge155_test.py", line 156, in test_evaluation
pyrouge_output = rouge.evaluate(system_id=11).strip()
File "/olhasempienv/lib/python3.6/site-packages/pyrouge/Rouge155.py", line 342, in evaluate
rouge_output = check_output(command, env=env).decode("UTF-8")
File "/usr/lib/python3.6/subprocess.py", line 336, in check_output
**kwargs).stdout
File "/usr/lib/python3.6/subprocess.py", line 403, in run
with Popen(*popenargs, **kwargs) as process:
File "/usr/lib/python3.6/subprocess.py", line 709, in init
restore_signals, start_new_session)
File "/usr/lib/python3.6/subprocess.py", line 1344, in _execute_child
raise child_exception_type(errno_num, err_msg, err_filename)
OSError: [Errno 8] Exec format error: '/home/olha/Desktop/GNLP/Project/fast_abs_rl/ROUGE-1.5.5/ROUGE-1.5.5.pl'

======================================================================
ERROR: test_options (pyrouge.tests.Rouge155_test.PyrougeTest)

Traceback (most recent call last):
File "/olhasempienv/lib/python3.6/site-packages/pyrouge/tests/Rouge155_test.py", line 218, in test_options
pyrouge_output = check_output_clean(pyrouge_command)
File "/olhasempienv/lib/python3.6/site-packages/pyrouge/tests/Rouge155_test.py", line 17, in
check_output_clean = lambda c: check_output(c).decode("UTF-8").strip()
File "/usr/lib/python3.6/subprocess.py", line 336, in check_output
**kwargs).stdout
File "/usr/lib/python3.6/subprocess.py", line 418, in run
output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['pyrouge_evaluate_plain_text_files', '-m', 'data/models_plain', '-s', 'data/systems_plain', '-sfp', 'D(\d+).M.100.T.A', '-mfp', 'D#ID#.M.100.T.[A-Z]', '-id', '1', '-rargs', '"-e /home/olhamuliava/Desktop/GNLP/Project/fast_abs_rl/ROUGE-1.5.5/data -c 90 -2 -1 -U -r 1000 -n 2 -w 1.2 -a -m data/config_test2.xml"']' returned non-zero exit status 1.

======================================================================
ERROR: test_rouge_for_plain_text (pyrouge.tests.Rouge155_test.PyrougeTest)

Traceback (most recent call last):
File "/olhasempienv/lib/python3.6/site-packages/pyrouge/tests/Rouge155_test.py", line 173, in test_rouge_for_plain_text
pyrouge_output = check_output_clean(pyrouge_command.split())
File "/olhasempienv/lib/python3.6/site-packages/pyrouge/tests/Rouge155_test.py", line 17, in
check_output_clean = lambda c: check_output(c).decode("UTF-8").strip()
File "/usr/lib/python3.6/subprocess.py", line 336, in check_output
**kwargs).stdout
File "/usr/lib/python3.6/subprocess.py", line 418, in run
output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['pyrouge_evaluate_plain_text_files', '-m', 'data/models_plain', '-s', 'data/systems_plain', '-sfp', 'D(\d+).M.100.T.A', '-mfp', 'D#ID#.M.100.T.[A-Z]', '-id', '1']' returned non-zero exit status 1.

---

Ran 11 tests in 0.396s

FAILED (errors=3)

Any ideas?

Any help would be appreciated!

how can I add stop words.

Is there such a thing as stop words and how can I customize them?

location of ROUGE binary

when installing on a RHEL (fedora varient Linux) system with pip3.6 install pyrouge --user installs pyrouge in the typical python package location for users /home/user.name/.local/lib/python3.6/site-packages/.

Running pyrouge_set_rouge_path /home/user.name/.local/lib/python3.6/site-packages/pyrouge from the shell, or r = Rouge155('/home/user.name/.local/lib/python3.6/site-packages/pyrouge') from python throws the following error.

Traceback (most recent call last):
  File "/home/user.name/.local/bin/pyrouge_set_rouge_path", line 18, in <module>
    main()
  File "/home/user.name/.local/bin/pyrouge_set_rouge_path", line 15, in main
    Rouge155(args.home_dir)
  File "/home/user.name/.local/lib/python3.6/site-packages/pyrouge/Rouge155.py", line 88, in __init__
    self.__set_rouge_dir(rouge_dir)
  File "/home/user.name/.local/lib/python3.6/site-packages/pyrouge/Rouge155.py", line 412, in __set_rouge_dir
    "/path/to/rouge/home.".format(self._bin_path))
Exception: ROUGE binary not found at /home/user.name/.local/lib/python3.6/site-packages/pyrouge/ROUGE-1.5.5.pl. Please set the correct path by running pyrouge_set_rouge_path /path/to/rouge/home.

Where are the pyrouge binaries?

Deleting temp files is time-consuming

Could you add some lines to delete the rouge-config folder and parsed summaries folder which are left in /Temp after convert_and_evaluate() ?
Clearing these folders is time-consuming (files are trivial and too many) and they easily occupied 8GB of disk C: .

Errror try to access module 'utils'

I install everything well as you can see

but when i try to call some library i get error like this

Installing the pyrouge on Microsoft Windows

FYI, I've had many issues to install the Python package pyrouge on Microsoft Windows, so I wrote a tutorial:
How to install the pyrouge on Microsoft Windows?

WordNet-2.0.exc.db error.

I get stuck by this error for two days,,,

2017-07-19 11:57:28,358 [MainThread  ] [INFO ]  Set ROUGE home directory to /home/eduOS/software/ROUGE-1.5.5.
Cannot open exception db file for reading: /home/eduOS/software/ROUGE-1.5.5/data/WordNet-2.0.exc.db
Traceback (most recent call last):
  File "cal_rouge.py", line 53, in <module>
    results_dict = rouge_eval(rd, dd)
  File "cal_rouge.py", line 22, in rouge_eval
    rouge_results = r.convert_and_evaluate()
  File "/usr/local/lib/python2.7/dist-packages/pyrouge-0.1.3-py2.7.egg/pyrouge/Rouge155.py", line 364, in convert_and_evaluate
    rouge_output = self.evaluate(system_id, rouge_args)
  File "/usr/local/lib/python2.7/dist-packages/pyrouge-0.1.3-py2.7.egg/pyrouge/Rouge155.py", line 339, in evaluate
    rouge_output = check_output(command).decode("UTF-8")
  File "/usr/lib/python2.7/subprocess.py", line 573, in check_output
    raise CalledProcessError(retcode, cmd, output=output)
subprocess.CalledProcessError: Command '[u'/home/eduOS/software/ROUGE-1.5.5/ROUGE-1.5.5.pl', '-e', '/home/eduOS/software/ROUGE-1.5.5/data', '-c', '95', '-2', '-1', '-U', '-r', '1000', '-n', '4', '-w', '1.2', '-a', u'-m', u'/tmp/tmpmV_cZG/rouge_conf.xml']' returned non-zero exit status 2

And I have also tried the pyrouge from andersjo, but its API is different. How can I tackle this problem? What's going on here? I have tried to delete the file but cannot find it there.

About the system file and model file

Thank you for your work!

I have a simple question. It is that when I calculate the ROUGE score, can I just use two files that each file contains multiple predictions and golds?

Thanks

Can't locate XML/Parser.pm in @INC

Hi,

I get this error when trying to run pyrouge.

Can't locate XML/Parser.pm in @inc (you may need to install the XML::Parser module) (@inc contains: ROUGE-1.5.5 /etc/perl /usr/local/lib/x86_64-linux-gnu/perl/5.26.1 /usr/local/share/perl/5.26.1 /usr/lib/x86_64-linux-gnu/perl5/5.26 /usr/share/perl5 /usr/lib/x86_64-linux-gnu/perl/5.26 /usr/share/perl/5.26 /usr/local/lib/site_perl /usr/lib/x86_64-linux-gnu/perl-base) at ROUGE-1.5.5/XML/DOM.pm line 41.
BEGIN failed--compilation aborted at ROUGE-1.5.5/XML/DOM.pm line 70.
Compilation failed in require at ROUGE-1.5.5/ROUGE-1.5.5.pl line 177.
BEGIN failed--compilation aborted at ROUGE-1.5.5/ROUGE-1.5.5.pl line 177.
Traceback (most recent call last):
File "official_rouge.py", line 56, in
main()
File "official_rouge.py", line 52, in main
print_results(evaluate(args.gold, args.pred))
File "official_rouge.py", line 31, in evaluate
output = r.convert_and_evaluate()
File "/home/sajad/anaconda3/lib/python3.6/site-packages/pyrouge/Rouge155.py", line 361, in convert_and_evaluate
rouge_output = self.evaluate(system_id, rouge_args)
File "/home/sajad/anaconda3/lib/python3.6/site-packages/pyrouge/Rouge155.py", line 336, in evaluate
rouge_output = check_output(command).decode("UTF-8")
File "/home/sajad/anaconda3/lib/python3.6/subprocess.py", line 336, in check_output
**kwargs).stdout
File "/home/sajad/anaconda3/lib/python3.6/subprocess.py", line 418, in run
output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['ROUGE-1.5.5/ROUGE-1.5.5.pl', '-e', 'ROUGE-1.5.5/data', '-c', '95', '-2', '-1', '-U', '-r', '1000', '-n', '4', '-w', '1.2', '-a', '-m', '/tmp/tmpub309057/rouge_conf.xml']' returned non-zero exit status 2.

I gotta say that I've installed XML::Parser via cpan XML::Parser.

Any ideas?

Thanks!

About time overhead

Hello,

Thank you for your contribution. But when I use your lib to calculate the rouge metric for about 4000 files, it takes me more than 12 hours, and I don't obtain the result. So I want to know Is it Normal?

Error with rouge_conf.xml: write_config_static does not find text files

I am working through the simple test case, and despite its simplicity I can't seems to fix the error that it gives me.

2020-08-12 16:02:52,959 [MainThread  ] [INFO ]  Writing summaries.
2020-08-12 16:02:52,960 [MainThread  ] [INFO ]  Processing summaries. Saving system files to /tmp/tmpswh42b4o/system and model files to /tmp/tmpswh42b4o/model.
2020-08-12 16:02:52,960 [MainThread  ] [INFO ]  Processing files in /content/gdrive/My Drive/test/system_summaries/.
2020-08-12 16:02:52,960 [MainThread  ] [INFO ]  Processing test.001.txt.
2020-08-12 16:02:52,961 [MainThread  ] [INFO ]  Saved processed files to /tmp/tmpswh42b4o/system.
2020-08-12 16:02:52,961 [MainThread  ] [INFO ]  Processing files in /content/gdrive/My Drive/test/model_summaries/.
2020-08-12 16:02:52,962 [MainThread  ] [INFO ]  Processing test.A.001.txt.
2020-08-12 16:02:52,963 [MainThread  ] [INFO ]  Saved processed files to /tmp/tmpswh42b4o/model.

Traceback (most recent call last):
  File "/content/gdrive/My Drive/test/rouge.py", line 9, in <module>
    output = r.convert_and_evaluate()
  File "/usr/local/lib/python3.6/dist-packages/pyrouge/Rouge155.py", line 361, in convert_and_evaluate
    rouge_output = self.evaluate(system_id, rouge_args)
  File "/usr/local/lib/python3.6/dist-packages/pyrouge/Rouge155.py", line 331, in evaluate
    self.write_config(system_id=system_id)
  File "/usr/local/lib/python3.6/dist-packages/pyrouge/Rouge155.py", line 315, in write_config
    self._config_file, system_id)
  File "/usr/local/lib/python3.6/dist-packages/pyrouge/Rouge155.py", line 277, in write_config_static
    system_filename_pattern.pattern, system_dir))
Exception: Did not find any files matching the pattern text.(\d+).txt in the system summaries directory /tmp/tmpswh42b4o/system.

This seems to be the case because it is not finding the rouge_conf.xml file to write to/with, since that is the next step of the output that I should see, according to the walkthrough. Is there a simple work around for this that I simply overlooked, or is there some glaring issue here? Any help would be greatly appreciated!

Pyrouge takes so much time to give the results

Hey, I run the package with the instructions on github/website, but it takes so much to give the results for Rouge scores (~20/25 minutes!) .

This is my code, which is actually a simple copy from the documentation:

from pyrouge import Rouge155

r = Rouge155()
r.system_dir = 'rad_decoder_sys/'
r.model_dir = 'rad_decoder_ref/'
r.system_filename_pattern = 'rademb-preds.(\d+).txt'
r.model_filename_pattern = 'rademb-gold.#ID#.txt'

output = r.convert_and_evaluate()
print(output)

When I try to run the script above, I end up with these logs:

2018-12-07 17:48:23,520 [MainThread ] [INFO ] Writing summaries.
2018-12-07 17:48:23,522 [MainThread ] [INFO ] Processing summaries. Saving system files to /var/folders/gz/4hxz5p9d235bxgbmb4phkk_00000gp/T/tmpre5wi1ih/system and model files to /var/folders/gz/4hxz5p9d235bxgbmb4phkk_00000gp/T/tmpre5wi1ih/model.
2018-12-07 17:48:23,522 [MainThread ] [INFO ] Processing files in rad_decoder_sys/.
2018-12-07 17:48:23,527 [MainThread ] [INFO ] Processing rademb-p.3614.txt.
2018-12-07 17:48:23,528 [MainThread ] [INFO ] Processing rademb-p.1003.txt.
...
2018-12-07 17:48:29,831 [MainThread ] [INFO ] Saved processed files to /var/folders/gz/4hxz5p9d235bxgbmb4phkk_00000gp/T/tmpre5wi1ih/model.
2018-12-07 17:48:57,667 [MainThread ] [INFO ] Written ROUGE configuration to /var/folders/gz/4hxz5p9d235bxgbmb4phkk_00000gp/T/tmp4z2m9wl7/rouge_conf.xml
2018-12-07 17:48:57,667 [MainThread ] [INFO ] Running ROUGE with command /Users/ss4164/PycharmProjects/rouge_calculator/official_rouge/RELEASE-1.5.5/ROUGE-1.5.5.pl -e /Users/ss4164/PycharmProjects/rouge_calculator/official_rouge/RELEASE-1.5.5/data -c 95 -2 -1 -U -r 1000 -n 4 -w 1.2 -a -m /var/folders/gz/4hxz5p9d235bxgbmb4phkk_00000gp/T/tmp4z2m9wl7/rouge_conf.xml

I tried with both PyCharm and terminal, but that didn't help.

Anyone having the same issue?

Thanks!

UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe9 in position 947: invalid continuation byte

i run these commands, successfully:

git clone https://github.com/bheinzerling/pyrouge
cd pyrouge
python setup.py install
pyrouge_set_rouge_path /absolute/path/to/ROUGE-1.5.5/directory
python -m pyrouge.test

and in the last command, i receive this:

Ran 11 tests in 6.322s
OK

then i run this piece of code:

from pyrouge import Rouge155
r = Rouge155()
r.system_dir = "/home/afsharizadeh/Desktop/summarization/summarization_dataset/DUC_2007/2007/all_sum/system_sum/"
r.model_dir = "/home/afsharizadeh/Desktop/summarization/summarization_dataset/DUC_2007/2007/all_sum/ref_sum/"
r.system_filename_pattern = 'sum.(\d+).txt'
r.model_filename_pattern = 'sum.[A-Z].#ID#.txt'

output = r.convert_and_evaluate()
print(output)
output_dict = r.output_to_dict(output)

then i get this error:

2017-08-27 17:22:12,119 [MainThread  ] [INFO ]  Writing summaries.
2017-08-27 17:22:12,121 [MainThread  ] [INFO ]  Processing summaries. Saving system files to /tmp/tmp192ti5r6/system and model files to /tmp/tmp192ti5r6/model.
2017-08-27 17:22:12,123 [MainThread  ] [INFO ]  Processing files in /home/afsharizadeh/Desktop/summarization/summarization_dataset/DUC_2007/2007/all_sum/system_sum/.
2017-08-27 17:22:12,125 [MainThread  ] [INFO ]  Processing sum.033.txt.
2017-08-27 17:22:12,128 [MainThread  ] [INFO ]  Processing sum.001.txt.
2017-08-27 17:22:12,130 [MainThread  ] [INFO ]  Processing sum.026.txt.
2017-08-27 17:22:12,132 [MainThread  ] [INFO ]  Processing sum.036.txt.
2017-08-27 17:22:12,135 [MainThread  ] [INFO ]  Processing sum.042.txt.
2017-08-27 17:22:12,137 [MainThread  ] [INFO ]  Processing sum.029.txt.
2017-08-27 17:22:12,139 [MainThread  ] [INFO ]  Processing sum.021.txt.
2017-08-27 17:22:12,141 [MainThread  ] [INFO ]  Processing sum.022.txt.
2017-08-27 17:22:12,144 [MainThread  ] [INFO ]  Processing sum.008.txt.
2017-08-27 17:22:12,146 [MainThread  ] [INFO ]  Processing sum.005.txt.
2017-08-27 17:22:12,150 [MainThread  ] [INFO ]  Processing sum.003.txt.
2017-08-27 17:22:12,152 [MainThread  ] [INFO ]  Processing sum.004.txt.
2017-08-27 17:22:12,155 [MainThread  ] [INFO ]  Processing sum.037.txt.
2017-08-27 17:22:12,159 [MainThread  ] [INFO ]  Processing sum.009.txt.
2017-08-27 17:22:12,162 [MainThread  ] [INFO ]  Processing sum.010.txt.
2017-08-27 17:22:12,165 [MainThread  ] [INFO ]  Processing sum.031.txt.
2017-08-27 17:22:12,168 [MainThread  ] [INFO ]  Processing sum.032.txt.
2017-08-27 17:22:12,171 [MainThread  ] [INFO ]  Processing sum.034.txt.
2017-08-27 17:22:12,174 [MainThread  ] [INFO ]  Processing sum.038.txt.
2017-08-27 17:22:12,177 [MainThread  ] [INFO ]  Processing sum.044.txt.
2017-08-27 17:22:12,179 [MainThread  ] [INFO ]  Processing sum.023.txt.
2017-08-27 17:22:12,182 [MainThread  ] [INFO ]  Processing sum.043.txt.
2017-08-27 17:22:12,185 [MainThread  ] [INFO ]  Processing sum.045.txt.
2017-08-27 17:22:12,187 [MainThread  ] [INFO ]  Processing sum.014.txt.
2017-08-27 17:22:12,190 [MainThread  ] [INFO ]  Processing sum.017.txt.
2017-08-27 17:22:12,193 [MainThread  ] [INFO ]  Processing sum.040.txt.
2017-08-27 17:22:12,195 [MainThread  ] [INFO ]  Processing sum.027.txt.
2017-08-27 17:22:12,198 [MainThread  ] [INFO ]  Processing sum.015.txt.
2017-08-27 17:22:12,200 [MainThread  ] [INFO ]  Processing sum.041.txt.
2017-08-27 17:22:12,203 [MainThread  ] [INFO ]  Processing sum.002.txt.
2017-08-27 17:22:12,205 [MainThread  ] [INFO ]  Processing sum.028.txt.
2017-08-27 17:22:12,208 [MainThread  ] [INFO ]  Processing sum.012.txt.
2017-08-27 17:22:12,210 [MainThread  ] [INFO ]  Processing sum.020.txt.
2017-08-27 17:22:12,213 [MainThread  ] [INFO ]  Processing sum.025.txt.
2017-08-27 17:22:12,216 [MainThread  ] [INFO ]  Processing sum.024.txt.
2017-08-27 17:22:12,218 [MainThread  ] [INFO ]  Processing sum.035.txt.
2017-08-27 17:22:12,221 [MainThread  ] [INFO ]  Processing sum.030.txt.
2017-08-27 17:22:12,224 [MainThread  ] [INFO ]  Processing sum.039.txt.
2017-08-27 17:22:12,226 [MainThread  ] [INFO ]  Processing sum.019.txt.
2017-08-27 17:22:12,229 [MainThread  ] [INFO ]  Processing sum.016.txt.
2017-08-27 17:22:12,232 [MainThread  ] [INFO ]  Processing sum.007.txt.
2017-08-27 17:22:12,234 [MainThread  ] [INFO ]  Processing sum.006.txt.
2017-08-27 17:22:12,237 [MainThread  ] [INFO ]  Processing sum.018.txt.
2017-08-27 17:22:12,241 [MainThread  ] [INFO ]  Processing sum.013.txt.
2017-08-27 17:22:12,243 [MainThread  ] [INFO ]  Processing sum.011.txt.
2017-08-27 17:22:12,246 [MainThread  ] [INFO ]  Saved processed files to /tmp/tmp192ti5r6/system.
2017-08-27 17:22:12,248 [MainThread  ] [INFO ]  Processing files in /home/afsharizadeh/Desktop/summarization/summarization_dataset/DUC_2007/2007/all_sum/ref_sum/.
2017-08-27 17:22:12,251 [MainThread  ] [INFO ]  Processing sum.C.027.txt.
2017-08-27 17:22:12,253 [MainThread  ] [INFO ]  Processing sum.D.014.txt.
2017-08-27 17:22:12,256 [MainThread  ] [INFO ]  Processing sum.C.018.txt.
2017-08-27 17:22:12,258 [MainThread  ] [INFO ]  Processing sum.B.005.txt.
2017-08-27 17:22:12,260 [MainThread  ] [INFO ]  Processing sum.A.019.txt.
2017-08-27 17:22:12,263 [MainThread  ] [INFO ]  Processing sum.B.004.txt.
2017-08-27 17:22:12,265 [MainThread  ] [INFO ]  Processing sum.A.007.txt.
2017-08-27 17:22:12,268 [MainThread  ] [INFO ]  Processing sum.C.008.txt.
2017-08-27 17:22:12,270 [MainThread  ] [INFO ]  Processing sum.A.013.txt.
2017-08-27 17:22:12,272 [MainThread  ] [INFO ]  Processing sum.A.003.txt.
2017-08-27 17:22:12,277 [MainThread  ] [INFO ]  Processing sum.C.020.txt.
2017-08-27 17:22:12,280 [MainThread  ] [INFO ]  Processing sum.A.021.txt.
2017-08-27 17:22:12,283 [MainThread  ] [INFO ]  Processing sum.D.012.txt.
2017-08-27 17:22:12,286 [MainThread  ] [INFO ]  Processing sum.C.019.txt.
2017-08-27 17:22:12,289 [MainThread  ] [INFO ]  Processing sum.A.033.txt.
2017-08-27 17:22:12,291 [MainThread  ] [INFO ]  Processing sum.C.042.txt.
2017-08-27 17:22:12,294 [MainThread  ] [INFO ]  Processing sum.B.032.txt.
2017-08-27 17:22:12,297 [MainThread  ] [INFO ]  Processing sum.C.029.txt.
2017-08-27 17:22:12,299 [MainThread  ] [INFO ]  Processing sum.D.002.txt.
2017-08-27 17:22:12,302 [MainThread  ] [INFO ]  Processing sum.A.045.txt.
2017-08-27 17:22:12,304 [MainThread  ] [INFO ]  Processing sum.B.001.txt.
2017-08-27 17:22:12,306 [MainThread  ] [INFO ]  Processing sum.D.037.txt.
2017-08-27 17:22:12,309 [MainThread  ] [INFO ]  Processing sum.A.004.txt.
2017-08-27 17:22:12,311 [MainThread  ] [INFO ]  Processing sum.B.014.txt.
2017-08-27 17:22:12,314 [MainThread  ] [INFO ]  Processing sum.C.026.txt.
2017-08-27 17:22:12,317 [MainThread  ] [INFO ]  Processing sum.A.031.txt.
2017-08-27 17:22:12,319 [MainThread  ] [INFO ]  Processing sum.D.005.txt.
2017-08-27 17:22:12,322 [MainThread  ] [INFO ]  Processing sum.B.038.txt.
2017-08-27 17:22:12,329 [MainThread  ] [INFO ]  Processing sum.B.027.txt.
2017-08-27 17:22:12,332 [MainThread  ] [INFO ]  Processing sum.C.010.txt.
2017-08-27 17:22:12,335 [MainThread  ] [INFO ]  Processing sum.B.041.txt.
2017-08-27 17:22:12,338 [MainThread  ] [INFO ]  Processing sum.C.030.txt.
2017-08-27 17:22:12,341 [MainThread  ] [INFO ]  Processing sum.B.007.txt.
2017-08-27 17:22:12,343 [MainThread  ] [INFO ]  Processing sum.C.023.txt.
2017-08-27 17:22:12,346 [MainThread  ] [INFO ]  Processing sum.C.002.txt.
2017-08-27 17:22:12,349 [MainThread  ] [INFO ]  Processing sum.B.033.txt.
2017-08-27 17:22:12,351 [MainThread  ] [INFO ]  Processing sum.D.023.txt.
2017-08-27 17:22:12,354 [MainThread  ] [INFO ]  Processing sum.C.014.txt.
2017-08-27 17:22:12,356 [MainThread  ] [INFO ]  Processing sum.D.007.txt.
2017-08-27 17:22:12,359 [MainThread  ] [INFO ]  Processing sum.D.008.txt.
2017-08-27 17:22:12,363 [MainThread  ] [INFO ]  Processing sum.D.032.txt.
2017-08-27 17:22:12,366 [MainThread  ] [INFO ]  Processing sum.C.005.txt.
2017-08-27 17:22:12,368 [MainThread  ] [INFO ]  Processing sum.B.023.txt.
2017-08-27 17:22:12,371 [MainThread  ] [INFO ]  Processing sum.B.035.txt.
2017-08-27 17:22:12,374 [MainThread  ] [INFO ]  Processing sum.A.016.txt.
2017-08-27 17:22:12,377 [MainThread  ] [INFO ]  Processing sum.D.001.txt.
2017-08-27 17:22:12,380 [MainThread  ] [INFO ]  Processing sum.C.017.txt.
2017-08-27 17:22:12,382 [MainThread  ] [INFO ]  Processing sum.A.010.txt.
2017-08-27 17:22:12,385 [MainThread  ] [INFO ]  Processing sum.B.021.txt.
2017-08-27 17:22:12,387 [MainThread  ] [INFO ]  Processing sum.B.010.txt.
2017-08-27 17:22:12,391 [MainThread  ] [INFO ]  Processing sum.A.009.txt.
2017-08-27 17:22:12,394 [MainThread  ] [INFO ]  Processing sum.A.011.txt.
2017-08-27 17:22:12,396 [MainThread  ] [INFO ]  Processing sum.D.045.txt.
2017-08-27 17:22:12,399 [MainThread  ] [INFO ]  Processing sum.A.018.txt.
2017-08-27 17:22:12,401 [MainThread  ] [INFO ]  Processing sum.D.028.txt.
2017-08-27 17:22:12,403 [MainThread  ] [INFO ]  Processing sum.D.020.txt.
2017-08-27 17:22:12,406 [MainThread  ] [INFO ]  Processing sum.C.031.txt.
2017-08-27 17:22:12,408 [MainThread  ] [INFO ]  Processing sum.B.025.txt.
2017-08-27 17:22:12,411 [MainThread  ] [INFO ]  Processing sum.C.028.txt.

2017-08-27 17:22:12,414 [MainThread  ] [INFO ]  Processing sum.D.009.txt.
2017-08-27 17:22:12,417 [MainThread  ] [INFO ]  Processing sum.C.013.txt.

---------------------------------------------------------------------------
UnicodeDecodeError                        Traceback (most recent call last)
<ipython-input-8-b3bc5a66e7f0> in <module>()
      6 r.model_filename_pattern = 'sum.[A-Z].#ID#.txt'
      7 
----> 8 output = r.convert_and_evaluate()
      9 print(output)
     10 output_dict = r.output_to_dict(output)

/home/afsharizadeh/anaconda3/lib/python3.6/site-packages/pyrouge/Rouge155.py in convert_and_evaluate(self, system_id, split_sentences, rouge_args)
    358         if split_sentences:
    359             self.split_sentences()
--> 360         self.__write_summaries()
    361         rouge_output = self.evaluate(system_id, rouge_args)
    362         return rouge_output

/home/afsharizadeh/anaconda3/lib/python3.6/site-packages/pyrouge/Rouge155.py in __write_summaries(self)
    487     def __write_summaries(self):
    488         self.log.info("Writing summaries.")
--> 489         self.__process_summaries(self.convert_summaries_to_rouge_format)
    490 
    491     @staticmethod

/home/afsharizadeh/anaconda3/lib/python3.6/site-packages/pyrouge/Rouge155.py in __process_summaries(self, process_func)
    481             "model files to {}.".format(new_system_dir, new_model_dir))
    482         process_func(self._system_dir, new_system_dir)
--> 483         process_func(self._model_dir, new_model_dir)
    484         self._system_dir = new_system_dir
    485         self._model_dir = new_model_dir

/home/afsharizadeh/anaconda3/lib/python3.6/site-packages/pyrouge/Rouge155.py in convert_summaries_to_rouge_format(input_dir, output_dir)
    200         """
    201         DirectoryProcessor.process(
--> 202             input_dir, output_dir, Rouge155.convert_text_to_rouge_format)
    203 
    204     @staticmethod

/home/afsharizadeh/anaconda3/lib/python3.6/site-packages/pyrouge/utils/file_utils.py in process(input_dir, output_dir, function)
     27             input_file = os.path.join(input_dir, input_file_name)
     28             with codecs.open(input_file, "r", encoding="UTF-8") as f:
---> 29                 input_string = f.read()
     30             output_string = function(input_string)
     31             output_file = os.path.join(output_dir, input_file_name)

/home/afsharizadeh/anaconda3/lib/python3.6/codecs.py in read(self, size)
    696     def read(self, size=-1):
    697 
--> 698         return self.reader.read(size)
    699 
    700     def readline(self, size=None):

/home/afsharizadeh/anaconda3/lib/python3.6/codecs.py in read(self, size, chars, firstline)
    499                 break
    500             try:
--> 501                 newchars, decodedbytes = self.decode(data, self.errors)
    502             except UnicodeDecodeError as exc:
    503                 if firstline:

UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe9 in position 947: invalid continuation byte

what is wrong?