recski/brise-plandok

Information extraction from text documents of the zoning plan of the City of Vienna

brise-plandok's Issues

potato/create_dataset fails unexpectedly if target data files exist

This command works fine if data/gold* files don't exist:

python create_dataset.py -d ~/sandbox/brise-nlp/annotation/2021_09/full_data -g fourlang -o -n gold

But if I rerun it to regenerate the graphs, the same command fails with this error:

/home/recski/miniconda3/envs/brise/lib/python3.7/site-packages/pandas/core/indexing.py:845: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
  self.obj[key] = _infer_fill_value(value)
/home/recski/miniconda3/envs/brise/lib/python3.7/site-packages/pandas/core/indexing.py:966: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
  self.obj[item] = s

Emptying the data folder solves the problem.

@Eszti please have a look when you can

Create script for checking if sentence already labeled in gold

Add attributes gold_exists and gold_attributes

convert.py does not open stream for -of other than xlsx

NOTE: this issue was detected on dev

Executing

python brise_plandok/convert.py \
    -i XLSX \
    -if ~/research/data/brise/ann/$1.xlsx \
    -o JSON \
    -of ~/research/data/brise/ann/$1.json

results in

Traceback (most recent call last):
  File "/home/eszter/research/brise-plandok/brise_plandok/convert.py", line 391, in <module>
    main()
  File "/home/eszter/research/brise-plandok/brise_plandok/convert.py", line 387, in main
    converter.convert(input_stream, output_stream)
  File "/home/eszter/research/brise-plandok/brise_plandok/convert.py", line 358, in convert
    self.write(doc, output_stream)
  File "/home/eszter/research/brise-plandok/brise_plandok/convert.py", line 348, in write
    self.write_json(doc, stream)
  File "/home/eszter/research/brise-plandok/brise_plandok/convert.py", line 324, in write_json
    stream.write(json.dumps(doc))
AttributeError: 'str' object has no attribute 'write'

Modify rule-based system according to label changes

See label changes here: https://github.com/recski/brise-nlp/issues/49 (Changes TBD in brise-plandok)

plandok.py leaves sections.sens.text null if cache is used

The following preprocessing step leaves the sections.sens.text attribute null if nlp_cache.json is used.

python brise_plandok/plandok.py sample_data/txt/*.txt > sample_data/json/sample.jsonl

If the cache file is deleted, and therefore regenerated on the next run, the json output is complete.

potato.create_dataset.py fails with new potato version + should now create csv

Two possible reasons identified: read_dataset expects a list but gets a numpy array, also potato now needs data files to have a .pickle extension. But using pickle is now deprecated, I suggest updating create_dataset.py so that it uses the new Dataset class in potato to save the data as csv.

refactor instances of xlsx generation to use shared module

see #13 (comment)

• Use a single source for categories & labels (e.g. merkmale_categories.csv ). Simplify label modification process
• utils for json I/O

Install puts 'export ALTO_JAR' 3 times into bash_profile invalid

After executing pip install . I have the following line in my .bash_profile:

export ALTO_JAR=/home/eszter/tuw_nlp_resources/alto-2.3.6-SNAPSHOT-all.jarexport ALTO_JAR=/home/eszter/tuw_nlp_resources/alto-2.3.6-SNAPSHOT-all.jarexport ALTO_JAR=/home/eszter/tuw_nlp_resources/alto-2.3.6-SNAPSHOT-all.jar