letuananh/chirptext

ChirpText is a collection of text processing tools for Python.

ChirpText is a collection of text processing tools for Python 3.

It is not meant to be a powerful tank like the popular NTLK but a small package which you can pip-install anywhere and write a few lines of code to process textual data.

Main features

• Simple file data manipulation using an enhanced open() function (txt, gz, binary, etc.)
• CSV helper functions
• Parse Japanese text with mecab library (Does not require mecab-python3 package even on Windows, only a binary release (i.e. mecab.exe) is required)
• Built-in "lite" text annotation formats (texttaglib TTL/CSV and TTL/JSON)
• Helper functions and useful data for processing English, Japanese, Chinese and Vietnamese.
• Application configuration files management which can make educated guess about config files' whereabouts
• Quick text-based report generation

Installation

chirptext is available on PyPI and can be installed using pip

pip install chirptext

Parsing Japanese text

chirptext supports parsing Japanese text using different parsers (mecab, Janome, and igo-python)

>>> from chirptext import deko
>>> sent = deko.parse('猫が好きです。')
>>> sent.tokens
['`猫`<0:1>', '`が`<1:2>', '`好き`<2:4>', '`です`<4:6>', '`。`<6:7>']
>>> sent.tokens.values()
['猫', 'が', '好き', 'です', '。']
>>> sent[0]
`猫`<0:1>
>>> sent[0].pos
'名詞'
>>> sent[1].lemma
'が'
>>> sent[2].reading
'スキ'

# tokenize
>>> deko.tokenize('猫が好きです。')
['猫', 'が', '好き', 'です', '。']

# split sentences
>>> deko.tokenize_sent("猫が好きです。\n犬も好きです。")
['猫が好きです。', '犬も好きです。']

# parse a document (i.e. multiple sentences)
>>> doc = deko.parse_doc("猫が好きです。\n犬も好きです。")
>>> for sent in doc:
...     print(sent, sent.tokens.values())
... 
猫が好きです。 ['猫', 'が', '好き', 'です', '。']
犬も好きです。 ['犬', 'も', '好き', 'です', '。']

Notes: At least one of the following tools must be installed to use chirptext Japanese parsing:

1. mecab: http://taku910.github.io/mecab/#download
2. Janome: available on PyPI, install with pip install Janome
3. igo-python: available on PyPI, install with pip install igo-python

Convenient IO APIs

>>> from chirptext import chio
>>> chio.write_tsv('data/test.tsv', [['a', 'b'], ['c', 'd']])
>>> chio.read_tsv('data/tes.tsv')
[['a', 'b'], ['c', 'd']]

>>> chio.write_file('data/content.tar.gz', 'Support writing to .tar.gz file')
>>> chio.read_file('data/content.tar.gz')
'Support writing to .tar.gz file'

>>> for row in chio.read_tsv_iter('data/test.tsv'):
...     print(row)
... 
['a', 'b']
['c', 'd']

Sample TextReport

# a string report
rp = TextReport()  # by default, TextReport will write to standard output, i.e. terminal
rp = TextReport(TextReport.STDOUT)  # same as above
rp = TextReport('~/tmp/my-report.txt')  # output to a file
rp = TextReport.null()  # ouptut to /dev/null, i.e. nowhere
rp = TextReport.string()  # output to a string. Call rp.content() to get the string
rp = TextReport(TextReport.STRINGIO)  # same as above

# TextReport will close the output stream automatically by using the with statement
with TextReport.string() as rp:
    rp.header("Lorem Ipsum Analysis", level="h0")
    rp.header("Raw", level="h1")
    rp.print(LOREM_IPSUM)
    rp.header("Top 5 most common letters")
    ct.summarise(report=rp, limit=5)
    print(rp.content())

Output

+---------------------------------------------------------------------------------- 
| Lorem Ipsum Analysis 
+---------------------------------------------------------------------------------- 
 
Raw 
------------------------------------------------------------ 
Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia deserunt mollit anim id est laborum. 
 
Top 5 most common letters
------------------------------------------------------------ 
i: 42 
e: 37 
t: 32 
o: 29 
a: 29 

Useful links

• Documentation: https://chirptext.readthedocs.io
• Source code: https://github.com/letuananh/chirptext/
• PyPI: https://pypi.org/project/chirptext/

chirptext's Issues

Asking for a new release on PyPi

Hi,

Version 0.1a18 is a bit outdated, could you update a newer version to PyPi?

I only need this commit, but since long time is passed i think most of the master change are stable.

chio module, move mode before encoding

Reason: encoding is almost always UTF-8 but default file access mode is changed quite often

Add support for Leipzig and Penn Treebank tagset

• Leipzig
  • Reference: https://www.eva.mpg.de/lingua/resources/glossing-rules.php
• Penn Treebank tag set
  • Version 1: https://www.sketchengine.eu/penn-treebank-tagset/
  • Version 2: https://www.sketchengine.eu/english-treetagger-pipeline-2

failed to import HIRAGANA and KATAKANA from chirptext.deko

This code from jamdict failed to run with chirptext 0.2a2

from chirptext.deko import HIRAGANA, KATAKANA
...

Revamp TTL APIs for more complex usecases

• simplify multi-tag handling (i.e. sense candidates, chunk languages, annotators, etc.)
• Built-in support for CoNLL
• use first tag slot for scalar tags (i.e. POS, lemma, surface, languages)
• Re-design TTL JSON

Add dialect support to TextReport

• Markdown
• RST
• org-mode
• HTML?
• LaTeX?

add chunk support to TTL for BELA

Chunks are

• taggable (i.e. languages, fonts, etc.)
• optional links to tokens (or inferred from (cfrom, cto)
• Ref: https://blipntu.github.io/belacon/

use TTL 2.0 for all deko-* parses

• dekomecab
• dekoigo
• dekohebi

support more Japanese parsers

Order of engines to use

1. mecab
2. janome
3. igo-python

Release chirptext 0.1

review ttl.Sentence.fix_cfrom_cto() function

review ttl.Sentence.fix_cfrom_cto() function

Determine whether to lose, migrate somewhere else, or refactor it for more general cases?

Update documentation about new deko API

Make sure that there is no hard-coded logging to INFO in any file

add encoding keyword to chio.write_csv and chio.read_csv

Consider to port back AppConfig.potentials() from version 0.2

chirptext/chirptext/leutile.py

Line 613 in 12a8997

LOC_TEMPLATE = ['{wd}/.{n}.{mode}', '{wd}/{n}.{mode}',

FileNotFoundError [WinError 2] The system cannot find the file specified

>>> from chirptext import deko
>>> sent = deko.parse('猫が好きです。')
>>> sent.tokens
===========================
FileNotFoundError                         Traceback (most recent call last)
<ipython-input-15-bd2eed5e1672> in <module>
      1 from chirptext import deko
----> 2 sent = deko.parse('猫が好きです。')
      3 sent.tokens

~\Anaconda3\lib\site-packages\chirptext\deko.py in txt2mecab(text, **kwargs)
    250 def txt2mecab(text, **kwargs):
    251     ''' Use mecab to parse one sentence '''
--> 252     mecab_out = _internal_mecab_parse(text, **kwargs).splitlines()
    253     tokens = [MeCabToken.parse(x) for x in mecab_out]
    254     return MeCabSent(text, tokens)

~\Anaconda3\lib\site-packages\chirptext\dekomecab.py in parse(content, *args, **kwargs)
     65         return MeCab.Tagger(*args).parse(content)
     66     else:
---> 67         return run_mecab_process(content, *args, **kwargs)
     68 
     69 

~\Anaconda3\lib\site-packages\chirptext\dekomecab.py in run_mecab_process(content, *args, **kwargs)
     55     output = subprocess.run(proc_args,
     56                             input=content.encode(encoding),
---> 57                             stdout=subprocess.PIPE)
     58     output_string = os.linesep.join(output.stdout.decode(encoding).splitlines())
     59     return output_string

~\Anaconda3\lib\subprocess.py in run(input, capture_output, timeout, check, *popenargs, **kwargs)
    464         kwargs['stderr'] = PIPE
    465 
--> 466     with Popen(*popenargs, **kwargs) as process:
    467         try:
    468             stdout, stderr = process.communicate(input, timeout=timeout)

~\Anaconda3\lib\subprocess.py in __init__(self, args, bufsize, executable, stdin, stdout, stderr, preexec_fn, close_fds, shell, cwd, env, universal_newlines, startupinfo, creationflags, restore_signals, start_new_session, pass_fds, encoding, errors, text)
    767                                 c2pread, c2pwrite,
    768                                 errread, errwrite,
--> 769                                 restore_signals, start_new_session)
    770         except:
    771             # Cleanup if the child failed starting.

~\Anaconda3\lib\subprocess.py in _execute_child(self, args, executable, preexec_fn, close_fds, pass_fds, cwd, env, startupinfo, creationflags, shell, p2cread, p2cwrite, c2pread, c2pwrite, errread, errwrite, unused_restore_signals, unused_start_new_session)
   1170                                          env,
   1171                                          os.fspath(cwd) if cwd is not None else None,
-> 1172                                          startupinfo)
   1173             finally:
   1174                 # Child is launched. Close the parent's copy of those pipe

FileNotFoundError: [WinError 2] The system cannot find the file specified