f1uctus/ttc

✍ 🗣 A Text-To-Conversation natural language processing toolkit [WIP].

Text-To-Conversation toolkit (TTC)

This NLP library can help you with:

• Extraction of characters' replicas from literary texts;
• Identification of the actors owning these replicas.

Demo (CLI)

Progress

We aim to achieve the following goals:

• Better accuracy on the actor classification task (it is near 80% or worse for now);
• Support for more languages (only Russian is supported at the moment).

Installation

Install with pip is just a usual pip install . from the project dir.

Usage

As a library

You can find an example of using the library in the cli.py file.

As a tool for the CLI

Test output on a text file:

ttc print-play path-to-the-text-file text-language

Notes

• Text must be encoded in UTF-8;
• Text must be sanitized (see #23);
• It is usually better to test on some middle-sized text (e.g a book chapter);
• Supported text-languages are:
  • ru (russian)

Development

Please install Poetry.

Spawn a new virtual environment for the project:

poetry shell

Install project dependencies:

poetry install [--with dev,large_models_ru]

Contributions are very welcome!

Implementation notes

russian/*/actor_classifier.py:

ttc's Issues

Input text must be sanitized before processing

See also: explosion/spaCy#7735

Any text to be processed by ttc should be sanitized first.
Please remove any duplicate spaces and trim any whitespace before and after the line breaks. Do not replace line breaks with spaces as the replica extraction algorithm relies on them.
Excessive whitespaces are not handled well by the underlying library, spaCy, because, for example, the dependency parsing algorithm's training dataset does not contain them. As a consequence, prediction results may lose in accuracy.

If your use case requires the text to be left as-is, you can still sanitize the input, and then map the output spans into the original text by means of accumulated indices.

Improve actor identification quality

Maybe use actual trained NN instead of rule-based algorithm.

Use Span Categorizer

Restructure some functions to use the all-new spaCy's Span Categorizer.