shepdl/handle

handle's Issues

Create default regex tests

Create tests of default regex patterns for the application that would be useful users unfamiliar with regexes. Consider each regex by what it means to a human reader, not what the character pattern looks like. For example, "keep contractions together", not "word as a set of word characters plus single quotes."

Some examples:

1. All unicode "word" characters
2. Hyphenation-preserving regexes
3. Contraction-preserving regexes

Acceptance tests:

1. Set of patterns with meaningful names
2. Comprehensive suite of tests for all regexes that builds a SpacePreservingLexer with each regex and tests several cases including edge cases.

Separate SpacePreservingLexer tests into tests of basic lexing function and tests that perform actual application logic

Create new sets of tests for SpacePreservingLexer that test basic functionality of preserving gaps between what matches the pattern. Extract those tests that test specific regexes to another test suite.

Test and decide on default regex for lexers

The default regex choice for lexers is inconsistent throughout the code. Make these consistent.

Remove punctuation in word count table

SpacePreservingLexer still includes punctuation and some other items in the word count table

Mark spaces and removed characters in Preview window

Mark spaces and other removed characters in Preview window.

Key requirements:

1. Removed characters can include accented characters which can appear in the middle of a word. It must be clear that a character will be removed.
2. This should be accessible to colorblind users.

Some ideas:

1. Excluded characters can be struck through like stopwords
2. Excluded characters could appear in a different color that is very different from the surrounding character if we can find a working color combination

Preview window shows incorrectly-encoded characters on Windows

On Windows, Japanese text encoded in UTF-8 does not render correctly in the preview window.

Removed documents still show in topic models

When you add and then remove a document, it will still be included in a topic model.

Steps to reproduce:

1. Add a few documents to the document import list
2. Remove some of them
3. Create a new topic model
4. Select the topic model in the list of topics models
5. Click the "Docs" tab

All the documents you added in step 1 will be listed

Save and load projects

Allow users to save the state of a project and load it back in. This has a few stages:

1. Design a file format for saving documents and topic models
2. Implement saving and loading
3. Write comprehensive suite of tests for saving and restoring tests