glaserl/glassplitter

A sentence splitter and tokenizer.

glassplitter's Issues

Warn and/or trim overly long sentences

DEBUG:main:Analyzing pdf 0909.3986
Token indices sequence length is longer than the specified maximum sequence length for this model (673 > 512). Running this sequence through the model will result in indexing errors

Split Sentences begin with an empty token, which is associated to the wrong span.

Low priority, but maybe it's an easy fix:

In my tests, split sentences always(?) begin and end with an empty '' token, that is associated with the previous span.

(The first sentence has no previous span, and there it attaches to the first span in the sentence)

Das betrifft anscheinend nur leere Tokens, die ich vermutlich sowieso, ignorieren kann.

tokenizer.split() occasionally returns the wrong span

Der folgende Test schlägt für "Different" fehl.
(Und für den leeren String davor auch, aber so sophisticated ist mein assert nicht.)

def test_glassplitter():
    tokenizer = Tokenizer(lang="en", clean=True, doc_type="pdf")
    example = ["I am a sentence with trailing space. ", "Different Box", " from the rest of the sentence.Hello there.", "StudNr something wrt TUD & KHG"]
    example_input = [(text, {"some_data": "schwund is immer"}) for text in example]

    for i, a in enumerate(tokenizer.split(example_input, trim=True)):
        logger.debug("Group %s", i)
        for j, (text, span) in enumerate(a):
            logger.debug("%s-%s: %s '%s'", i, j, span, text)
            assert text in example[span]

Add splitter interface for use with sklearn

The TF-IDF Vectorizer expects a function/callable with the signature
str -> list[str]
I request a function (let's call it tokenize_flat or something) that expects a list/iterable of strings (instead of tuples) and returns a list/iterable of strings. (instead of a nested list of enumerated strings)

My kinda messy workaround for reference:

tokenizer = Tokenizer(lang="en", clean=True, doc_type="pdf")

def tokenize_only(tokenizer: Tokenizer, text_boxes):
    """Tokenize text-boxes."""
    spans = ((item["text"], i) for i, item in enumerate(text_boxes))
    sentence_list = tokenizer.split(spans, trim=True)

    for sentence in sentence_list:
        yield [a for a, i in sentence]

def tokenizer_wrapped(tokenizer: Tokenizer) -> Callable:
    def tokenize(arg: str):
        return list(reduce(operator.add, tokenize_only(tokenizer, arg)))
    return tokenize

# print("Vectorizing %d documents..." % len(data))
tfidf_vectorizer = TfidfVectorizer(
    max_df=0.5, # Percentage of docs
    min_df=3, # Absolute number of docs to contain feature (word)
    max_features=5000,
    use_idf=True,
    sublinear_tf=True,
    preprocessor=lambda x: x, #HACK
    tokenizer=tokenizer_wrapped(tokenizer),
    # tokenizer=str.split,
    token_pattern=None,
    stop_words="english",
    ngram_range=(1, 2),
)

Batch Tokenization