hunan-rostomyan/index

Documentation indexing tool

index's Issues

Don't get rid of all dashes

For example: 'self-signed'

=> 'self signed' (remove punctuation)
=> ['self', 'signed'] (tokenize)
=> ['signed'] (remove stops)

Ideally we'd want to preserve the whole thing. One idea DK suggested is to transform the query into:

=> ['self', 'signed', 'self-signed'] (tokenize, dashed preserved)
=> ['signed', 'self-signed'] (remove stops)