jiegec/tantivy-jieba

An adapter that bridges between tantivy and jieba-rs.

tantivy-jieba

An adapter that bridges between tantivy and jieba-rs.

Usage

Add dependency tantivy-jieba to your Cargo.toml.

Example

use tantivy::tokenizer::*;
let mut tokenizer = tantivy_jieba::JiebaTokenizer {};
let mut token_stream = tokenizer.token_stream("测试");
assert_eq!(token_stream.next().unwrap().text, "测试");
assert!(token_stream.next().is_none());

Register tantivy tokenizer

use tantivy::schema::Schema;
use tantivy::tokenizer::*;
use tantivy::Index;
let tokenizer = tantivy_jieba::JiebaTokenizer {};
let index = Index::create_in_ram(schema);
index.tokenizers()
     .register("jieba", tokenizer);

License

tantivy-jieba's Issues

begin <= end (12 <= 6) when slicing

fn highlight(snippet: Snippet) -> String {
let mut result = String::new();
let mut start_from = 0;

for fragment_range in snippet.highlighted() {
    result.push_str(&snippet.fragment()[**start_from..fragment_range.start**]);
    result.push_str("<font color='red'>");
    result.push_str(&snippet.fragment()[fragment_range.clone()]);
    result.push_str("</font>");
    start_from = fragment_range.end;
}

result.push_str(&snippet.fragment()[start_from..]);
result

}

Unable to use token_stream

Hi,

I followed your example:

use tantivy::tokenizer::*;
let tokenizer = tantivy_jieba::JiebaTokenizer {};
let mut token_stream = tokenizer.token_stream("测试");
assert_eq!(token_stream.next().unwrap().text, "测试");
assert!(token_stream.next().is_none());

but it gave me error in this line:
tokenizer.token_stream("测试");

The error msg is:

no method named `token_stream` found for struct `JiebaTokenizer` in the current scope
items from traits can only be used if the trait is in scope`

How can I fix that?

Many thanks

offset_to and offset_from need to be byte offsets

Hi,

coming from this issue: quickwit-oss/tantivy#1134

tokens.push(Token {
                offset_from: token.start,
                offset_to: token.end,
                position: token.start,
                text: String::from(&text[(indices[token.start].0)..(indices[token.end].0)]),
                position_length: token.end - token.start,
});

this should probably be

tokens.push(Token {
                offset_from: indices[token.start].0,
                offset_to: indices[token.end].0,
                position: token.start,
                text: String::from(&text[(indices[token.start].0)..(indices[token.end].0)]),
                position_length: token.end - token.start,
});

the trait bound `JiebaTokenizer: Tokenizer` is not satisfied

use tantivy::schema::Schema;
use tantivy::tokenizer::*;
use tantivy::Index;
let tokenizer = tantivy_jieba::JiebaTokenizer {};
let mut schema_builder = Schema::builder();
let schema = schema_builder.build();
let index = Index::create_in_ram(schema);
index.tokenizers()
.register("jieba", tokenizer);

the trait bound JiebaTokenizer: Tokenizer is not satisfied
the following other types implement trait Tokenizer:
FacetTokenizer
NgramTokenizer
RawTokenizer
SimpleTokenizer
WhitespaceTokenizer
required for TextAnalyzer to implement