Documented `id_to_token` doesn't exist for UnicodeCodepointTokenizer

[fecundf]

Documented `id_to_token` doesn't exist for UnicodeCodepointTokenizer

[mehtamansi29]

Hi @fecundf -

I am getting this error "NameError: name 'sample_words' is not defined" while running your code snippet. Could you please provide me full code snippet for replicating the issue. Attached gist file for reference.

[fecundf]

Much simpler showing the problem (I edited the gist)

# This works
tokenizer=keras_nlp.tokenizers.WordPieceTokenizer(vocabulary=['Hi','there','people','[UNK]'])
print(tokenizer.id_to_token(2))

# This shows the missing id_to_token method
tokenizer=keras_nlp.tokenizers.UnicodeCodepointTokenizer()
print(tokenizer.id_to_token(2))

[fecundf]

even better example

# This works
tokenizer=keras_nlp.tokenizers.WordPieceTokenizer(vocabulary=['Hi','there','people','[UNK]'])
print(tokenizer.id_to_token(tokenizer.token_to_id("people"))) # shows "people"
print(tokenizer.id_to_token(2)) # also shows "people"

# This shows the missing id_to_token, token_to_id methods
tokenizer=keras_nlp.tokenizers.UnicodeCodepointTokenizer()
print(tokenizer.id_to_token(tokenizer.token_to_id("A"))) # should show "A"
print(tokenizer.id_to_token(65)) # should show "A"

[mehtamansi29]

Hi @fecundf

I have reproduced the issue by the latest code snippet and it seems like id_to_token method is not implemented in UnicodeCodePointTokernizer Class.
We need to check with keras-developer team. Thanks...!