Comments (5)
In my case, I do:
tokens = ['[UNK]', '[CLS]', '[SEP]', 'want', '##ed', 'wa', 'un', 'runn', '##ing', ',']
text = ' '.join([x for x in tokens])
fine_text = text.replace(' ##', '')
from transformers.
@thomwolf could you point to the specific section of run_squad.py
that handles this, I'm having trouble
EDIT: is it this bit from processors/squad.py
?
tok_to_orig_index = []
orig_to_tok_index = []
all_doc_tokens = []
for (i, token) in enumerate(example.doc_tokens):
orig_to_tok_index.append(len(all_doc_tokens))
sub_tokens = tokenizer.tokenize(token)
for sub_token in sub_tokens:
tok_to_orig_index.append(i)
all_doc_tokens.append(sub_token)
from transformers.
Yes. I don't plan to include a reverse conversion of tokens in the tokenizer.
For an example on how to keep track of the original characters position, please read the run_squad.py
example.
from transformers.
You can remove ' ##' but you cannot know if there was a space around punctuations tokens or uppercase words.
from transformers.
Apostrophe is considered as a punctuation mark, but often it is an integrated part of the word. Regular .tokenize()
always converts apostrophe to the stand alone token, so the information to which word it belongs is lost. If the original sentence contains apostrophes, it is impossible to recreate the original sentence from its' tokens (for example when apostrophe is a last symbol in some word convert_tokens_to_string()
will join it with the following one). In order to overcome this, one can check the surroundings of the apostrophe and add ##
immediately after the tokenization. For example:
sent = "The Smiths' used their son's car"
tokens = tokenizer.tokenize(sent)
now if you fix tokens
to look like:
original =>['the', 'smith', '##s', "'", 'used', 'their', 'son', "'", 's', 'car']
fixed => ['the', 'smith', '##s', "##'", 'used', 'their', 'son', "##'", '##s', 'car']
you will be able to restore the original words.
from transformers.
Related Issues (20)
- Support multiple `timm` as backbones with `forward_intermediates` method HOT 4
- Badam support HOT 1
- Can't load tokenizer HOT 1
- assisted_decoding does not correctly update static KV cache HOT 3
- Mamba + LoRA: after merge_and_unload() does not work well HOT 2
- [Bug] Error when trying to run two models in a machine with created ZeRO config variables. HOT 1
- Cannot load codeqwen Tokenizer HOT 4
- Unable to use set_weights() and reload model later
- Error on TPU: Invalid --2a886c8_slice_builder_worker_addresses specified. Expected 4 worker addresses, got 1. HOT 1
- idefics2-8b-AWQ failed when doing multiple calls HOT 2
- Cannot load AWQ model on GPU with deepspeed zero3 HOT 2
- If a training job job failed MLFlow will not be reported and MLFlow shows job still running HOT 1
- Add `device` parameter for PEFT `load_adapter` HOT 1
- [Finetuning OneFormer] Seems not to use multiple GPUs, with both DataParallel and Accelerate HOT 1
- Problem with pretrained Transformers in Offline mode HOT 3
- `generate` method does not work with `generation_config.use_cache=False` in `Llama-2` model HOT 1
- Question about VisionEncoderDecoderModel HOT 2
- RuntimeError: Expected one of cpu, cuda, ipu, xpu, mkldnn, opengl, opencl, ideep, hip, ve, fpga, ort, xla, lazy, vulkan, mps, meta, hpu, mtia, privateuseone device type at start of device string: auto HOT 3
- Static cache is locked after torch.compile with model.generate HOT 1
- Accelerator incorrectly gather batches if features do not have batch dim
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from transformers.