Comments (8)
@ricardopieper
Can you share the model file for testing?
from nlpaug.
Yes, here it is:
http://143.107.183.175:22980/download.php?file=embeddings/fasttext/cbow_s50.zip
You can find more here: http://nilc.icmc.usp.br/embeddings
from nlpaug.
One of the offending lines start with "R$ 0,00", it explodes because 0,00 can't be parsed to float.
from nlpaug.
I changed the Fasttext model loader class with this code:
def read(self, file_path, max_num_vector=None):
with open(file_path, 'r', encoding='utf-8') as f:
header = f.readline()
self.vocab_size, self.emb_size = map(int, header.split())
for i, line in enumerate(f):
tokens = line.split()
word = " ".join(tokens[0:(len(tokens) - self.emb_size):])
values = np.array([float(val) for val in tokens[(self.emb_size*-1):]])
The idea is that, if we know the size of the word vectors, we can just load the last N splitted values and consider the rest as the word itself.
from nlpaug.
made a pull request describing the fix #8 feel free to evaluate the solution
from nlpaug.
One of the offending lines start with "R$ 0,00", it explodes because 0,00 can't be parsed to float.
Which pre-trained embeddings do you use and hitting this bug?
from nlpaug.
@makcedward the same I mentioned earlier. Here's a bit more context:
For portuguese word embeddings, we like to use USP's models (USP = Universidade de Sao Paulo, Brazil). They provide models with varied size. In particular we're using fasttext, though we could use any other (for our particular case, fasttext seems to be a bit better).
The one I'm using is this one: http://143.107.183.175:22980/download.php?file=embeddings/fasttext/cbow_s50.zip
You can find more here: http://nilc.icmc.usp.br/embeddings
Also I'm afraid the same fix has to be applied to all other models (glove, word2vec, etc), but I haven't checked.
from nlpaug.
@makcedward the same I mentioned earlier. Here's a bit more context:
For portuguese word embeddings, we like to use USP's models (USP = Universidade de Sao Paulo, Brazil). They provide models with varied size. In particular we're using fasttext, though we could use any other (for our particular case, fasttext seems to be a bit better).
The one I'm using is this one: http://143.107.183.175:22980/download.php?file=embeddings/fasttext/cbow_s50.zip
You can find more here: http://nilc.icmc.usp.br/embeddings
Also I'm afraid the same fix has to be applied to all other models (glove, word2vec, etc), but I haven't checked.
After studying pre-trained embeddings from http://nilc.icmc.usp.br/embeddings, found that word2vec, glove and fasttext embeddings follow same fasttext's (FB official embeddings) file format.
Will suggest to use FasttextAug to load those library. On the other hand,
One of the offending lines start with "R$ 0,00", it explodes because 0,00 can't be parsed to float.
Which pre-trained embeddings do you use and hitting this bug?
Will apply the following change to read content correctly.
def read(self, file_path, max_num_vector=None):
with open(file_path, 'r', encoding='utf-8') as f:
header = f.readline()
self.vocab_size, self.emb_size = map(int, header.split())
for i, line in enumerate(f):
tokens = line.split()
values = [val for val in tokens[(self.emb_size * -1):]]
value_pos = line.find(' '.join(values))
word = line[:value_pos-1]
values = np.array([float(val) for val in values])
from nlpaug.
Related Issues (20)
- ContextualWordEmbsAug: extra whitespace before and after punctuaion
- lambada augmenter: The following `model_kwargs` are not used by the model: ['num_workers'] HOT 2
- Unable to generate new data
- Lambada Model giving gibberish output
- Short description about different parameters used inside augmenter and what it does?
- RuntimeError: DataLoader worker (pid(s) 3954) exited unexpectedly
- LookupError when trying to augment data with SynonymAug
- What is the correct way to specify include_detail=True?
- does `ContextualWordEmbsAug` support XLM-Roberta models HOT 1
- The tokenizer class you load from this checkpoint is 'RobertaTokenizer'
- Augmenting Long Documents
- TypeError when utilizing GPU on Apple silicon for language models
- Actively maintained? HOT 3
- Error with English to French Backtranslation
- Bug on 1.1.11
- instancing more than one naw spelling aug with diferent dict_path doesn't work
- Feature Request: Progress bar during augmentation
- Not able to presever the words.
- Is support for facebook/bart-base removed in the latest version?
- Using nas.ContextualWordEmbsForSentenceAug with gpt2 or distilgpt2 returns nonsense
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from nlpaug.