Comments (9)
@o0windseed0o can you paste your complete config file here? It's probably an iteration boundary related bug somewhere in our code.
from matchzoo.
@uduse Thanks for your reply! Please see the following.
{
"net_name": "match_pyramid",
"global":{
"model_type": "PY",
"weights_file": "examples/QA/weights/matchpyramid_classify.weights",
"save_weights_iters": 10,
"num_iters": 200,
"display_interval": 10,
"test_weights_iters": 200,
"optimizer": "adam",
"learning_rate": 0.0001
},
"inputs": {
"share": {
"text1_corpus": "./data/QA/corpus_preprocessed.txt",
"text2_corpus": "./data/QA/corpus_preprocessed.txt",
"use_dpool": true,
"embed_size": 100,
"train_embed": true,
"vocab_size": 28780,
"target_mode": "classification",
"class_num": 2,
"text1_maxlen": 25,
"text2_maxlen": 50
},
"train": {
"input_type": "PointGenerator",
"phase": "TRAIN",
"use_iter": false,
"query_per_iter": 20,
"batch_per_iter": 5,
"batch_size": 100,
"relation_file": "./data/QA/relation_train.txt"
},
"valid": {
"input_type": "PointGenerator",
"phase": "EVAL",
"batch_size": 100,
"relation_file": "./data/QA/relation_train.txt"
},
"test": {
"input_type": "PointGenerator",
"phase": "EVAL",
"batch_size": 100,
"relation_file": "./data/QA/relation_test.txt"
},
"predict": {
"input_type": "PointGenerator",
"phase": "PREDICT",
"batch_size": 50,
"relation_file": "./data/QA/relation_test.txt"
}
},
"outputs": {
"predict": {
"save_format": "TEXTNET",
"save_path": "predict.test.medqa_matchpyramid_classify.txt"
}
},
"model": {
"model_path": "matchzoo/models/",
"model_py": "matchpyramid.MatchPyramid",
"setting": {
"kernel_count": 32,
"kernel_size": [3, 3],
"dpool_size": [3, 10],
"dropout_rate": 0.5
}
},
"losses": [
{
"object_name": "categorical_crossentropy",
"object_params": {}
}
],
"metrics": [ "accuracy" ]
}
There are several paramters in the config file that I don't know how to set, such as query_per_iter and bath_per_iter. Are there any instructions or introductions on how to write config files?
As to the error, if it's not related to the config file, it might be the operation on generating batches, since the missing samples are always the last ones.
from matchzoo.
Have you figured out what caused the problem? Or can anybody tell me which py file should be checked, related to the batch generator?
from matchzoo.
@o0windseed0o haven't got a chance to dive into the problem yet. You might want to look at PointGenerator
class.
from matchzoo.
@uduse I have checked the PointerGenerator class, and I think the problem should be around the while True loop in the get_batch_generator function. There are no operations on how to deal with those samples that cannot be built up to a batch. Please check from here when you are free. Thank you!
from matchzoo.
Need to figure out whether a bug exist.
from matchzoo.
Today I come to the same situation: I have 5000 prediction samples, and it output 4998 predicts.
But no matter how I change my batch_size, the output is always 4998.
Then later I found that there's duplicate samples in my prediction sample.
@o0windseed0o Not sure if you have the duplicate sample.
@bwanglzu By the way, can the model fixed the duplicate situation?
from matchzoo.
Apparently there's something wrong in Generator
, I guess @faneshion and @yangliuy are the right person to ask.
@Genie-Liu and you provide a bit more context?
from matchzoo.
@faneshion any ideas?
from matchzoo.
Related Issues (20)
- ImportError: cannot import name 'losses_utils' HOT 2
- FileNotFoundError: [Errno 2] No such file or directory: '\\site-packages\\matchzoo\\datasets\\toy\\train.csv' HOT 2
- Is there a tutorial on the ANMM model? HOT 2
- Is there any method to change csv to data_pack? HOT 2
- Do I have to make label=0 data when I make new datapack?
- Using callbacks for early stopping in DSSM
- Hello, as long as I use a structure with a convolutional layer, memory overflow will occur for small data. How to solve the memory overflow? HOT 1
- ChineseTokenize疑惑 HOT 1
- The use of the GPU.
- Import Error HOT 1
- How to setting learning rate in the model params??
- matchzoo.contrib.models.ESIM(), model.save, raise ValueError: substring not found HOT 1
- keras should be replaced by tf.keras HOT 6
- GPU-Utils is low 1%
- set_up.py missing tensorflow HOT 1
- DSSM model.predict() scores rank does not match with the rank by dot layer cosine similarity
- Can Deep Component (Representation-focused model) is there in matchzoo?
- Bug/enhancement HOT 2
- TypeSpec error while DRMM model build
- MatchACNN
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from matchzoo.