Comments (2)
Hi Riaan,
There are several possible reasons:
-
I sometimes observed gradient explosions in Theano (in general). Most of them are caused by log-softmax or log-sigmoid operations. Calculating the gradient wrt. these operations can result in explosion. These can be resolved with better coding, such that Theano is aware of automatically optimizing log-softmax / log-sigmoid operations into softplus (http://theano.readthedocs.io/en/0.8.x/crei2013/theano.html). I don't know if TF has this feature, but this can be done manually.
-
For the rationale model, I've seen the issue you observed (assuming it is not due to the above reason). Since the gradient is sampled and approximated, my understanding is the model may experience "bad gradient" due to sampling variance.
I used a big batch size of 256 for the beer review dataset. Another better but more complicated solution is to use variance reduction tricks. This is a common practice for reinforcement learning, especially policy gradient learning. You can find a lot materials / tutorials online. As an example, this follow-up paper by Jiwei (https://arxiv.org/pdf/1612.08220.pdf) uses such trick.
- If you work on a different dataset, you need to tune the lambda regularizers. The model is quite sensitive to the values. My personal experience is to fix lambda_2 to a small value (0~1). And then find a good choice of lambda_1 such that the model doesn't end up with producing all 1's or all 0's for z.
Hope these can help!
Tao
from rcnn.
They have helped, thanks! The implementation of the dependent model can be found here:
https://github.com/RiaanZoetmulder/Master-Thesis/tree/master/rationale
I have some results on the small dataset. Curious to see what you think of them :)
from rcnn.
Related Issues (17)
- Specifications of machine to run the codes HOT 1
- Error while running main.py HOT 1
- Disparity in BM25 performance HOT 3
- when run main.py in "qa", get the result p_norm:['nan', 'nan','nan','nan','nan'] HOT 10
- Why the condition in LSTM/GRU model is x.ndim > 1 but 2? HOT 7
- Model not saved HOT 3
- Medical dataset HOT 2
- Generator converging to wrong output HOT 1
- qa/api.py example usage HOT 2
- Producing Rationales HOT 3
- Observation: code is only for python 2?
- Dataset layout? HOT 1
- dev loss HOT 1
- Rationale evaluation
- Where is model.py in the sentiment directory?
- UnicodeDecodeError
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from rcnn.