Comments (4)
I'm sorry that I don't have enough time to work on this project right now, but I will come back to this and finish TODO lists as soon as possible.
I'm not sure that I understand your idea correctly. Does it mean one forward/backward pass with N*M-sized data blob or M forward/backward passes with N-sized data blob?
In the former case, we have to consider the memory limit. For example, 100-sized 20 mini-batches is roughly equivalent to 2000 mini-batches in feedforward networks in terms of memory usage.
In the latter case, we have to compute the sum of gradients over several forward/backward passes. The main problem is that every layer does not preserve gradient(diff) after one backward pass, because one backward pass always involves one weight update in caffe.
I hope we will find a clever way to deal with this issue.
Thank you for sharing your idea!
from caffe-lstm.
Thank you for your detail explanation. My question was about the former approach as you said. Some researches seem to use 4-size(quite small) minibatch for RNN update. If '4' is not so bad for mini batch size, memory requirement doesn't seem to be as a big problem. For the latter one M-fwd and 1-bwd seems to be sufficient to BP. Isn't it? 1) accumulate gradient for each weight during grouped forward passes 2) and back propagate to achieve large sized mini-batch with 'physical' small-batch.
from caffe-lstm.
The former approach seems to be a simple and good option.
Thank you for your comment!
from caffe-lstm.
This implementation supports mini-batch update now.
from caffe-lstm.
Related Issues (20)
- How to leverage the large-scale unlabeled data to train the LSTM model?
- Question about 'lr_mult' HOT 3
- Meaning of label in lstm_short example HOT 2
- why the batch_size is a parameter of the layer HOT 3
- Use of default clip markers as [0,1,1, 1...,1] HOT 7
- why not the lstm_swquence.bin file HOT 1
- Loss layer calculation
- Is caffe command line available?
- 'clipping_threshold' LSTM parameter same at 'clip_gradient' Caffe parameter? HOT 7
- Does the batch_size have to be at least as large as the sequence length ? HOT 2
- Issue with linking HOT 5
- How to do Many-to-One LSTM learning? HOT 7
- How to build many to one LSTM HOT 1
- Variable-sized input sequence HOT 1
- Suggestions for implementing CWRNN in Caffe with c++
- Multi-inputs to one output HOT 1
- lstm_layer.cu:172] Check failed: error == cudaSuccess (9 vs. 0) invalid configuration argument
- Using LSTM for text generation in caffe
- bug? HOT 2
- running on android
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from caffe-lstm.