Comments (1)
You can create one table which contains variable lengths. Both the old Writer/Sampler and the new Trajectory Writer/Sampler allow it (if you provide a signature to the dataset, simply set the outer/time dimension to None
in Python).
But now you additionally have the problem of batching and training from a stream of variable episode lengths.
You have roughly 3 options from a tf.data
perspective:
- padded-batch (what you're using)
- bucket-by-sequence-length (what you're suggesting except move the bucketing into tf.data streams and out of tables; but using multiple tables works too!)
- chunk-and-shuffle (chop all episodes into fixed sized pieces and shuffle those, not bothering to train on full eps)
Chunk-and-shuffle often works well in dense reward scenarios. Here's an example:
stream = dataset.unbatch()
# Since we're generally reading from a never-ending replay buffer, we
# can drop remainder here and get the benefit of usually having known
# episode lengths and batch sizes.
chunks = stream.batch(chunk_size, drop_remainder=True)
# reshuffle_each_iteration will probably never be used since replay
# buffers are based on never-ending datasets, but we add it just in case.
shuffled = chunks.shuffle(
shuffle_buffer_size, reshuffle_each_iteration=True, seed=seed)
batched = shuffled.batch(batch_size, drop_remainder=True)
return batched
The bucket-by-sequence-length approach can be combined with padding-up-to-fixed-lengths-per-bucket (allowing fixed shaped graphs, which is useful if you're training with TPUs). The tf.data
function that helps you do this is bucket_by_sequence_length.
In all cases, you can accelerate by creating multiple reverb datasets (flexible_batch_size=1 in each) and using tf.data's interleave (ACME's reverb dataset code does this).
from reverb.
Related Issues (20)
- Error in `SampleToInsertRatio` docstring HOT 3
- table signature (shapes) must be same for each field of structured data? HOT 2
- Sampling from a loaded replay checkpoint HOT 4
- TensorFlow ROCm support? HOT 1
- Maintaining episode-level priority while sampling individual transitions from a replay buffer with MinHeap remover
- TFclient Error Message about data loss HOT 1
- import reverb error,raise TypeError, 'Regexp cannot be negated' HOT 2
- print samples with high priority HOT 6
- Access to keys? HOT 1
- Initialize or Clean memory for on-policy HOT 1
- c++ use python to import reverb error
- Use With Vectorized Environments HOT 6
- C++ Documentation
- [Conceptual Question] Item creation and sampling trajectories with specified period HOT 1
- Can I change the rate limiter after I have created the table HOT 2
- Unnecessary assert in prioritized selector HOT 1
- How to change Table max_time_sampled param when reloading ?
- How to use multiple clients - for multiple envs in parallel settings ?
- How to interpret warning on Table access with/without gRPC HOT 2
- Reverb fails to install HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from reverb.