Comments (7)
I think it can be closed; it's reasonably well documented already via the AWS docs and the :limit
parameter.
from faraday.
scan
will not actually scan the whole table. It will return a (strict) array of elements, with some metadata. One of the meta-data key is last-prim-kvs
. If this is not null, it means the scan isn't complete and you'll have to call scan
again supplying this last-prim-kvs
to continue scanning.
I have code looking a bit like that:
(defn scan-table
[callback]
(loop [kvs :none]
(let [query (if (= :none kvs) {} {:last-prim-kvs kvs})
results (faraday/scan opts table-name query)
next-kvs (:last-prim-kvs (meta results))
]
(callback results)
(when-not (nil? next-kvs) (recur next-kvs)))))
Don't forget to eventually add a :limit
option and use a rate limiter to avoid blowing up through your provisionned capacity.
from faraday.
Here's a similar approach to @geekingfrog, but operating semi-lazily:
(defn lazy-scan
([client-opts table] (lazy-scan client-opts table nil))
([client-opts table opts]
(lazy-seq
(let [results (faraday/scan client-opts table opts)
next-kvs (:last-prim-kvs (meta results))]
(if next-kvs
(lazy-cat results (lazy-scan client-opts table (assoc opts :last-prim-kvs next-kvs)))
results)))))
I say semi-lazily, because production will stay ahead of consumption, but in chunks related somewhat to whatever initial :limit
value is specified in opts
.
from faraday.
@brosenan and @rwilson are we satisfied here? Can this one be closed? Do you suggest something be adding to the docs around scan?
from faraday.
Sounds good @rwilson
from faraday.
I think that the original poster's question have been misunderstood. He went to read the source code, found the function merge-more
, expressed concerns about what it does, but nobody answered about that function w.r.t. lazyness.
from faraday.
@brosenan To get just a piece of the query's result, you need to specify {:limit n, :span-reqs {:max m}}
in the options. You will get up to (* n m)
items in your results, i.e. m
requests of up to n
items concatenated together in the result.
By default, (-> options :span-reqs :max)
is set to 5 in the current version of Faraday (1.9.0
).
from faraday.
Related Issues (20)
- CI integration HOT 5
- Remove support for `:expected` HOT 3
- attribute_not_exists not respected consistently HOT 3
- Support for Streams HOT 8
- missing ScanIndexForward option with far/scan HOT 1
- Request: custom keywordization routine HOT 3
- Upgrade to 1.11.16+ of aws-sdk-java for task-level IAM role support in AWS ECS HOT 9
- Add support for DynamoDB Transactions HOT 6
- Upgrade to 1.11.459 of aws-sdk-java for management of On Demand tables
- Upload all doc versions to http://taoensso.github.io/faraday/{version}
- Porting to AWS SDK v2 HOT 16
- missing page-size/max-result-size option with query HOT 1
- Support empty strings HOT 4
- not listing tables HOT 14
- Setting `region` in client-opts causes ClassCastException
- String index names cause type error HOT 3
- Local docker example from readme only with `-p 8000:8000` arg HOT 1
- Possibly confusing dependencies found HOT 3
- FYI: have pushed some superficial changes HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from faraday.