Comments (12)
Hi Moe,
I'm not familiar with the AWS async DynamoDB client. Could you explain a little what benefit you'd get from wrapping the async client over, say, just using Clojure's own async capabilities like core.async
?
Would definitely be open to a non-breaking PR if it brought something valuable.
Cheers! :-)
from faraday.
I think non-blocking I/O with DynamoDB would be highly beneficial - e.g. see here Martin Trojer's blog post on core.async+netty http://martintrojer.github.io/clojure/2013/07/07/coreasync-and-blocking-io/.
from faraday.
@tlipski Hi Tomek, what makes you think you can't use core.async with the current DynamoDB client?
from faraday.
We can - but it will block a thread, right?
from faraday.
Well, a logical thread. core.async lets you decide whether you'd like to spend an actual system thread, or a go "thread" (parking FSM).
The async DynamoDB client appears to just operate with futures, so actually gives you fewer options from what I can tell (I may be missing something, just skimmed the docs).
Keep in mind that one of the nice things about core.async is that it lets you make any synchronous activity asynchronous - and in a particularly pleasant and flexible way.
Does that make sense?
from faraday.
Well, I think you need to have a separate thread if your code uses java.io underneath (and that's what is being in AWS sync client I think).
But maybe I am missing something obvious - can you show some code example where no threads are blocked during communication between the JVM and AWS endpoint?
from faraday.
The Martin Trojer article is a good overview.
As Tomek said, you need an actual system thread on which to perform the I/O. The one thing in core.async which is useful for a case like this, is thread
, in so far as it allows you to retrieve the result without blocking the consumer, but it uses an unbounded thread pool / doesn't give you any control of how operations are scheduled. Using it as a general mechanism for handling blocking I/O would be a mistake.
In addition to futures, the async Dynamo API allows callbacks to be provided, which means no deref-ing/blocking in the caller. Separately, I don't necessarily think channels are the greatest fit for many APIs, because of their focus on the streaming of multiple values, and poorly thought-through error handling. I was considering using Manifold for result delivery, because it's lightweight, and has little to no conceptual overhead. I'm curious to know what you think.
from faraday.
Well, I think you need to have a separate thread if your code uses java.io underneath (and that's what is being in AWS sync client I think).
@tlipski Sure, in the case of go
blocks, you'll be blocking one of the threads in the pool that services the go state machine. You'd need to be careful doing that, which is why explicit thread
s are normally recommended for i/o. I believe it's the intention at some point for go
to issue warnings against blocking i/o calls.
In addition to futures, the async Dynamo API allows callbacks to be provided, which means no deref-ing/blocking in the caller.
Had missed that - thought they were just providing futures/promises for everything. Could you show me what docs you're looking at?
I was considering using Manifold for result delivery, because it's lightweight, and has little to no conceptual overhead.
I wouldn't be keen on Manifold in Faraday, sorry. But my guess is that we'd probably be fine with just futures+promises, possibly some core.async for paged requests if we wanted to expose a streaming view on long-running jobs.
Let's start by making sure that we're looking at the same docs and go from there? All I could find was this, but it's deprecated: http://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/dynamodb/AmazonDynamoDBAsyncClient.html
Would also want to check their implementation to make sure we'd actually be getting some real benefit with the callback interface over just letting clients use futures/promises/core.async as they like.
Does that make sense?
Thanks a lot for the input, much appreciated! :-)
from faraday.
Ahh, found the v2 docs: http://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/dynamodbv2/package-summary.html
from faraday.
@ptaoussanis Thanks for the response. I think promise
and future
aren't a good fit for pervasively asynchronous applications, because they cannot be realized without blocking. They would provide the option of having a very thin (third-party) wrapper around Faraday which could convert them into Manifold-like deferred objects, but this would require even more processing in threads (by Manifold).
It would be preferable, I think, to either return a core.async
channel from every function, or just to accept continuations in the I/O-performing functions. Either of those could be converted into something higher-level downstream in a non-blocking way.
Alternatively, the PR could just cover the defprotocol & the factoring of the request-object generating methods, and the alternate API could live somewhere else.
from faraday.
It would be preferable, I think, to either return a core.async channel from every function, or just to accept continuations in the I/O-performing functions.
If the DDB async API can accept callbacks, then I'd maybe suggest we just accept callbacks ourselves? Then folks can decide for themselves what async architecture they'd like to employ (be it delivering a promise, put!
ing to a core.async channel, etc.) This'd also let us avoid picking up any extra dependencies.
Alternatively, the PR could just cover the defprotocol & the factoring of the request-object generating methods, and the alternate API could live somewhere else.
My hands are quite full atm; would be happy to leave this to your judgement.
from faraday.
Closing this for now, we can open a fresh issue later if you (or someone else) would like to bring an async API directly to Faraday.
from faraday.
Related Issues (20)
- CI integration HOT 5
- Remove support for `:expected` HOT 3
- attribute_not_exists not respected consistently HOT 3
- Support for Streams HOT 8
- missing ScanIndexForward option with far/scan HOT 1
- Request: custom keywordization routine HOT 3
- Is the scan function safe for large tables? HOT 7
- Upgrade to 1.11.16+ of aws-sdk-java for task-level IAM role support in AWS ECS HOT 9
- Add support for DynamoDB Transactions HOT 6
- Upgrade to 1.11.459 of aws-sdk-java for management of On Demand tables
- Upload all doc versions to http://taoensso.github.io/faraday/{version}
- Porting to AWS SDK v2 HOT 16
- missing page-size/max-result-size option with query HOT 1
- Support empty strings HOT 4
- not listing tables HOT 14
- Setting `region` in client-opts causes ClassCastException
- String index names cause type error HOT 3
- Local docker example from readme only with `-p 8000:8000` arg HOT 1
- Possibly confusing dependencies found HOT 3
- FYI: have pushed some superficial changes HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from faraday.