Comments (16)
Just FYI @barkanido , stewardship of Faraday has largely been passed on at this point. I suspect @ptaoussanis is happy to weigh in on important matters here and there, but isn't going to take on a big project like an API port.
@kipz and @joelittlejohn have done a lot of the great maintenance work lately for example.
It'd probably be helpful to build up a case of everything Faraday is missing out on by not porting so we can make an informed decision as a community. Auto-pagination for example sounds interesting, but non-blocking IO can be achieved fairly simply by the Faraday user (or by Faraday) on the Clojure side. It's unclear that we need that from the AWS API (maybe you can set me straight on that).
from faraday.
I think the killer app for 2.0 is the async support. I do agree that 'non-blocking IO can be achieved fairly simply by the Faraday user (or by Faraday) on the Clojure side' however it's also the nature of non-blocking solutions that the further you can be non-blocking all the way down the better. Right now, when you wrap blocking libraries for non-blocking use, you inevitably need to reserve more resources to do so. Project Loom will solve this.
I'm not sure there has been mass adoption of the AWS SDK 2.x though, particularly in the Clojure community. Making this move too early can cause a lot of headaches for people building apps that need to interact with a few different AWS services (Dynamo plus some others). The fact that 2.x exists and was not created for fun isn't really a compelling reason to migrate. I don't think the features that are missing from 2.x are a problem for us, it's more a question of what will cause least disruption for everyone and do we have enough users that need this? There's also a movement in the community away from the Java SDKs towards e.g. cognitect/aws-api so we might decide to just go that way instead.
Overall I'm not inclined to go ahead and start migrating. I'd like to see this issue get more comments and support from users before doing it.
from faraday.
Thanks @joelittlejohn . I do agree that this is something that needs a discussion in a wider forum of users.
from faraday.
Closing this one as I don't think it is a goal. I think it's more likely we would migrate to cognitect/aws-api in future.
from faraday.
Good to know that there is a solid community behind Faraday.
It is just that I tend to believe that although the major version 1 of AWS API is still maintained. Version 2 wasn't released just for fun. AWS will deprecate v1 at some point and move all new development to v2. So if the community would like to stay in sync with the underlying AWS API they would have to take this decision.
It is not an easy one of course. For example, expose the blocking and non-blocking API side by side requires substantial work.
As for Non blocking IO specifically, to us it is very important. IMHO, doing it better then (or just as) netty already does in the new version is just reinventing the wheel.
I also think that for some users, non blocking IO is essential, especially for applications that are IO bounded and need to parallelise most of their work in order to achieve acceptable throughput (web servers that are connected to dynamodb are good example).
My first thought was to try and do it for the community. However, I totally understand if this kind a path is not something you are willing to take currently.
What do you think?
from faraday.
I tend to agree with @belucid here, but would prefer that auto-pagination also be left to the user (or some companion library?).
How best to deal with threading & memory management of large queries/scans can be very application specific.
At my work, we have built quite a few features on top of Faraday to manage pagination, batching, table configuration convergence amongst other things, and am grateful that it's easy to implement on top of current Faraday API. We're not against contributing them to an appropriate project should there be appetite for such a thing, and maybe other folks feel the same about they're usage.
The more Faraday has opinions on such things, the larger a surface area we have to support as a community, and IMHO, these opinions can sometimes restrict the library's flexibility.
Having said all that, as @belucid says, it probably makes sense to start building a case for looking at v2, and to see if there's enough momentum in the community to take it on.
from faraday.
FWIW, I'm running into nasty exceptions like this
Unable to unmarshall exception response with the unmarshallers provided
which are related to using AWS SDKv1 together with Java >=17
from faraday.
Thanks @pesterhazy. Looks like we finally have a compelling reason to upgrade.
from faraday.
FYI I'm also running into nasty exceptions like this when using batch-write-item
:
Execution error (AmazonDynamoDBException) at com.amazonaws.http.AmazonHttpClient$RequestExecutor/handleErrorResponse (AmazonHttpClient.java:1879).
Provided list of item keys contains duplicates (Service: AmazonDynamoDBv2; Status Code: 400; Error Code: ValidationException; Request ID: 0FNP3B7UDUESMP6MBC3U3676CJVV4KQNSO5AEMVJF66Q9ASUAAJG; Proxy: null)
even though there aren't duplicate keys. We're using Java11.
When using Cognitects AWS API it batch writes the records with no issues.
from faraday.
This is a strange one @LouiseKlodt. I'm not aware of anything that would cause you to see duplicate keys that relates to using the AWS SDK v1.x on Java 11.
Are you able to reduce log the data that was being written when this exception was thrown and inspect it? (and maybe post here if it is not sensitive).
from faraday.
Hi @joelittlejohn, sorry for the slow response. I tried reproducing the issue a few days ago, but it's working OK now, so either something has resolved itself, or I repeatedly made some mistake. So please disregard my comment above. Sorry for the noise! Thanks!
from faraday.
Related Issues (20)
- CI integration HOT 5
- Remove support for `:expected` HOT 3
- attribute_not_exists not respected consistently HOT 3
- Support for Streams HOT 8
- missing ScanIndexForward option with far/scan HOT 1
- Request: custom keywordization routine HOT 3
- Is the scan function safe for large tables? HOT 7
- Upgrade to 1.11.16+ of aws-sdk-java for task-level IAM role support in AWS ECS HOT 9
- Add support for DynamoDB Transactions HOT 6
- Upgrade to 1.11.459 of aws-sdk-java for management of On Demand tables
- Upload all doc versions to http://taoensso.github.io/faraday/{version}
- missing page-size/max-result-size option with query HOT 1
- Support empty strings HOT 4
- not listing tables HOT 14
- Setting `region` in client-opts causes ClassCastException
- String index names cause type error HOT 3
- Local docker example from readme only with `-p 8000:8000` arg HOT 1
- Possibly confusing dependencies found HOT 3
- FYI: have pushed some superficial changes HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from faraday.