Comments (6)
zsxwing
commented on May 19, 2024
5
This is resolved by c8169bd
from delta.
tdas
commented on May 19, 2024
4
Using something like DynamoDB as locking service is definitely one of ideas we will be playing with to implement the multi-cluster mode (tracked by #41 ). For this issue, we are focusing on releasing something quickly that enables the community to start using Delta Lake with S3.
from delta.
binary132
commented on May 19, 2024
What about going through a locking interface which could be implemented within Spark or by a service, such as DynamoDB? Then future work could be merged to enable multi-cluster ACID.
from delta.
gourav-sg
commented on May 19, 2024
with EMR 5.24.0 we can have multiple master nodes (obviously they are fail-safe ones, but will be interesting to test).
from delta.
gourav-sg
commented on May 19, 2024
What should be the best way to start testing this in AWS S3? I can also start updating few documentation around it so that others can refer to it as well, please do let me know.
from delta.
zsxwing
commented on May 19, 2024
@gourav-sg We are working the document right now. Once we update it, I will ping you in this ticket. Thanks!
from delta.
Related Issues (20)
- [Kernel][Refactoring] Separate out the delta file and stats parsing logic in `DefaultJsonHandler`
- [BUG][SPARK] Using a temp view in a MERGE INTO spark sql statement leads to MISSING_ATTRIBUTES.RESOLVED_ATTRIBUTE_MISSING_FROM_INPUT error
- [BUG] Missing Partition Filters on renamed Columns after Join HOT 4
- [BUG][Spark] Deletion Vectors can cause `AMBIGUOUS_REFERENCE` errors on MERGE HOT 1
- [BUG][Kernel] Delta clone does not copy the table comment, only columns HOT 2
- [Question] PsSpark: downgrade Delta table protocol by rewriting the table (Delta 2.2.0)
- NoClassDefFoundError scala/Serializable, How to build spark session with delta with java 8
- ipad support HOT 1
- [BUG][Spark][Uniform] Reorg Table for Uniform upgrades minReaderVersion to 3, documentation says it should be 2 HOT 1
- [Feature Request][Standalone] `TimestampNTZ` datatype is not supported HOT 2
- [Question][Standalone] How to add `constraints` when creating tables with delta-standalone? HOT 3
- [BUG][Spark] Inconsistent casting behavior in insert
- [Kernel][Feature Request] Revisit how to treat unsupported expressions on partition columns that are part of the query filter
- [Feature Request]: Support for CREATE DATABASE
- [BUG] [SPARK] Unintended Rewrite of Other Partitions During Partition-Level Delta Table Update
- [BUG] [FLINK] Delta log not updated with with the parquet files created after the last checkpoint before Flink Job completed HOT 4
- [Feature Request][INFRA] Improve the java checkstyle checks to log the errors to the sbt console
- [Feature Request] Deletion Vectors metrics for reads
- [Feature Request][Spark] Optimize automated batching HOT 1
- [BUG][Kernel] Delta kernel read checkpoint parquet file failed with exception "list not found in optional group readerFeatures (LIST)" HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from delta.