Comments (5)
We dont have any support for that right now. The only API we have write now is Spark DataFrame Reader Writer APIs, and we will have SQL support that will be executable through Spark SQL.
It's important to understand that Delta Lake is a data layout format. It does not run a service or a process, so the question of any API endpoint does not arise.
from delta.
I am closing this issue. Please reopen it if you have any follow up questions.
from delta.
@tdas A year later, but I have a question in the same "zone" as the OP. I understand "that Delta Lake is a data layout format" (quoted above). Is there now a standard for "exposing" the large Delta Lake tables(we intend using HDFS for storage) - reason we'd like to run visualisations on Delta Lake tables created by Spark scripts, rather than:
querying HDFS -> porting Delta Lake table snippet to Spark Parquet -> writing temp parquet data-> visualise using a 3rd party platform -> delete temp Parquet data
Or is there another way you suggest we go about this? Thanks!
from delta.
This would be really helpful and really powerful. We could store schemas in one place - Delta Tables - rather than more than one place. We just need to be able to fetch the schema from a Delta table to do this :(
from delta.
@rjurney Delta Lake is just a table format, similar to Parquet. It doesn't have any service. If you would like to read Delta tables through REST APIs, you can try Delta Sharing.
from delta.
Related Issues (20)
- [Kernel][Documentation/Examples] Add usage of time-travel APIs (by version and by timestamp) to the user guide and examples
- [BUG][Spark] DeltaIllegalStateException: [DELTA_SPARK_SESSION_NOT_SET] Active SparkSession not set on Spark 3.5.0/Delta 3.0.0
- [Feature Request][Kernel] Add support for predicate pushdown into Parquet reader
- [Feature Request][Kernel] Support `kernel-api` module issuing multiple read requests in a one read call during state reconstruction
- [Feature Request][Kernel] Add better logging in snapshot loading to debug performance issues
- [BUG] mergeSchema=true do not update metadata and nullability of fields HOT 1
- [Feature Request][Kernel] Add readerFeatures, writerFeatures, Metadata.createdTime to ScanStateRow
- [Kernel] Support getting the version and snapshot with atOrAfter and beforeOrAt semantics
- [PROTOCOL RFC] Column Mapping Usage Tracking
- HadoopFileSystemLogStore.listFrom [Feature Request - Suggestion - Performance Fix]
- [Feature Request][Kernel] Improve the last checkpoint finding code
- [Kernel] Clarify the `ScanBuilder.withFilter` API
- [PROTOCOL RFC] Config value to control the amount of characters in commitInfo.operationParameters.predicate
- [Kernel] Add unittests for reading cloned (shallow/deep) Delta tables in Kernel HOT 1
- [BUG] Clustered Spark fails to write _delta_log via a Notebook without granting the Notebook data access HOT 1
- [Feature Request] Options to disable expiration
- [BUG] delta lake 3.1.0 and delta-hive-assembly issue HOT 1
- [Flink] Runtime of Flink test suite is too long HOT 4
- [Feature Request] Improve UniForm Hudi support for Lists/Maps and Schema Evolution
- [BUG][Flink] Flink K8s Autoscaler metrics not detected
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from delta.