apache / arrow-adbc Goto Github PK
View Code? Open in Web Editor NEWDatabase connectivity API standard and libraries for Apache Arrow
Home Page: https://arrow.apache.org/adbc/
License: Apache License 2.0
Database connectivity API standard and libraries for Apache Arrow
Home Page: https://arrow.apache.org/adbc/
License: Apache License 2.0
Removed in apache/arrow#13382
I noticed that the ADBC metadata information assumes a fixed hierarchy:
What would the advice be for datasources that don't fit this, like Snowflake/Trino/Dremio where the hierarchy might be:
datasource.database.schema.table
postgres1.mydb.public.emps
If we provide AdbcDriverManagerConfig.cmake
, CMake users can find ADBC driver manager easily: find_package(AdbcDriverManager)
Now, driver implementers can use any name for entrypoint function. How about introducing a naming rule for entrypoint function such as AdbcDriverInit()
or ${driver_name}_init()
? If we have a naming rule, users don't need to entrypoint option.
We may not be able to use constant name such as AdbcDriverInit()
because LoadLibraryEx()
doesn't have a local bind flag like RTLD_LOCAL
for dlopen()
.
Related: #84
Generally, plugin system uses its own plugin directory. For example, PostgreSQL uses ${prefix}/${version}/lib/
and MySQL uses ${prefix}/lib/mysql/plugin/
.
How about adding support for driver directory like them? If we use MODULE library not SHARED library (see #84), it's strange that adbc_driver_postgres.so
(not lib
prefix) exists in /usr/lib/
.
We don't want to/can't futz with shared libraries when distributing packages. Instead, since drivers have an entrypoint, just expose the entrypoint in Python?
e.g. adbc_driver_sqlite
will bundle the driver and expose adbc_driver_sqlite._entrypoint(driver_address: int) -> int
~= AdbcStatusCode _entrypoint(struct AdbcDriver* driver)
. Then the driver manager can just import the package and call the entrypoint. (There'll need to be some support for this on the C++ side too)
That way we can more easily distribute pip-installable packages
CI:
Packaging:
Arrowdantic: https://github.com/jorgecarleitao/arrowdantic/
Turbodbc: https://github.com/blue-yonder/turbodbc/
We should improve the documentation/justification for this interface, describe better what happens when it's not supported, and make sure it lines up with what potential users of the interface expect.
In particular, it should line up with Spark's DataSourceV2. Looking at ReadSupport, the main thing is that we need to return the schema and partitions at the same time. So we might want to return something like this:
struct AdbcPartitions {
struct ArrowSchema result_schema;
size_t num_partitions;
uint8_t** partitions;
void* private_data;
};
AdbcStatusCode AdbcPartitionsRelease(struct AdbcPartitions*, struct AdbcError*);
Also, should deserializing a partition descriptor give you a statement, or just directly give you a result reader?
Also see #61 which proposes refactoring the Execute API.
Things to consider
Example: libpq disallows concurrent queries through a single PGconn, so multiple AdbcStatements can't be used if they share a connection (and the semantics of that get murky anyways) - but what should the behavior be?
We should support Windows, MacOS, and Linux.
On Windows: use delvewheel
On MacOS: use delocate
On Linux: use auditwheel
We should start considering CI as well
The existing bindings aren't fully complete.
We should have two packages: a low-level package with no/minimal dependencies that mirrors the C API closely; and eventually, a high-level package that implements a PEP 249 compliant API with extensions (similar to turbodbc).
For the common case of executing a single SQL string, let's have a method on the connection object for executing the query directly without the need for an intermediate Statement object
If we provide adbc-driver-manager.pc
, users can find ADBC driver manager easily.
We can't use ../
for files that are included into .gem
:
$ cd ruby
$ rake install
red-adbc 1.0.0 built to pkg/red-adbc-1.0.0.gem.
rake aborted!
Running `gem install .../ruby/pkg/red-adbc-1.0.0.gem` failed with the following output:
ERROR: While executing gem ... (Gem::Package::PathError)
installing into parent path /tmp/local/lib/ruby/gems/3.2.0+2/gems/LICENSE.txt of /tmp/local/lib/ruby/gems/3.2.0+2/gems/red-adbc-1.0.0 is not allowed
...
JDBC, ODBC, Flight SQL (implicitly): auto-commit
PEP 249: manual commit
We should define what the default is and add a function to set this on the connection.
Generally, plugin for C/C++ is implemented as MODULE library.
See also https://cmake.org/cmake/help/latest/command/add_library.html#normal-libraries for MODULE and SHARED libraries.
For example, PostgreSQL's extensions and MySQL's storage engines are implemented as MODULE library.
If we use MODULE library, driver module file name is something like adbc_driver_postgresql.so
on Linux. It doesn't have lib
prefix and shared object version such as 900.0.0
.
Targets:
c/driver_manager/
c/drivers/sqlite/
- not needed?c/drivers/postgres/
glib/
- not exists yetTarget distributions:
ADBC_VERSION_0_0_1
to ADBC_VERSION_1_0_0
AdbcDriver
struct is all up-to-dateI did a little prototyping around this to try to hit nanoarrow with some more real-world data...my interest in SQLite3 is mostly around the GeoPackage (https://geopackage.org), which is a common spatial file format based on SQLite3. The experiment is currently living in its own repo ( https://github.com/paleolimbot/minigpkg/blob/main/src/minigpkg/nanoarrow_sqlite3.h#L112-L127 ), and while I don't think I currently have the bandwidth to implement the ADBC API, I'd be happy to circle back to this at some point since an SQLite3 driver that depends only on sqlite3 seems like it would be useful here.
As noted in the current driver (https://github.com/apache/arrow-adbc/blob/main/c/drivers/sqlite/sqlite.cc#L139-L141), SQLite really needs an adaptive builder or an adaptive schema guesser to be useful.
We should have a structure like
/
adbc.h
c/
driver_manager
driver
validation
java/
...
(also, the Java "testsuite" package should be renamed to "validation")
Implemented for C/Python and Java:
IMO, we should have these features:
Not yet:
Cursor.rowcount
)Backend.current_database
)paramstyle
)ClickHouse supports Arrow result sets, can easily insert Arrow data into tables, and has HTTP and gRPC interfaces: https://clickhouse.com/docs/en/interfaces/formats/#data-format-arrow-stream
private_manager
and release
need to be populated and tested on WindowsWe should try to use the 'native' type of the C API. Apparently, this will also ease interoperability with R.
From #64. The definitions were updated but we should ensure drivers comply (the test suite should validate this as much as possible).
See also: #84 (comment)
If available, it would be great to be able to retrieve any information about parameter binding that is available. Some potential information that might be available:
Since it's not reliable to always be able to get this information, there would need to be a way to indicate that there was no error, but the information is not available.
We should fill out the error codes. Things to consider:
Do we want to extend AdbcError with space for database-specific error codes?
See lidavidm/arrow#9
Similar to AdbcConnectionGetTables, there should be a way to query the columns available in a table with their names, types, NULL-ness etc.
I was basically modeling this off of Flight SQL: https://github.com/apache/arrow/blob/master/format/FlightSql.proto
In this case GetTables has a parameter to optionally fetch the Arrow schema of the tables as well. And then SQL-specific type info is encoded as column metadata.
I recall we talked about using something like MSSQL information_schema instead. This was also debated for Flight SQL. We decided against it since databases don't necessarily implement this, or implement slightly different things, and there's no easy way for a client to figure this out. Also, then you're dependent on knowing the database's SQL dialect in order to introspect it. While explicit calls like GetTables are tedious and less flexible, they don't require so much bootstrapping. (This is important for Flight SQL which does not want to have to implement client-side drivers; a Flight SQL client should be compatible with any database implementing the protocol. ADBC doesn't necessarily have the same constraint, though.)
GetTables
needs to also (optionally) return the schema. Or we could have a separate method that models the schema explicitly? The difference would be that we could explicitly model the SQL-database metadata (which Flight SQL stuffs into Arrow field metadata)
Will make implementing Ibis backends easier since we can reuse more of the code for SQLAlchemy-based backends
Targets:
c/driver_manager/
c/drivers/sqlite/
- not needed?c/drivers/postgres/
glib/
- not exists yetSELECT *
from a table without providing a query (makes it easier to provide non-query-engine based backends, e.g. a Parquet file backend)Rather than the separate Execute
/ GetStream
functions, it might be better to follow something similar to FlightSQL's interface or Go's database/sql
API.
Have two functions:
Execute(struct AdbcConnection*, const char*, struct AdbcResult*, struct AdbcError*)
where AdbcResult
would contain an optional LastInsertedID and Number of Rows affectedQuery(struct AdbcConnection*, const char*, struct ArrowArrayStream*, struct AdbcError*)
where the ArrowArrayStream
is populated with the result set.Corresponding methods would exist for a Statement just without the need for taking the const char*
as it would already be prepared in the statement.
Some benefits of this idea:
AdbcStatementExecuteUpdate
can set rows_affected
, which is a bit ancient given that most SQL systems support the RETURNING
clause now, which makes the separation between ExecuteQuery
and ExecuteUpdate
redundant. I propose to have only one method to run a query, possibly along with allowing the out
to be nullptr
in AdbcStatementExecuteQuery
. rows_affected
should be removed.
Pgeon: https://github.com/0x0L/pgeon
ConnectorX: https://sfu-db.github.io/connector-x/intro.html
For C/Java parity.
We need to set up:
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.