datastax / astrapy Goto Github PK
View Code? Open in Web Editor NEWAstraPy is a Pythonic interface for DataStax Astra DB and the Data API
Home Page: https://github.com/datastax/astrapy
License: Apache License 2.0
AstraPy is a Pythonic interface for DataStax Astra DB and the Data API
Home Page: https://github.com/datastax/astrapy
License: Apache License 2.0
A faulty commit has made into main.
Add a CHANGES.txt and move to a release cadence where several PRs are accumulated in each release.
Also move the status on setup.py (=> PyPI) to Beta
I would like to thoroughly test if the requests_toolbelt dependency can be bumped to ^1.0.0
(I'm almost sure it can).
this is critical to get LangChain integration (otherwise there are version conflicts)
The new JSON API has renamed "X-Cassandra-Token" to "Token". astrapy should update accordingly
(the old name is supported for compatibility but better to stay up to date)
In:
try:
if return_type == "json":
return r.json()
else:
return r
shouldn't it be r.text
at the end?
The API has merged the fix to a naming inconsistency, in short the pagination calls (once fix is deployed, i.e. post API v 1.0.0-BETA-3) will return pageState
instead of pagingState
.
For astrapy, this means adapting the pagination method to look for the new name (or, perhaps even better, to fall back to the old name before stating there's no next page altogether)
Check each API response for the right shape (perhaps with automation to define the expected ok-shape)
and raise a verbose error if not
e.g. create collection currently can silently fail if 50 indexes (free tier)
This makes it into the desired "one pip install" goal. Users would be instructed to "pip install astrapy" and they would get all required packages to go the CQL/CassIO route if they so wish.
Note for py 3.12: CassIO requires cassandra-driver. The latter would get pip-installed all right on 3.12 but then when imported there would be errors. We can ignore these (going to be addressed in ~1 month or so presumably). The important thing is that this added dependency does not break the main astrapy route, and it does not.
Only, as there is no ready wheel, the pip install proces is a bit longer on 3.12 (it maye be 2-3 minutes). But this would be resolved with a newer cassandra-driver supposedly going out soon.
Something like a ruff/black/mypy
suite as an easy-to-use make command would encourage good writing style IMO
some irregularities in what methods expose options, filter and so on. Check e.g. what methods would admit options and so on
General problem: assuming prod in the devops API (url)
vs keeping it simple for the happy path of prod usage
.
Idea from a conversation:
There is no way to "spawn" a DBOps object from the general AstraDB. If there were one, then I would tell you:
regular usage would have users "spawn" an Ops from their main
AstraDB
instance, which would have an.ops()
method spawning anAstraDBOps
. That method would sound like this:
def ops(self, dev_ops_url=None):
if dev_ops_url is None and self.base_url = <production api url>:
dev_ops_url = <prod OPS url>
return AstraDBOps(self.api_key, dev_ops_url=dev_ops_url)```
Fix small formatting issues
Back out to requests for now until time for further investigation
Remove defaults, require the params be passed if required by API
Instead of <astrapy.db.AstraDBCollection at 0x79205e617c70>
we would like to repr as e.g.
Astra DB Collection[name="Blabla", dimension=123]
or whatever.
The current example on the README assume the developer connects to AstraDB.
I would be nice to also have example to connect to plain cassandra deployed by K8ssandra.
Or maybe this is not in the goal of this library? The name astrapy
looks like it is targeted at AstraDB?
Regression seems due to the propagation of a ValueError in the API response, when we were expecting a success response that we could parse.
Split into requirements.txt and requirements-dev.txt
The latter with pytest, ruff, etc
The former with the reqs we want to bundle as package deps ... plus the special -e .
to self-install in a local dev env.
Solves the risk if misaligning setup.py and requirements.txt (also keeping dev stuff out of the build)
As of 0.6.0,
astra_db.create_collection("novectors")
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/home/stefano/personal/WORK_Datastax/Code/other_github/astrapy/astrapy/db.py", line 798, in create_collection
self._request(
File "/home/stefano/personal/WORK_Datastax/Code/other_github/astrapy/astrapy/db.py", line 725, in _request
raise ValueError(json.dumps(responsebody["errors"]))
ValueError: [{"message": "Cannot invoke \"java.lang.Integer.intValue()\" because the return value of \"io.stargate.sgv2.jsonapi.api.model.command.impl.CreateCollectionCommand$Options$VectorSearchConfig.dimension()\" is null"}]
the reason is that the method creates an incomplete options = {"vector": {}}
, which makes the API angry.
Features/packages that depend on 3.8 and higher are present.
So we don't have to make two separate calls
Suggestion: db_id
=> database_id
everywhere.
Also (check with a broader set of folks) token
=> api_key
(that was being discussed at length)
Fix name inconsistency between name and collection (name is probably preferred, even better collection_name IMO):
astra_client_vectordb1.create_collection(name="collection_test", size=5)
astra_db_collection = AstraDbCollection(collection="collection_test")
Defaulting the namespace to "default_namespace"
in the __init__
params is tricky.
One might use a pattern such as passing namespace = os.environ.get["NAMESPACE"]
and end up with errors.
Better to have a None
in the init signature and in the code convert Nones
to "default_namespace"
Wrapper around find() with more "pythonic" syntactic sugar.
Errors in the requests.request method in utils.py
do not work as intended: the try/except block there makes the make_request
method return a JSON instead of a request object, so that callers (such as _request
of AstraDBCollection) fail when they try the .json()
method on these.
I just saw this thanks to someAstra API hiccup:
create_collection_response = self.astra_db.create_collection(
File "[...]/astrapy/db.py", line 360, in create_collection
response = self._request(
File "[...]/astrapy/db.py", line 329, in _request
responsebody = response.json()
AttributeError: 'dict' object has no attribute 'json'
Option 1: remove the try/except altogether in make_request. Onus is on the client.
Option 2:
make_request
returns the request.json() when this works, otherwise returns {"errors": [...]}
, ready to be used by the caller just like the errors genuinely coming from the APIresponsebody = make_request(...)
and call it a dayThis will involve a switch to httpx
which has a number of other advantages over requests
which is currently being used.
Based on JSON api developments
Building on top of the methods that are 1:1 to HTTP calls.
Would dissolve the responses and present a smooth iterable of individual Documents, lazily managing usage of the underlying calls as the iterable gets exhausted.
Emulate a single-document upsert logic:
Name upsert
, signature TDB.
(no need for "single" or "one" in the name, all methods without _many
are implicitly one-doc methods).
This would build on existing methods that are 1:1 to HTTP calls (and can assume error handling is in place)
optional, maybe with a wrapper class or Logger.
Opt=in to log the vector values, a mechanism (on by default) to purge them from outputs for readability
The db and collection are independent fixtures right now, which makes it hard to e.g. select a single test method because it may happen that the collection is not created or similar (e.g. test_collection with -k test_insert_many
or similar).
Make fixtures module-scoped and perhaps nest them or something, so that each method runs by itself all right
(also do not assume any order between test function calls)
Currently the DEFAULT_HOST for the devops is harcoded and it's the prod one.
Dev shoud have DEFAULT_HOST = "https://api.dev.cloud.datastax.com"
.
So possibly a dict from the base_url to the devops corresponding url could be handy?
Unless there's a chance that the latter will even change. Anyway leaving a way for the user to specify a custom devops base host+url would be necessary.
It is used only in testing, no need to install it in the package
Currently the vector API breaks on find if empty dicts are passed for example for sort.
In other words, ..."sort": {}...
in the payload causes errors, while omitting it works all right.
While this will be addressed, I suggest to clean out empty dicts from such places before issuing requests. I don't think it would hurt and it would get all the stack to work sooner.
Such as, the caller takes responsibility of guaranteeing e.g. that all IDs are new to the collection, etc (might induce skipping some checks and/or directly switching to the insert_many API-native method, etc)
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.