meltanolabs/tap-duckdb

tap-duckdb's Issues

chore: Upstream breaking changes to SQLConnector - please pin sdk

This PR meltano/sdk#1394 makes some small breaking changes to SQLConnector.

Recommendation: Pin the SDK <=0.19.0 in your pyproject.toml. Looks like you've already pinned at 0.14.0, so just make sure to review this issue before you update to >0.19.0

Please take a look at the PR description for a deeper discussion. Here is a summary of what's changed.

Overview:

• The SQLAlchemy connection on the SQLConnector is now protected, and belongs only to the SQLConnector or it's subclasses. Streams and other objects/functions should stop accessing the SQLConnector's engine or connection.
• Within the SQLConnector, DB connections are opened in only one way: by using the _connect() method as a context manager.
• The SQLConnector uses a single engine throughout its lifetime. Connections are opened as needed and closed after each operation.

Specific code changes:

• _connection is gone altogether. You cannot pass it to SQLConnector and you cannot access it off of the connector. We are not caching or passing around an open DB connection anywhere anymore.
• create_sqlalchemy_engine and create_sqlalchemy_connection have been altered to use the single cached engine, and are also deprecated. Ideally, subclasses & other objects would stop using those methods.

Suggestions:

• If you have any database-interacting logic that happens outside of the SQLConnector (eg if your SQLStream subclass has a get_records method that gets the SQLConnector.connection and does something with it), please move that logic onto a method on your subclass of SQLConnector and call that method from the Stream/other object.
• Wherever you're accessing the connection on SQLConnector/a subclass, please do so by using the _connect() method as a context manager. It's directly equivalent to SQLAlchemy.Engine.connect(). Here's an example of its use.

Tests are failing

Looks like we just need to update the dependencies and it'd start working (poetry update so it seems)

Upgrade Python Version

Hello Meltano Team,

Is it possible to upgrade python version requirement to allow 3.11.x?

Thanks
Guilherme

Did not recognize type 'hugeint'

/home/visch/git/prodder/.meltano/extractors/tap-duckdb/venv/lib/python3.8/site-packages/singer_sdk/connectors/sql.py:371: SAWarning: Did not recognize type 'hugeint' of column 'estimated_seconds'