codenotary/immudb-py

immudb Python SDK

immudb-py's Issues

Update Python SDK - v0.8 release

[Feat] Python SDK 100% grpc api coverage

Description
Python SDK must support all GRPC api.

Action items

• All methods can be invoked with a client
• All methods that support streaming can be used

VerificationException with immudb 1.2.1

Hi, I wanted to report a immudb.exceptions.VerificationException exception when using verifiedSet with the latest release (1.1.0). Here's a script to reproduce:

from immudb import ImmudbClient

client = ImmudbClient()
client.login("immudb", "immudb")
client.verifiedSet(b"hello", b"world")

Examples from immudb-client-examples that use verifiedSet raise the same exception.

Looking at the release notes for immudb 1.2.1, it looks like there's a breaking change that may be responsible.

Introduction of the entry metadata requires a slightly modified version of the proof that also needs to consider such metadata. An updated version of the SDK will be needed to correctly verify proofs for new entries.

Enhancement Issue: Unique constraint in SQL

Hi,

I have been asked to create an enhancement issue regarding "UNIQUE" constraint in SQL.

An example usage of Unique constraint would be:

CREATE TABLE table1 (id INTEGER, title VARCHAR UNIQUE, active BOOLEAN NOT NULL, PRIMARY KEY id)

[Feat] Python SDK 1.3.2 release - backward compability

Better exception handling

Currently there is no way to get matching error code from error exception.

From my investigation:

This exception have mainly 3 "normal" fields now

debug_error_string {"created":"@1649922703.326925387","description":"Error received from peer ipv6:[::1]:3322","file":"src/core/lib/surface/call.cc","file_line":903,"grpc_message":"invalid user name or password","grpc_status":2}

details invalid user name or password

code StatusCode.UNKNOWN - it matches internal codes from GRPC, and not the code provided by server (08004)

Or in some cases
https://github.com/codenotary/immudb/blob/master/pkg/errors/map.go

I looked depper into it and i see that the proper status code is in some unparsed trailing_metadata field
(_Metadatum(key='grpc-status-details-bin', value=b'\x08\x02\x12\x1dinvalid user name or password\x1aE\n+type.googleapis.com/immudb.schema.ErrorInfo\x12\x16\n\x0508004\x12\rkey not found\x1a-\n+type.googleapis.com/immudb.schema.RetryInfo'),)

.....schema.ErrorInfo\x12\x16\n\x05------->08004<-------\x12\rkey not found\x1a-\n+type.google......

User should have easy way to catch errors and don't compare them to string by himself.
SDK should also have some generic way to deal with Exceptions, or at least map existing error codes

https://github.com/codenotary/immudb/blob/master/pkg/errors/meta.go

New parameter for sqlQuery()

A select query fetches only up to a 1000 records.
Can this be modified to fetch r number of records?
For e.g.:
results = client.sqlQuery(queryStr,r=100000)

changed method names

in README.md it is client.useDatabase, in client.py databaseUse. Same for createDatabase vs. databaseCreate.

Support for additional immudb functions

Immudb-py lacks support for several functions, compared to the Golang client. Esp. the
.) Delete()
.) ExpirableSet()
.) DescribeTable()
would be important for test coverage resp. usability.

Documentation on Python SDK - v0.8 release - with code snippets

test_sql.py failed

tests/immu/test_sql.py failed with error below:

E           grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
E           	status = StatusCode.UNIMPLEMENTED
E           	details = "unknown method SQLExec for service immudb.schema.ImmuService"
E           	debug_error_string = "{"created":"@1627255447.060995531","description":"Error received from peer ipv6:[::1]:3322","file":"src/core/lib/surface/call.cc","file_line":1069,"grpc_message":"unknown method SQLExec for service immudb.schema.ImmuService","grpc_status":12}"
E           >

fix immudb-py tests

ref.: #70

Permission for history

When using a get for the same key, I get results. When using immuclient as the same user, history works. When using history from the Python connector, I get:

grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:<
status = StatusCode.UNKNOWN
details = "you do not have permission for this operation"
debug_error_string = "{"created":"@1607898051.288363185","description":"Error received from peer ipv6:[::1]:3322","file":"src/core/lib/surface/call.cc","file_line":1062,"grpc_message":"you do not have permission for this operation","grpc_status":2}"

Note: I was unable to install via pip (older version without history instead was installed) so I grabbed current master from GitHub.

StatusCode.UNIMPLEMENTED

Hi,

When trying the immudb. I can not get it to work. I run the db on docker with: sudo docker run -it -p 3322:3322 -p 9497:9497 --name immudb codenotary/immudb:latest and this is my client code:

import immudb
from immudb.client import ImmudbClient

ic = ImmudbClient("0.0.0.0:3322") 
ic.login("immudb","immudb")

if "testdb" not in [o.databasename for o in ic.databaseList().dblist.databases]:
    ic.databaseCreate(b"testdb")
print(ic.databaseList().dblist.databases)

ic.databaseUse(b"testdb")
print('using testdb')
key="Hello".encode('utf8')
value="Immutable world!".encode('utf8')

# set a key/value pair
ic.set(key,value)

I do get a read out of the databases. But then I get the following error:

    raise _InactiveRpcError(state)
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
        status = StatusCode.UNIMPLEMENTED
        details = "unknown method SetSV for service immudb.schema.ImmuService"
        debug_error_string = "{"created":"@1606746880.907274017","description":"Error received from peer ipv4:0.0.0.0:3322","file":"src/core/lib/surface/call.cc","file_line":1061,"grpc_message":"unknown method SetSV for service immudb.schema.ImmuService","grpc_status":12}"
>

I'm even running the code in docker and still get the same error. here the Dockerfile:

FROM python:3.7.2-slim

RUN apt-get update && apt-get remove -y --purge man-db && apt-get dist-upgrade -y && apt-get autoremove -y
RUN apt-get update && apt-get install gcc wget git python3-dev g++ -y
RUN apt install python3-pip git -y
RUN pip3 install --upgrade pip

RUN python3 -m pip install --upgrade setuptools
RUN pip3 install --no-cache-dir  --force-reinstall -Iv grpcio #==1.32

RUN pip3 install immudb-py

WORKDIR /app
COPY run_immudb.py /app/run_immudb.py

CMD python3 run_immudb.py

Any help would be much appreciated

Support for immudb 1.2.4

Transactions support

Hello, do you plan to support SQL transactions?

Map Permissions to proper classes / enums

Currently users of immudb-py have to map Permission and PermissionAction by themselves

Should be probably covered by SDK

class Permission(IntEnum):
    NONE = 0
    READ = 1
    READ_WRITE = 2
    ADMIN = 254
    SYS_ADMIN = 255
class PermissionAction(IntEnum):
    GRANT = 0
    REVOKE = 1

Dataclasses issue when importing from immudb

When deploying to lambda using serverless, there is an error with dataclasses. Upon searching, the most common solution is to remove the installation of dataclasses since it's not a necessity for python above 3.6 but deploying with serverless only gives an option to add required packages and no option to remove any. When installing immudb-py, it forces an installation of dataclasses 0.6 which is not necessary and only gives errors. Tried using python 3.8 and 3.10 with both the same results.

[ERROR] AttributeError: module 'typing' has no attribute '_ClassVar'
Traceback (most recent call last):
File "/var/lang/lib/python3.10/importlib/init.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "", line 1050, in _gcd_import
File "", line 1027, in _find_and_load
File "", line 1006, in _find_and_load_unlocked
File "", line 688, in _load_unlocked
File "", line 883, in exec_module
File "", line 241, in _call_with_frames_removed
File "/var/task/app.py", line 13, in
from immudb import ImmudbClient
File "/var/task/immudb/init.py", line 13, in
from immudb.client import ImmudbClient
File "/var/task/immudb/client.py", line 19, in
from immudb import datatypes
File "/var/task/immudb/datatypes.py", line 26, in
class SetResponse:
File "/var/task/dataclasses.py", line 958, in dataclass
return wrap(_cls)
File "/var/task/dataclasses.py", line 950, in wrap
return _process_class(cls, init, repr, eq, order, unsafe_hash, frozen)
File "/var/task/dataclasses.py", line 800, in _process_class
cls_fields = [_get_field(cls, name, type)
File "/var/task/dataclasses.py", line 800, in
cls_fields = [_get_field(cls, name, type)
File "/var/task/dataclasses.py", line 659, in _get_field
if (_is_classvar(a_type, typing)
File "/var/task/dataclasses.py", line 550, in _is_classvar
return type(a_type) is typing._ClassVar

Asyncio support

Hello! Do you have any plans to support asyncio?

Support for SinceTx parameter in history() method

Currently immudb-py doesn't support SinceTx parameter for history() method

async await support .

Need an asynchronous support in immudb-py

Add support and documentation samples for revision-based queries

Revision-based queries were added in: codenotary/immudb#1204

Documentation update: codenotary/immudb.io#154 - this documentation only contains examples in go, please fill in corresponding python examples.

Module import error occured after installing immudb-py with setup.py

Reproducer:

setup.py

from setuptools import setup
setup(
  install_requires=[
    'immudb-py',
  ],
)

test.py

from immudb import ImmudbClient

reporoduce log

$ python setup.py install 
(...)
$ python -m pip freeze
appier==1.32.0
cachetools==5.3.3
certifi==2024.2.2
charset-normalizer==3.3.2
dataclasses==0.8
ecdsa==0.18.0
google-api==0.1.12
google-api-core==2.18.0
google-auth==2.29.0
googleapis-common-protos==1.63.0
grpcio==1.62.1
idna==3.6
immudb-py==1.4.0
proto-plus==1.24.0.dev0
protobuf==3.20.3
pyasn1==0.5.1
pyasn1-modules==0.3.0
requests==2.31.0
rsa==4.9
six==1.16.0
UNKNOWN==0.0.0
urllib3==2.2.1

$ python test.py
Traceback (most recent call last):
  File "/home/khwarizmi/test/immudb-py/test.py", line 1, in <module>
    from immudb import ImmudbClient
  File "/home/khwarizmi/test/immudb-py/env/lib64/python3.9/site-packages/immudb_py-1.4.0-py3.9.egg/immudb/__init__.py", line 13, in <module>
    from immudb.client import ImmudbClient
  File "/home/khwarizmi/test/immudb-py/env/lib64/python3.9/site-packages/immudb_py-1.4.0-py3.9.egg/immudb/client.py", line 18, in <module>
    from immudb import grpcutils
  File "/home/khwarizmi/test/immudb-py/env/lib64/python3.9/site-packages/immudb_py-1.4.0-py3.9.egg/immudb/grpcutils.py", line 20, in <module>
    import immudb.grpc.schema_pb2_grpc as schema_pb2_grpc
  File "/home/khwarizmi/test/immudb-py/env/lib64/python3.9/site-packages/immudb_py-1.4.0-py3.9.egg/immudb/grpc/schema_pb2_grpc.py", line 6, in <module>
    from . import schema_pb2 as schema__pb2
  File "/home/khwarizmi/test/immudb-py/env/lib64/python3.9/site-packages/immudb_py-1.4.0-py3.9.egg/immudb/grpc/schema_pb2.py", line 16, in <module>
    from google.api import annotations_pb2 as google_dot_api_dot_annotations__pb2
ModuleNotFoundError: No module named 'google.api'

Note:

If I installed immudb-py using pip install, then this error not occured.

Detail of this issue:

It seems that this issue is caused by the combination of versions of googleapis-common-protos and protobuf.
googleapis-common-protos==1.62 (released 2023/12/08) has this patch: googleapis/python-api-common-protos@713e388
I guess, way to make namespace in library changed in this patch. (reference: https://packaging.python.org/en/latest/guides/packaging-namespace-packages/)
It seems that there is an issue where namespaces cannot be properly created when installing a combination of googleapis-common-protos 1.62 or later and older protobuf with setuptools.

simple test 1:

test.py:

from google.api import distribution_pb2
print("error doesn't occured.")

setup.py:

from setuptools import setup
setup(
  install_requires=[
    'googleapis-common-protos',
    ''protobuf>=3.13.0,<4.0.0'     # same as immudb-py
  ],
)

log:

$ python setup.py install
(...)
$ python -m pip freeze
googleapis-common-protos==1.63.0
protobuf==3.20.3
UNKNOWN==0.0.0
$ python test.py
Traceback (most recent call last):
  File "/home/khwarizmi/test/immudb-py/test.py", line 1, in <module>
    from google.api import distribution_pb2
ModuleNotFoundError: No module named 'google.api'

simple test 2

setup.py:

from setuptools import setup
setup(
  install_requires=[
    'googleapis-common-protos',
  ],
)

log:

$ python setup.py install
(...)
$ python -m pip freeze
googleapis-common-protos==1.63.0
protobuf==4.25.3
UNKNOWN==0.0.0

$ python test.py
error doesn't occured.

What I want you to do:

My tool using immudb-py is expected to be installed with setuptools. So, I encounterd this issue.
immudb-py has the following dependency on protobuf: https://github.com/codenotary/immudb-py/blob/master/setup.py#L34
Is this dependency mandatory?
If it is not mandatory, then removing it should solve this issue.
Please consider making this change.

UpdateDatabaseV2

Can we get the proper example of how to update immudb settings with UpdateDatabaseV2 using python lang.

[Feat] Docstring documentation for Python SDK

Description
Document Python SDK so it is easily usable with any IDE and intelli sense. How documentation should look like ? Python docstrings

Action Items

• every public function has description including arguments, return values, errors etc.
• every public class has description including arguments, return values, errors etc.
• every non trivial function (or with lot of arguments) has short code sample in description

Session support in immudb-py

stale files in pip distribution

The pip distribution has some extra files which are no longer in master:
handler/safeGet.py
handler/safeSet.py
consistency.py
inclusion.py
item.py
proofs.py

sqlQuery untrusted input

[Feat] immudb-py toxiproxy for timeout testing

https://github.com/vchain-us/audit-change-mgmt/issues/556

Implement all functions found in immudb4j

At this time, the java SDK implements these functions:

• createDatabase
• useDatabase
• databases (list databases)
• set
• get
• safeGet
• safeSet
• setAll
• getAll
• root (provides the last validated root)

These function should be implemented in the python SDK also.

grpc.RpcError exception

grpc._channel._InactiveRpcError in tests should be replaced with grpc.RpcError.

Reference:

• grpc/examples/python/errors
• article about grpc errors

pytest fails with newer immudb version

github action tests are performed against 1.4.0, and there are some tests failing.

• tests/immu/test_sql.py::TestSql::test_exec_query
• tests/immu/test_sql_count.py::TestSQLCount::test_sql_count
• tests/immu/test_sql_dict_response.py::Test::test_sql_diffent_columns_response
• tests/immu/test_sql_timestamp.py::TestSqlTimestamp::test_exec_query_timestamp
• tests/immu/test_sql_transaction.py::TestSQLTransaction::test_sql_simple_transaction_multiline
• tests/immu/test_sql_transaction.py::TestSQLTransaction::test_sql_simple_transaction_flat
• tests/immu/test_sql_verify.py::TestVerifySQL::test_exec_query
• tests/immu/test_sql_verify.py::TestVerifySQL::test_varchar
• tests/immu/test_sql_verify.py::TestVerifySQL::test_boolean
• tests/immu/test_sql_verify.py::TestVerifySQL::test_blob
• tests/immu/test_sql_verify.py::TestVerifySQL::test_ts

We should update immudb version in the test to 1.9.0 and fix the tests or the library accordingly

python SDK update

• Some functions are named reverse than in Go SDK.
• We should get rid of binary input parameters (i.e. b’test’)
• Publish to conda as well
• Example on list users not working
• Example on how to execute sql queries, list tables, list databases
• Default port in examples should be the same default port for immudb
• Improve output messages: okay statuses after connection, or better output when inserting rows.

starttestcontainers.sh / ci.yml

The health-cmd in these files is obsolete. Since containers are now from scratch, there is no /bin/sh, and the check won't execute, since it is in shell form, not exec array. However, images have health command already defined, so probably we could skip that at all.
https://github.com/codenotary/immudb-py/blob/master/tests/starttestcontainers.sh

Support for VerifiedGetSince method

Currently there is no option to get verified data since some transaction, and immudb already supports it

Tags for releases

I suggest to have tags pointing to the releases again.