data-engineering-collective/minimalkv

A minimal key-value store interface for binary data (maintained fork of simplekv).

minimal key-value storage api

minimalkv is an API for very basic key-value stores used for small, frequently accessed data or large binary blobs. Its basic interface is easy to implement and it supports a number of backends, including the filesystem, SQLAlchemy, MongoDB, Redis and Amazon S3/Google Storage.

Installation

minimalkv is available on PyPI and can be installed through pip:

pip install minimalkv

or via conda on conda-forge:

conda install -c conda-forge minimalkv

Documentation

The documentation for minimalkv is available at https://minimalkv.readthedocs.io.

License

minimalkv is licensed under the terms of the BSD-3-Clause license.

minimalkv's Issues

Handling of colon in Azure KeyValueStores

Hi everyone,
currently minimalkv prohibits using colon ":" in KeyValueStores due to it not being in VALID_KEY_RE_EXTENDED.
Azure Stores would support colons in their path. As far as I know this is currently impossible to read using minimalkv. I.e. if another application writes a file with a colon, you cannot read this file with minimalkv (no expert here though, maybe there is a way to quote this).

I can create a PR to allow for colons only for Azure stores. Would this be ok or is there a decision not to allow this to keep the allowed filenames uniform between different store classes?

pyupgrade pre-commit hook

Add a pyupgrade pre-commit hook once #15 has been merged.

Remove `BotoStore` and reliance on old `boto`

The Boto3Store is working fine. We should remove the BotoStore, probably as part of a 2.0.0 release.

Enable GCS tests in Github Actions

This requires setting up the fake-gcs-server in GHA.

Setup conda-forge packages

Once we have a first PyPI release, we should also submit a recipe to conda-forge.

Teach git-lfs to `GitCommitStore`

Hardcoded version in `docs/conf.py`

Here:

minimalkv/docs/conf.py

Line 45 in ed026cd

version = "0.14.1"

Use fsspec for store defintion

Many stores, like GoogleCloudStore, are written from scratch. It would be much easier to use the abstractions provided through fsspec, like gcsfs.

Store now using fsspec:

• GoogleCloudStore: #51
• Boto3Store: #66

Boto3 Store is untested code

In SimpleKV the Boto3 store was merged while the CI had been down for months: mbr/simplekv#106

So there was never a successful test-run on the CI. I spent an hour on it but couldn't figure out how to get it to connect to a Minio instance without bigger changes to the code. This is due to the client being created implicitly (contrary to explicitly like in the Boto store). I couldn't find a way to point the client to a local endpoint without modifying the global ~/.aws/config file.

The boto store still works, but the boto library hasn't seen any updates since 2018, so we should move to boto3 at some point definitely.

Ambivalent license

In the README.rst it is stated that the minimalkv license is MIT. However, the LICENSE file itself looks like the BSD-3 license. So which license is minimalkv distributed under? Would you mind making this a bit more clear?

Thank you for a great software package!

`get_store_from_url` fails due to missing import `redis`

Without redis installed, we currently get

_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
src/quantcore/thek/utils.py:698: in inner_f
    return f(*args, **kwargs)
src/quantcore/thek/view.py:134: in __init__
    if self.exists and partition_on is None:
src/quantcore/thek/kartothek_dataset.py:107: in exists
    return DatasetMetadata.exists(uuid=self.uuid, store=self.store())
/opt/conda/envs/quantcore.thek/lib/python3.7/site-packages/minimalkv/_get_store.py:55: in get_store_from_url
    return get_store(**url2dict(url))
/opt/conda/envs/quantcore.thek/lib/python3.7/site-packages/minimalkv/_get_store.py:145: in get_store
    store = create_store(type, params)
/opt/conda/envs/quantcore.thek/lib/python3.7/site-packages/minimalkv/_store_creation.py:23: in create_store
    return _create_store_hfs(type, params)
/opt/conda/envs/quantcore.thek/lib/python3.7/site-packages/minimalkv/_store_creation.py:114: in _create_store_hfs
    from minimalkv._hstores import HFilesystemStore
/opt/conda/envs/quantcore.thek/lib/python3.7/site-packages/minimalkv/_hstores.py:6: in <module>
    from minimalkv.memory.redisstore import RedisStore
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
    import re
    from io import BytesIO
    from typing import IO, Iterator, List, Optional, Union
>   from redis import StrictRedis
E   ModuleNotFoundError: No module named 'redis'

when running get_store_from_url in an environment without redis-py installed. This is not surprising, as we don't put the import into the class / function scopes as we e.g. do for the GoogleCloudStore. Similarly for the GitCommitStore. The question here IMO is how this worked before.

Build documentation on readthedocs

Move code from __init__ to (private) submodules

Is there any reason / benefit of defining objects directly in the __init__?

Add remaining tests to Github Actions

Currently being skipped:

• azure
• boto3 (test through minio)
• mongodb
• redis
• sqlalchemy

Delete, bc we cannot test it:

• boto (doesn't work with Python3)

This is how it used to be done on simplekv travis:
https://github.com/mbr/simplekv/blob/master/.travis.yml

before_script:
- bash .travis/start_minio.sh
- docker run -p 10000:10000 mcr.microsoft.com/azure-storage/azurite azurite-blob --blobHost 0.0.0.0 &
- docker run -d --name fake-gcs-server -p 4443:4443 fsouza/fake-gcs-server -scheme http
- psql -c 'create database simplekv_test;' -U postgres
- psql -c 'ALTER ROLE travis CONNECTION LIMIT -1;' -U postgres
- mysql -e 'create database simplekv_test;'
- mysql -e 'set global max_connections = 200;'

Add flake8 to pre-commit hooks

Followup to #31,

`test_git_store` takes approx 15min on 4 cores

This seems a bit excessive.

File-like interface of GoogleCloudStore fsspec implementation doesn't behave like previous releases

The FSSpecStoreEntry returned by open doesn't behave in the same way as previously. Will provide an MVE later.

This also seems to be a credentials issue. Explicitly specifying the OAuth scope at least changes the error message.

Will elaborate later.

Restrict possible return types of `_put` and `_delete` methods

I forgot: Do we (want to) restrict the return types of _put and _delete methods implemented by the backends? IIUC they generally return the key at which data was saved (or deleted).

Originally posted by @mlondschien in #25 (comment)

Support registering new backends

As a user of the minimalkv framework, I want to create and register new backends and use them through the get_store_from_url function without changing the minimalkv library. This is currently not possible, as the extract_params function hard-codes the known storage types:

https://github.com/data-engineering-collective/minimalkv/blob/main/minimalkv/_urls.py#L70-L122

What I'm imagining is a registration function that makes minimalkv aware of this new storage type, similarly to how fsspec does it.

Enforce black-formatting via pre-commit

mypy pre-commit Hook

Add a mypy pre-commit hook once #15 has been merged.

Convert docstrings to numpydoc style

... and check the docstring type with flake8. This could be done automatically (up to some fixes 🤞) using pyment.

Handle import of optional dependencies consistently

@xhochy suggested this pattern for optional dependencies like gcsfs:

minimalkv/minimalkv/net/gcstore.py

Lines 7 to 12 in cff85c0

	try:
	from gcsfs import GCSFileSystem
	has_gcsfs = True
	except ImportError:
	has_gcsfs = False

We should apply this to all the dependencies, e.g. dulwich:

minimalkv/minimalkv/git.py

Lines 6 to 7 in cff85c0

	from dulwich.objects import Blob, Commit, Tree
	from dulwich.repo import Repo

Add Codecov to CI

Seems relevant given #10.