benbd86/v3io-fs

v3io fsspec (filesystem_spec) drivers

v3iofs

An fsspec driver for v3io.

Examples

Python

>>> from v3iofs import V3ioFS  # Will register 'v3io' protocol in ffspec
>>> fs = V3ioFS('api.app.yh48.iguazio-cd2.com', v3io_access_key='s3cr3t')
>>> fs.ls('/container/path')

Pandas

# Use V3IO_ACCESS_KEY & V3IO_API from environment
>> df = pd.read_csv('v3io://container/path/to/file.csv')

Dask

>>> from v3iofs import V3ioFS
>>> from dask import bag

# Use V3IO_ACCESS_KEY & V3IO_API from environment
>>> url = 'v3io://container/path'
>>> file = bag.read_text(url)
>>> data, _ = file.compute()

# Pass key in storage_options
>>> storage_options={
...     'v3io_api': 'webapi.app.iguazio.com',
...     'v3io_access_key': 's3cr3t',
... }
>>> file = bag.read_text(url, storage_options=storage_options)
>>> data, _ = file.compute()

Development

Testing

You need to set V3IO_ACCESS_KEY and V3IO_API environment variables. Then run make test to run the tests.

Environment

Deployment requirements are in requirements.txt and development requirements are in dev-requirements.txt.

$ python -m venv venv
$ ./venv/bin/python -m pip install -r dev-requirements.txt