redisbloom/redisbloom-py

Python client for Redisbloom

Python client for RedisBloom

Deprecation notice

As of redis-py 4.0.0 this library is deprecated. It's features have been merged into redis-py. Please either install it from pypy or the repo.

---

redisbloom-py is a package that gives developers easy access to several probabilistic data structures. The package extends redis-py's interface with RedisBloom's API.

Installation

$ pip install redisbloom

Usage example

# Using Bloom Filter
from redisbloom.client import Client
rb = Client()
rb.bfCreate('bloom', 0.01, 1000)
rb.bfAdd('bloom', 'foo')        # returns 1
rb.bfAdd('bloom', 'foo')        # returns 0
rb.bfExists('bloom', 'foo')     # returns 1
rb.bfExists('bloom', 'noexist') # returns 0

# Using Cuckoo Filter
from redisbloom.client import Client
rb = Client()
rb.cfCreate('cuckoo', 1000)
rb.cfAdd('cuckoo', 'filter')        # returns 1
rb.cfAddNX('cuckoo', 'filter')      # returns 0
rb.cfExists('cuckoo', 'filter')     # returns 1
rb.cfExists('cuckoo', 'noexist')    # returns 0

# Using Count-Min Sketch
from redisbloom.client import Client
rb = Client()
rb.cmsInitByDim('dim', 1000, 5)
rb.cmsIncrBy('dim', ['foo'], [5])
rb.cmsIncrBy('dim', ['foo', 'bar'], [5, 15])
rb.cmsQuery('dim', 'foo', 'bar')    # returns [10, 15]

# Using Top-K
from redisbloom.client import Client
rb = Client()
rb.topkReserve('topk', 3, 20, 3, 0.9)
rb.topkAdd('topk', 'A', 'B', 'C', 'D', 'E', 'A', 'A', 'B',
                   'C', 'G', 'D', 'B', 'D', 'A', 'E', 'E')
rb.topkQuery('topk', 'A', 'B', 'C', 'D')    # returns [1, 1, 0, 1]
rb.topkCount('topk', 'A', 'B', 'C', 'D')    # returns [4, 3, 2, 3]
rb.topkList('topk')                         # returns ['A', 'B', 'E']
rb.topkListWithCount('topk')                # returns ['A', 4, 'B', 3, 'E', 3]

API

For complete documentation about RedisBloom's commands, refer to RedisBloom's website.

License

BSD 3-Clause

Development

1. Create a virtualenv to manage your python dependencies, and ensure it's active. virtualenv -v venv
2. Install pypoetry to manage your dependencies. pip install poetry
3. Install dependencies. poetry install

tox runs all tests as its default target. Running tox by itself will run unit tests. Ensure you have a running redis, with the module loaded.

redisbloom-py's Issues

How to use bfInfo from redisbloom-py

Hello folks. There is a bfInfo function inside Client class from redisbloom. But whenever I try to use that command I got results like <redisbloom.client.BFInfo object at 0x7ff5e00d3290>.
Can you give me example please how can I print the entire information from real BF INFO command here using that function?

redis.exceptions.ResponseError: unknown command `BF.RESERVE`, with args beginning with: `bloom`, `0.01`, `1000`

rb = Client(host=custom_config.QINGSHIHE_HOST, port=custom_config.REDIS_PORT)
rb.bfCreate('bloom', 0.01, 1000)

fail:
redis.exceptions.ResponseError: unknown command BF.RESERVE, with args beginning with: bloom, 0.01, 1000

Support redis-py 4.1.0

It would be useful to use CMS in cluster-mode to count very large sets. That is only supported in redis-py 4.1.0 or greater.

Why 'cfAddNX' so many misjudgement?

I am doing some test of Cuckoo Filter with data set of range(1, 1e8)

redis_cuckoo.cfCreate(CK_FID, 100000000, bucket_size=10, ) cnt = 0 duplicate_cnt = 0 for i in range(100000000): exist = not bool(redis_cuckoo.cfAddNX(CK_FID, i)) cnt += 1 if cnt % 10000 == 0: print(cnt, duplicate_cnt) if exist: duplicate_cnt += 1

After nearly 1000k elements was added , error count was about ~22k, which make no sense for me .
can anybody give me some guidance. TYVM!

Latest release (0.4.0) installs incompatible version of redis-py

The latest released version of this package pins the redis dependency to >=2.1.0. Recently, version 4.0.0 of redis-py was released and introduces some breaking changes for this package.

The latest code in master appropriately pins the redis dependency to "^3.5.3" which would fix the issue.

To get around this, I've needed to pin the redis dependency to "^3.5.3" wherever our code requires redisbloom.

Enhancement to the redisbloom

@gkorland
User should be able to create create the object once and configure the application later to support it using init_app() method like other libraries SQLAlchemy, Bcrypt etc.

from redisbloom.client import Client

rb = Client()

def create_app():
     app = Flask(__name__)
     rb.init_app(app)
     return app

redisbloom should support configuration of Client object using 'app' configuration.

"invalid offset - no link found" on bfLoadChunk (Python 3.7.3)

When I try to dump and restore a Bloom filter using bfScandump and bfLoadChunk I get the following error:

>>> from redisbloom.client import Client
>>> r = Client()
>>> i, d = r.bfScandump('bloom', 0)
>>> r.bfLoadChunk('bloom', 0, d)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.7/dist-packages/redisbloom/client.py", line 257, in bfLoadChunk
    return self.execute_command(self.BF_LOADCHUNK, *params)
  File "/usr/local/lib/python3.7/dist-packages/redis/client.py", line 775, in execute_command
    return self.parse_response(connection, command_name, **options)
  File "/usr/local/lib/python3.7/dist-packages/redis/client.py", line 789, in parse_response
    response = connection.read_response()
  File "/usr/local/lib/python3.7/dist-packages/redis/connection.py", line 642, in read_response
    raise response
redis.exceptions.ResponseError: invalid offset - no link found
>>>

'bloom' is an existing Bloom filter.

I'm using Python 3.7.3 (2.7.16 gives the same error)

Support for T-Digest data structure

Given RedisBloom/RedisBloom#285 there is the following set of commands we should enable:

• TDIGEST.CREATE: Allocate a new histogram
• TDIGEST.RESET: Empty out a histogram and re-initialize it
• TDIGEST.ADD: Add a value to the t-Digest with the specified count
• TDIGEST.MERGE: Merge one t-Digest into another
• TDIGEST.CDF: Returns the fraction of all points added which are ≤ x.
• TDIGEST.QUANTILE: Returns an estimate of the cutoff such that a specified fraction of the data added to the t-Digest would be less than or equal to the cutoff.
• TDIGEST.MIN: Get the minimum value from the histogram. Will return DBL_MAX if the histogram is empty
• TDIGEST.MAX: Get the maximum value from the histogram. Will return DBL_MIN if the histogram is empty
• TDIGEST.INFO : Returns compression, capacity, total merged and unmerged nodes, the total compressions
  made up to date on that key, and merged and unmerged weight.

With the in-depth params in https://oss.redislabs.com/redisbloom/master/TDigest_Commands/

Unknown command `BF.RESERVE`

I am getting the error:

redis.exceptions.ResponseError: unknown command `BF.RESERVE`, with args beginning with: `bloom`, `0.01`, `1000`,

This happens when running the #Using Bloom Filter example from the documentation - I get the same error with other scripts as well.

Redis Server is up and receives the request:

57121:M 12 Sep 2020 11:08:12.964 - Accepted ::1:53467
57121:M 12 Sep 2020 11:08:12.970 - Client closed connection

Full stack trace:

Traceback (most recent call last):
  File "example.py", line 5, in <module>
    rb.bfCreate('bloom', 0.01, 1000)
  File "/Users/henrik/tmp/bloom/venv/lib/python3.7/site-packages/redisbloom/client.py", line 242, in bfCreate
    return self.execute_command(self.BF_RESERVE, *params)
  File "/Users/henrik/tmp/bloom/venv/lib/python3.7/site-packages/redis/client.py", line 901, in execute_command
    return self.parse_response(conn, command_name, **options)
  File "/Users/henrik/tmp/bloom/venv/lib/python3.7/site-packages/redis/client.py", line 915, in parse_response
    response = connection.read_response()
  File "/Users/henrik/tmp/bloom/venv/lib/python3.7/site-packages/redis/connection.py", line 756, in read_response
    raise response
redis.exceptions.ResponseError: unknown command `BF.RESERVE`, with args beginning with: `bloom`, `0.01`, `1000`,

pip freeze:

hiredis==1.1.0
redis==3.5.3
redisbloom==0.4.0
rmtest==0.7.0
six==1.15.0

Python 3.7.3.

format and check the style of Python code in PEP8 compliant manner

Blackened code looks the same regardless of the project and this is why I believe we should format and check for black style on commits.

Add support for BF.INFO and CF.INFO

mentioned on RedisBloom/RedisBloom#243

CF.DEL not work

Add and expose at runtime in a standard format the __version__ attribute

We should be able expose the package version at runtime via __version__ attribute while ensuring a single source of truth for version number.

Here's a discussion on how to standardize this info: https://stackoverflow.com/questions/458550/standard-way-to-embed-version-into-python-package

Example of redistimeseries-py:

attribute
https://github.com/RedisTimeSeries/redistimeseries-py/blob/master/redistimeseries/_version.py

setup.py way of reading it:
https://github.com/RedisTimeSeries/redistimeseries-py/blob/master/setup.py#L8