projecteru/redis-trib.py

Redis Cluster lib in Python

redis-trib.py's Issues

rebalance and reshard support?

I want to use 'rebalance' and 'reshard', but I can't find in your project(redis-trib.py)

Where should I find it?
If not, do you plan to make it?

does it support python 3?

i think many projects use python3 nowadays,
so will it support python 3?
and i also want to know if this tool supports redis 4.x.

Problem slicing in function _add_slots() in pypy 7.0.0 (Python 2.7.13)

Stumbled on this yesterday and took some time to find as the code looked fine and required debugging to see the issue. Basically it seems like pypy converts uses of range to xrange in the background. This surfaces as an issue in the function _add_slots which is attempts to slice the list of slots provided to it by the _add_slots_range function supply a range as the slots_list parameter, yet pypy silently overriding this with xrange blows up as you can slice a generator.

Note: this is using redistrib version 0.6.1 installed via pip in virtualenv on Ubuntu 19.04

Attached two images to make it clear, along with my monkey patch hack to fix the issue quickly - though I'm sure someone else can do better.

Does redis-trib.py support redis 4.0 cluster ?

There is a error:

clusternode.py", line 13, in init
self.port = int(port)
ValueError: invalid literal for int() with base 10: '6379@16379'

Does redis-trib.py work with redis passwords?

not working whith python3.9

python 3.5, 3.7, 3.8 is fine.
test with conda.

Collecting redis-trib
  Using cached redis_trib-0.6.2-py3-none-any.whl
Collecting click==6.7
  Using cached click-6.7-py2.py3-none-any.whl (71 kB)
Collecting six==1.11.0
  Using cached six-1.11.0-py2.py3-none-any.whl (10 kB)
Collecting retrying==1.3.3
  Using cached retrying-1.3.3-py3-none-any.whl
Collecting hiredis==0.2.0
  Using cached hiredis-0.2.0.tar.gz (46 kB)
Collecting Werkzeug==0.14.1
  Using cached Werkzeug-0.14.1-py2.py3-none-any.whl (322 kB)
Building wheels for collected packages: hiredis
  Building wheel for hiredis (setup.py) ... error
  ERROR: Command errored out with exit status 1:
   command: /opt/miniconda3/envs/shake39/bin/python -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/bd/76t00dq94z57wqxmyznnbn180000gp/T/pip-install-43ofoctq/hiredis_43c050b304ac4a028d66702afcf64012/setup.py'"'"'; __file__='"'"'/private/var/folders/bd/76t00dq94z57wqxmyznnbn180000gp/T/pip-install-43ofoctq/hiredis_43c050b304ac4a028d66702afcf64012/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d /private/var/folders/bd/76t00dq94z57wqxmyznnbn180000gp/T/pip-wheel-21iqsjm8
       cwd: /private/var/folders/bd/76t00dq94z57wqxmyznnbn180000gp/T/pip-install-43ofoctq/hiredis_43c050b304ac4a028d66702afcf64012/
  Complete output (19 lines):
  /private/var/folders/bd/76t00dq94z57wqxmyznnbn180000gp/T/pip-install-43ofoctq/hiredis_43c050b304ac4a028d66702afcf64012/setup.py:9: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
    import sys, imp, os, glob
  running bdist_wheel
  running build
  running build_py
  creating build
  creating build/lib.macosx-10.9-x86_64-3.9
  creating build/lib.macosx-10.9-x86_64-3.9/hiredis
  copying hiredis/version.py -> build/lib.macosx-10.9-x86_64-3.9/hiredis
  copying hiredis/__init__.py -> build/lib.macosx-10.9-x86_64-3.9/hiredis
  running build_clib
  building 'hiredis_for_hiredis_py' library
  creating build/temp.macosx-10.9-x86_64-3.9
  creating build/temp.macosx-10.9-x86_64-3.9/vendor
  creating build/temp.macosx-10.9-x86_64-3.9/vendor/hiredis
  clang -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -fwrapv -O2 -Wall -fPIC -O2 -isystem /opt/miniconda3/envs/shake39/include -arch x86_64 -fPIC -O2 -isystem /opt/miniconda3/envs/shake39/include -arch x86_64 -c vendor/hiredis/read.c -o build/temp.macosx-10.9-x86_64-3.9/vendor/hiredis/read.o
  clang -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -fwrapv -O2 -Wall -fPIC -O2 -isystem /opt/miniconda3/envs/shake39/include -arch x86_64 -fPIC -O2 -isystem /opt/miniconda3/envs/shake39/include -arch x86_64 -c vendor/hiredis/sds.c -o build/temp.macosx-10.9-x86_64-3.9/vendor/hiredis/sds.o
  /opt/concourse/worker/volumes/live/77507ea3-3744-435d-535a-9a2fa7c142cb/volume/python-split_1627662353336/_build_env/bin/llvm-ar rcs build/temp.macosx-10.9-x86_64-3.9/libhiredis_for_hiredis_py.a build/temp.macosx-10.9-x86_64-3.9/vendor/hiredis/read.o build/temp.macosx-10.9-x86_64-3.9/vendor/hiredis/sds.o
  error: command '/opt/concourse/worker/volumes/live/77507ea3-3744-435d-535a-9a2fa7c142cb/volume/python-split_1627662353336/_build_env/bin/llvm-ar' failed: No such file or directory
  ----------------------------------------
  ERROR: Failed building wheel for hiredis
  Running setup.py clean for hiredis
Failed to build hiredis
Installing collected packages: six, Werkzeug, retrying, hiredis, click, redis-trib
    Running setup.py install for hiredis ... error
    ERROR: Command errored out with exit status 1:
     command: /opt/miniconda3/envs/shake39/bin/python -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/bd/76t00dq94z57wqxmyznnbn180000gp/T/pip-install-43ofoctq/hiredis_43c050b304ac4a028d66702afcf64012/setup.py'"'"'; __file__='"'"'/private/var/folders/bd/76t00dq94z57wqxmyznnbn180000gp/T/pip-install-43ofoctq/hiredis_43c050b304ac4a028d66702afcf64012/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /private/var/folders/bd/76t00dq94z57wqxmyznnbn180000gp/T/pip-record-wt9oi5zd/install-record.txt --single-version-externally-managed --compile --install-headers /opt/miniconda3/envs/shake39/include/python3.9/hiredis
         cwd: /private/var/folders/bd/76t00dq94z57wqxmyznnbn180000gp/T/pip-install-43ofoctq/hiredis_43c050b304ac4a028d66702afcf64012/
    Complete output (19 lines):
    /private/var/folders/bd/76t00dq94z57wqxmyznnbn180000gp/T/pip-install-43ofoctq/hiredis_43c050b304ac4a028d66702afcf64012/setup.py:9: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
      import sys, imp, os, glob
    running install
    running build
    running build_py
    creating build
    creating build/lib.macosx-10.9-x86_64-3.9
    creating build/lib.macosx-10.9-x86_64-3.9/hiredis
    copying hiredis/version.py -> build/lib.macosx-10.9-x86_64-3.9/hiredis
    copying hiredis/__init__.py -> build/lib.macosx-10.9-x86_64-3.9/hiredis
    running build_clib
    building 'hiredis_for_hiredis_py' library
    creating build/temp.macosx-10.9-x86_64-3.9
    creating build/temp.macosx-10.9-x86_64-3.9/vendor
    creating build/temp.macosx-10.9-x86_64-3.9/vendor/hiredis
    clang -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -fwrapv -O2 -Wall -fPIC -O2 -isystem /opt/miniconda3/envs/shake39/include -arch x86_64 -fPIC -O2 -isystem /opt/miniconda3/envs/shake39/include -arch x86_64 -c vendor/hiredis/read.c -o build/temp.macosx-10.9-x86_64-3.9/vendor/hiredis/read.o
    clang -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -fwrapv -O2 -Wall -fPIC -O2 -isystem /opt/miniconda3/envs/shake39/include -arch x86_64 -fPIC -O2 -isystem /opt/miniconda3/envs/shake39/include -arch x86_64 -c vendor/hiredis/sds.c -o build/temp.macosx-10.9-x86_64-3.9/vendor/hiredis/sds.o
    /opt/concourse/worker/volumes/live/77507ea3-3744-435d-535a-9a2fa7c142cb/volume/python-split_1627662353336/_build_env/bin/llvm-ar rcs build/temp.macosx-10.9-x86_64-3.9/libhiredis_for_hiredis_py.a build/temp.macosx-10.9-x86_64-3.9/vendor/hiredis/read.o build/temp.macosx-10.9-x86_64-3.9/vendor/hiredis/sds.o
    error: command '/opt/concourse/worker/volumes/live/77507ea3-3744-435d-535a-9a2fa7c142cb/volume/python-split_1627662353336/_build_env/bin/llvm-ar' failed: No such file or directory
    ----------------------------------------
ERROR: Command errored out with exit status 1: /opt/miniconda3/envs/shake39/bin/python -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/bd/76t00dq94z57wqxmyznnbn180000gp/T/pip-install-43ofoctq/hiredis_43c050b304ac4a028d66702afcf64012/setup.py'"'"'; __file__='"'"'/private/var/folders/bd/76t00dq94z57wqxmyznnbn180000gp/T/pip-install-43ofoctq/hiredis_43c050b304ac4a028d66702afcf64012/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /private/var/folders/bd/76t00dq94z57wqxmyznnbn180000gp/T/pip-record-wt9oi5zd/install-record.txt --single-version-externally-managed --compile --install-headers /opt/miniconda3/envs/shake39/include/python3.9/hiredis Check the logs for full command output.

when I try to start a single cluster node, it failed with hiredis error

redis-info
........

Cluster

cluster_enabled:1

Keyspace

10.1.100.219:7018> cluster info
cluster_state:fail
cluster_slots_assigned:0
cluster_slots_ok:0
cluster_slots_pfail:0
cluster_slots_fail:0
cluster_known_nodes:1
cluster_size:0
cluster_current_epoch:0
cluster_my_epoch:0
cluster_stats_messages_sent:0
cluster_stats_messages_received:0

$ redis-trib.py start 10.1.100.219:7018
Redis-trib 0.4.1 Copyright (c) HunanTV Platform developers
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/2.7/bin/redis-trib.py", line 11, in
sys.exit(main())
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/redistrib/console.py", line 81, in main
getattr(sys.modules[name], sys.argv[1])(_sys.argv[2:])
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/redistrib/console.py", line 14, in start
command.start_cluster(__parse_host_port(host_port))
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/redistrib/command.py", line 87, in start_cluster
_add_slots(t, 0, SLOT_COUNT, max_slots)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/redistrib/command.py", line 81, in _add_slots
addslots(t, begin, end)
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/redistrib/command.py", line 71, in addslots
m = t.talk('cluster', 'addslots', _xrange(begin, end))
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/redistrib/clusternode.py", line 99, in talk
return self.talk_raw(pack_command(_args))
File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/redistrib/clusternode.py", line 95, in talk_raw
raise r
hiredis.ReplyError: ERR Protocol error: expected '$', got '8'

不错

不错 ！

installation issues

pip install redis-trib
Collecting redis-trib
Using cached https://files.pythonhosted.org/packages/51/53/c59f606785dd687af6e61d12587f96d1d3f191c41932d13f5544c86bffc6/redis-trib-0.5.2.tar.gz
Complete output from command python setup.py egg_info:
Traceback (most recent call last):
File "", line 1, in
File "/tmp/pip-install-0nhYwk/redis-trib/setup.py", line 5, in
with open('requirements.txt', 'r') as reqin:
IOError: [Errno 2] No such file or directory: 'requirements.txt'

Run slowly

when i use flow command migrate slots :
redis-trib.py migrate --src-addr 10.105.91.60:11001 --dst-addr 10.105.91.60:11003 10923-16383

Just migrate 2014177 keys | 634 slots for a new node costed about 3 hours.

Whether considering about optimizing? @neuront