stephan-hof / pyrocksdb Goto Github PK
View Code? Open in Web Editor NEWPython bindings for RocksDB
License: BSD 3-Clause "New" or "Revised" License
pyrocksdb's People
Contributors
Stargazers
Watchers
Forkers
vmx alexandrem merlin83 gisupc janssen freeseacher rayleyva peterhj grove zhaoyan1117 balena johnistan chester89 tempbottle ummae adamchainz congnghia0609 awesome-python syslot eminarcissus chenghao oddconcepts twmht shpala divyekapoor jaydeng2837 forsterzhang oldeucryptoboi agneinlc hypocalese alienflash01 lumosnysm njn007 mosquito naik-amey rensjepyrocksdb's Issues
Assertion `data[n] != kNoCompression' failed
This happens when I was dumping data to RockStore, see code here:
https://github.com/yinhm/datafeed/blob/tick/datafeed/datastore.py#L497
around 740K keys dumped, this hit this error.
python: table/format.cc:191: rocksdb::Status rocksdb::UncompressBlockContents(const char*, size_t, rocksdb::BlockContents*): Assertion `data[n] != kNoCompression' failed.
It happens couple of times when I was running unittest, restart will fix it.
While this time I did restart twice to get it fixed.
The successful one LOG are here(tailed):
2014/04/26-17:12:16.835834 7f357508f700 Expanding@1 1+1 (17843648+3718462 bytes) to 2+1 (19149831+3718462 bytes)
2014/04/26-17:12:16.835852 7f357508f700 (Original Log Time 2014/04/26-17:12:16.835844) [CF 0] Compacting 2@1 + 1@2 files, score 1.83 slots available 0
2014/04/26-17:12:16.835855 7f357508f700 (Original Log Time 2014/04/26-17:12:16.835849) Compaction start summary: Base version 2 Base level 1, seek compaction:0, inputs: [24(17MB) 25(1275KB) ],[5(3631KB) ]
2014/04/26-17:12:21.159493 7f357508f700 Table was constructed:
[basic properties]: # data blocks=18108; # entries=679188; raw key size=18338085; raw average key size=27.000013; raw value size=56379081; raw average value size=83.009536; data block size=16777713; index block size=531228; filter block size=903079; (estimated) table size=18212020; filter policy name=rocksdb.BuiltinBloomFilter;
[user collected properties]: kDeletedKeys=0;
2014/04/26-17:12:21.227518 7f357508f700 Generated table #29: 679188 keys, 18013951 bytes
2014/04/26-17:12:21.657262 7f357508f700 Table was constructed:
[basic properties]: # data blocks=1734; # entries=65414; raw key size=1766181; raw average key size=27.000046; raw value size=5405124; raw average value size=82.629468; data block size=1742797; index block size=48636; filter block size=87470; (estimated) table size=1878903; filter policy name=rocksdb.BuiltinBloomFilter;
[user collected properties]: kDeletedKeys=0;
2014/04/26-17:12:21.664842 7f357508f700 Generated table #30: 65414 keys, 1860860 bytes
2014/04/26-17:12:21.666518 7f357508f700 (Original Log Time 2014/04/26-17:12:21.665545) Compacted 2@1 + 1@2 files => 19874811 bytes
2014/04/26-17:12:21.666522 7f357508f700 (Original Log Time 2014/04/26-17:12:21.666480) compacted to: files[0 0 2 0 0 0 0 ], 8.9 MB/sec, level 2, files in(2, 1) out(2) MB in(18.3, 3.5) out(19.0), read-write-amplify(2.2) write-amplify(1.0) OK
2014/04/26-17:12:21.667129 7f357508f700 Delete ....../datafeed/var/rdb/000005.sst type=2 #5 -- OK
2014/04/26-17:12:21.669623 7f357508f700 Delete ....../datafeed/var/rdb/000024.sst type=2 #24 -- OK
2014/04/26-17:12:21.669805 7f357508f700 Delete .....t/datafeed/var/rdb/000025.sst type=2 #25 -- OK
The failed one LOGs stopped at:
first time trying to restart:
2014/04/26-17:12:07.015352 7f0cc1d83740 Recovering log #15
2014/04/26-17:12:07.015649 7f0cc1d83740 Level-0 table #18: started
2014/04/26-17:12:07.019641 7f0cc1d83740 Table was constructed:
[basic properties]: # data blocks=13; # entries=452; raw key size=12216; raw average key size=27.026549; raw value size=37303; raw average value size=82.528761; data block size=14739; index block size=381; filter block size=629; (estimated) table size=15749; filter policy name=rocksdb.BuiltinBloomFilter;
[user collected properties]: kDeletedKeys=0;
2014/04/26-17:12:07.020826 7f0cc1d83740 Level-0 table #18: 16174 bytes OK
2014/04/26-17:12:07.022957 7f0cc1d83740 Deleting manifest 6 current manifest 19
2014/04/26-17:12:07.023168 7f0cc1d83740 Delete ......datafeed/var/rdb/000015.log type=0 #15 -- OK
2014/04/26-17:12:07.023197 7f0cc1d83740 Delete ....../datafeed/var/rdb/000017.sst type=2 #17 -- OK
2014/04/26-17:12:07.024267 7f0cb43ec700 Compactions
Level Files Size(MB) Score Time(sec) Read(MB) Write(MB) Rn(MB) Rnp1(MB) Wnew(MB) RW-Amplify Read(MB/s) Write(MB/s) Rn Rnp1 Wnp1 NewW Count msComp msStall Ln-stall Stall-cnt
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
0 5 15 1.5 0 0 0 0 0 0 0.0 0.0 3.0 0 0 1 1 1 2 0.0 0.0 0
1 1 3 0.3 0 0 0 0 0 0 0.0 0.0 0.0 0 0 0 0 0 0 0.0 0.0 0
2 1 4 0.0 0 0 0 0 0 0 0.0 0.0 0.0 0 0 0 0 0 0 0.0 0.0 0
Uptime(secs): 0.0 total, 0.0 interval
Writes cumulative: 0 total, 0 batches, 0.0 per batch, 0.00 ingest GB
WAL cumulative: 0 WAL writes, 0 WAL syncs, 0.00 writes per sync, 0.00 GB written
Compaction IO cumulative (GB): 0.00 new, 0.00 read, 0.00 write, 0.00 read+write
Compaction IO cumulative (MB/sec): 0.0 new, 0.0 read, 1.7 write, 1.7 read+write
Amplification cumulative: 16174.0 write, 16174.0 compaction
Writes interval: 0 total, 0 batches, 0.0 per batch, 0.0 ingest MB
WAL interval: 0 WAL writes, 0 WAL syncs, 0.00 writes per sync, 0.00 MB written
Compaction IO interval (MB): 0.00 new, 0.00 read, 0.02 write, 0.02 read+write
Compaction IO interval (MB/sec): 0.0 new, 0.0 read, 1.7 write, 1.7 read+write
Amplification interval: 16174.0 write, 16174.0 compaction
Stalls(secs): 0.000 level0_slowdown, 0.000 level0_numfiles, 0.000 memtable_compaction, 0.000 leveln_slowdown
Stalls(count): 0 level0_slowdown, 0 level0_numfiles, 0 memtable_compaction, 0 leveln_slowdown
2014/04/26-17:12:07.024342 7f0cb43ec700 (Original Log Time 2014/04/26-17:12:07.024321) [CF 0] Compacting 5@0 + 1@1 files, score 1.25 slots available 0
2014/04/26-17:12:07.024349 7f0cb43ec700 (Original Log Time 2014/04/26-17:12:07.024336) Compaction start summary: Base version 2 Base level 0, seek compaction:0, inputs: [18(15KB) 16(3943KB) 14(3993KB) 12(3778KB) 10(3537KB) ],[8(3435KB) ]
Second time, also failed:
2014/04/26-17:12:09.157656 7f74f37a0740 Column family "default", log number is 16
2014/04/26-17:12:09.157705 7f74f37a0740 Recovering log #20
2014/04/26-17:12:09.160369 7f74f37a0740 Deleting manifest 19 current manifest 22
2014/04/26-17:12:09.160478 7f74f37a0740 Delete ....../datafeed/var/rdb/000020.log type=0 #20 -- OK
2014/04/26-17:12:09.161592 7f74e5e09700 Compactions
Level Files Size(MB) Score Time(sec) Read(MB) Write(MB) Rn(MB) Rnp1(MB) Wnew(MB) RW-Amplify Read(MB/s) Write(MB/s) Rn Rnp1 Wnp1 NewW Count msComp msStall
Ln-stall Stall-cnt
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
--------------------
0 5 15 1.5 0 0 0 0 0 0 0.0 0.0 0.0 0 0 0 0 0 0 0.0 0.0 0
1 1 3 0.3 0 0 0 0 0 0 0.0 0.0 0.0 0 0 0 0 0 0 0.0 0.0 0
2 1 4 0.0 0 0 0 0 0 0 0.0 0.0 0.0 0 0 0 0 0 0 0.0 0.0 0
Uptime(secs): 0.0 total, 0.0 interval
Writes cumulative: 0 total, 0 batches, 0.0 per batch, 0.00 ingest GB
WAL cumulative: 0 WAL writes, 0 WAL syncs, 0.00 writes per sync, 0.00 GB written
Compaction IO cumulative (GB): 0.00 new, 0.00 read, 0.00 write, 0.00 read+write
Compaction IO cumulative (MB/sec): 0.0 new, 0.0 read, 0.0 write, 0.0 read+write
Amplification cumulative: 0.0 write, 0.0 compaction
Writes interval: 0 total, 0 batches, 0.0 per batch, 0.0 ingest MB
WAL interval: 0 WAL writes, 0 WAL syncs, 0.00 writes per sync, 0.00 MB written
Compaction IO interval (MB): 0.00 new, 0.00 read, 0.00 write, 0.00 read+write
Compaction IO interval (MB/sec): 0.0 new, 0.0 read, 0.0 write, 0.0 read+write
Amplification interval: 0.0 write, 0.0 compaction
Stalls(secs): 0.000 level0_slowdown, 0.000 level0_numfiles, 0.000 memtable_compaction, 0.000 leveln_slowdown
Stalls(count): 0 level0_slowdown, 0 level0_numfiles, 0 memtable_compaction, 0 leveln_slowdown
2014/04/26-17:12:09.161653 7f74e5e09700 (Original Log Time 2014/04/26-17:12:09.161634) [CF 0] Compacting 5@0 + 1@1 files, score 1.25 slots available 0
2014/04/26-17:12:09.161661 7f74e5e09700 (Original Log Time 2014/04/26-17:12:09.161648) Compaction start summary: Base version 2 Base level 0, seek compaction:0, inputs: [18(15KB) 16(3943KB) 14(3993KB) 12(3778KB) 10(3537KB) ],[8(3435KB) ]
2014/04/26-17:12:12.972832 7f74e5e09700 Table was constructed:
[basic properties]: # data blocks=15470; # entries=579520; raw key size=15647049; raw average key size=27.000016; raw value size=48182953; raw average value size=83.142865; data block size=16777569; index block size=451211; filter block size=778409; (estimated) table size=18007189; filter policy name=rocksdb.BuiltinBloomFilter;
[user collected properties]: kDeletedKeys=0;
2014/04/26-17:12:13.039513 7f74e5e09700 Generated table #24: 579520 keys, 17843648 bytes
2014/04/26-17:12:13.301210 7f74e5e09700 Table was constructed:
[basic properties]: # data blocks=1071; # entries=40432; raw key size=1091667; raw average key size=27.000074; raw value size=3340079; raw average value size=82.609789; data block size=1232307; index block size=30100; filter block size=54587; (estimated) table size=1316994; filter policy name=rocksdb.BuiltinBloomFilter;
[user collected properties]: kDeletedKeys=0;
2014/04/26-17:12:13.039513 7f74e5e09700 Generated table #24: 579520 keys, 17843648 bytes
2014/04/26-17:12:13.301210 7f74e5e09700 Table was constructed:
[basic properties]: # data blocks=1071; # entries=40432; raw key size=1091667; raw average key size=27.000074; raw value size=3340079; raw average value size=82.609789; data block size=1232307; index block size=30100; filter block size=54587; (estimated) table size=1316994; filter policy name=rocksdb.BuiltinBloomFilter;
[user collected properties]: kDeletedKeys=0;
2014/04/26-17:12:13.306339 7f74e5e09700 Generated table #25: 40432 keys, 1306183 bytes
2014/04/26-17:12:13.308060 7f74e5e09700 (Original Log Time 2014/04/26-17:12:13.307053) Compacted 5@0 + 1@1 files => 19149831 bytes
2014/04/26-17:12:13.308065 7f74e5e09700 (Original Log Time 2014/04/26-17:12:13.308024) compacted to: files[0 2 1 0 0 0 0 ], 9.2 MB/sec, level 1, files in(5, 1) out(2) MB in(14.9, 3.4) out(18.3), read-write-amplify(2.4) write-amplify(1.2) OK
2014/04/26-17:12:13.308593 7f74e5e09700 Delete ....../datafeed/var/rdb/000008.sst type=2 #8 -- OK
2014/04/26-17:12:13.309492 7f74e5e09700 Delete ....../datafeed/var/rdb/000010.sst type=2 #10 -- OK
2014/04/26-17:12:13.310447 7f74e5e09700 Delete ....../datafeed/var/rdb/000012.sst type=2 #12 -- OK
2014/04/26-17:12:13.311463 7f74e5e09700 Delete ....../datafeed/var/rdb/000014.sst type=2 #14 -- OK
2014/04/26-17:12:13.312466 7f74e5e09700 Delete ....../datafeed/var/rdb/000016.sst type=2 #16 -- OK
2014/04/26-17:12:13.312514 7f74e5e09700 Delete ....../datafeed/var/rdb/000018.sst type=2 #18 -- OK
2014/04/26-17:12:13.312549 7f74e5e09700 Expanding@1 1+1 (17843648+3718462 bytes) to 2+1 (19149831+3718462 bytes)
2014/04/26-17:12:13.312580 7f74e5e09700 (Original Log Time 2014/04/26-17:12:13.312566) [CF 0] Compacting 2@1 + 1@2 files, score 1.83 slots available 0
2014/04/26-17:12:13.312586 7f74e5e09700 (Original Log Time 2014/04/26-17:12:13.312576) Compaction start summary: Base version 3 Base level 1, seek compaction:0, inputs: [24(17MB) 25(1275KB) ],[5(3631KB) ]
It seems recovered after some sst files got deleted, I do not sure it this related to pyrocksdb, or a rocksdb I was hitting.
filter_deletes method deprecated
Recent rocksdb commit has deprecated 'filter_deletes' methods
- commit: facebook/rocksdb@7b79238
Because of this pip install pyrocksdb is failing on recently installed rocksdb.
Cannot install pyrocksdb, with python3.5.2, due to warning: the result of the conversion is unspecified because ‘-1’ is outside the range of type ‘rocksdb::CompressionType’ [-Wconversion]
I am on Linux Mint 18 KDE, was able to install rocksdb as shared library. gcc -v output
Using built-in specs.
COLLECT_GCC=gcc
COLLECT_LTO_WRAPPER=/usr/lib/gcc/x86_64-linux-gnu/5/lto-wrapper
Target: x86_64-linux-gnu
Configured with: ../src/configure -v --with-pkgversion='Ubuntu 5.4.0-6ubuntu1~16.04.2' --with-bugurl=file:///usr/share/doc/gcc-5/README.Bugs --enable-languages=c,ada,c++,java,go,d,fortran,objc,obj-c++ --prefix=/usr --program-suffix=-5 --enable-shared --enable-linker-build-id --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --libdir=/usr/lib --enable-nls --with-sysroot=/ --enable-clocale=gnu --enable-libstdcxx-debug --enable-libstdcxx-time=yes --with-default-libstdcxx-abi=new --enable-gnu-unique-object --disable-vtable-verify --enable-libmpx --enable-plugin --with-system-zlib --disable-browser-plugin --enable-java-awt=gtk --enable-gtk-cairo --with-java-home=/usr/lib/jvm/java-1.5.0-gcj-5-amd64/jre --enable-java-home --with-jvm-root-dir=/usr/lib/jvm/java-1.5.0-gcj-5-amd64 --with-jvm-jar-dir=/usr/lib/jvm-exports/java-1.5.0-gcj-5-amd64 --with-arch-directory=amd64 --with-ecj-jar=/usr/share/java/eclipse-ecj.jar --enable-objc-gc --enable-multiarch --disable-werror --with-arch-32=i686 --with-abi=m64 --with-multilib-list=m32,m64,mx32 --enable-multilib --with-tune=generic --enable-checking=release --build=x86_64-linux-gnu --host=x86_64-linux-gnu --target=x86_64-linux-gnu
Thread model: posix
gcc version 5.4.0 20160609 (Ubuntu 5.4.0-6ubuntu1~16.04.2)
But when i do pip install pyrocksdb, i get this error
running install
running build
running build_py
creating build
creating build/lib.linux-x86_64-3.5
creating build/lib.linux-x86_64-3.5/rocksdb
copying rocksdb/interfaces.py -> build/lib.linux-x86_64-3.5/rocksdb
copying rocksdb/errors.py -> build/lib.linux-x86_64-3.5/rocksdb
copying rocksdb/__init__.py -> build/lib.linux-x86_64-3.5/rocksdb
creating build/lib.linux-x86_64-3.5/rocksdb/tests
copying rocksdb/tests/test_options.py -> build/lib.linux-x86_64-3.5/rocksdb/tests
copying rocksdb/tests/test_db.py -> build/lib.linux-x86_64-3.5/rocksdb/tests
copying rocksdb/tests/__init__.py -> build/lib.linux-x86_64-3.5/rocksdb/tests
running egg_info
writing dependency_links to pyrocksdb.egg-info/dependency_links.txt
writing requirements to pyrocksdb.egg-info/requires.txt
writing pyrocksdb.egg-info/PKG-INFO
writing top-level names to pyrocksdb.egg-info/top_level.txt
warning: manifest_maker: standard file '-c' not found
reading manifest file 'pyrocksdb.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
writing manifest file 'pyrocksdb.egg-info/SOURCES.txt'
copying rocksdb/_rocksdb.cpp -> build/lib.linux-x86_64-3.5/rocksdb
copying rocksdb/_rocksdb.pyx -> build/lib.linux-x86_64-3.5/rocksdb
copying rocksdb/backup.pxd -> build/lib.linux-x86_64-3.5/rocksdb
copying rocksdb/cache.pxd -> build/lib.linux-x86_64-3.5/rocksdb
copying rocksdb/comparator.pxd -> build/lib.linux-x86_64-3.5/rocksdb
copying rocksdb/db.pxd -> build/lib.linux-x86_64-3.5/rocksdb
copying rocksdb/env.pxd -> build/lib.linux-x86_64-3.5/rocksdb
copying rocksdb/filter_policy.pxd -> build/lib.linux-x86_64-3.5/rocksdb
copying rocksdb/iterator.pxd -> build/lib.linux-x86_64-3.5/rocksdb
copying rocksdb/logger.pxd -> build/lib.linux-x86_64-3.5/rocksdb
copying rocksdb/memtablerep.pxd -> build/lib.linux-x86_64-3.5/rocksdb
copying rocksdb/merge_operator.pxd -> build/lib.linux-x86_64-3.5/rocksdb
copying rocksdb/options.pxd -> build/lib.linux-x86_64-3.5/rocksdb
copying rocksdb/slice_.pxd -> build/lib.linux-x86_64-3.5/rocksdb
copying rocksdb/slice_transform.pxd -> build/lib.linux-x86_64-3.5/rocksdb
copying rocksdb/snapshot.pxd -> build/lib.linux-x86_64-3.5/rocksdb
copying rocksdb/status.pxd -> build/lib.linux-x86_64-3.5/rocksdb
copying rocksdb/std_memory.pxd -> build/lib.linux-x86_64-3.5/rocksdb
copying rocksdb/table_factory.pxd -> build/lib.linux-x86_64-3.5/rocksdb
copying rocksdb/universal_compaction.pxd -> build/lib.linux-x86_64-3.5/rocksdb
creating build/lib.linux-x86_64-3.5/rocksdb/cpp
copying rocksdb/cpp/comparator_wrapper.hpp -> build/lib.linux-x86_64-3.5/rocksdb/cpp
copying rocksdb/cpp/filter_policy_wrapper.hpp -> build/lib.linux-x86_64-3.5/rocksdb/cpp
copying rocksdb/cpp/memtable_factories.hpp -> build/lib.linux-x86_64-3.5/rocksdb/cpp
copying rocksdb/cpp/merge_operator_wrapper.hpp -> build/lib.linux-x86_64-3.5/rocksdb/cpp
copying rocksdb/cpp/slice_transform_wrapper.hpp -> build/lib.linux-x86_64-3.5/rocksdb/cpp
copying rocksdb/cpp/utils.hpp -> build/lib.linux-x86_64-3.5/rocksdb/cpp
copying rocksdb/cpp/write_batch_iter_helper.hpp -> build/lib.linux-x86_64-3.5/rocksdb/cpp
running build_ext
building 'rocksdb._rocksdb' extension
creating build/temp.linux-x86_64-3.5
creating build/temp.linux-x86_64-3.5/rocksdb
x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/usr/include/python3.5m -I/home/lovesh/.virtualenvs/db-bench/include/python3.5m -c rocksdb/_rocksdb.cpp -o build/temp.linux-x86_64-3.5/rocksdb/_rocksdb.o -std=c++11 -O3 -Wall -Wextra -Wconversion -fno-strict-aliasing
cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++
rocksdb/_rocksdb.cpp: In function ‘PyObject* __pyx_pf_7rocksdb_8_rocksdb_19PyBloomFilterPolicy_4create_filter(__pyx_obj_7rocksdb_8_rocksdb_PyBloomFilterPolicy*, PyObject*)’:
rocksdb/_rocksdb.cpp:6580:105: warning: conversion to ‘int’ from ‘std::vector<rocksdb::Slice>::size_type {aka long unsigned int}’ may alter its value [-Wconversion]
__pyx_t_6->CreateFilter(py_rocks::vector_data(__pyx_v_c_keys), __pyx_v_c_keys.size(), (&__pyx_v_dst));
^
rocksdb/_rocksdb.cpp: In function ‘PyObject* __pyx_pf_7rocksdb_8_rocksdb_7Options_14filter_deletes___get__(__pyx_obj_7rocksdb_8_rocksdb_Options*)’:
rocksdb/_rocksdb.cpp:19539:57: error: ‘struct rocksdb::Options’ has no member named ‘filter_deletes’
__pyx_t_1 = __Pyx_PyBool_FromLong(__pyx_v_self->opts->filter_deletes); if (unlikely(!__pyx_t_1)) __PYX_ERR(0, 1142, __pyx_L1_error)
^
rocksdb/_rocksdb.cpp:581:36: note: in definition of macro ‘__Pyx_PyBool_FromLong’
#define __Pyx_PyBool_FromLong(b) ((b) ? __Pyx_NewRef(Py_True) : __Pyx_NewRef(Py_False))
^
rocksdb/_rocksdb.cpp: In function ‘int __pyx_pf_7rocksdb_8_rocksdb_7Options_14filter_deletes_2__set__(__pyx_obj_7rocksdb_8_rocksdb_Options*, PyObject*)’:
rocksdb/_rocksdb.cpp:19599:23: error: ‘struct rocksdb::Options’ has no member named ‘filter_deletes’
__pyx_v_self->opts->filter_deletes = __pyx_t_1;
^
rocksdb/_rocksdb.cpp: In function ‘PyObject* __Pyx_PyInt_From_rocksdb_3a__3a_CompressionType(rocksdb::CompressionType)’:
rocksdb/_rocksdb.cpp:37381:74: warning: the result of the conversion is unspecified because ‘-1’ is outside the range of type ‘rocksdb::CompressionType’ [-Wconversion]
const rocksdb::CompressionType neg_one = (rocksdb::CompressionType) -1, const_zero = (rocksdb::CompressionType) 0;
^
error: command 'x86_64-linux-gnu-gcc' failed with exit status 1
Support for column families
This would be very nice to have in pyrocksdb.
Close db
The rocksdb has a "close db" - https://github.com/facebook/rocksdb/wiki/Basic-Operations#closing-a-database - though it is strange to call it a "delete db" - or, do they really mean delete the db?
In pyrocksdb, could not find a close db api/function?
Error while installing pyrocksdb on Ubuntu
cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++ [enabled by default]
rocksdb/_rocksdb.cpp: In function ‘PyObject* __pyx_pf_7rocksdb_8_rocksdb_19PyBloomFilterPolicy_6create_filter(__pyx_obj_7rocksdb_8_rocksdb_PyBloomFilterPolicy*, PyObject*)’:
rocksdb/_rocksdb.cpp:5239:99: warning: conversion to ‘int’ from ‘std::vector<rocksdb::Slice>::size_type {aka long unsigned int}’ may alter its value [-Wconversion]
__pyx_v_self->policy->CreateFilter(py_rocks::vector_data(__pyx_v_c_keys), __pyx_v_c_keys.size(), (&__pyx_v_dst));
^
rocksdb/_rocksdb.cpp: In function ‘int __pyx_pf_7rocksdb_8_rocksdb_7Options_16prefix_extractor_2__set__(__pyx_obj_7rocksdb_8_rocksdb_Options*, PyObject*)’:
rocksdb/_rocksdb.cpp:17345:40: error: cannot convert ‘std::shared_ptr<rocksdb::SliceTransform>’ to ‘const rocksdb::SliceTransform*’ in assignment
__pyx_v_self->opts->prefix_extractor = ((struct __pyx_vtabstruct_7rocksdb_8_rocksdb_PySliceTransform *)__pyx_v_self->py_prefix_extractor->__pyx_vtab)->get_transformer(__pyx_v_self->py_prefix_extractor);
^
error: command 'x86_64-linux-gnu-gcc' failed with exit status 1
----------------------------------------
Cleaning up...
Command /usr/bin/python -c "import setuptools, tokenize;__file__='/tmp/pip-0vH8ot-build/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-Wrtox1-record/install-record.txt --single-version-externally-managed --compile failed with error code 1 in /tmp/pip-0vH8ot-build
Traceback (most recent call last):
File "/usr/bin/pip", line 9, in <module>
load_entry_point('pip==1.5.4', 'console_scripts', 'pip')()
File "/usr/lib/python2.7/dist-packages/pip/__init__.py", line 185, in main
return command.main(cmd_args)
File "/usr/lib/python2.7/dist-packages/pip/basecommand.py", line 161, in main
text = '\n'.join(complete_log)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 42: ordinal not in range(128)Support unstable ZSTD
I don't know if you want to bring this in yet, but the current RocksDB has a compression stub now for ZSTD as kZSTDNotFinalCompression.
zstd is quite a bit faster than zlib with a similar compression ratio. It's not yet stable/mature. Same author as LZ4.
pyrocksdb install issue
Hello,
Please help. I just installed rocksdb and followed instructions in your pyrocksdb page (and consulted with http://www.guguncube.com/2925/rocksdb-installation-and-performance-tests as well). I am getting the following error.
Please help me fix it.
Platform version: Ubuntu 14.04
thanks,
Buvana
Error message follows-----
(pyrocks_test):~/rocksdb/rocksdb/pyrocks_test$ pip install pyrocksdb
Downloading/unpacking pyrocksdb
Downloading pyrocksdb-0.4.tar.gz (139kB): 139kB downloaded
Running setup.py (path:/nas1/buvana/rocksdb/rocksdb/pyrocks_test/build/pyrocksdb/setup.py) egg_info for package pyrocksdb
Requirement already satisfied (use --upgrade to upgrade): setuptools in ./lib/python2.7/site-packages (from pyrocksdb)
Installing collected packages: pyrocksdb
Running setup.py install for pyrocksdb
building 'rocksdb._rocksdb' extension
x86_64-linux-gnu-gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC -I/usr/include/python2.7 -c rocksdb/_rocksdb.cpp -o build/temp.linux-x86_64-2.7/rocksdb/_rocksdb.o -std=c++11 -O3 -Wall -Wextra -Wconversion -fno-strict-aliasing
cc1plus: warning: command line option \u2018-Wstrict-prototypes\u2019 is valid for C/ObjC but not for C++ [enabled by default]
rocksdb/_rocksdb.cpp: In function \u2018PyObject* __pyx_pf_7rocksdb_8_rocksdb_19PyBloomFilterPolicy_4create_filter(__pyx_obj_7rocksdb_8_rocksdb_PyBloomFilterPolicy*, PyObject*)\u2019:
rocksdb/_rocksdb.cpp:5639:88: warning: conversion to \u2018int\u2019 from \u2018std::vector<rocksdb::Slice>::size_type {aka long unsigned int}\u2019 may alter its value [-Wconversion]
__pyx_t_6->CreateFilter(py_rocks::vector_data(__pyx_v_c_keys), __pyx_v_c_keys.size(), (&__pyx_v_dst));
^
rocksdb/_rocksdb.cpp: In function \u2018PyObject* __pyx_pf_7rocksdb_8_rocksdb_7Options_26expanded_compaction_factor___get__(__pyx_obj_7rocksdb_8_rocksdb_Options*)\u2019:
rocksdb/_rocksdb.cpp:13229:56: error: \u2018struct rocksdb::Options\u2019 has no member named \u2018expanded_compaction_factor\u2019
__pyx_t_1 = __Pyx_PyInt_From_int(__pyx_v_self->opts->expanded_compaction_factor); if (unlikely(!__pyx_t_1)) {__pyx_filename = __pyx_f[0]; __pyx_lineno = 873; __pyx_clineno = __LINE__; goto __pyx_L1_error;}
^
rocksdb/_rocksdb.cpp: In function \u2018int __pyx_pf_7rocksdb_8_rocksdb_7Options_26expanded_compaction_factor_2__set__(__pyx_obj_7rocksdb_8_rocksdb_Options*, PyObject*)\u2019:
rocksdb/_rocksdb.cpp:13292:23: error: \u2018struct rocksdb::Options\u2019 has no member named \u2018expanded_compaction_factor\u2019
__pyx_v_self->opts->expanded_compaction_factor = __pyx_t_1;
^
rocksdb/_rocksdb.cpp: In function \u2018PyObject* __pyx_pf_7rocksdb_8_rocksdb_7Options_24source_compaction_factor___get__(__pyx_obj_7rocksdb_8_rocksdb_Options*)\u2019:
rocksdb/_rocksdb.cpp:13351:56: error: \u2018struct rocksdb::Options\u2019 has no member named \u2018source_compaction_factor\u2019
__pyx_t_1 = __Pyx_PyInt_From_int(__pyx_v_self->opts->source_compaction_factor); if (unlikely(!__pyx_t_1)) {__pyx_filename = __pyx_f[0]; __pyx_lineno = 879; __pyx_clineno = __LINE__; goto __pyx_L1_error;}
^
rocksdb/_rocksdb.cpp: In function \u2018int __pyx_pf_7rocksdb_8_rocksdb_7Options_24source_compaction_factor_2__set__(__pyx_obj_7rocksdb_8_rocksdb_Options*, PyObject*)\u2019:
rocksdb/_rocksdb.cpp:13414:23: error: \u2018struct rocksdb::Options\u2019 has no member named \u2018source_compaction_factor\u2019
__pyx_v_self->opts->source_compaction_factor = __pyx_t_1;
^
rocksdb/_rocksdb.cpp: In function \u2018PyObject* __pyx_pf_7rocksdb_8_rocksdb_7Options_30max_grandparent_overlap_factor___get__(__pyx_obj_7rocksdb_8_rocksdb_Options*)\u2019:
rocksdb/_rocksdb.cpp:13473:56: error: \u2018struct rocksdb::Options\u2019 has no member named \u2018max_grandparent_overlap_factor\u2019
__pyx_t_1 = __Pyx_PyInt_From_int(__pyx_v_self->opts->max_grandparent_overlap_factor); if (unlikely(!__pyx_t_1)) {__pyx_filename = __pyx_f[0]; __pyx_lineno = 885; __pyx_clineno = __LINE__; goto __pyx_L1_error;}
^
rocksdb/_rocksdb.cpp: In function \u2018int __pyx_pf_7rocksdb_8_rocksdb_7Options_30max_grandparent_overlap_factor_2__set__(__pyx_obj_7rocksdb_8_rocksdb_Options*, PyObject*)\u2019:
rocksdb/_rocksdb.cpp:13536:23: error: \u2018struct rocksdb::Options\u2019 has no member named \u2018max_grandparent_overlap_factor\u2019
__pyx_v_self->opts->max_grandparent_overlap_factor = __pyx_t_1;
^
rocksdb/_rocksdb.cpp: In function \u2018PyObject* __pyx_pf_7rocksdb_8_rocksdb_7Options_14filter_deletes___get__(__pyx_obj_7rocksdb_8_rocksdb_Options*)\u2019:
rocksdb/_rocksdb.cpp:18153:57: error: \u2018struct rocksdb::Options\u2019 has no member named \u2018filter_deletes\u2019
__pyx_t_1 = __Pyx_PyBool_FromLong(__pyx_v_self->opts->filter_deletes); if (unlikely(!__pyx_t_1)) {__pyx_filename = __pyx_f[0]; __pyx_lineno = 1142; __pyx_clineno = __LINE__; goto __pyx_L1_error;}
^
rocksdb/_rocksdb.cpp:447:36: note: in definition of macro \u2018__Pyx_PyBool_FromLong\u2019
#define __Pyx_PyBool_FromLong(b) ((b) ? (Py_INCREF(Py_True), Py_True) : (Py_INCREF(Py_False), Py_False))
^
rocksdb/_rocksdb.cpp: In function \u2018int __pyx_pf_7rocksdb_8_rocksdb_7Options_14filter_deletes_2__set__(__pyx_obj_7rocksdb_8_rocksdb_Options*, PyObject*)\u2019:
rocksdb/_rocksdb.cpp:18216:23: error: \u2018struct rocksdb::Options\u2019 has no member named \u2018filter_deletes\u2019
__pyx_v_self->opts->filter_deletes = __pyx_t_1;
^
error: command 'x86_64-linux-gnu-gcc' failed with exit status 1
Complete output from command /nas1/buvana/rocksdb/rocksdb/pyrocks_test/bin/python -c "import setuptools, tokenize;__file__='/nas1/buvana/rocksdb/rocksdb/pyrocks_test/build/pyrocksdb/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-Lxk83M-record/install-record.txt --single-version-externally-managed --compile --install-headers /nas1/buvana/rocksdb/rocksdb/pyrocks_test/include/site/python2.7:
running install
running build
running build_py
creating build
creating build/lib.linux-x86_64-2.7
creating build/lib.linux-x86_64-2.7/rocksdb
copying rocksdb/init.py -> build/lib.linux-x86_64-2.7/rocksdb
copying rocksdb/interfaces.py -> build/lib.linux-x86_64-2.7/rocksdb
copying rocksdb/errors.py -> build/lib.linux-x86_64-2.7/rocksdb
creating build/lib.linux-x86_64-2.7/rocksdb/tests
copying rocksdb/tests/init.py -> build/lib.linux-x86_64-2.7/rocksdb/tests
copying rocksdb/tests/test_options.py -> build/lib.linux-x86_64-2.7/rocksdb/tests
copying rocksdb/tests/test_db.py -> build/lib.linux-x86_64-2.7/rocksdb/tests
running egg_info
writing dependency_links to pyrocksdb.egg-info/dependency_links.txt
writing requirements to pyrocksdb.egg-info/requires.txt
writing pyrocksdb.egg-info/PKG-INFO
writing top-level names to pyrocksdb.egg-info/top_level.txt
warning: manifest_maker: standard file '-c' not found
reading manifest file 'pyrocksdb.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
writing manifest file 'pyrocksdb.egg-info/SOURCES.txt'
copying rocksdb/_rocksdb.cpp -> build/lib.linux-x86_64-2.7/rocksdb
copying rocksdb/_rocksdb.pyx -> build/lib.linux-x86_64-2.7/rocksdb
copying rocksdb/backup.pxd -> build/lib.linux-x86_64-2.7/rocksdb
copying rocksdb/cache.pxd -> build/lib.linux-x86_64-2.7/rocksdb
copying rocksdb/comparator.pxd -> build/lib.linux-x86_64-2.7/rocksdb
copying rocksdb/db.pxd -> build/lib.linux-x86_64-2.7/rocksdb
copying rocksdb/env.pxd -> build/lib.linux-x86_64-2.7/rocksdb
copying rocksdb/filter_policy.pxd -> build/lib.linux-x86_64-2.7/rocksdb
copying rocksdb/iterator.pxd -> build/lib.linux-x86_64-2.7/rocksdb
copying rocksdb/logger.pxd -> build/lib.linux-x86_64-2.7/rocksdb
copying rocksdb/memtablerep.pxd -> build/lib.linux-x86_64-2.7/rocksdb
copying rocksdb/merge_operator.pxd -> build/lib.linux-x86_64-2.7/rocksdb
copying rocksdb/options.pxd -> build/lib.linux-x86_64-2.7/rocksdb
copying rocksdb/slice_.pxd -> build/lib.linux-x86_64-2.7/rocksdb
copying rocksdb/slice_transform.pxd -> build/lib.linux-x86_64-2.7/rocksdb
copying rocksdb/snapshot.pxd -> build/lib.linux-x86_64-2.7/rocksdb
copying rocksdb/status.pxd -> build/lib.linux-x86_64-2.7/rocksdb
copying rocksdb/std_memory.pxd -> build/lib.linux-x86_64-2.7/rocksdb
copying rocksdb/table_factory.pxd -> build/lib.linux-x86_64-2.7/rocksdb
copying rocksdb/universal_compaction.pxd -> build/lib.linux-x86_64-2.7/rocksdb
creating build/lib.linux-x86_64-2.7/rocksdb/cpp
copying rocksdb/cpp/comparator_wrapper.hpp -> build/lib.linux-x86_64-2.7/rocksdb/cpp
copying rocksdb/cpp/filter_policy_wrapper.hpp -> build/lib.linux-x86_64-2.7/rocksdb/cpp
copying rocksdb/cpp/memtable_factories.hpp -> build/lib.linux-x86_64-2.7/rocksdb/cpp
copying rocksdb/cpp/merge_operator_wrapper.hpp -> build/lib.linux-x86_64-2.7/rocksdb/cpp
copying rocksdb/cpp/slice_transform_wrapper.hpp -> build/lib.linux-x86_64-2.7/rocksdb/cpp
copying rocksdb/cpp/utils.hpp -> build/lib.linux-x86_64-2.7/rocksdb/cpp
copying rocksdb/cpp/write_batch_iter_helper.hpp -> build/lib.linux-x86_64-2.7/rocksdb/cpp
running build_ext
building 'rocksdb._rocksdb' extension
creating build/temp.linux-x86_64-2.7
creating build/temp.linux-x86_64-2.7/rocksdb
x86_64-linux-gnu-gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC -I/usr/include/python2.7 -c rocksdb/_rocksdb.cpp -o build/temp.linux-x86_64-2.7/rocksdb/_rocksdb.o -std=c++11 -O3 -Wall -Wextra -Wconversion -fno-strict-aliasing
cc1plus: warning: command line option \u2018-Wstrict-prototypes\u2019 is valid for C/ObjC but not for C++ [enabled by default]
rocksdb/_rocksdb.cpp: In function \u2018PyObject* __pyx_pf_7rocksdb_8_rocksdb_19PyBloomFilterPolicy_4create_filter(pyx_obj_7rocksdb_8_rocksdb_PyBloomFilterPolicy, PyObject)\u2019:
rocksdb/_rocksdb.cpp:5639:88: warning: conversion to \u2018int\u2019 from \u2018std::vectorrocksdb::Slice::size_type {aka long unsigned int}\u2019 may alter its value [-Wconversion]
__pyx_t_6->CreateFilter(py_rocks::vector_data(__pyx_v_c_keys), __pyx_v_c_keys.size(), (&__pyx_v_dst));
^
rocksdb/_rocksdb.cpp: In function \u2018PyObject* pyx_pf_7rocksdb_8_rocksdb_7Options_26expanded_compaction_factor___get(__pyx_obj_7rocksdb_8_rocksdb_Options*)\u2019:
rocksdb/_rocksdb.cpp:13229:56: error: \u2018struct rocksdb::Options\u2019 has no member named \u2018expanded_compaction_factor\u2019
pyx_t_1 = __Pyx_PyInt_From_int(__pyx_v_self->opts->expanded_compaction_factor); if (unlikely(!__pyx_t_1)) {__pyx_filename = __pyx_f[0]; __pyx_lineno = 873; __pyx_clineno = __LINE; goto __pyx_L1_error;}
^
rocksdb/_rocksdb.cpp: In function \u2018int pyx_pf_7rocksdb_8_rocksdb_7Options_26expanded_compaction_factor_2__set(pyx_obj_7rocksdb_8_rocksdb_Options, PyObject)\u2019:
rocksdb/_rocksdb.cpp:13292:23: error: \u2018struct rocksdb::Options\u2019 has no member named \u2018expanded_compaction_factor\u2019
__pyx_v_self->opts->expanded_compaction_factor = __pyx_t_1;
^
rocksdb/_rocksdb.cpp: In function \u2018PyObject* pyx_pf_7rocksdb_8_rocksdb_7Options_24source_compaction_factor___get(__pyx_obj_7rocksdb_8_rocksdb_Options*)\u2019:
rocksdb/_rocksdb.cpp:13351:56: error: \u2018struct rocksdb::Options\u2019 has no member named \u2018source_compaction_factor\u2019
pyx_t_1 = __Pyx_PyInt_From_int(__pyx_v_self->opts->source_compaction_factor); if (unlikely(!__pyx_t_1)) {__pyx_filename = __pyx_f[0]; __pyx_lineno = 879; __pyx_clineno = __LINE; goto __pyx_L1_error;}
^
rocksdb/_rocksdb.cpp: In function \u2018int pyx_pf_7rocksdb_8_rocksdb_7Options_24source_compaction_factor_2__set(pyx_obj_7rocksdb_8_rocksdb_Options, PyObject)\u2019:
rocksdb/_rocksdb.cpp:13414:23: error: \u2018struct rocksdb::Options\u2019 has no member named \u2018source_compaction_factor\u2019
__pyx_v_self->opts->source_compaction_factor = __pyx_t_1;
^
rocksdb/_rocksdb.cpp: In function \u2018PyObject* pyx_pf_7rocksdb_8_rocksdb_7Options_30max_grandparent_overlap_factor___get(__pyx_obj_7rocksdb_8_rocksdb_Options*)\u2019:
rocksdb/_rocksdb.cpp:13473:56: error: \u2018struct rocksdb::Options\u2019 has no member named \u2018max_grandparent_overlap_factor\u2019
pyx_t_1 = __Pyx_PyInt_From_int(__pyx_v_self->opts->max_grandparent_overlap_factor); if (unlikely(!__pyx_t_1)) {__pyx_filename = __pyx_f[0]; __pyx_lineno = 885; __pyx_clineno = __LINE; goto __pyx_L1_error;}
^
rocksdb/_rocksdb.cpp: In function \u2018int pyx_pf_7rocksdb_8_rocksdb_7Options_30max_grandparent_overlap_factor_2__set(pyx_obj_7rocksdb_8_rocksdb_Options, PyObject)\u2019:
rocksdb/_rocksdb.cpp:13536:23: error: \u2018struct rocksdb::Options\u2019 has no member named \u2018max_grandparent_overlap_factor\u2019
__pyx_v_self->opts->max_grandparent_overlap_factor = __pyx_t_1;
^
rocksdb/_rocksdb.cpp: In function \u2018PyObject* pyx_pf_7rocksdb_8_rocksdb_7Options_14filter_deletes___get(__pyx_obj_7rocksdb_8_rocksdb_Options*)\u2019:
rocksdb/_rocksdb.cpp:18153:57: error: \u2018struct rocksdb::Options\u2019 has no member named \u2018filter_deletes\u2019
pyx_t_1 = __Pyx_PyBool_FromLong(__pyx_v_self->opts->filter_deletes); if (unlikely(!__pyx_t_1)) {__pyx_filename = __pyx_f[0]; __pyx_lineno = 1142; __pyx_clineno = __LINE; goto __pyx_L1_error;}
^
rocksdb/_rocksdb.cpp:447:36: note: in definition of macro \u2018__Pyx_PyBool_FromLong\u2019
#define __Pyx_PyBool_FromLong(b) ((b) ? (Py_INCREF(Py_True), Py_True) : (Py_INCREF(Py_False), Py_False))
^
rocksdb/_rocksdb.cpp: In function \u2018int pyx_pf_7rocksdb_8_rocksdb_7Options_14filter_deletes_2__set(pyx_obj_7rocksdb_8_rocksdb_Options, PyObject)\u2019:
rocksdb/_rocksdb.cpp:18216:23: error: \u2018struct rocksdb::Options\u2019 has no member named \u2018filter_deletes\u2019
__pyx_v_self->opts->filter_deletes = __pyx_t_1;
^
error: command 'x86_64-linux-gnu-gcc' failed with exit status 1
Cleaning up...
Command /nas1/buvana/rocksdb/rocksdb/pyrocks_test/bin/python -c "import setuptools, tokenize;file='/nas1/buvana/rocksdb/rocksdb/pyrocks_test/build/pyrocksdb/setup.py';exec(compile(getattr(tokenize, 'open', open)(file).read().replace('\r\n', '\n'), file, 'exec'))" install --record /tmp/pip-Lxk83M-record/install-record.txt --single-version-externally-managed --compile --install-headers /nas1/buvana/rocksdb/rocksdb/pyrocks_test/include/site/python2.7 failed with error code 1 in /nas1/buvana/rocksdb/rocksdb/pyrocks_test/build/pyrocksdb
Traceback (most recent call last):
File "/nas1/buvana/rocksdb/rocksdb/pyrocks_test/bin/pip", line 11, in
sys.exit(main())
File "/nas1/buvana/rocksdb/rocksdb/pyrocks_test/local/lib/python2.7/site-packages/pip/init.py", line 185, in main
return command.main(cmd_args)
File "/nas1/buvana/rocksdb/rocksdb/pyrocks_test/local/lib/python2.7/site-packages/pip/basecommand.py", line 161, in main
text = '\n'.join(complete_log)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 42: ordinal not in range(128)
(pyrocks_test)buvana@datamon3:~/rocksdb/rocksdb/pyrocks_test$
Segfault happening: Assertion `data[n] != kNoCompression' failed.
I'm using rocksdb 2.8 and pyrocksdb 0.2.
python: table/format.cc:192: rocksdb::Status rocksdb::UncompressBlockContents(const char*, size_t, rocksdb::BlockContents*): Assertion `data[n] != kNoCompression' failed.I did notice this issue
#6
I have the following in the opts
opts.block_cache = rocksdb.LRUCache(2 * (1024 ** 3)) |FEED_FETCH_BACKOFF = [.1, .5, 1, 2, 4, 8, 16, 32, 60]
opts.block_cache_compressed = rocksdb.LRUCache(500 * (1024 ** 2))When I comment out the second line the segfault does not happen. Should only one of "block_cache" and "block_cache_compressed" be configured?
Does pyrocksdb support multiple embedded databases
Unable to install pyrocksdb against RocksDB master
I followed the instructions from here: http://pyrocksdb.readthedocs.org/en/latest/installation.html
When I try pip install git+git://github.com/stephan-hof/pyrocksdb.git I get the following output with error:
running install
running build
running build_py
creating build
creating build/lib.linux-x86_64-3.4
creating build/lib.linux-x86_64-3.4/rocksdb
copying rocksdb/__init__.py -> build/lib.linux-x86_64-3.4/rocksdb
copying rocksdb/interfaces.py -> build/lib.linux-x86_64-3.4/rocksdb
copying rocksdb/errors.py -> build/lib.linux-x86_64-3.4/rocksdb
creating build/lib.linux-x86_64-3.4/rocksdb/tests
copying rocksdb/tests/__init__.py -> build/lib.linux-x86_64-3.4/rocksdb/tests
copying rocksdb/tests/test_db.py -> build/lib.linux-x86_64-3.4/rocksdb/tests
copying rocksdb/tests/test_options.py -> build/lib.linux-x86_64-3.4/rocksdb/tests
running egg_info
creating pyrocksdb.egg-info
writing dependency_links to pyrocksdb.egg-info/dependency_links.txt
writing top-level names to pyrocksdb.egg-info/top_level.txt
writing requirements to pyrocksdb.egg-info/requires.txt
writing pyrocksdb.egg-info/PKG-INFO
writing manifest file 'pyrocksdb.egg-info/SOURCES.txt'
warning: manifest_maker: standard file '-c' not found
reading manifest file 'pyrocksdb.egg-info/SOURCES.txt'
writing manifest file 'pyrocksdb.egg-info/SOURCES.txt'
copying rocksdb/_rocksdb.cpp -> build/lib.linux-x86_64-3.4/rocksdb
running build_ext
building 'rocksdb._rocksdb' extension
creating build/temp.linux-x86_64-3.4
creating build/temp.linux-x86_64-3.4/rocksdb
x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -g -fstack-protector --param=ssp-buffer-size=4 -Wformat -Werror=format-security -D_FORTIFY_SOURCE=2 -fPIC -I/home/iulian/py3env/include -I/usr/include/python3.4m -c rocksdb/_rocksdb.cpp -o build/temp.linux-x86_64-3.4/rocksdb/_rocksdb.o -std=gnu++11 -O3 -Wall -Wextra -Wconversion -fno-strict-aliasing
cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++ [enabled by default]
rocksdb/_rocksdb.cpp:289:27: fatal error: rocksdb/slice.h: No such file or directory
#include "rocksdb/slice.h"
^
compilation terminated.
error: command 'x86_64-linux-gnu-gcc' failed with exit status 1
----------------------------------------
Command "/home/iulian/py3env/bin/python -c "import setuptools, tokenize;__file__='/tmp/pip-mndoekm3-build/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-3fwa2_ff-record/install-record.txt --single-version-externally-managed --compile --install-headers /home/iulian/py3env/include/site/python3.4" failed with error code 1 in /tmp/pip-mndoekm3-build
I tried to checkout various versions of RocksDB (3.5, 3.6, 3.8, 3.9) and still no luck.
I am using Python 3.4 on a 64 bit Ubuntu 14.04.
Support WriteBatch iterator
WriteBatch is an iterable object, but that's not currently exposed through pyrocksdb.
It seems to just be WriteBatch.iterate(handler) from Java and Perl examples.
Failing to install pyrocksdb against rocksdb 2.8
Hi Stephan,
We tried to install pyrocksdb from the doc 'http://pyrocksdb.readthedocs.org/en/v0.2.1/installation.html' in a ubuntu server with the following dependencies and failed,
ubuntu 14.04
gcc 4.8.2
rocksdb 2.8
pyrocksdb 0.2.1
Error faced:
root@devserver:/tmp/rocksdb# sudo pip install git+git://github.com/stephan-hof/[email protected]
Downloading/unpacking git+git://github.com/stephan-hof/pyrocksdb.git@v0.2.1
Cloning git://github.com/stephan-hof/pyrocksdb.git (to v0.2.1) to /tmp/pip-Q187Rq-build
Running setup.py (path:/tmp/pip-Q187Rq-build/setup.py) egg_info for package from git+git://github.com/stephan-hof/pyrocksdb.git@v0.2.1
Compiling rocksdb/_rocksdb.pyx because it changed.
Cythonizing rocksdb/_rocksdb.pyx
Requirement already satisfied (use --upgrade to upgrade): setuptools in /usr/lib/python2.7/dist-packages (from pyrocksdb==0.2.1)
Requirement already satisfied (use --upgrade to upgrade): Cython>=0.20 in /usr/local/lib/python2.7/dist-packages (from pyrocksdb==0.2.1)
Installing collected packages: pyrocksdb
Running setup.py install for pyrocksdb
building 'rocksdb._rocksdb' extension
x86_64-linux-gnu-gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC -I/usr/include/python2.7 -c rocksdb/_rocksdb.cpp -o build/
temp.linux-x86_64-2.7/rocksdb/_rocksdb.o -std=gnu++11 -O3 -Wall -Wextra -Wconversion -fno-strict-aliasing
cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++ [enabled by default]
rocksdb/_rocksdb.cpp: In function ‘PyObject* __pyx_pf_7rocksdb_8_rocksdb_19PyBloomFilterPolicy_6create_filter(__pyx_obj_7rocksdb_8_rocksdb_PyBloomFilterPolicy*, PyOb
ject*)’:
rocksdb/_rocksdb.cpp:5874:99: warning: conversion to ‘int’ from ‘std::vector<rocksdb::Slice>::size_type {aka long unsigned int}’ may alter its value [-Wconversion]
__pyx_v_self->policy->CreateFilter(py_rocks::vector_data(__pyx_v_c_keys), __pyx_v_c_keys.size(), (&__pyx_v_dst));
^
rocksdb/_rocksdb.cpp: In function ‘int __pyx_pf_7rocksdb_8_rocksdb_2DB___cinit__(__pyx_obj_7rocksdb_8_rocksdb_DB*, PyObject*, __pyx_obj_7rocksdb_8_rocksdb_Options*,
PyObject*)’:
rocksdb/_rocksdb.cpp:21807:24: error: base operand of ‘->’ has non-pointer type ‘rocksdb::Options’
__pyx_t_6 = __pyx_t_5->info_log;
^
error: command 'x86_64-linux-gnu-gcc' failed with exit status 1
Complete output from command /usr/bin/python -c "import setuptools, tokenize;__file__='/tmp/pip-Q187Rq-build/setup.py';exec(compile(getattr(tokenize, 'open', open)(_
_file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-bdobOT-record/install-record.txt --single-version-externally-managed --compile:
running install
running build
running build_py
creating build
creating build/lib.linux-x86_64-2.7
creating build/lib.linux-x86_64-2.7/rocksdb
copying rocksdb/interfaces.py -> build/lib.linux-x86_64-2.7/rocksdb
copying rocksdb/__init__.py -> build/lib.linux-x86_64-2.7/rocksdb
copying rocksdb/errors.py -> build/lib.linux-x86_64-2.7/rocksdb
creating build/lib.linux-x86_64-2.7/rocksdb/tests
copying rocksdb/tests/test_db.py -> build/lib.linux-x86_64-2.7/rocksdb/tests
copying rocksdb/tests/__init__.py -> build/lib.linux-x86_64-2.7/rocksdb/tests
copying rocksdb/tests/test_options.py -> build/lib.linux-x86_64-2.7/rocksdb/tests
running egg_info
creating pyrocksdb.egg-info
writing requirements to pyrocksdb.egg-info/requires.txt
writing pyrocksdb.egg-info/PKG-INFO
writing top-level names to pyrocksdb.egg-info/top_level.txt
writing dependency_links to pyrocksdb.egg-info/dependency_links.txt
writing manifest file 'pyrocksdb.egg-info/SOURCES.txt'
warning: manifest_maker: standard file '-c' not found
reading manifest file 'pyrocksdb.egg-info/SOURCES.txt'
writing manifest file 'pyrocksdb.egg-info/SOURCES.txt'
copying rocksdb/_rocksdb.cpp -> build/lib.linux-x86_64-2.7/rocksdb
running build_ext
building 'rocksdb._rocksdb' extension
creating build/temp.linux-x86_64-2.7
creating build/temp.linux-x86_64-2.7/rocksdb
x86_64-linux-gnu-gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC -I/usr/include/python2.7 -c rocksdb/_rocksdb.cpp -o build/temp.linux-x86_64-2.7/rocksdb/_rocksdb.o -std=gnu++11 -O3 -Wall -Wextra -Wconversion -fno-strict-aliasing
cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++ [enabled by default]
rocksdb/_rocksdb.cpp: In function ‘PyObject* __pyx_pf_7rocksdb_8_rocksdb_19PyBloomFilterPolicy_6create_filter(__pyx_obj_7rocksdb_8_rocksdb_PyBloomFilterPolicy*, PyObject*)’:
rocksdb/_rocksdb.cpp:5874:99: warning: conversion to ‘int’ from ‘std::vector<rocksdb::Slice>::size_type {aka long unsigned int}’ may alter its value [-Wconversion]
__pyx_v_self->policy->CreateFilter(py_rocks::vector_data(__pyx_v_c_keys), __pyx_v_c_keys.size(), (&__pyx_v_dst));
^
rocksdb/_rocksdb.cpp: In function ‘int __pyx_pf_7rocksdb_8_rocksdb_2DB___cinit__(__pyx_obj_7rocksdb_8_rocksdb_DB*, PyObject*, __pyx_obj_7rocksdb_8_rocksdb_Options*, PyObject*)’:
rocksdb/_rocksdb.cpp:21807:24: error: base operand of ‘->’ has non-pointer type ‘rocksdb::Options’
__pyx_t_6 = __pyx_t_5->info_log;
^
error: command 'x86_64-linux-gnu-gcc' failed with exit status 1
----------------------------------------Push on PyPI
Would be nice to see your package there!
Cheers,
Installation Error - fatal error: rocksdb/slice.h: No such file or directory
Hi Stephan,
upon installing pyrocksdb, I get the error below:
It has to do with the rocksdb libraries.
I'm not sure if should place certain rocksdb files somewhere in order for this to work.
sudo pip install git+git://github.com/stephan-hof/pyrocksdb.git
Downloading/unpacking git+git://github.com/stephan-hof/pyrocksdb.git
Cloning git://github.com/stephan-hof/pyrocksdb.git to /tmp/pip-mHdZe8-build
Running setup.py egg_info for package from git+git://github.com/stephan-hof/pyrocksdb.git
missing cimport in module 'pair': /usr/local/lib/python2.7/dist-packages/Cython/Includes/libcpp/deque.pxd
Compiling rocksdb/_rocksdb.pyx because it changed.
Cythonizing rocksdb/_rocksdb.pyx
Requirement already satisfied (use --upgrade to upgrade): setuptools in /usr/lib/python2.7/dist-packages (from pyrocksdb==0.1)
Requirement already satisfied (use --upgrade to upgrade): Cython>=0.20 in /usr/local/lib/python2.7/dist-packages (from pyrocksdb==0.1)
Installing collected packages: pyrocksdb
Running setup.py install for pyrocksdb
missing cimport in module 'pair': /usr/local/lib/python2.7/dist-packages/Cython/Includes/libcpp/deque.pxd
building 'rocksdb._rocksdb' extension
x86_64-linux-gnu-gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC -I/usr/include/python2.7 -c rocksdb/_rocksdb.cpp -o build/temp.linux-x86_64-2.7/rocksdb/_rocksdb.o -std=gnu++11 -O3 -Wall -Wextra -Wconversion -fno-strict-aliasing
cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++ [enabled by default]
rocksdb/_rocksdb.cpp:348:27: fatal error: rocksdb/slice.h: No such file or directory
#include "rocksdb/slice.h"
^
compilation terminated.
error: command 'x86_64-linux-gnu-gcc' failed with exit status 1
Complete output from command /usr/bin/python -c "import setuptools;__file__='/tmp/pip-mHdZe8-build/setup.py';exec(compile(open(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-N8DNKA-record/install-record.txt --single-version-externally-managed:
missing cimport in module 'pair': /usr/local/lib/python2.7/dist-packages/Cython/Includes/libcpp/deque.pxd
running install
running build
running build_py
creating build
creating build/lib.linux-x86_64-2.7
creating build/lib.linux-x86_64-2.7/rocksdb
copying rocksdb/interfaces.py -> build/lib.linux-x86_64-2.7/rocksdb
copying rocksdb/init.py -> build/lib.linux-x86_64-2.7/rocksdb
copying rocksdb/errors.py -> build/lib.linux-x86_64-2.7/rocksdb
creating build/lib.linux-x86_64-2.7/rocksdb/tests
copying rocksdb/tests/test_options.py -> build/lib.linux-x86_64-2.7/rocksdb/tests
copying rocksdb/tests/init.py -> build/lib.linux-x86_64-2.7/rocksdb/tests
copying rocksdb/tests/test_db.py -> build/lib.linux-x86_64-2.7/rocksdb/tests
running egg_info
creating pyrocksdb.egg-info
writing requirements to pyrocksdb.egg-info/requires.txt
writing pyrocksdb.egg-info/PKG-INFO
writing top-level names to pyrocksdb.egg-info/top_level.txt
writing dependency_links to pyrocksdb.egg-info/dependency_links.txt
writing manifest file 'pyrocksdb.egg-info/SOURCES.txt'
warning: manifest_maker: standard file '-c' not found
reading manifest file 'pyrocksdb.egg-info/SOURCES.txt'
writing manifest file 'pyrocksdb.egg-info/SOURCES.txt'
copying rocksdb/_rocksdb.cpp -> build/lib.linux-x86_64-2.7/rocksdb
running build_ext
building 'rocksdb._rocksdb' extension
creating build/temp.linux-x86_64-2.7
creating build/temp.linux-x86_64-2.7/rocksdb
x86_64-linux-gnu-gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC -I/usr/include/python2.7 -c rocksdb/_rocksdb.cpp -o build/temp.linux-x86_64-2.7/rocksdb/_rocksdb.o -std=gnu++11 -O3 -Wall -Wextra -Wconversion -fno-strict-aliasing
cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++ [enabled by default]
rocksdb/_rocksdb.cpp:348:27: fatal error: rocksdb/slice.h: No such file or directory
#include "rocksdb/slice.h"
^
compilation terminated.
error: command 'x86_64-linux-gnu-gcc' failed with exit status 1
Cleaning up...
Command /usr/bin/python -c "import setuptools;file='/tmp/pip-mHdZe8-build/setup.py';exec(compile(open(file).read().replace('\r\n', '\n'), file, 'exec'))" install --record /tmp/pip-N8DNKA-record/install-record.txt --single-version-externally-managed failed with error code 1 in /tmp/pip-mHdZe8-build
Storing complete log in /home/a/.pip/pip.log
Cant install package on Windows 7 - cl.exe does not understand /Wextra compiler flag
Last night I tried to install pyrocks on a Windows 7 machine and I ran into a problem - linker says invalid numeric argument /Wextra. Seems like VC compiler does not support this flag.
There's a possibility some other flags aren't accepted too.
If I manage to make it work on Windows, will you accept a PR?
_rocksdb.so: undefined symbol: _ZNK7rocksdb24AssociativeMergeOperator9FullMergeERKNS_5SliceEPS2_RKSt5dequeISsSaISsEEPSsPNS_6LoggerE
rocksdb and network filesystem
Is this the best place to ask general questions?
I know rocksdb is not a distributed db, but is it designed to work with a network filesystem ie. machine runs rocksdb but db is on filesystem on network?
MacOs Install instructions
Is there some MacOs Install instructions ?
Support new bulk loading feature
'experimental DB::AddFile() to plug sst files into empty DB':
facebook/rocksdb@f03b5c9
This feature is useful for combining the results of non-overlapping ranges, when bulk loading. It's not yet stable.
error: command 'gcc' failed with exit status 1 on max osx mojave 10.14.2
Hello, I am trying to install pyrocksdb on Mac os x and getting this error.
pip install pyrocksdb 1 ↵ 8278 15:48:43
Collecting pyrocksdb
Downloading https://files.pythonhosted.org/packages/92/02/ea89030b42c545df7ae8cfccd567399de1118f26bf23f629fb5391c3e2f8/pyrocksdb-0.4.tar.gz (139kB)
100% |████████████████████████████████| 143kB 330kB/s
Requirement already satisfied: setuptools in /anaconda3/lib/python3.6/site-packages (from pyrocksdb) (39.1.0)
Building wheels for collected packages: pyrocksdb
Running setup.py bdist_wheel for pyrocksdb ... error
Complete output from command /anaconda3/bin/python -u -c "import setuptools, tokenize;file='/private/var/folders/13/qn9rxyhd6ws0vbyr7qq_741m0000gn/T/pip-install-23echd02/pyrocksdb/setup.py';f=getattr(tokenize, 'open', open)(file);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, file, 'exec'))" bdist_wheel -d /private/var/folders/13/qn9rxyhd6ws0vbyr7qq_741m0000gn/T/pip-wheel-tb0hawoa --python-tag cp36:
running bdist_wheel
running build
running build_py
creating build
creating build/lib.macosx-10.7-x86_64-3.6
creating build/lib.macosx-10.7-x86_64-3.6/rocksdb
copying rocksdb/interfaces.py -> build/lib.macosx-10.7-x86_64-3.6/rocksdb
copying rocksdb/init.py -> build/lib.macosx-10.7-x86_64-3.6/rocksdb
copying rocksdb/errors.py -> build/lib.macosx-10.7-x86_64-3.6/rocksdb
creating build/lib.macosx-10.7-x86_64-3.6/rocksdb/tests
copying rocksdb/tests/test_options.py -> build/lib.macosx-10.7-x86_64-3.6/rocksdb/tests
copying rocksdb/tests/test_db.py -> build/lib.macosx-10.7-x86_64-3.6/rocksdb/tests
copying rocksdb/tests/init.py -> build/lib.macosx-10.7-x86_64-3.6/rocksdb/tests
running egg_info
writing pyrocksdb.egg-info/PKG-INFO
writing dependency_links to pyrocksdb.egg-info/dependency_links.txt
writing requirements to pyrocksdb.egg-info/requires.txt
writing top-level names to pyrocksdb.egg-info/top_level.txt
reading manifest file 'pyrocksdb.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
writing manifest file 'pyrocksdb.egg-info/SOURCES.txt'
copying rocksdb/_rocksdb.cpp -> build/lib.macosx-10.7-x86_64-3.6/rocksdb
copying rocksdb/rocksdb.pyx -> build/lib.macosx-10.7-x86_64-3.6/rocksdb
copying rocksdb/backup.pxd -> build/lib.macosx-10.7-x86_64-3.6/rocksdb
copying rocksdb/cache.pxd -> build/lib.macosx-10.7-x86_64-3.6/rocksdb
copying rocksdb/comparator.pxd -> build/lib.macosx-10.7-x86_64-3.6/rocksdb
copying rocksdb/db.pxd -> build/lib.macosx-10.7-x86_64-3.6/rocksdb
copying rocksdb/env.pxd -> build/lib.macosx-10.7-x86_64-3.6/rocksdb
copying rocksdb/filter_policy.pxd -> build/lib.macosx-10.7-x86_64-3.6/rocksdb
copying rocksdb/iterator.pxd -> build/lib.macosx-10.7-x86_64-3.6/rocksdb
copying rocksdb/logger.pxd -> build/lib.macosx-10.7-x86_64-3.6/rocksdb
copying rocksdb/memtablerep.pxd -> build/lib.macosx-10.7-x86_64-3.6/rocksdb
copying rocksdb/merge_operator.pxd -> build/lib.macosx-10.7-x86_64-3.6/rocksdb
copying rocksdb/options.pxd -> build/lib.macosx-10.7-x86_64-3.6/rocksdb
copying rocksdb/slice.pxd -> build/lib.macosx-10.7-x86_64-3.6/rocksdb
copying rocksdb/slice_transform.pxd -> build/lib.macosx-10.7-x86_64-3.6/rocksdb
copying rocksdb/snapshot.pxd -> build/lib.macosx-10.7-x86_64-3.6/rocksdb
copying rocksdb/status.pxd -> build/lib.macosx-10.7-x86_64-3.6/rocksdb
copying rocksdb/std_memory.pxd -> build/lib.macosx-10.7-x86_64-3.6/rocksdb
copying rocksdb/table_factory.pxd -> build/lib.macosx-10.7-x86_64-3.6/rocksdb
copying rocksdb/universal_compaction.pxd -> build/lib.macosx-10.7-x86_64-3.6/rocksdb
creating build/lib.macosx-10.7-x86_64-3.6/rocksdb/cpp
copying rocksdb/cpp/comparator_wrapper.hpp -> build/lib.macosx-10.7-x86_64-3.6/rocksdb/cpp
copying rocksdb/cpp/filter_policy_wrapper.hpp -> build/lib.macosx-10.7-x86_64-3.6/rocksdb/cpp
copying rocksdb/cpp/memtable_factories.hpp -> build/lib.macosx-10.7-x86_64-3.6/rocksdb/cpp
copying rocksdb/cpp/merge_operator_wrapper.hpp -> build/lib.macosx-10.7-x86_64-3.6/rocksdb/cpp
copying rocksdb/cpp/slice_transform_wrapper.hpp -> build/lib.macosx-10.7-x86_64-3.6/rocksdb/cpp
copying rocksdb/cpp/utils.hpp -> build/lib.macosx-10.7-x86_64-3.6/rocksdb/cpp
copying rocksdb/cpp/write_batch_iter_helper.hpp -> build/lib.macosx-10.7-x86_64-3.6/rocksdb/cpp
running build_ext
building 'rocksdb._rocksdb' extension
creating build/temp.macosx-10.7-x86_64-3.6
creating build/temp.macosx-10.7-x86_64-3.6/rocksdb
gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/anaconda3/include -arch x86_64 -I/anaconda3/include -arch x86_64 -Irocksdb -I./rocksdb -I/anaconda3/include/python3.6m -c rocksdb/_rocksdb.cpp -o build/temp.macosx-10.7-x86_64-3.6/rocksdb/_rocksdb.o -std=c++11 -O3 -Wall -Wextra -Wconversion -fno-strict-aliasing
warning: include path for stdlibc++ headers not found; pass '-std=libc++' on the command line to use the libc++ standard library instead [-Wstdlibcxx-not-found]
rocksdb/_rocksdb.cpp:619:10: fatal error: 'ios' file not found
#include "ios"
^~~~~
1 warning and 1 error generated.
error: command 'gcc' failed with exit status 1
Failed building wheel for pyrocksdb
Running setup.py clean for pyrocksdb
Failed to build pyrocksdb
Installing collected packages: pyrocksdb
Running setup.py install for pyrocksdb ... error
Complete output from command /anaconda3/bin/python -u -c "import setuptools, tokenize;file='/private/var/folders/13/qn9rxyhd6ws0vbyr7qq_741m0000gn/T/pip-install-23echd02/pyrocksdb/setup.py';f=getattr(tokenize, 'open', open)(file);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, file, 'exec'))" install --record /private/var/folders/13/qn9rxyhd6ws0vbyr7qq_741m0000gn/T/pip-record-62bei6uy/install-record.txt --single-version-externally-managed --compile:
running install
running build
running build_py
creating build
creating build/lib.macosx-10.7-x86_64-3.6
creating build/lib.macosx-10.7-x86_64-3.6/rocksdb
copying rocksdb/interfaces.py -> build/lib.macosx-10.7-x86_64-3.6/rocksdb
copying rocksdb/init.py -> build/lib.macosx-10.7-x86_64-3.6/rocksdb
copying rocksdb/errors.py -> build/lib.macosx-10.7-x86_64-3.6/rocksdb
creating build/lib.macosx-10.7-x86_64-3.6/rocksdb/tests
copying rocksdb/tests/test_options.py -> build/lib.macosx-10.7-x86_64-3.6/rocksdb/tests
copying rocksdb/tests/test_db.py -> build/lib.macosx-10.7-x86_64-3.6/rocksdb/tests
copying rocksdb/tests/init.py -> build/lib.macosx-10.7-x86_64-3.6/rocksdb/tests
running egg_info
writing pyrocksdb.egg-info/PKG-INFO
writing dependency_links to pyrocksdb.egg-info/dependency_links.txt
writing requirements to pyrocksdb.egg-info/requires.txt
writing top-level names to pyrocksdb.egg-info/top_level.txt
reading manifest file 'pyrocksdb.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
writing manifest file 'pyrocksdb.egg-info/SOURCES.txt'
copying rocksdb/_rocksdb.cpp -> build/lib.macosx-10.7-x86_64-3.6/rocksdb
copying rocksdb/rocksdb.pyx -> build/lib.macosx-10.7-x86_64-3.6/rocksdb
copying rocksdb/backup.pxd -> build/lib.macosx-10.7-x86_64-3.6/rocksdb
copying rocksdb/cache.pxd -> build/lib.macosx-10.7-x86_64-3.6/rocksdb
copying rocksdb/comparator.pxd -> build/lib.macosx-10.7-x86_64-3.6/rocksdb
copying rocksdb/db.pxd -> build/lib.macosx-10.7-x86_64-3.6/rocksdb
copying rocksdb/env.pxd -> build/lib.macosx-10.7-x86_64-3.6/rocksdb
copying rocksdb/filter_policy.pxd -> build/lib.macosx-10.7-x86_64-3.6/rocksdb
copying rocksdb/iterator.pxd -> build/lib.macosx-10.7-x86_64-3.6/rocksdb
copying rocksdb/logger.pxd -> build/lib.macosx-10.7-x86_64-3.6/rocksdb
copying rocksdb/memtablerep.pxd -> build/lib.macosx-10.7-x86_64-3.6/rocksdb
copying rocksdb/merge_operator.pxd -> build/lib.macosx-10.7-x86_64-3.6/rocksdb
copying rocksdb/options.pxd -> build/lib.macosx-10.7-x86_64-3.6/rocksdb
copying rocksdb/slice.pxd -> build/lib.macosx-10.7-x86_64-3.6/rocksdb
copying rocksdb/slice_transform.pxd -> build/lib.macosx-10.7-x86_64-3.6/rocksdb
copying rocksdb/snapshot.pxd -> build/lib.macosx-10.7-x86_64-3.6/rocksdb
copying rocksdb/status.pxd -> build/lib.macosx-10.7-x86_64-3.6/rocksdb
copying rocksdb/std_memory.pxd -> build/lib.macosx-10.7-x86_64-3.6/rocksdb
copying rocksdb/table_factory.pxd -> build/lib.macosx-10.7-x86_64-3.6/rocksdb
copying rocksdb/universal_compaction.pxd -> build/lib.macosx-10.7-x86_64-3.6/rocksdb
creating build/lib.macosx-10.7-x86_64-3.6/rocksdb/cpp
copying rocksdb/cpp/comparator_wrapper.hpp -> build/lib.macosx-10.7-x86_64-3.6/rocksdb/cpp
copying rocksdb/cpp/filter_policy_wrapper.hpp -> build/lib.macosx-10.7-x86_64-3.6/rocksdb/cpp
copying rocksdb/cpp/memtable_factories.hpp -> build/lib.macosx-10.7-x86_64-3.6/rocksdb/cpp
copying rocksdb/cpp/merge_operator_wrapper.hpp -> build/lib.macosx-10.7-x86_64-3.6/rocksdb/cpp
copying rocksdb/cpp/slice_transform_wrapper.hpp -> build/lib.macosx-10.7-x86_64-3.6/rocksdb/cpp
copying rocksdb/cpp/utils.hpp -> build/lib.macosx-10.7-x86_64-3.6/rocksdb/cpp
copying rocksdb/cpp/write_batch_iter_helper.hpp -> build/lib.macosx-10.7-x86_64-3.6/rocksdb/cpp
running build_ext
building 'rocksdb._rocksdb' extension
creating build/temp.macosx-10.7-x86_64-3.6
creating build/temp.macosx-10.7-x86_64-3.6/rocksdb
gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/anaconda3/include -arch x86_64 -I/anaconda3/include -arch x86_64 -Irocksdb -I./rocksdb -I/anaconda3/include/python3.6m -c rocksdb/_rocksdb.cpp -o build/temp.macosx-10.7-x86_64-3.6/rocksdb/_rocksdb.o -std=c++11 -O3 -Wall -Wextra -Wconversion -fno-strict-aliasing
warning: include path for stdlibc++ headers not found; pass '-std=libc++' on the command line to use the libc++ standard library instead [-Wstdlibcxx-not-found]
rocksdb/_rocksdb.cpp:619:10: fatal error: 'ios' file not found
#include "ios"
^~~~~
1 warning and 1 error generated.
error: command 'gcc' failed with exit status 1
----------------------------------------
Command "/anaconda3/bin/python -u -c "import setuptools, tokenize;file='/private/var/folders/13/qn9rxyhd6ws0vbyr7qq_741m0000gn/T/pip-install-23echd02/pyrocksdb/setup.py';f=getattr(tokenize, 'open', open)(file);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, file, 'exec'))" install --record /private/var/folders/13/qn9rxyhd6ws0vbyr7qq_741m0000gn/T/pip-record-62bei6uy/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /private/var/folders/13/qn9rxyhd6ws0vbyr7qq_741m0000gn/T/pip-install-23echd02/pyrocksdb/
Install problem on centos 5
Below is the output - cannot see what the problem is. Thanks!
$ g++ --version
g++ (GCC) 4.8.1
$ gcc --version
gcc (GCC) 4.8.1
$ cd rocksdb
$ git checkout 2.8.fb
Note: checking out '2.8.fb'.
You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by performing another checkout.
If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -b with the checkout command again. Example:
git checkout -b new_branch_name
HEAD is now at f01a04e... Update HISTORY.md
$ export CPLUS_INCLUDE_PATH=$CPLUS_INCLUDE_PATH:/home/scratch/software/rocksdb/include
$ export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/home/scratch/software/rocksdb
$ export LIBRARY_PATH=$LIBRARY_PATH:/home/scratch/software/rocksdb
$ pip install git+git://github.com/stephan-hof/[email protected]
Downloading/unpacking git+git://github.com/stephan-hof/[email protected]
Cloning git://github.com/stephan-hof/pyrocksdb.git (to v0.2) to /tmp/pip-HZQ_kU-build
Running setup.py (path:/tmp/pip-HZQ_kU-build/setup.py) egg_info for package from git+git://github.com/stephan-hof/[email protected]
missing cimport in module 'pair': /home/anaconda/lib/python2.7/site-packages/Cython/Includes/libcpp/deque.pxd
Compiling rocksdb/_rocksdb.pyx because it changed.
Cythonizing rocksdb/_rocksdb.pyx
Requirement already satisfied (use --upgrade to upgrade): setuptools in /home/anaconda/lib/python2.7/site-packages/setuptools-3.6-py2.7.egg (from pyrocksdb==0.2)
Requirement already satisfied (use --upgrade to upgrade): Cython>=0.20 in /home/anaconda/lib/python2.7/site-packages (from pyrocksdb==0.2)
Installing collected packages: pyrocksdb
Running setup.py install for pyrocksdb
missing cimport in module 'pair': /home/anaconda/lib/python2.7/site-packages/Cython/Includes/libcpp/deque.pxd
building 'rocksdb._rocksdb' extension
gcc -pthread -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/home/anaconda/include/python2.7 -c rocksdb/_rocksdb.cpp -o build/temp.linux-x86_64-2.7/rocksdb/_rocksdb.o -std=gnu++11 -O3 -Wall -Wextra -Wconversion -fno-strict-aliasing
cc1plus: warning: command line option "-Wstrict-prototypes" is valid for Ada/C/ObjC but not for C++
cc1plus: error: unrecognized command line option "-std=gnu++11"
error: command 'gcc' failed with exit status 1
Complete output from command /home/anaconda/bin/python -c "import setuptools, tokenize;__file__='/tmp/pip-HZQ_kU-build/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-JAnhVo-record/install-record.txt --single-version-externally-managed --compile:
missing cimport in module 'pair': /home/anaconda/lib/python2.7/site-packages/Cython/Includes/libcpp/deque.pxd
running install
running build
running build_py
creating build
creating build/lib.linux-x86_64-2.7
creating build/lib.linux-x86_64-2.7/rocksdb
copying rocksdb/interfaces.py -> build/lib.linux-x86_64-2.7/rocksdb
copying rocksdb/errors.py -> build/lib.linux-x86_64-2.7/rocksdb
copying rocksdb/init.py -> build/lib.linux-x86_64-2.7/rocksdb
creating build/lib.linux-x86_64-2.7/rocksdb/tests
copying rocksdb/tests/test_db.py -> build/lib.linux-x86_64-2.7/rocksdb/tests
copying rocksdb/tests/test_options.py -> build/lib.linux-x86_64-2.7/rocksdb/tests
copying rocksdb/tests/init.py -> build/lib.linux-x86_64-2.7/rocksdb/tests
running egg_info
creating pyrocksdb.egg-info
writing requirements to pyrocksdb.egg-info/requires.txt
writing pyrocksdb.egg-info/PKG-INFO
writing top-level names to pyrocksdb.egg-info/top_level.txt
writing dependency_links to pyrocksdb.egg-info/dependency_links.txt
writing manifest file 'pyrocksdb.egg-info/SOURCES.txt'
warning: manifest_maker: standard file '-c' not found
reading manifest file 'pyrocksdb.egg-info/SOURCES.txt'
writing manifest file 'pyrocksdb.egg-info/SOURCES.txt'
copying rocksdb/_rocksdb.cpp -> build/lib.linux-x86_64-2.7/rocksdb
running build_ext
building 'rocksdb._rocksdb' extension
creating build/temp.linux-x86_64-2.7
creating build/temp.linux-x86_64-2.7/rocksdb
gcc -pthread -fno-strict-aliasing -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/home/anaconda/include/python2.7 -c rocksdb/_rocksdb.cpp -o build/temp.linux-x86_64-2.7/rocksdb/_rocksdb.o -std=gnu++11 -O3 -Wall -Wextra -Wconversion -fno-strict-aliasing
cc1plus: warning: command line option "-Wstrict-prototypes" is valid for Ada/C/ObjC but not for C++
cc1plus: error: unrecognized command line option "-std=gnu++11"
error: command 'gcc' failed with exit status 1
Cleaning up...
Command /home/anaconda/bin/python -c "import setuptools, tokenize;file='/tmp/pip-HZQ_kU-build/setup.py';exec(compile(getattr(tokenize, 'open', open)(file).read().replace('\r\n', '\n'), file, 'exec'))" install --record /tmp/pip-JAnhVo-record/install-record.txt --single-version-externally-managed --compile failed with error code 1 in /tmp/pip-HZQ_kU-build
Storing debug log for failure in /home/.pip/pip.log
repair damaged DB?
Is it possible to repair DB which is not ok?
For example when we are having some files unreadable or missing...
How?
Allow bytearray as input
Currently the bytes_to_slice does not support bytearray; bytearray is just a standard/compatible pointer.
batch.put(key, bytearray("data"))
rocksdb 3.x
@Stefan Is rocksdb 3.x support planned for pyrocksdb?
Undefined symbol in librt.so
import rocksdb
Traceback (most recent call last):
File "", line 1, in
File "/tmp/pyrocks_test/lib/python2.6/site-packages/rocksdb/init.py", line 1, in
from ._rocksdb import *
ImportError: /usr/lib/librocksdb.so: undefined symbol: clock_gettime
I think this will go away if you add "rt" as a library in setup.py
I was testing with a rocksdb rpm generated from trunk after this diff:
adsharma/rocksdb@6a4b16b
method to close db?
Imho closing DB maybe should be part of python api.
def _close_db(self):
del self.db
gc.collect()undefined symbol: _ZTIN7rocksdb10ComparatorE
python
Python 2.7.12 (default, Nov 19 2016, 06:48:10)
[GCC 5.4.0 20160609] on linux2
Type "help", "copyright", "credits" or "license" for more information.
import rocksdb
Traceback (most recent call last):
File "", line 1, in
File "/usr/local/lib/python2.7/dist-packages/rocksdb/init.py", line 1, in
from ._rocksdb import *
ImportError: /usr/local/lib/python2.7/dist-packages/rocksdb/_rocksdb.so: undefined symbol: _ZTIN7rocksdb10ComparatorE
Unable to close database connection
Hi, thanks for the awesome API to rocksdb.
I don't see a close() method for the rocksdb.DB class. I wind up being unable to manage the opening/closing of the connection and frequently fight with locked connections:
rocksdb.errors.RocksIOError: IO error: lock mydb/LOCK: No locks available
Is there a proper way to handle closing the database connection? Or do we need to implement the close method from the C++ driver? I would offer, but have a bit of a learning curve on the Cython front.
Thanks!
rocksdb on a cluster
On a cluster with network attached storage, would it be possible for each machine to run a rocksdb db instance in-memory such that:
- the data is loaded (written) to each rocksdb instance at startup
- alternatively, pre-create the rocksdb db for each machine and
store on the network storage; at startup, load the db into each machine's memory - once setup, the rocksdb instance on each machine runs independently of the network storage and each rocksdb runs independently of each other ie. not looking for distributed db functionality.
Hope this makes sense!
Support preset compression dictionary
From the blog:
Allow preset compression dictionary for improved compression of block-based tables. This is supported for zlib, zstd, and lz4. The compression dictionary’s size is configurable via CompressionOptions::max_dict_bytes.
looks like project cant be build on latest version of rocksdb
is project alive ?
can't build against rocksdb version 3.5 -- header file not found
Looks like the header files have moved:
rocksdb/_rocksdb.cpp:259:37: fatal error: utilities/backupable_db.h: No such file or directory
#include "utilities/backupable_db.h"
^
compilation terminated.
error: command 'x86_64-linux-gnu-gcc' failed with exit status 1
Memory leaks
I tried to test pyrocksdb to store large cache data. I'm processing and write with batch about 10M keys with ~500 bytes of data.
Memory usage growth from 50MB to 900MB and after fixing possible leaks in my code, I did't see any changes.
After that, I tried plyvel just to check if I'm right. With plyvel and leveldb memory usage was 10-100MB.
Also, I can't use memory_profiler package, script never ends. Without memory_profile script usually works for 3-4 minutes.
pyrocksdb==0.3
rocksdb: stable 3.6.1 (bottled)
Python 2.7.8
So, can you check for possible memory leaks?
Unable to install pyrocksdb against rocksdb 3.3
Hi,
I was able to successfully build pyrocksdb against rocksdb up until 3.2.
Installation does not work in my experience with the newly released 3.3.
Here is how I reproduced (Ubuntu 14.04 LTS):
apt-get install build-essential libgflags-dev libsnappy-dev zlib1g-dev libbz2-dev python-pip python-dev git
cd /usr/local/src
git clone https://github.com/facebook/rocksdb.git
cd rocksdb
git checkout 3.3.fb
make librocksdb.so
cp librocksdb.so /usr/local/lib
mkdir -p /usr/local/include/rocksdb/
cp -r ./include/* /usr/local/include/
pip install "Cython>=0.20"
pip install git+git://github.com/stephan-hof/pyrocksdb.git
# pip install git+git://github.com/stephan-hof/pyrocksdb.git
Downloading/unpacking git+git://github.com/stephan-hof/pyrocksdb.git
Cloning git://github.com/stephan-hof/pyrocksdb.git to /tmp/pip-zRO8z4-build
Running setup.py (path:/tmp/pip-zRO8z4-build/setup.py) egg_info for package from git+git://github.com/stephan-hof/pyrocksdb.git
missing cimport in module 'pair': /usr/local/lib/python2.7/dist-packages/Cython/Includes/libcpp/deque.pxd
Compiling rocksdb/_rocksdb.pyx because it changed.
Cythonizing rocksdb/_rocksdb.pyx
Requirement already satisfied (use --upgrade to upgrade): setuptools in /usr/lib/python2.7/dist-packages (from pyrocksdb==0.3)
Requirement already satisfied (use --upgrade to upgrade): Cython>=0.20 in /usr/local/lib/python2.7/dist-packages (from pyrocksdb==0.3)
Installing collected packages: pyrocksdb
Running setup.py install for pyrocksdb
missing cimport in module 'pair': /usr/local/lib/python2.7/dist-packages/Cython/Includes/libcpp/deque.pxd
building 'rocksdb._rocksdb' extension
x86_64-linux-gnu-gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC -I/usr/include/python2.7 -c rocksdb/_rocksdb.cpp -o build/temp.linux-x86_64-2.7/rocksdb/_rocksdb.o -std=gnu++11 -O3 -Wall -Wextra -Wconversion -fno-strict-aliasing
cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++ [enabled by default]
rocksdb/_rocksdb.cpp: In function ‘PyObject* __pyx_pf_7rocksdb_8_rocksdb_19PyBloomFilterPolicy_6create_filter(__pyx_obj_7rocksdb_8_rocksdb_PyBloomFilterPolicy*, PyObject*)’:
rocksdb/_rocksdb.cpp:5584:99: warning: conversion to ‘int’ from ‘std::vector<rocksdb::Slice>::size_type {aka long unsigned int}’ may alter its value [-Wconversion]
__pyx_v_self->policy->CreateFilter(py_rocks::vector_data(__pyx_v_c_keys), __pyx_v_c_keys.size(), (&__pyx_v_dst));
^
rocksdb/_rocksdb.cpp: In function ‘int __pyx_pf_7rocksdb_8_rocksdb_27TotalOrderPlainTableFactory___init__(__pyx_obj_7rocksdb_8_rocksdb_TotalOrderPlainTableFactory*, PyObject*, PyObject*, PyObject*)’:
rocksdb/_rocksdb.cpp:8988:44: error: ‘NewTotalOrderPlainTableFactory’ is not a member of ‘rocksdb’
__pyx_v_self->__pyx_base.factory.reset(rocksdb::NewTotalOrderPlainTableFactory(__pyx_t_1, __pyx_t_2, __pyx_t_3));
^
error: command 'x86_64-linux-gnu-gcc' failed with exit status 1
Complete output from command /usr/bin/python -c "import setuptools, tokenize;__file__='/tmp/pip-zRO8z4-build/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-vwXPvW-record/install-record.txt --single-version-externally-managed --compile:
missing cimport in module 'pair': /usr/local/lib/python2.7/dist-packages/Cython/Includes/libcpp/deque.pxd
running install
running build
running build_py
creating build
creating build/lib.linux-x86_64-2.7
creating build/lib.linux-x86_64-2.7/rocksdb
copying rocksdb/interfaces.py -> build/lib.linux-x86_64-2.7/rocksdb
copying rocksdb/errors.py -> build/lib.linux-x86_64-2.7/rocksdb
copying rocksdb/__init__.py -> build/lib.linux-x86_64-2.7/rocksdb
creating build/lib.linux-x86_64-2.7/rocksdb/tests
copying rocksdb/tests/test_options.py -> build/lib.linux-x86_64-2.7/rocksdb/tests
copying rocksdb/tests/test_db.py -> build/lib.linux-x86_64-2.7/rocksdb/tests
copying rocksdb/tests/__init__.py -> build/lib.linux-x86_64-2.7/rocksdb/tests
running egg_info
creating pyrocksdb.egg-info
writing requirements to pyrocksdb.egg-info/requires.txt
writing pyrocksdb.egg-info/PKG-INFO
writing top-level names to pyrocksdb.egg-info/top_level.txt
writing dependency_links to pyrocksdb.egg-info/dependency_links.txt
writing manifest file 'pyrocksdb.egg-info/SOURCES.txt'
warning: manifest_maker: standard file '-c' not found
reading manifest file 'pyrocksdb.egg-info/SOURCES.txt'
writing manifest file 'pyrocksdb.egg-info/SOURCES.txt'
copying rocksdb/_rocksdb.cpp -> build/lib.linux-x86_64-2.7/rocksdb
running build_ext
building 'rocksdb._rocksdb' extension
creating build/temp.linux-x86_64-2.7
creating build/temp.linux-x86_64-2.7/rocksdb
x86_64-linux-gnu-gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC -I/usr/include/python2.7 -c rocksdb/_rocksdb.cpp -o build/temp.linux-x86_64-2.7/rocksdb/_rocksdb.o -std=gnu++11 -O3 -Wall -Wextra -Wconversion -fno-strict-aliasing
cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++ [enabled by default]
rocksdb/_rocksdb.cpp: In function ‘PyObject* __pyx_pf_7rocksdb_8_rocksdb_19PyBloomFilterPolicy_6create_filter(__pyx_obj_7rocksdb_8_rocksdb_PyBloomFilterPolicy*, PyObject*)’:
rocksdb/_rocksdb.cpp:5584:99: warning: conversion to ‘int’ from ‘std::vector<rocksdb::Slice>::size_type {aka long unsigned int}’ may alter its value [-Wconversion]
__pyx_v_self->policy->CreateFilter(py_rocks::vector_data(__pyx_v_c_keys), __pyx_v_c_keys.size(), (&__pyx_v_dst));
^
rocksdb/_rocksdb.cpp: In function ‘int __pyx_pf_7rocksdb_8_rocksdb_27TotalOrderPlainTableFactory___init__(__pyx_obj_7rocksdb_8_rocksdb_TotalOrderPlainTableFactory*, PyObject*, PyObject*, PyObject*)’:
rocksdb/_rocksdb.cpp:8988:44: error: ‘NewTotalOrderPlainTableFactory’ is not a member of ‘rocksdb’
__pyx_v_self->__pyx_base.factory.reset(rocksdb::NewTotalOrderPlainTableFactory(__pyx_t_1, __pyx_t_2, __pyx_t_3));
^
error: command 'x86_64-linux-gnu-gcc' failed with exit status 1
----------------------------------------
Cleaning up...
Command /usr/bin/python -c "import setuptools, tokenize;__file__='/tmp/pip-zRO8z4-build/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-vwXPvW-record/install-record.txt --single-version-externally-managed --compile failed with error code 1 in /tmp/pip-zRO8z4-build
Traceback (most recent call last):
File "/usr/bin/pip", line 9, in <module>
load_entry_point('pip==1.5.4', 'console_scripts', 'pip')()
File "/usr/lib/python2.7/dist-packages/pip/__init__.py", line 185, in main
return command.main(cmd_args)
File "/usr/lib/python2.7/dist-packages/pip/basecommand.py", line 161, in main
text = '\n'.join(complete_log)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 42: ordinal not in range(128)
is this repo active ?
Hi,
It seems that pyrocksdb is the only driver in Python for RocksDB.
Is very sad that it seems to be abandoned ?
What about the TTL support ?
Enable LZ4 Compression
LZ4 is now enabled in RocksDB (it appears).
TypeError: Can't instantiate abstract class StaticPrefix
Hey Stephan,
i tried out pyrocksdb and i appreciate very much your work!
I have an issue with the prefix scan:
In [1]: import rocksdb
In [2]: class StaticPrefix(rocksdb.interfaces.SliceTransform):
...: def name(self):
...: return b'static'
...:
In [3]: def transform(self, src):
...: return (0, 5)
...:
In [4]: def in_domain(self, src):
...: return len(src) >= 5
...:
In [5]: def in_range(self, dst):
...: return len(dst) == 5
...:
In [6]:
In [6]: opts = rocksdb.Options()
In [7]: opts.create_if_missing=True
In [8]: opts.prefix_extractor = StaticPrefix()
---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
<ipython-input-8-9fbe2376d9cd> in <module>()
----> 1 opts.prefix_extractor = StaticPrefix()
TypeError: Can't instantiate abstract class StaticPrefix with abstract methods in_domain, in_range, transform```
am i doing something wrong?command 'x86_64-linux-gnu-gcc' failed with exit status 1
sudo /usr/bin/python3.6 -m pip install --force-reinstall --ignore-installed --no-binary --no-cache-dir git+git://github.com/stephan-hof/pyrocksdb.git
The directory '/home/testnet/.cache/pip/http' or its parent directory is not owned by the current user and the cache has been disabled. Please check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
The directory '/home/testnet/.cache/pip' or its parent directory is not owned by the current user and caching wheels has been disabled. check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
Collecting git+git://github.com/stephan-hof/pyrocksdb.git
Cloning git://github.com/stephan-hof/pyrocksdb.git to /tmp/pip-f03sqdob-build
Collecting setuptools (from pyrocksdb==0.5)
Downloading setuptools-38.5.1-py2.py3-none-any.whl (489kB)
100% |################################| 491kB 1.7MB/s
Installing collected packages: setuptools, pyrocksdb
Running setup.py install for pyrocksdb ... error
Complete output from command /usr/bin/python3.6 -u -c "import setuptools, tokenize;__file__='/tmp/pip-f03sqdob-build/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-f88nl2o3-record/install-record.txt --single-version-externally-managed --compile:
running install
running build
running build_py
creating build
creating build/lib.linux-x86_64-3.6
creating build/lib.linux-x86_64-3.6/rocksdb
copying rocksdb/__init__.py -> build/lib.linux-x86_64-3.6/rocksdb
copying rocksdb/interfaces.py -> build/lib.linux-x86_64-3.6/rocksdb
copying rocksdb/errors.py -> build/lib.linux-x86_64-3.6/rocksdb
creating build/lib.linux-x86_64-3.6/rocksdb/tests
copying rocksdb/tests/test_db.py -> build/lib.linux-x86_64-3.6/rocksdb/tests
copying rocksdb/tests/__init__.py -> build/lib.linux-x86_64-3.6/rocksdb/tests
copying rocksdb/tests/test_options.py -> build/lib.linux-x86_64-3.6/rocksdb/tests
running egg_info
creating pyrocksdb.egg-info
writing pyrocksdb.egg-info/PKG-INFO
writing dependency_links to pyrocksdb.egg-info/dependency_links.txt
writing requirements to pyrocksdb.egg-info/requires.txt
writing top-level names to pyrocksdb.egg-info/top_level.txt
writing manifest file 'pyrocksdb.egg-info/SOURCES.txt'
reading manifest file 'pyrocksdb.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
writing manifest file 'pyrocksdb.egg-info/SOURCES.txt'
copying rocksdb/_rocksdb.cpp -> build/lib.linux-x86_64-3.6/rocksdb
copying rocksdb/_rocksdb.pyx -> build/lib.linux-x86_64-3.6/rocksdb
copying rocksdb/backup.pxd -> build/lib.linux-x86_64-3.6/rocksdb
copying rocksdb/cache.pxd -> build/lib.linux-x86_64-3.6/rocksdb
copying rocksdb/comparator.pxd -> build/lib.linux-x86_64-3.6/rocksdb
copying rocksdb/db.pxd -> build/lib.linux-x86_64-3.6/rocksdb
copying rocksdb/env.pxd -> build/lib.linux-x86_64-3.6/rocksdb
copying rocksdb/filter_policy.pxd -> build/lib.linux-x86_64-3.6/rocksdb
copying rocksdb/iterator.pxd -> build/lib.linux-x86_64-3.6/rocksdb
copying rocksdb/logger.pxd -> build/lib.linux-x86_64-3.6/rocksdb
copying rocksdb/memtablerep.pxd -> build/lib.linux-x86_64-3.6/rocksdb
copying rocksdb/merge_operator.pxd -> build/lib.linux-x86_64-3.6/rocksdb
copying rocksdb/options.pxd -> build/lib.linux-x86_64-3.6/rocksdb
copying rocksdb/slice_.pxd -> build/lib.linux-x86_64-3.6/rocksdb
copying rocksdb/slice_transform.pxd -> build/lib.linux-x86_64-3.6/rocksdb
copying rocksdb/snapshot.pxd -> build/lib.linux-x86_64-3.6/rocksdb
copying rocksdb/status.pxd -> build/lib.linux-x86_64-3.6/rocksdb
copying rocksdb/std_memory.pxd -> build/lib.linux-x86_64-3.6/rocksdb
copying rocksdb/table_factory.pxd -> build/lib.linux-x86_64-3.6/rocksdb
copying rocksdb/universal_compaction.pxd -> build/lib.linux-x86_64-3.6/rocksdb
creating build/lib.linux-x86_64-3.6/rocksdb/cpp
copying rocksdb/cpp/comparator_wrapper.hpp -> build/lib.linux-x86_64-3.6/rocksdb/cpp
copying rocksdb/cpp/filter_policy_wrapper.hpp -> build/lib.linux-x86_64-3.6/rocksdb/cpp
copying rocksdb/cpp/memtable_factories.hpp -> build/lib.linux-x86_64-3.6/rocksdb/cpp
copying rocksdb/cpp/merge_operator_wrapper.hpp -> build/lib.linux-x86_64-3.6/rocksdb/cpp
copying rocksdb/cpp/slice_transform_wrapper.hpp -> build/lib.linux-x86_64-3.6/rocksdb/cpp
copying rocksdb/cpp/utils.hpp -> build/lib.linux-x86_64-3.6/rocksdb/cpp
copying rocksdb/cpp/write_batch_iter_helper.hpp -> build/lib.linux-x86_64-3.6/rocksdb/cpp
running build_ext
building 'rocksdb._rocksdb' extension
creating build/temp.linux-x86_64-3.6
creating build/temp.linux-x86_64-3.6/rocksdb
x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -Irocksdb -I./rocksdb -I/usr/include/python3.6m -c rocksdb/_rocksdb.cpp -o build/temp.linux-x86_64-3.6/rocksdb/_rocksdb.o -std=c++11 -O3 -Wall -Wextra -Wconversion -fno-strict-aliasing
cc1plus: warning: command line option '-Wstrict-prototypes' is valid for C/ObjC but not for C++
rocksdb/_rocksdb.cpp:589:27: fatal error: rocksdb/slice.h: No such file or directory
compilation terminated.
error: command 'x86_64-linux-gnu-gcc' failed with exit status 1
----------------------------------------
Command "/usr/bin/python3.6 -u -c "import setuptools, tokenize;__file__='/tmp/pip-f03sqdob-build/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-f88nl2o3-record/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /tmp/pip-f03sqdob-build/
Ubuntu 16.04.3 LTS (GNU/Linux 4.4.0-112-generic x86_64)
Optimization: Support batch size arguments for iterables
I'm going to take a peek at the performance on this -- if it's not worth it, I'll just close out the issue.
There may be overhead in the iterkeys/iteritems/itervalues calls when traversing a large set of data, which could be reduced by running a loop of iteration calls in the cython interface before returning results.
My goal is to show a significant increase in speed when traversing through more than a hundred thousand records. I'll just implement an iterbatch() method for the moment.
Build problem on mac osx and python3
Hi,
On mac osx (Yosemite 10.10.3) with gcc:
Configured with: --prefix=/Library/Developer/CommandLineTools/usr --with-gxx-include-dir=/usr/include/c++/4.2.1
Apple LLVM version 6.1.0 (clang-602.0.53) (based on LLVM 3.6.0svn)
Target: x86_64-apple-darwin14.3.0
Thread model: posix
And Clang
Apple LLVM version 6.1.0 (clang-602.0.53) (based on LLVM 3.6.0svn)
Target: x86_64-apple-darwin14.3.0
Thread model: posix
I am able to build and test pyrocksdb with python 2.7 but when attempting:
sudo python setup.py install with python3.4 I get the following error:
running install
running bdist_egg
running egg_info
writing dependency_links to pyrocksdb.egg-info/dependency_links.txt
writing pyrocksdb.egg-info/PKG-INFO
writing top-level names to pyrocksdb.egg-info/top_level.txt
writing requirements to pyrocksdb.egg-info/requires.txt
reading manifest file 'pyrocksdb.egg-info/SOURCES.txt'
writing manifest file 'pyrocksdb.egg-info/SOURCES.txt'
installing library code to build/bdist.macosx-10.6-intel/egg
running install_lib
running build_py
running build_ext
building 'rocksdb._rocksdb' extension
/usr/bin/clang -fno-strict-aliasing -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -arch i386 -arch x86_64 -g -I/Library/Frameworks/Python.framework/Versions/3.4/include/python3.4m -c rocksdb/_rocksdb.cpp -o build/temp.macosx-10.6-intel-3.4/rocksdb/_rocksdb.o -std=gnu++11 -O3 -Wall -Wextra -Wconversion -fno-strict-aliasing
In file included from rocksdb/_rocksdb.cpp:298:
/usr/local/include/rocksdb/env.h:43:12: error: no member named 'unique_ptr' in namespace 'std'
using std::unique_ptr;
~~~~~^
Followed by other errors due to this one and other missing stuff from std lib in c++11.
However, if i run the above clang command in the terminal it succeeds.
I have tried this on a second mac and had exactly the same problem. I have been looking around for what the issue might be but come up blank. I realize this is not a issue with pyrocksdb itself but I hoped that someone might have had this problem, or something similar, and dealt with it already.
Thanks
Open database with TTL
RocksDB supports TTL settings(according to https://github.com/facebook/rocksdb/wiki/Time-to-Live) though I cannot find related options in pyrocksdb. is it not supported?
Does pyrocksdb supports against rocksdb3.6?
Hi @stephan-hof ,
Does pyrocksdb supports against rocksdb3.6?, if so it was failed when tried in the following environment.
ubuntu 14.04
gcc 4.8.2
rocksdb 3.6
Cython 0.20
pyrocksdb 0.2.1
Can you please look over the error faced:
root@vmserver:/tmp/rocksdb# sudo pip install git+git://github.com/stephan-hof/[email protected]
Downloading/unpacking git+git://github.com/stephan-hof/pyrocksdb.git@v0.2.1
Cloning git://github.com/stephan-hof/pyrocksdb.git (to v0.2.1) to /tmp/pip-wnupVg-build
Running setup.py (path:/tmp/pip-wnupVg-build/setup.py) egg_info for package from git+git://github.com/stephan-hof/pyrocksdb.git@v0.2.1
missing cimport in module 'pair': /usr/local/lib/python2.7/dist-packages/Cython/Includes/libcpp/deque.pxd
Compiling rocksdb/_rocksdb.pyx because it changed.
Cythonizing rocksdb/_rocksdb.pyx
Requirement already satisfied (use --upgrade to upgrade): setuptools in /usr/lib/python2.7/dist-packages (from pyrocksdb==0.2.1)
Requirement already satisfied (use --upgrade to upgrade): Cython>=0.20 in /usr/local/lib/python2.7/dist-packages (from pyrocksdb==0.2.1)
Installing collected packages: pyrocksdb
Running setup.py install for pyrocksdb
missing cimport in module 'pair': /usr/local/lib/python2.7/dist-packages/Cython/Includes/libcpp/deque.pxd
building 'rocksdb._rocksdb' extension
x86_64-linux-gnu-gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC -I/usr/include/python2.7 -c rocksdb/_rocksdb.cpp -o build/
temp.linux-x86_64-2.7/rocksdb/_rocksdb.o -std=gnu++11 -O3 -Wall -Wextra -Wconversion -fno-strict-aliasing
cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++ [enabled by default]
In file included from rocksdb/_rocksdb.cpp:368:0:
/usr/include/utilities/backupable_db.h:11:2: warning: #warning This file was moved to rocksdb/utilities/backupable_db.h [-Wcpp]
#warning This file was moved to rocksdb/utilities/backupable_db.h
^
rocksdb/_rocksdb.cpp: In function ‘PyObject* __pyx_pf_7rocksdb_8_rocksdb_19PyBloomFilterPolicy_6create_filter(__pyx_obj_7rocksdb_8_rocksdb_PyBloomFilterPolicy*, PyOb
ject*)’:
rocksdb/_rocksdb.cpp:5589:99: warning: conversion to ‘int’ from ‘std::vector<rocksdb::Slice>::size_type {aka long unsigned int}’ may alter its value [-Wconversion]
__pyx_v_self->policy->CreateFilter(py_rocks::vector_data(__pyx_v_c_keys), __pyx_v_c_keys.size(), (&__pyx_v_dst));
^
rocksdb/_rocksdb.cpp: In function ‘int __pyx_pf_7rocksdb_8_rocksdb_17PlainTableFactory___init__(__pyx_obj_7rocksdb_8_rocksdb_PlainTableFactory*, PyObject*, PyObject*, PyObject*, PyObject*)’:
rocksdb/_rocksdb.cpp:8837:116: error: invalid initialization of reference of type ‘const rocksdb::PlainTableOptions&’ from expression of type ‘uint32_t {aka unsigned int}’
__pyx_v_self->__pyx_base.factory.reset(rocksdb::NewPlainTableFactory(__pyx_t_1, __pyx_t_2, __pyx_t_3, __pyx_t_4));
^
In file included from rocksdb/_rocksdb.cpp:360:0:
/usr/include/rocksdb/table.h:227:22: error: in passing argument 1 of ‘rocksdb::TableFactory* rocksdb::NewPlainTableFactory(const rocksdb::PlainTableOptions&)’
extern TableFactory* NewPlainTableFactory(const PlainTableOptions& options =
^
rocksdb/_rocksdb.cpp: In function ‘int __pyx_pf_7rocksdb_8_rocksdb_27TotalOrderPlainTableFactory___init__(__pyx_obj_7rocksdb_8_rocksdb_TotalOrderPlainTableFactory*, PyObject*, PyObject*, PyObject*)’:
rocksdb/_rocksdb.cpp:8993:44: error: ‘NewTotalOrderPlainTableFactory’ is not a member of ‘rocksdb’
__pyx_v_self->__pyx_base.factory.reset(rocksdb::NewTotalOrderPlainTableFactory(__pyx_t_1, __pyx_t_2, __pyx_t_3));
^
rocksdb/_rocksdb.cpp: In function ‘PyObject* __pyx_pf_7rocksdb_8_rocksdb_7Options_10block_size___get__(__pyx_obj_7rocksdb_8_rocksdb_Options*)’:
rocksdb/_rocksdb.cpp:10822:58: error: ‘struct rocksdb::Options’ has no member named ‘block_size’
__pyx_t_1 = __Pyx_PyInt_FromSize_t(__pyx_v_self->opts->block_size); if (unlikely(!__pyx_t_1)) {__pyx_filename = __pyx_f[0]; __pyx_lineno = 717; __pyx_clineno = __LINE__; goto __pyx_L1_error;}
^
rocksdb/_rocksdb.cpp: In function ‘int __pyx_pf_7rocksdb_8_rocksdb_7Options_10block_size_2__set__(__pyx_obj_7rocksdb_8_rocksdb_Options*, PyObject*)’:
rocksdb/_rocksdb.cpp:10885:23: error: ‘struct rocksdb::Options’ has no member named ‘block_size’
__pyx_v_self->opts->block_size = __pyx_t_1;
^
rocksdb/_rocksdb.cpp: In function ‘PyObject* __pyx_pf_7rocksdb_8_rocksdb_7Options_22block_restart_interval___get__(__pyx_obj_7rocksdb_8_rocksdb_Options*)’:
rocksdb/_rocksdb.cpp:10944:56: error: ‘struct rocksdb::Options’ has no member named ‘block_restart_interval’
__pyx_t_1 = __Pyx_PyInt_From_int(__pyx_v_self->opts->block_restart_interval); if (unlikely(!__pyx_t_1)) {__pyx_filename = __pyx_f[0]; __pyx_lineno = 723; __pyx_clineno = __LINE__; goto __pyx_L1_error;}
^
rocksdb/_rocksdb.cpp: In function ‘int __pyx_pf_7rocksdb_8_rocksdb_7Options_22block_restart_interval_2__set__(__pyx_obj_7rocksdb_8_rocksdb_Options*, PyObject*)’:
rocksdb/_rocksdb.cpp:11007:23: error: ‘struct rocksdb::Options’ has no member named ‘block_restart_interval’
__pyx_v_self->opts->block_restart_interval = __pyx_t_1;
^
rocksdb/_rocksdb.cpp: In function ‘PyObject* __pyx_pf_7rocksdb_8_rocksdb_7Options_19whole_key_filtering___get__(__pyx_obj_7rocksdb_8_rocksdb_Options*)’:
rocksdb/_rocksdb.cpp:11430:57: error: ‘struct rocksdb::Options’ has no member named ‘whole_key_filtering’
__pyx_t_1 = __Pyx_PyBool_FromLong(__pyx_v_self->opts->whole_key_filtering); if (unlikely(!__pyx_t_1)) {__pyx_filename = __pyx_f[0]; __pyx_lineno = 754; __pyx_clineno = __LINE__; goto __pyx_L1_error;}
^
rocksdb/_rocksdb.cpp:444:36: note: in definition of macro ‘__Pyx_PyBool_FromLong’
#define __Pyx_PyBool_FromLong(b) ((b) ? (Py_INCREF(Py_True), Py_True) : (Py_INCREF(Py_False), Py_False))
^
rocksdb/_rocksdb.cpp: In function ‘int __pyx_pf_7rocksdb_8_rocksdb_7Options_19whole_key_filtering_2__set__(__pyx_obj_7rocksdb_8_rocksdb_Options*, PyObject*)’:
rocksdb/_rocksdb.cpp:11493:23: error: ‘struct rocksdb::Options’ has no member named ‘whole_key_filtering’
__pyx_v_self->opts->whole_key_filtering = __pyx_t_1;
^
rocksdb/_rocksdb.cpp: In function ‘PyObject* __pyx_pf_7rocksdb_8_rocksdb_7Options_21target_file_size_base___get__(__pyx_obj_7rocksdb_8_rocksdb_Options*)’:
rocksdb/_rocksdb.cpp:12162:77: warning: conversion to ‘int’ from ‘uint64_t {aka long unsigned int}’ may alter its value [-Wconversion]
__pyx_t_1 = __Pyx_PyInt_From_int(__pyx_v_self->opts->target_file_size_base); if (unlikely(!__pyx_t_1)) {__pyx_filename = __pyx_f[0]; __pyx_lineno = 790; __pyx_clineno = __LINE__; goto __pyx_L1_error;}
^
rocksdb/_rocksdb.cpp: In function ‘PyObject* __pyx_pf_7rocksdb_8_rocksdb_7Options_21db_stats_log_interval___get__(__pyx_obj_7rocksdb_8_rocksdb_Options*)’:
rocksdb/_rocksdb.cpp:13382:56: error: ‘struct rocksdb::Options’ has no member named ‘db_stats_log_interval’
__pyx_t_1 = __Pyx_PyInt_From_int(__pyx_v_self->opts->db_stats_log_interval); if (unlikely(!__pyx_t_1)) {__pyx_filename = __pyx_f[0]; __pyx_lineno = 850; __pyx_clineno = __LINE__; goto __pyx_L1_error;}
^
rocksdb/_rocksdb.cpp: In function ‘int __pyx_pf_7rocksdb_8_rocksdb_7Options_21db_stats_log_interval_2__set__(__pyx_obj_7rocksdb_8_rocksdb_Options*, PyObject*)’:
rocksdb/_rocksdb.cpp:13445:23: error: ‘struct rocksdb::Options’ has no member named ‘db_stats_log_interval’
__pyx_v_self->opts->db_stats_log_interval = __pyx_t_1;
^
rocksdb/_rocksdb.cpp: In function ‘PyObject* __pyx_pf_7rocksdb_8_rocksdb_7Options_23disable_seek_compaction___get__(__pyx_obj_7rocksdb_8_rocksdb_Options*)’:
rocksdb/_rocksdb.cpp:13748:57: error: ‘struct rocksdb::Options’ has no member named ‘disable_seek_compaction’
__pyx_t_1 = __Pyx_PyBool_FromLong(__pyx_v_self->opts->disable_seek_compaction); if (unlikely(!__pyx_t_1)) {__pyx_filename = __pyx_f[0]; __pyx_lineno = 868; __pyx_clineno = __LINE__; goto __pyx_L1_error;}
^
rocksdb/_rocksdb.cpp:444:36: note: in definition of macro ‘__Pyx_PyBool_FromLong’
#define __Pyx_PyBool_FromLong(b) ((b) ? (Py_INCREF(Py_True), Py_True) : (Py_INCREF(Py_False), Py_False))
^
rocksdb/_rocksdb.cpp: In function ‘int __pyx_pf_7rocksdb_8_rocksdb_7Options_23disable_seek_compaction_2__set__(__pyx_obj_7rocksdb_8_rocksdb_Options*, PyObject*)’:
rocksdb/_rocksdb.cpp:13811:23: error: ‘struct rocksdb::Options’ has no member named ‘disable_seek_compaction’
__pyx_v_self->opts->disable_seek_compaction = __pyx_t_1;
^
rocksdb/_rocksdb.cpp: In function ‘PyObject* __pyx_pf_7rocksdb_8_rocksdb_7Options_14no_block_cache___get__(__pyx_obj_7rocksdb_8_rocksdb_Options*)’:
rocksdb/_rocksdb.cpp:15090:57: error: ‘struct rocksdb::Options’ has no member named ‘no_block_cache’
__pyx_t_1 = __Pyx_PyBool_FromLong(__pyx_v_self->opts->no_block_cache); if (unlikely(!__pyx_t_1)) {__pyx_filename = __pyx_f[0]; __pyx_lineno = 934; __pyx_clineno = __LINE__; goto __pyx_L1_error;}
^
rocksdb/_rocksdb.cpp:444:36: note: in definition of macro ‘__Pyx_PyBool_FromLong’
#define __Pyx_PyBool_FromLong(b) ((b) ? (Py_INCREF(Py_True), Py_True) : (Py_INCREF(Py_False), Py_False))
^
rocksdb/_rocksdb.cpp: In function ‘int __pyx_pf_7rocksdb_8_rocksdb_7Options_14no_block_cache_2__set__(__pyx_obj_7rocksdb_8_rocksdb_Options*, PyObject*)’:
rocksdb/_rocksdb.cpp:15153:23: error: ‘struct rocksdb::Options’ has no member named ‘no_block_cache’
__pyx_v_self->opts->no_block_cache = __pyx_t_1;
^
rocksdb/_rocksdb.cpp: In function ‘PyObject* __pyx_pf_7rocksdb_8_rocksdb_7Options_20block_size_deviation___get__(__pyx_obj_7rocksdb_8_rocksdb_Options*)’:
rocksdb/_rocksdb.cpp:16920:56: error: ‘struct rocksdb::Options’ has no member named ‘block_size_deviation’
__pyx_t_1 = __Pyx_PyInt_From_int(__pyx_v_self->opts->block_size_deviation); if (unlikely(!__pyx_t_1)) {__pyx_filename = __pyx_f[0]; __pyx_lineno = 1024; __pyx_clineno = __LINE__; goto __pyx_L1_error;}
^
rocksdb/_rocksdb.cpp: In function ‘int __pyx_pf_7rocksdb_8_rocksdb_7Options_20block_size_deviation_2__set__(__pyx_obj_7rocksdb_8_rocksdb_Options*, PyObject*)’:
rocksdb/_rocksdb.cpp:16983:23: error: ‘struct rocksdb::Options’ has no member named ‘block_size_deviation’
__pyx_v_self->opts->block_size_deviation = __pyx_t_1;
^
rocksdb/_rocksdb.cpp: In function ‘int __pyx_pf_7rocksdb_8_rocksdb_7Options_13filter_policy_2__set__(__pyx_obj_7rocksdb_8_rocksdb_Options*, PyObject*)’:
rocksdb/_rocksdb.cpp:19602:23: error: ‘struct rocksdb::Options’ has no member named ‘filter_policy’
__pyx_v_self->opts->filter_policy = ((struct __pyx_vtabstruct_7rocksdb_8_rocksdb_PyFilterPolicy *)__pyx_v_self->py_filter_policy->__pyx_vtab)->get_policy(__pyx_v_self->py_filter_policy);
^
rocksdb/_rocksdb.cpp: In function ‘int __pyx_pf_7rocksdb_8_rocksdb_7Options_11block_cache_2__set__(__pyx_obj_7rocksdb_8_rocksdb_Options*, PyObject*)’:
rocksdb/_rocksdb.cpp:19950:27: error: ‘struct rocksdb::Options’ has no member named ‘block_cache’
__pyx_v_self->opts->block_cache.reset();
^
rocksdb/_rocksdb.cpp:20015:25: error: ‘struct rocksdb::Options’ has no member named ‘block_cache’
__pyx_v_self->opts->block_cache = ((struct __pyx_vtabstruct_7rocksdb_8_rocksdb_PyCache *)__pyx_v_self->py_block_cache->__pyx_vtab)->get_cache(__pyx_v_self->py_block_cache);
^
rocksdb/_rocksdb.cpp: In function ‘int __pyx_pf_7rocksdb_8_rocksdb_7Options_22block_cache_compressed_2__set__(__pyx_obj_7rocksdb_8_rocksdb_Options*, PyObject*)’:
rocksdb/_rocksdb.cpp:20194:27: error: ‘struct rocksdb::Options’ has no member named ‘block_cache_compressed’
__pyx_v_self->opts->block_cache_compressed.reset();
^
rocksdb/_rocksdb.cpp:20267:23: error: ‘struct rocksdb::Options’ has no member named ‘block_cache_compressed’
__pyx_v_self->opts->block_cache_compressed = ((struct __pyx_vtabstruct_7rocksdb_8_rocksdb_PyCache *)((struct __pyx_obj_7rocksdb_8_rocksdb_PyCache *)__pyx_v_value)->__pyx_vtab)->get_cache(((struct __pyx_obj_7rocksdb_8_rocksdb_PyCache *)__pyx_v_value));
^
rocksdb/_rocksdb.cpp: In function ‘rocksdb::ReadOptions __pyx_f_7rocksdb_8_rocksdb_2DB_build_read_opts(__pyx_obj_7rocksdb_8_rocksdb_DB*, PyObject*)’:
rocksdb/_rocksdb.cpp:25316:16: error: ‘struct rocksdb::ReadOptions’ has no member named ‘prefix_seek’
__pyx_v_opts.prefix_seek = __pyx_t_2;
^
rocksdb/_rocksdb.cpp: In function ‘PyObject* __pyx_f_7rocksdb_8_rocksdb_12BaseIterator_set_prefix(__pyx_obj_7rocksdb_8_rocksdb_BaseIterator*, rocksdb::ReadOptions&, __pyx_opt_args_7rocksdb_8_rocksdb_12BaseIterator_set_prefix*)’:
rocksdb/_rocksdb.cpp:26349:16: error: ‘struct rocksdb::ReadOptions’ has no member named ‘prefix’
__pyx_v_opts.prefix = (&__pyx_v_self->c_prefix);
^
error: command 'x86_64-linux-gnu-gcc' failed with exit status 1
Complete output from command /usr/bin/python -c "import setuptools, tokenize;__file__='/tmp/pip-wnupVg-build/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-45kNfa-record/install-record.txt --single-version-externally-managed --compile:
missing cimport in module 'pair': /usr/local/lib/python2.7/dist-packages/Cython/Includes/libcpp/deque.pxd
running install
running build
running build_py
creating build
creating build/lib.linux-x86_64-2.7
creating build/lib.linux-x86_64-2.7/rocksdb
copying rocksdb/errors.py -> build/lib.linux-x86_64-2.7/rocksdb
copying rocksdb/interfaces.py -> build/lib.linux-x86_64-2.7/rocksdb
copying rocksdb/__init__.py -> build/lib.linux-x86_64-2.7/rocksdb
creating build/lib.linux-x86_64-2.7/rocksdb/tests
copying rocksdb/tests/test_options.py -> build/lib.linux-x86_64-2.7/rocksdb/tests
copying rocksdb/tests/test_db.py -> build/lib.linux-x86_64-2.7/rocksdb/tests
copying rocksdb/tests/__init__.py -> build/lib.linux-x86_64-2.7/rocksdb/tests
running egg_info
creating pyrocksdb.egg-info
writing requirements to pyrocksdb.egg-info/requires.txt
writing pyrocksdb.egg-info/PKG-INFO
writing top-level names to pyrocksdb.egg-info/top_level.txt
writing dependency_links to pyrocksdb.egg-info/dependency_links.txt
writing manifest file 'pyrocksdb.egg-info/SOURCES.txt'
warning: manifest_maker: standard file '-c' not found
reading manifest file 'pyrocksdb.egg-info/SOURCES.txt'
writing manifest file 'pyrocksdb.egg-info/SOURCES.txt'
copying rocksdb/_rocksdb.cpp -> build/lib.linux-x86_64-2.7/rocksdb
running build_ext
building 'rocksdb._rocksdb' extension
creating build/temp.linux-x86_64-2.7
creating build/temp.linux-x86_64-2.7/rocksdb
x86_64-linux-gnu-gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC -I/usr/include/python2.7 -c rocksdb/_rocksdb.cpp -o build/temp.linux-x86_64-2.7/rocksdb/_rocksdb.o -std=gnu++11 -O3 -Wall -Wextra -Wconversion -fno-strict-aliasing
cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++ [enabled by default]
In file included from rocksdb/_rocksdb.cpp:368:0:
/usr/include/utilities/backupable_db.h:11:2: warning: #warning This file was moved to rocksdb/utilities/backupable_db.h [-Wcpp]
#warning This file was moved to rocksdb/utilities/backupable_db.h
^
rocksdb/_rocksdb.cpp: In function ‘PyObject* __pyx_pf_7rocksdb_8_rocksdb_19PyBloomFilterPolicy_6create_filter(__pyx_obj_7rocksdb_8_rocksdb_PyBloomFilterPolicy*, PyObject*)’:
rocksdb/_rocksdb.cpp:5589:99: warning: conversion to ‘int’ from ‘std::vector<rocksdb::Slice>::size_type {aka long unsigned int}’ may alter its value [-Wconversion]
__pyx_v_self->policy->CreateFilter(py_rocks::vector_data(__pyx_v_c_keys), __pyx_v_c_keys.size(), (&__pyx_v_dst));
^
rocksdb/_rocksdb.cpp: In function ‘int __pyx_pf_7rocksdb_8_rocksdb_17PlainTableFactory___init__(__pyx_obj_7rocksdb_8_rocksdb_PlainTableFactory*, PyObject*, PyObject*, PyObject*, PyObject*)’:
rocksdb/_rocksdb.cpp:8837:116: error: invalid initialization of reference of type ‘const rocksdb::PlainTableOptions&’ from expression of type ‘uint32_t {aka unsigned int}’
__pyx_v_self->__pyx_base.factory.reset(rocksdb::NewPlainTableFactory(__pyx_t_1, __pyx_t_2, __pyx_t_3, __pyx_t_4));
^
In file included from rocksdb/_rocksdb.cpp:360:0:
/usr/include/rocksdb/table.h:227:22: error: in passing argument 1 of ‘rocksdb::TableFactory* rocksdb::NewPlainTableFactory(const rocksdb::PlainTableOptions&)’
extern TableFactory* NewPlainTableFactory(const PlainTableOptions& options =
^
rocksdb/_rocksdb.cpp: In function ‘int __pyx_pf_7rocksdb_8_rocksdb_27TotalOrderPlainTableFactory___init__(__pyx_obj_7rocksdb_8_rocksdb_TotalOrderPlainTableFactory*, PyObject*, PyObject*, PyObject*)’:
rocksdb/_rocksdb.cpp:8993:44: error: ‘NewTotalOrderPlainTableFactory’ is not a member of ‘rocksdb’
__pyx_v_self->__pyx_base.factory.reset(rocksdb::NewTotalOrderPlainTableFactory(__pyx_t_1, __pyx_t_2, __pyx_t_3));
^
rocksdb/_rocksdb.cpp: In function ‘PyObject* __pyx_pf_7rocksdb_8_rocksdb_7Options_10block_size___get__(__pyx_obj_7rocksdb_8_rocksdb_Options*)’:
rocksdb/_rocksdb.cpp:10822:58: error: ‘struct rocksdb::Options’ has no member named ‘block_size’
__pyx_t_1 = __Pyx_PyInt_FromSize_t(__pyx_v_self->opts->block_size); if (unlikely(!__pyx_t_1)) {__pyx_filename = __pyx_f[0]; __pyx_lineno = 717; __pyx_clineno = __LINE__; goto __pyx_L1_error;}
^
rocksdb/_rocksdb.cpp: In function ‘int __pyx_pf_7rocksdb_8_rocksdb_7Options_10block_size_2__set__(__pyx_obj_7rocksdb_8_rocksdb_Options*, PyObject*)’:
rocksdb/_rocksdb.cpp:10885:23: error: ‘struct rocksdb::Options’ has no member named ‘block_size’
__pyx_v_self->opts->block_size = __pyx_t_1;
^
rocksdb/_rocksdb.cpp: In function ‘PyObject* __pyx_pf_7rocksdb_8_rocksdb_7Options_22block_restart_interval___get__(__pyx_obj_7rocksdb_8_rocksdb_Options*)’:
rocksdb/_rocksdb.cpp:10944:56: error: ‘struct rocksdb::Options’ has no member named ‘block_restart_interval’
__pyx_t_1 = __Pyx_PyInt_From_int(__pyx_v_self->opts->block_restart_interval); if (unlikely(!__pyx_t_1)) {__pyx_filename = __pyx_f[0]; __pyx_lineno = 723; __pyx_clineno = __LINE__; goto __pyx_L1_error;}
^
rocksdb/_rocksdb.cpp: In function ‘int __pyx_pf_7rocksdb_8_rocksdb_7Options_22block_restart_interval_2__set__(__pyx_obj_7rocksdb_8_rocksdb_Options*, PyObject*)’:
rocksdb/_rocksdb.cpp:11007:23: error: ‘struct rocksdb::Options’ has no member named ‘block_restart_interval’
__pyx_v_self->opts->block_restart_interval = __pyx_t_1;
^
rocksdb/_rocksdb.cpp: In function ‘PyObject* __pyx_pf_7rocksdb_8_rocksdb_7Options_19whole_key_filtering___get__(__pyx_obj_7rocksdb_8_rocksdb_Options*)’:
rocksdb/_rocksdb.cpp:11430:57: error: ‘struct rocksdb::Options’ has no member named ‘whole_key_filtering’
__pyx_t_1 = __Pyx_PyBool_FromLong(__pyx_v_self->opts->whole_key_filtering); if (unlikely(!__pyx_t_1)) {__pyx_filename = __pyx_f[0]; __pyx_lineno = 754; __pyx_clineno = __LINE__; goto __pyx_L1_error;}
^
rocksdb/_rocksdb.cpp:444:36: note: in definition of macro ‘__Pyx_PyBool_FromLong’
#define __Pyx_PyBool_FromLong(b) ((b) ? (Py_INCREF(Py_True), Py_True) : (Py_INCREF(Py_False), Py_False))
^
rocksdb/_rocksdb.cpp: In function ‘int __pyx_pf_7rocksdb_8_rocksdb_7Options_19whole_key_filtering_2__set__(__pyx_obj_7rocksdb_8_rocksdb_Options*, PyObject*)’:
rocksdb/_rocksdb.cpp:11493:23: error: ‘struct rocksdb::Options’ has no member named ‘whole_key_filtering’
__pyx_v_self->opts->whole_key_filtering = __pyx_t_1;
^
rocksdb/_rocksdb.cpp: In function ‘PyObject* __pyx_pf_7rocksdb_8_rocksdb_7Options_21target_file_size_base___get__(__pyx_obj_7rocksdb_8_rocksdb_Options*)’:
rocksdb/_rocksdb.cpp:12162:77: warning: conversion to ‘int’ from ‘uint64_t {aka long unsigned int}’ may alter its value [-Wconversion]
__pyx_t_1 = __Pyx_PyInt_From_int(__pyx_v_self->opts->target_file_size_base); if (unlikely(!__pyx_t_1)) {__pyx_filename = __pyx_f[0]; __pyx_lineno = 790; __pyx_clineno = __LINE__; goto __pyx_L1_error;}
^
rocksdb/_rocksdb.cpp: In function ‘PyObject* __pyx_pf_7rocksdb_8_rocksdb_7Options_21db_stats_log_interval___get__(__pyx_obj_7rocksdb_8_rocksdb_Options*)’:
rocksdb/_rocksdb.cpp:13382:56: error: ‘struct rocksdb::Options’ has no member named ‘db_stats_log_interval’
__pyx_t_1 = __Pyx_PyInt_From_int(__pyx_v_self->opts->db_stats_log_interval); if (unlikely(!__pyx_t_1)) {__pyx_filename = __pyx_f[0]; __pyx_lineno = 850; __pyx_clineno = __LINE__; goto __pyx_L1_error;}
^
rocksdb/_rocksdb.cpp: In function ‘int __pyx_pf_7rocksdb_8_rocksdb_7Options_21db_stats_log_interval_2__set__(__pyx_obj_7rocksdb_8_rocksdb_Options*, PyObject*)’:
rocksdb/_rocksdb.cpp:13445:23: error: ‘struct rocksdb::Options’ has no member named ‘db_stats_log_interval’
__pyx_v_self->opts->db_stats_log_interval = __pyx_t_1;
^
rocksdb/_rocksdb.cpp: In function ‘PyObject* __pyx_pf_7rocksdb_8_rocksdb_7Options_23disable_seek_compaction___get__(__pyx_obj_7rocksdb_8_rocksdb_Options*)’:
rocksdb/_rocksdb.cpp:13748:57: error: ‘struct rocksdb::Options’ has no member named ‘disable_seek_compaction’
__pyx_t_1 = __Pyx_PyBool_FromLong(__pyx_v_self->opts->disable_seek_compaction); if (unlikely(!__pyx_t_1)) {__pyx_filename = __pyx_f[0]; __pyx_lineno = 868; __pyx_clineno = __LINE__; goto __pyx_L1_error;}
^
rocksdb/_rocksdb.cpp:444:36: note: in definition of macro ‘__Pyx_PyBool_FromLong’
#define __Pyx_PyBool_FromLong(b) ((b) ? (Py_INCREF(Py_True), Py_True) : (Py_INCREF(Py_False), Py_False))
^
rocksdb/_rocksdb.cpp: In function ‘int __pyx_pf_7rocksdb_8_rocksdb_7Options_23disable_seek_compaction_2__set__(__pyx_obj_7rocksdb_8_rocksdb_Options*, PyObject*)’:
rocksdb/_rocksdb.cpp:13811:23: error: ‘struct rocksdb::Options’ has no member named ‘disable_seek_compaction’
__pyx_v_self->opts->disable_seek_compaction = __pyx_t_1;
^
rocksdb/_rocksdb.cpp: In function ‘PyObject* __pyx_pf_7rocksdb_8_rocksdb_7Options_14no_block_cache___get__(__pyx_obj_7rocksdb_8_rocksdb_Options*)’:
rocksdb/_rocksdb.cpp:15090:57: error: ‘struct rocksdb::Options’ has no member named ‘no_block_cache’
__pyx_t_1 = __Pyx_PyBool_FromLong(__pyx_v_self->opts->no_block_cache); if (unlikely(!__pyx_t_1)) {__pyx_filename = __pyx_f[0]; __pyx_lineno = 934; __pyx_clineno = __LINE__; goto __pyx_L1_error;}
^
rocksdb/_rocksdb.cpp:444:36: note: in definition of macro ‘__Pyx_PyBool_FromLong’
#define __Pyx_PyBool_FromLong(b) ((b) ? (Py_INCREF(Py_True), Py_True) : (Py_INCREF(Py_False), Py_False))
^
rocksdb/_rocksdb.cpp: In function ‘int __pyx_pf_7rocksdb_8_rocksdb_7Options_14no_block_cache_2__set__(__pyx_obj_7rocksdb_8_rocksdb_Options*, PyObject*)’:
rocksdb/_rocksdb.cpp:15153:23: error: ‘struct rocksdb::Options’ has no member named ‘no_block_cache’
__pyx_v_self->opts->no_block_cache = __pyx_t_1;
^
rocksdb/_rocksdb.cpp: In function ‘PyObject* __pyx_pf_7rocksdb_8_rocksdb_7Options_20block_size_deviation___get__(__pyx_obj_7rocksdb_8_rocksdb_Options*)’:
rocksdb/_rocksdb.cpp:16920:56: error: ‘struct rocksdb::Options’ has no member named ‘block_size_deviation’
__pyx_t_1 = __Pyx_PyInt_From_int(__pyx_v_self->opts->block_size_deviation); if (unlikely(!__pyx_t_1)) {__pyx_filename = __pyx_f[0]; __pyx_lineno = 1024; __pyx_clineno = __LINE__; goto __pyx_L1_error;}
^
rocksdb/_rocksdb.cpp: In function ‘int __pyx_pf_7rocksdb_8_rocksdb_7Options_20block_size_deviation_2__set__(__pyx_obj_7rocksdb_8_rocksdb_Options*, PyObject*)’:
rocksdb/_rocksdb.cpp:16983:23: error: ‘struct rocksdb::Options’ has no member named ‘block_size_deviation’
__pyx_v_self->opts->block_size_deviation = __pyx_t_1;
^
rocksdb/_rocksdb.cpp: In function ‘int __pyx_pf_7rocksdb_8_rocksdb_7Options_13filter_policy_2__set__(__pyx_obj_7rocksdb_8_rocksdb_Options*, PyObject*)’:
rocksdb/_rocksdb.cpp:19602:23: error: ‘struct rocksdb::Options’ has no member named ‘filter_policy’
__pyx_v_self->opts->filter_policy = ((struct __pyx_vtabstruct_7rocksdb_8_rocksdb_PyFilterPolicy *)__pyx_v_self->py_filter_policy->__pyx_vtab)->get_policy(__pyx_v_self->py_filter_policy);
^
rocksdb/_rocksdb.cpp: In function ‘int __pyx_pf_7rocksdb_8_rocksdb_7Options_11block_cache_2__set__(__pyx_obj_7rocksdb_8_rocksdb_Options*, PyObject*)’:
rocksdb/_rocksdb.cpp:19950:27: error: ‘struct rocksdb::Options’ has no member named ‘block_cache’
__pyx_v_self->opts->block_cache.reset();
^
rocksdb/_rocksdb.cpp:20015:25: error: ‘struct rocksdb::Options’ has no member named ‘block_cache’
__pyx_v_self->opts->block_cache = ((struct __pyx_vtabstruct_7rocksdb_8_rocksdb_PyCache *)__pyx_v_self->py_block_cache->__pyx_vtab)->get_cache(__pyx_v_self->py_block_cache);
^
rocksdb/_rocksdb.cpp: In function ‘int __pyx_pf_7rocksdb_8_rocksdb_7Options_22block_cache_compressed_2__set__(__pyx_obj_7rocksdb_8_rocksdb_Options*, PyObject*)’:
rocksdb/_rocksdb.cpp:20194:27: error: ‘struct rocksdb::Options’ has no member named ‘block_cache_compressed’
__pyx_v_self->opts->block_cache_compressed.reset();
^
rocksdb/_rocksdb.cpp:20267:23: error: ‘struct rocksdb::Options’ has no member named ‘block_cache_compressed’
__pyx_v_self->opts->block_cache_compressed = ((struct __pyx_vtabstruct_7rocksdb_8_rocksdb_PyCache *)((struct __pyx_obj_7rocksdb_8_rocksdb_PyCache *)__pyx_v_value)->__pyx_vtab)->get_cache(((struct __pyx_obj_7rocksdb_8_rocksdb_PyCache *)__pyx_v_value));
^
rocksdb/_rocksdb.cpp: In function ‘rocksdb::ReadOptions __pyx_f_7rocksdb_8_rocksdb_2DB_build_read_opts(__pyx_obj_7rocksdb_8_rocksdb_DB*, PyObject*)’:
rocksdb/_rocksdb.cpp:25316:16: error: ‘struct rocksdb::ReadOptions’ has no member named ‘prefix_seek’
__pyx_v_opts.prefix_seek = __pyx_t_2;
^
rocksdb/_rocksdb.cpp: In function ‘PyObject* __pyx_f_7rocksdb_8_rocksdb_12BaseIterator_set_prefix(__pyx_obj_7rocksdb_8_rocksdb_BaseIterator*, rocksdb::ReadOptions&, __pyx_opt_args_7rocksdb_8_rocksdb_12BaseIterator_set_prefix*)’:
rocksdb/_rocksdb.cpp:26349:16: error: ‘struct rocksdb::ReadOptions’ has no member named ‘prefix’
__pyx_v_opts.prefix = (&__pyx_v_self->c_prefix);
^
error: command 'x86_64-linux-gnu-gcc' failed with exit status 1
----------------------------------------
Cleaning up...
Command /usr/bin/python -c "import setuptools, tokenize;__file__='/tmp/pip-wnupVg-build/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-45kNfa-record/install-record.txt --single-version-externally-managed --compile failed with error code 1 in /tmp/pip-wnupVg-build
Traceback (most recent call last):
File "/usr/bin/pip", line 9, in <module>
load_entry_point('pip==1.5.4', 'console_scripts', 'pip')()
File "/usr/lib/python2.7/dist-packages/pip/__init__.py", line 185, in main
return command.main(cmd_args)
File "/usr/lib/python2.7/dist-packages/pip/basecommand.py", line 161, in main
text = '\n'.join(complete_log)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 42: ordinal not in range(128)compilation error? debian wheezy.
/usr/lib/pymodules/python2.7/Cython/Includes/libcpp/__init__.pxd
/usr/lib/pymodules/python2.7/Cython/Includes/cpython/bytes.pxd
./rocksdb/db.pxd
/usr/lib/pymodules/python2.7/Cython/Includes/cpython/unicode.pxd
./rocksdb/std_memory.pxd
/usr/lib/pymodules/python2.7/Cython/Includes/libc/stdint.pxd
./rocksdb/comparator.pxd
./rocksdb/slice_transform.pxd
./rocksdb/snapshot.pxd
./rocksdb/filter_policy.pxd
/usr/lib/pymodules/python2.7/Cython/Includes/libcpp/vector.pxd
./rocksdb/backup.pxd
./rocksdb/options.pxd
./rocksdb/table_factory.pxd
('missing cimport', u'/usr/lib/pymodules/python2.7/Cython/Includes/libcpp/deque.pxd')
pair
Compiling rocksdb/_rocksdb.pyx because it changed.
Cythonizing rocksdb/_rocksdb.pyx
Error compiling Cython file:
...
from interfaces import SliceTransform as ISliceTransform
import traceback
import errors
cdef extern from "cpp/utils.hpp" namespace "py_rocks":
cdef const Slice* vector_data(vector[Slice]&)
^
rocksdb/_rocksdb.pyx:52:20: Syntax error in C variable declaration
Traceback (most recent call last):
File "setup.py", line 44, in
ext_modules=cythonize([mod1]),
File "/usr/lib/pymodules/python2.7/Cython/Build/Dependencies.py", line 517, in cythonize
cythonize_one(pyx_file, c_file, quiet, options)
File "/usr/lib/pymodules/python2.7/Cython/Build/Dependencies.py", line 540, in cythonize_one
raise CompileError(None, pyx_file)
Cython.Compiler.Errors.CompileError: rocksdb/_rocksdb.pyx
pip install failure
On Ubuntu 12.04, after following the instructions at http://pyrocksdb.readthedocs.org/en/latest/installation.html to build rocksdb and pyrocksdb, the following error is seen at the last step (pip install git+git…). The installation aborts.
(rdbtest)[rithal@rithal-ubuntu64 ~/rdbtest]$ pip install git+git://github.com/stephan-hof/pyrocksdb.git
Downloading/unpacking git+git://github.com/stephan-hof/pyrocksdb.git
Cloning git://github.com/stephan-hof/pyrocksdb.git to /tmp/pip-JuTIW1-build
Running setup.py egg_info for package from git+git://github.com/stephan-hof/pyrocksdb.git
Requirement already satisfied (use --upgrade to upgrade): setuptools in ./lib/python2.7/site-packages/setuptools-0.6c11-py2.7.egg (from pyrocksdb==0.3)
Requirement already satisfied (use --upgrade to upgrade): Cython>=0.20 in ./lib/python2.7/site-packages (from pyrocksdb==0.3)
Installing collected packages: pyrocksdb
Running setup.py install for pyrocksdb
error: can't copy 'rocksdb/_rocksdb.c': doesn't exist or not a regular file
Complete output from command /home/rithal/rdbtest/bin/python -c "import setuptools;__file__='/tmp/pip-JuTIW1-build/setup.py';exec(compile(open(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-2AzIUx-record/install-record.txt --single-version-externally-managed --install-headers /home/rithal/rdbtest/include/site/python2.7:
running install
running build
running build_py
creating build
creating build/lib.linux-x86_64-2.7
creating build/lib.linux-x86_64-2.7/rocksdb
copying rocksdb/interfaces.py -> build/lib.linux-x86_64-2.7/rocksdb
copying rocksdb/errors.py -> build/lib.linux-x86_64-2.7/rocksdb
copying rocksdb/init.py -> build/lib.linux-x86_64-2.7/rocksdb
creating build/lib.linux-x86_64-2.7/rocksdb/tests
copying rocksdb/tests/test_options.py -> build/lib.linux-x86_64-2.7/rocksdb/tests
copying rocksdb/tests/test_db.py -> build/lib.linux-x86_64-2.7/rocksdb/tests
copying rocksdb/tests/init.py -> build/lib.linux-x86_64-2.7/rocksdb/tests
running egg_info
creating pyrocksdb.egg-info
writing requirements to pyrocksdb.egg-info/requires.txt
writing pyrocksdb.egg-info/PKG-INFO
writing top-level names to pyrocksdb.egg-info/top_level.txt
writing dependency_links to pyrocksdb.egg-info/dependency_links.txt
writing manifest file 'pyrocksdb.egg-info/SOURCES.txt'
warning: manifest_maker: standard file '-c' not found
reading manifest file 'pyrocksdb.egg-info/SOURCES.txt'
writing manifest file 'pyrocksdb.egg-info/SOURCES.txt'
error: can't copy 'rocksdb/_rocksdb.c': doesn't exist or not a regular file
Command /home/rithal/rdbtest/bin/python -c "import setuptools;file='/tmp/pip-JuTIW1-build/setup.py';exec(compile(open(file).read().replace('\r\n', '\n'), file, 'exec'))" install --record /tmp/pip-2AzIUx-record/install-record.txt --single-version-externally-managed --install-headers /home/rithal/rdbtest/include/site/python2.7 failed with error code 1 in /tmp/pip-JuTIW1-build
Storing complete log in /home/rithal/.pip/pip.log
method to clear db?
It would be nice to have really fast method to delete all db contents, which can be runned while db is opened, from python. For example for unittesting.
db.clear() would be nice name for that.
i do not like the idea of running rm -rf from shell,
and deleting every key yielded from list of keys is not nice too.
undefined symbol: clock_gettime
I faced with follow error. ;-(
(pyrocks)[email protected]:~$ python
Python 2.7.3 (default, Sep 26 2013, 20:03:06)
[GCC 4.6.3] on linux2
Type "help", "copyright", "credits" or "license" for more information.
import rocksdb
Traceback (most recent call last):
File "", line 1, in
File "/home/user/virtualenv/pyrocks/local/lib/python2.7/site-packages/rocksdb/init.py", line 1, in
from ._rocksdb import *
ImportError: /usr/local/lib/librocksdb.so: undefined symbol: clock_gettime
Error installing in Ubuntu
I can not install pyrockdb.
This is the log. Please, some help.
ubuntu@machine:/pyrocks_test$ sudo -H pip install pyrocksdb
Collecting pyrocksdb
Downloading pyrocksdb-0.4.tar.gz (139kB)
100% |████████████████████████████████| 143kB 3.6MB/s
Requirement already satisfied: setuptools in /usr/local/lib/python2.7/dist-packages (from pyrocksdb)
Building wheels for collected packages: pyrocksdb
Running setup.py bdist_wheel for pyrocksdb ... error
Complete output from command /usr/bin/python -u -c "import setuptools, tokenize;file='/tmp/pip-build-9xH9m7/pyrocksdb/setup.py';f=getattr(tokenize, 'open', open)(file);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, file, 'exec'))" bdist_wheel -d /tmp/tmpKxhl9Rpip-wheel- --python-tag cp27:
missing cimport in module 'pair': /usr/local/lib/python2.7/dist-packages/Cython/Includes/libcpp/deque.pxd
running bdist_wheel
running build
running build_py
creating build
creating build/lib.linux-x86_64-2.7
creating build/lib.linux-x86_64-2.7/rocksdb
copying rocksdb/errors.py -> build/lib.linux-x86_64-2.7/rocksdb
copying rocksdb/init.py -> build/lib.linux-x86_64-2.7/rocksdb
copying rocksdb/interfaces.py -> build/lib.linux-x86_64-2.7/rocksdb
creating build/lib.linux-x86_64-2.7/rocksdb/tests
copying rocksdb/tests/test_options.py -> build/lib.linux-x86_64-2.7/rocksdb/tests
copying rocksdb/tests/init.py -> build/lib.linux-x86_64-2.7/rocksdb/tests
copying rocksdb/tests/test_db.py -> build/lib.linux-x86_64-2.7/rocksdb/tests
running egg_info
writing requirements to pyrocksdb.egg-info/requires.txt
writing pyrocksdb.egg-info/PKG-INFO
writing top-level names to pyrocksdb.egg-info/top_level.txt
writing dependency_links to pyrocksdb.egg-info/dependency_links.txt
reading manifest file 'pyrocksdb.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
writing manifest file 'pyrocksdb.egg-info/SOURCES.txt'
copying rocksdb/_rocksdb.cpp -> build/lib.linux-x86_64-2.7/rocksdb
copying rocksdb/rocksdb.pyx -> build/lib.linux-x86_64-2.7/rocksdb
copying rocksdb/backup.pxd -> build/lib.linux-x86_64-2.7/rocksdb
copying rocksdb/cache.pxd -> build/lib.linux-x86_64-2.7/rocksdb
copying rocksdb/comparator.pxd -> build/lib.linux-x86_64-2.7/rocksdb
copying rocksdb/db.pxd -> build/lib.linux-x86_64-2.7/rocksdb
copying rocksdb/env.pxd -> build/lib.linux-x86_64-2.7/rocksdb
copying rocksdb/filter_policy.pxd -> build/lib.linux-x86_64-2.7/rocksdb
copying rocksdb/iterator.pxd -> build/lib.linux-x86_64-2.7/rocksdb
copying rocksdb/logger.pxd -> build/lib.linux-x86_64-2.7/rocksdb
copying rocksdb/memtablerep.pxd -> build/lib.linux-x86_64-2.7/rocksdb
copying rocksdb/merge_operator.pxd -> build/lib.linux-x86_64-2.7/rocksdb
copying rocksdb/options.pxd -> build/lib.linux-x86_64-2.7/rocksdb
copying rocksdb/slice.pxd -> build/lib.linux-x86_64-2.7/rocksdb
copying rocksdb/slice_transform.pxd -> build/lib.linux-x86_64-2.7/rocksdb
copying rocksdb/snapshot.pxd -> build/lib.linux-x86_64-2.7/rocksdb
copying rocksdb/status.pxd -> build/lib.linux-x86_64-2.7/rocksdb
copying rocksdb/std_memory.pxd -> build/lib.linux-x86_64-2.7/rocksdb
copying rocksdb/table_factory.pxd -> build/lib.linux-x86_64-2.7/rocksdb
copying rocksdb/universal_compaction.pxd -> build/lib.linux-x86_64-2.7/rocksdb
creating build/lib.linux-x86_64-2.7/rocksdb/cpp
copying rocksdb/cpp/comparator_wrapper.hpp -> build/lib.linux-x86_64-2.7/rocksdb/cpp
copying rocksdb/cpp/filter_policy_wrapper.hpp -> build/lib.linux-x86_64-2.7/rocksdb/cpp
copying rocksdb/cpp/memtable_factories.hpp -> build/lib.linux-x86_64-2.7/rocksdb/cpp
copying rocksdb/cpp/merge_operator_wrapper.hpp -> build/lib.linux-x86_64-2.7/rocksdb/cpp
copying rocksdb/cpp/slice_transform_wrapper.hpp -> build/lib.linux-x86_64-2.7/rocksdb/cpp
copying rocksdb/cpp/utils.hpp -> build/lib.linux-x86_64-2.7/rocksdb/cpp
copying rocksdb/cpp/write_batch_iter_helper.hpp -> build/lib.linux-x86_64-2.7/rocksdb/cpp
running build_ext
building 'rocksdb._rocksdb' extension
creating build/temp.linux-x86_64-2.7
creating build/temp.linux-x86_64-2.7/rocksdb
x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fno-strict-aliasing -Wdate-time -D_FORTIFY_SOURCE=2 -g -fstack-protector-strong -Wformat -Werror=format-security -fPIC -I/usr/include/python2.7 -c rocksdb/_rocksdb.cpp -o build/temp.linux-x86_64-2.7/rocksdb/_rocksdb.o -std=c++11 -O3 -Wall -Wextra -Wconversion -fno-strict-aliasing
cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++
rocksdb/_rocksdb.cpp:350:27: fatal error: rocksdb/slice.h: No such file or directory
compilation terminated.
error: command 'x86_64-linux-gnu-gcc' failed with exit status 1
Failed building wheel for pyrocksdb
Running setup.py clean for pyrocksdb
Failed to build pyrocksdb
Installing collected packages: pyrocksdb
Running setup.py install for pyrocksdb ... error
Complete output from command /usr/bin/python -u -c "import setuptools, tokenize;file='/tmp/pip-build-9xH9m7/pyrocksdb/setup.py';f=getattr(tokenize, 'open', open)(file);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, file, 'exec'))" install --record /tmp/pip-lK65nj-record/install-record.txt --single-version-externally-managed --compile:
missing cimport in module 'pair': /usr/local/lib/python2.7/dist-packages/Cython/Includes/libcpp/deque.pxd
running install
running build
running build_py
creating build
creating build/lib.linux-x86_64-2.7
creating build/lib.linux-x86_64-2.7/rocksdb
copying rocksdb/errors.py -> build/lib.linux-x86_64-2.7/rocksdb
copying rocksdb/init.py -> build/lib.linux-x86_64-2.7/rocksdb
copying rocksdb/interfaces.py -> build/lib.linux-x86_64-2.7/rocksdb
creating build/lib.linux-x86_64-2.7/rocksdb/tests
copying rocksdb/tests/test_options.py -> build/lib.linux-x86_64-2.7/rocksdb/tests
copying rocksdb/tests/init.py -> build/lib.linux-x86_64-2.7/rocksdb/tests
copying rocksdb/tests/test_db.py -> build/lib.linux-x86_64-2.7/rocksdb/tests
running egg_info
writing requirements to pyrocksdb.egg-info/requires.txt
writing pyrocksdb.egg-info/PKG-INFO
writing top-level names to pyrocksdb.egg-info/top_level.txt
writing dependency_links to pyrocksdb.egg-info/dependency_links.txt
reading manifest file 'pyrocksdb.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
writing manifest file 'pyrocksdb.egg-info/SOURCES.txt'
copying rocksdb/_rocksdb.cpp -> build/lib.linux-x86_64-2.7/rocksdb
copying rocksdb/rocksdb.pyx -> build/lib.linux-x86_64-2.7/rocksdb
copying rocksdb/backup.pxd -> build/lib.linux-x86_64-2.7/rocksdb
copying rocksdb/cache.pxd -> build/lib.linux-x86_64-2.7/rocksdb
copying rocksdb/comparator.pxd -> build/lib.linux-x86_64-2.7/rocksdb
copying rocksdb/db.pxd -> build/lib.linux-x86_64-2.7/rocksdb
copying rocksdb/env.pxd -> build/lib.linux-x86_64-2.7/rocksdb
copying rocksdb/filter_policy.pxd -> build/lib.linux-x86_64-2.7/rocksdb
copying rocksdb/iterator.pxd -> build/lib.linux-x86_64-2.7/rocksdb
copying rocksdb/logger.pxd -> build/lib.linux-x86_64-2.7/rocksdb
copying rocksdb/memtablerep.pxd -> build/lib.linux-x86_64-2.7/rocksdb
copying rocksdb/merge_operator.pxd -> build/lib.linux-x86_64-2.7/rocksdb
copying rocksdb/options.pxd -> build/lib.linux-x86_64-2.7/rocksdb
copying rocksdb/slice.pxd -> build/lib.linux-x86_64-2.7/rocksdb
copying rocksdb/slice_transform.pxd -> build/lib.linux-x86_64-2.7/rocksdb
copying rocksdb/snapshot.pxd -> build/lib.linux-x86_64-2.7/rocksdb
copying rocksdb/status.pxd -> build/lib.linux-x86_64-2.7/rocksdb
copying rocksdb/std_memory.pxd -> build/lib.linux-x86_64-2.7/rocksdb
copying rocksdb/table_factory.pxd -> build/lib.linux-x86_64-2.7/rocksdb
copying rocksdb/universal_compaction.pxd -> build/lib.linux-x86_64-2.7/rocksdb
creating build/lib.linux-x86_64-2.7/rocksdb/cpp
copying rocksdb/cpp/comparator_wrapper.hpp -> build/lib.linux-x86_64-2.7/rocksdb/cpp
copying rocksdb/cpp/filter_policy_wrapper.hpp -> build/lib.linux-x86_64-2.7/rocksdb/cpp
copying rocksdb/cpp/memtable_factories.hpp -> build/lib.linux-x86_64-2.7/rocksdb/cpp
copying rocksdb/cpp/merge_operator_wrapper.hpp -> build/lib.linux-x86_64-2.7/rocksdb/cpp
copying rocksdb/cpp/slice_transform_wrapper.hpp -> build/lib.linux-x86_64-2.7/rocksdb/cpp
copying rocksdb/cpp/utils.hpp -> build/lib.linux-x86_64-2.7/rocksdb/cpp
copying rocksdb/cpp/write_batch_iter_helper.hpp -> build/lib.linux-x86_64-2.7/rocksdb/cpp
running build_ext
building 'rocksdb._rocksdb' extension
creating build/temp.linux-x86_64-2.7
creating build/temp.linux-x86_64-2.7/rocksdb
x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fno-strict-aliasing -Wdate-time -D_FORTIFY_SOURCE=2 -g -fstack-protector-strong -Wformat -Werror=format-security -fPIC -I/usr/include/python2.7 -c rocksdb/_rocksdb.cpp -o build/temp.linux-x86_64-2.7/rocksdb/_rocksdb.o -std=c++11 -O3 -Wall -Wextra -Wconversion -fno-strict-aliasing
cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++
rocksdb/_rocksdb.cpp:350:27: fatal error: rocksdb/slice.h: No such file or directory
compilation terminated.
error: command 'x86_64-linux-gnu-gcc' failed with exit status 1
----------------------------------------
Command "/usr/bin/python -u -c "import setuptools, tokenize;file='/tmp/pip-build-9xH9m7/pyrocksdb/setup.py';f=getattr(tokenize, 'open', open)(file);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, file, 'exec'))" install --record /tmp/pip-lK65nj-record/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /tmp/pip-build-9xH9m7/pyrocksdb/
do pyrocksdb support for rocksdb 4.x ?
do pyrocksdb support for rocksdb 4.x ?
Reverse seek
Assume the following code:
db.put(b"key1", bytes(1))
db.put(b"key2", bytes(2))
db.put(b"key3", bytes(3))
db.put(b"key4", bytes(4))
it = db.iteritems()
it = reversed(it)
it.seek("key")
print it.__iter__().__next__()
I would expect that this prints (b'key4', b'\x00\x00\x00\x00') but the output is (b'key1', b'\x00').
Is there any way to make seek search backwards (find the last matching entry)?
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.