Comments (2)
Experiment I ran annotating parser.pyx off the experiment/lmdb
branch:
# parser.pyx
cdef class Parser(object):
cdef bytearray _buffer
cdef int _buffer_pos
cdef object _read_fn
cdef int MAX_BUFSIZE
cdef str CRLF
cdef int CRLF_LEN
def __cinit__(self, read_fn):
self._buffer = bytearray()
self._buffer_pos = 0
self._read_fn = read_fn
self.MAX_BUFSIZE = 1024 * 1024
self.CRLF = '\r\n'
self.CRLF_LEN = 2
cdef bytearray _readline(self):
cdef bytearray result
cdef bytearray buffer
cdef int crlf_position
buffer = self._buffer[self._buffer_pos:]
crlf_position = buffer.find(self.CRLF)
if crlf_position == -1:
raise StopIteration()
result = buffer[:crlf_position]
self._buffer_pos += crlf_position + self.CRLF_LEN
return result
cdef void _read_into_buffer(self):
cdef bytes data
# FIXME: implement a maximum size for the buffer to prevent a crash due to bad clients
data = self._read_fn(self.MAX_BUFSIZE)
self._buffer.extend(data)
cdef bytearray _read(self, int n_bytes):
cdef bytearray result
cdef bytearray buffer
buffer = self._buffer[self._buffer_pos:]
if len(buffer) < n_bytes:
raise StopIteration()
result = buffer[:n_bytes]
# FIXME: ensure self.CRLF is next
self._buffer_pos += n_bytes + self.CRLF_LEN
return result
def get_instructions(self):
cdef bytearray instructions
cdef int array_length
cdef list instruction_set
cdef bytearray line
cdef str instruction
self._read_into_buffer()
while self._buffer:
self._buffer_pos = 0
instructions = self._readline()
# the Redis protocol says that all commands are arrays, however,
# Redis's own tests have commands like PING being sent as a Simple String
if instructions.startswith('+'):
self._buffer = self._buffer[self._buffer_pos:]
yield str(instructions[1:].strip()).split()
elif instructions.startswith('*'):
# array of instructions
array_length = int(instructions[1:]) # skip '*' char
instruction_set = []
for _ in range(array_length):
line = self._readline()
str_len = int(line[1:]) # skip '$' char
instruction = str(self._read(str_len))
instruction_set.append(instruction)
self._buffer = self._buffer[self._buffer_pos:]
yield instruction_set
else:
# inline instructions, saw them in the Redis tests
for line in instructions.split(self.CRLF):
self._buffer = self._buffer[self._buffer_pos:]
yield str(line.strip()).split()
# setup.py
from distutils.core import setup
from Cython.Build import cythonize
setup(
ext_modules = cythonize(["dredis/parser.pyx"])
)
Before Cython:
$ ./redis-benchmark -r 10000 -n 10000 -q -t ping
PING_INLINE: 18903.59 requests per second
PING_BULK: 17035.78 requests per second
After Cython:
$ ./redis-benchmark -r 10000 -n 10000 -q -t ping
PING_INLINE: 20325.20 requests per second
PING_BULK: 19880.71 requests per second
Takeaway: ~7% improvement by annotating only the parser file.
from dredis.
Sorry, I had cProfile enabled and it messed the results. Here's it corrected (without cProfile):
# before cython
$ ./redis-benchmark -r 10000 -n 10000 -q -t ping
PING_INLINE: 26525.20 requests per second
PING_BULK: 23980.82 requests per second
# after cython
$ ./redis-benchmark -r 10000 -n 10000 -q -t ping
PING_INLINE: 27777.78 requests per second
PING_BULK: 27472.53 requests per second
~4-5% improvement
from dredis.
Related Issues (20)
- Add support to output RDB
- Create Dockerfile
- Implement authentication (AUTH)
- Add slowlog support
- Expire/TTL support
- Run `black` as pre-commit hook
- Implement DUMP
- Implement RESTORE
- Add RENAME command
- Negative scores are stored out of order (after positive scores) HOT 2
- Add TLS suggestions to documentation
- Add ZSCAN support
- Add support to HSCAN
- Add CONFIG GET/SET commands
- Add codecov
- Add NX parameter support to the ZADD command
- Redis-py's zscan_iter has unexpected behavior
- Port to Python 3
- Windows Support HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from dredis.