davebryson/py-abci

Build Tendermint blockchain applications in Python

Py-ABCI

Build Tendermint blockchain applications in Python. It's fun. This library provides the core functionality needed to create Tendermint ABCI applications.

Supported Tendermint Version

• Tendermint 0.34.24
• ABCI 0.17.0

Installation

Requires Python >= 3.9

pip install abci

You'll need a binary version of the Tendermint engine. Available here: https://github.com/tendermint/tendermint/releases

Make sure the Tendermint version you download matches the current support version of this library

Quick Start - demo

A very simple demo application is included and available from the command line as counter. You can find the code here: https://github.com/davebryson/py-abci/blob/master/src/example/counter.py

To try it out:

1. Make sure you have the Tendermint binary setup locally and in your path. To test it's working open a terminal window and type:

>> tendermint version

It should output your version of Tendermint that should match the currently supported version of this library.

2. Next, initialize Tendermint by running:

>> tendermint init

3. Start the Tendermint node:

>> tendermint node

The node will start, but will be waiting for you application to start.

4. Open another terminal, and start the counter application. The counter will be available from within the Python environment where you installed abci

>> counter

You'll see the application start, and in the Tendermint terminal, you'll see the output of blocks being produced

5. Now, open a 3rd terminal window to send some transaction to the blockchain. To do this we'll use the curl application to send transaction to the local blockchain over http. For example:

>> curl http://localhost:26657/broadcast_tx_commit?tx=0x01
>> curl http://localhost:26657/broadcast_tx_commit?tx=0x02

The counter application expects you to send transactions as numbers encoded as hex in order: 1,2,3... It will reject and out-of-order numbers. You can always see the latest accepted value by sending the request:

>> curl http://localhost:26657/abci_query

To shut down the application enter CTRL-C

Get Started

To start building your own application:

1. Extend the abci.application.BaseApplication class
2. Implement the Tendermint ABCI callbacks - see https://docs.tendermint.com/v0.34/spec/abci for details on how they work
3. Start it:

from abci.server import ABCIServer

app = ABCIServer(app=MyApplication())
app.run()

See the counter.py application in the example directory https://github.com/davebryson/py-abci/blob/master/src/example/counter.py for a full example.

Developing on the code base

If you're working directly on the code base. Install a local editable version:

pip install --editable '.[test]'

Updating Protobuf code

You should only re-generate the protobuf code if you're updating the associated protobuf files, and/or contributing to this code base. You do not need to rebuild protos to create apps.

A note on protobuf: The primary code directory is abci, but you'll notice additional directories: gogoproto, tendermint, and protos.

The gogoproto and tendermint directories are the protobuf generated code used by abci. It adds proper Python modules and preserves all the import statements used by Tendermint for the various protobuf files spread across their codebase. The protos directory is the source .proto files.

To (re)build the protobuf files:

1. Install protoc so it's available in your PATH as a command
2. Run make update-proto

py-abci's Issues

Getting "Rejected bad transaction" when doing a transaction

Hi,
I am new to tendermint and i have took your application as base and started working on it. I am running tendermint node and for abci server i am running counter.py ,when i do a transaction(through curl hits) it is i'm getting something like this
I[10-16|05:18:23.063] Rejected bad transaction module=mempool tx=E63539B2EC0BF863354CAF73130E40FCDE0EBBA5 res="&{CheckTx:code:1 fee:<> }"
I[10-16|05:18:23.063] HTTPRestRPC module=rpc-server method=/broadcast_tx_commit args="[<[]uint8 Value>]" returns="[<*core_types.ResultBroadcastTxCommit Value> ]"
I[10-16|05:18:23.063] Served RPC HTTP response module=rpc-server method=GET url="/broadcast_tx_commit?tx=%22thisisvaldgsdidtranasaction%22" status=200 duration=4 remoteAddr=10.40.20.186:49658
the transaction is not getting committed. what can be the problem here ?

Got an error while sending transaction to example app

I started counter.py from examples folder, than started tendermint and when I tried to send transaction via curl I've go an error in app, can you help me with it?

Log from app console:

INFO ABCIServer started on port: 26658
INFO ... connection from Tendermint: 127.0.0.1:60668 ...
INFO ... connection from Tendermint: 127.0.0.1:60669 ...
INFO ... connection from Tendermint: 127.0.0.1:60670 ...
INFO ... connection from Tendermint: 127.0.0.1:60671 ...
Traceback (most recent call last):
File "src/gevent/greenlet.py", line 716, in gevent._greenlet.Greenlet.run
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/gevent/baseserver.py", line 26, in _handle_and_close_when_done
return handle(*args_tuple)
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/abci/server.py", line 169, in __handle_connection
for message in messages:
File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/abci/encoding.py", line 59, in read_messages
m.ParseFromString(data)
google.protobuf.message.DecodeError: Error parsing message
2018-08-06T16:07:50Z <Greenlet "Greenlet-0" at 0x106824d48: _handle_and_close_when_done(<bound method ABCIServer.__handle_connection of <a, <bound method StreamServer.do_close of <StreamServ, (<gevent._socket3.socket [closed] object, fd=-1, )> failed with DecodeError

Contributions?

Hi! We would like to know whether you are in a position to process contributions? We are considering using py-abci for a project that will need to go through dev, staging, and production phases. For this reason we will most certainly be in a position to make various contributions, ranging from unit tests, continuous integration, packaging, to whatever may need to be improved.

We would rather contribute to an existing community project than starting our own, but at the same time we would need to operate with considerable momentum.

Hence, if you are available for a conversation about possible ways to benefit from contributions we would be most happy to talk with you.

Best regards,
Sylvain

ABCI server crashes in case of large data messages

Description

I am trying to send data message to tendermint which have large payload size (>1024 bytes). This causes the ABCI server to crash when parsing the message as it reads 1024 bytes and attempts to parse the same.

If this repo is open to contribution then I can generate a PR which would generalize to messages of any size?

Update types.proto & types_pb2.py for latest tendermint (0.15)

The go abci types.proto has changed and the current py-abci does not work with the Tendermint 0.15.

Will add more details and steps to reproduce the problem soon.

Also related: bigchaindb/bigchaindb#2005 (comment)

In the meantime, this issue seems related: tendermint/abci#179

Related: code types (enum CodeType) were removed: tendermint/abci@42a8e32#diff-f325a92d451852ea8923aaa3a50731be

What's the status of this project?

I noticed that this project hasn't had any commits since 1/30/18.

Just pulled the most recent version of tendermint and tried to run the counter example from this project and ran into this errors:
E[04-06|20:25:55.657] Stopping abci.socketClient for error: EOF module=abci-client connection=query

Is this no longer supported? What's the latest version of tendermint/abci that it's supported for?

abcisocket is failing to connect

Good Day,
I am trying to run the Counter Example for py-abci. The issue I am facing is that the abcisocket is failing to connect. I have tried different ports, but I am not sure where to start to track down the issue. Initially I was running an old version of tendermint, but even after updating, I still faced the same issue. I would appreciate your guidance and direction

------------------------------------------------------------------
Some relevant stats about the operating environment
Running the Windows Subsystem Linux
Python 3.6.8
iramaj@LAPTOP-241H1D7P:~/.tendermint$ go env GOARCH="amd64" GOBIN="" GOCACHE="/home/iramaj/.cache/go-build" GOEXE="" GOFLAGS="" GOHOSTARCH="amd64" GOHOSTOS="linux" GOOS="linux" GOPATH="/home/iramaj/go" GOPROXY="" GORACE="" GOROOT="/usr/local/go" GOTMPDIR="" GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64" GCCGO="gccgo" CC="gcc" CXX="g++" CGO_ENABLED="1" GOMOD="" CGO_CFLAGS="-g -O2" CGO_CPPFLAGS="" CGO_CXXFLAGS="-g -O2" CGO_FFLAGS="-g -O2" CGO_LDFLAGS="-g -O2" PKG_CONFIG="pkg-config" GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build921347695=/tmp/go-build -gno-record-gcc-switches"

---------------------------- Running the counter app-----------------------------
iramaj@LAPTOP-241H1D7P:~/Tendermint$ python3 counter.py -p 46658 INFO ABCIServer started on port: 46658 ^Z [1]+ Stopped python3 counter.py -p 46658 iramaj@LAPTOP-241H1D7P:~/Tendermint$ python3 counter.py -p 26658 INFO ABCIServer started on port: 26658

------------starting the tendermint node------------------------------------------
iramaj@LAPTOP-241H1D7P:~/.tendermint$ tendermint node E[2019-08-16|17:26:13.189] abci.socketClient failed to connect to tcp://127.0.0.1:26658. Retrying... module=abci-client connection=query err="dial tcp 127.0.0.1:26658: connect: connection refused" E[2019-08-16|17:26:17.194] abci.socketClient failed to connect to tcp://127.0.0.1:26658. Retrying... module=abci-client connection=query err="dial tcp 127.0.0.1:26658: connect: connection refused" E[2019-08-16|17:26:21.198] abci.socketClient failed to connect to tcp://127.0.0.1:26658. Retrying... module=abci-client connection=query err="dial tcp 127.0.0.1:26658: connect: connection refused" E[2019-08-16|17:26:25.203] abci.socketClient failed to connect to tcp://127.0.0.1:26658. Retrying... module=abci-client connection=query err="dial tcp 127.0.0.1:26658: connect: connection refused" ^Z [1]+ Stopped tendermint node iramaj@LAPTOP-241H1D7P:~/.tendermint$ tendermint version 0.32.2-eec6d33e

[packaging] When should the proto files be compiled?

Background

Depending on how #9 is resolved it could be that an additional protobuf file (gogo.proto) is needed. Just like the tendermint/abci proto file, it needs to be compiled into a python file. @davebryson pointed out in #9 (comment) that it could be compiled during the setup phase (via setup.py). Since the tendermint/abci proto file is pre-compiled and the resulting python file (types_pb2.py) is simply included in the package I suggested in PR #14 that the question of when to compile the gogo.proto file be tackled in a joint issue with the case of the tendermint/abci proto file.

Useful Links

• stackoverflow post
• How does google/protobuf/python does it?
• bblfsh/client-python approach, pointed out by @davebryson in #9 (comment)
• protobuf-setuptools

Push new pypi release

Need to push updated pypi release once the recent changes are tested enough.

ABCIServer.run() raises an `AttributeError`

  File "/home/tux/.cache/pypoetry/virtualenvs/tendies-UGdA5WR3-py3.9/lib/python3.9/site-packages/abci/server.py", line 139, in
run
    gevent.signal(signal.SIGQUIT, evt.set)
AttributeError: module 'gevent' has no attribute 'signal'

Any idea what this is? I'm running with Python3.9

tm-bench gives zero Txs/sec against the counter.py example

I tried running tm-bench tool against the running counter.py and tendermint node to measure the TPS (transactions per second). I'm always getting zero, while the provided kvstore example ABCI app is giving me some numbers.

Setup 1: counter.py app

tm-bench -T 10 -r 1000 localhost:26657

Stats          Avg       StdDev     Max     Total     
Txs/sec        0         0          0       0         
Blocks/sec     0.900     0.300      1       9 

Setup 2: kvstore app that ships with tendermint

tm-bench -T 10 -r 1000 localhost:26657

Stats          Avg       StdDev     Max      Total     
Txs/sec        971       335        1219     9710      
Blocks/sec     0.900     0.300      1        9  

Question: Why is Txs/sec always zero for the counter.py app?

SimpleCounter not working

I'm trying to get in touch with Tendermint, using your abci packet. So, the first thing I've tried is to run the counter.py example - and it doesn't work (there are errors from the Tendermint and counter.py). I'm working on the:

$ uname -a
Linux local 4.13.0-26-generic #29~16.04.2-Ubuntu SMP Tue Jan 9 22:00:44 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

Here's the list of commands I'm using to run all the stuff:

1. Install abci packet into the new virtual environment:

$ mkdir Tendermint_python && cd Tendermint_python
$ virtualenv --python python3 --no-site-packages venv
$ source venv/bin/activate && pip install abci

2. Copy the counter.py into the counter.py file
3. Initialize Tendermint

$ cd Tendermint_python
$ tendermint --home ".tendermint" init

4. Run Tendermint node

$ tendermint --home ".tendermint" node

5. Open another terminal and run counter.py

$ cd Tendermint_python && source venv/bin/activate
$ python counter.py

Here's the logs from the Tendermint:

E[01-17|14:27:39.616] Stopping abci.socketClient for error: proto: bad wiretype for field types.ResponseSetOption.Code: got wiretype 2, want 0 module=abci-client connection=query

And counter.py's logs:

INFO      ABCIServer started on port: 46658
Traceback (most recent call last):
  File "/home/pavlovdog/Desktop/Tendermint_Bitcoin/venv/lib/python3.5/site-packages/gevent/greenlet.py", line 536, in run
    result = self._run(*self.args, **self.kwargs)
  File "/home/pavlovdog/Desktop/Tendermint_Bitcoin/venv/lib/python3.5/site-packages/gevent/baseserver.py", line 26, in _handle_and_close_when_done
    return handle(*args_tuple)
  File "/home/pavlovdog/Desktop/Tendermint_Bitcoin/venv/lib/python3.5/site-packages/abci/server.py", line 132, in __handle_connection
    inbound = socket.recv(1024)
  File "/home/pavlovdog/Desktop/Tendermint_Bitcoin/venv/lib/python3.5/site-packages/gevent/_socket3.py", line 332, in recv
    return _socket.socket.recv(self._sock, *args)
ConnectionResetError: [Errno 104] Connection reset by peer
Wed Jan 17 17:28:53 2018 <Greenlet at 0x7f78626732a8: _handle_and_close_when_done(<bound method ABCIServer.__handle_connection of <a, <bound method StreamServer.do_close of <StreamServ, (<gevent._socket3.socket [closed]  object, fd=-1, )> failed with ConnectionResetError

What am I doing wrong? Thanks in advance!

tendermint v0.34

hi, do you have any plan for upgrade py-abci to support tendermint latest version? (v0.34) thanx

Running the counter examples gives bad wiretype for field, 2 instead of 1

Versions:

$ tendermint version
0.15.0
$ abci-cli version
0.9.0

Starting the example

$ python3 countermint.py
INFO      ABCIServer started on port: 46658

Starting tendermint

$ tendermint unsafe_reset_all
$ tendermint init            
$ tendermint node            
E[01-23|10:43:58.703] Stopping abci.socketClient for error: proto: bad wiretype for field types.ResponseSetOption.Code: got wiretype 2, want 0 module=abci-client connection=query


E[01-23|10:40:55.782] Stopping abci.socketClient for error: proto: bad wiretype for field types.ResponseSetOption.Code: got wiretype 2, want 0 module=abci-client connection=query

Tendermint connection managment

Tendermint will make 3 connections to an abci application: mempool, consensus, query.
If an error happens in 1, it will leave the others open - which means you don't have all the available connections.

Can we better manage these connections with Gevent? Will Tendermint attempt to reconnect?

branch for tendermint 0.31.5

Hi,

I'd like to commit a branch and do a PR for tendermint 0.31.5 support (ABCI 16.0).
would love to push it to your repository.

best

Gevent fails on Windows

This line may be an issue on Windows: gevent.signal_handler(signal.SIGQUIT, evt.set). But I don't have windows to test it...

Support CometBFT

Since CometBFT is the successor of Tendermint, do you plan to also support CometBFT in future?

And thanks for creating py-abci ! Helps our project a lot.

release on github that matches PyPI release

Hey @davebryson! Would it be possible to create a release on Github that matches the PyPI release?

This is to make it easier for quick reference when one is wondering what the PyPI release points to in terms of commits.

ModuleNotFoundError: No module named 'protobuf'

I installed protobuf:

$ sudo python3 -m pip install --upgrade protobuf==3.6.1
The directory '/home/identos/.cache/pip/http' or its parent directory is not owned by the current user and the cache has been disabled. Please check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
The directory '/home/identos/.cache/pip' or its parent directory is not owned by the current user and caching wheels has been disabled. check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
Collecting protobuf==3.6.1
Downloading https://files.pythonhosted.org/packages/c2/f9/28787754923612ca9bfdffc588daa05580ed70698add063a5629d1a4209d/protobuf-3.6.1-cp36-cp36m-manylinux1_x86_64.whl (1.1MB)
100% |████████████████████████████████| 1.1MB 804kB/s
Requirement already satisfied, skipping upgrade: setuptools in /usr/local/lib/python3.6/dist-packages (from protobuf==3.6.1) (40.4.3)
Requirement already satisfied, skipping upgrade: six>=1.9 in /usr/local/lib/python3.6/dist-packages (from protobuf==3.6.1) (1.11.0)
Installing collected packages: protobuf
Found existing installation: protobuf 3.0.0a3
Uninstalling protobuf-3.0.0a3:
Successfully uninstalled protobuf-3.0.0a3
Successfully installed protobuf-3.6.1

Here is the output of protoc version:
$ protoc --version
libprotoc 3.6.1

However, when I want to run counter.py I have the following error:
$python3 counter.py
Traceback (most recent call last):
File "counter.py", line 22, in
import abci.utils as util
File "/usr/local/lib/python3.6/dist-packages/abci-0.5.1-py3.6.egg/abci/init.py", line 2, in
File "/usr/local/lib/python3.6/dist-packages/abci-0.5.1-py3.6.egg/abci/server.py", line 21, in
File "/usr/local/lib/python3.6/dist-packages/abci-0.5.1-py3.6.egg/abci/application.py", line 3, in
File "/usr/local/lib/python3.6/dist-packages/abci-0.5.1-py3.6.egg/abci/types_pb2.py", line 16, in
ModuleNotFoundError: No module named 'protobuf'

Any idea how to solve this error?

need to re-export *all* abci types

the application is missing some re-exports for the protobuf types. Specifically: event

Need to update tag and pip

Need to update tag and push to pip based on latest pull request.

Query function with GET parameters

BaseApplication provides def query.
Is there any way you can get the path parameters there?
For example I would like to create more functionality there like /abci_query?type=get_tx&hash=xy

Error parsing message

Hi, in the counter.py example, it is mentioned that transactions to the app are sent using "curl http://localhost:46657/broadcast_tx_commit?tx=0x01", but when I use this command, I am not getting any output. So, I changed port to 26658 ( "curl http://localhost:26658/broadcast_tx_commit?tx=0x01" ) as the ABCI server is running on port 26658. But, now I keep getting the below error on server side

Can anyone please tell me how to figure out ths?

How to send binary data transactions

Since the struct pack returns a binary string as an output
>>> encode_number(1) b'\x00\x00\x00\x01'

and decode also takes this
>>> decode_number(b'\x00\x00\x00\x01') 1

while the transcations given via curl are given in Hex like
curl http://localhost:26657/broadcast_tx_commit?tx=0x01

How do you convert your binary string to hex easily for curl command ?

Update python setup

• Add pyproject.toml
• Include 'wheel' with sdist
• move to src/

Consider replacing wildcard imports with explicit imports

This can help troubleshooting and detecting errors among other things.

PEP 8 discourages wildcard imports:

Wildcard imports (from <module> import *) should be avoided, as they make it unclear which names are present in the namespace, confusing both readers and many automated tools. There is one defensible use case for a wildcard import, which is to republish an internal interface as part of a public API (for example, overwriting a pure Python implementation of an interface with the definitions from an optional accelerator module and exactly which definitions will be overwritten isn't known in advance).

Make the license more visible / apparent

As per my earlier comment on issue #1 (comment) :

The license on the code should be made more visible / apparent. Currently it's a bit hidden inside the setup.py file (license='Apache2'). Much better would be to use the GitHub conventions so it's more obvious; see

https://help.github.com/articles/adding-a-license-to-a-repository/

Latest version of ABCI uses int64 vs bigendian

ABCI messages use varint as delimiters. The latest version of ABCI changed from bigendian to int64. This is causing more messaging issues.

CTRL-C signals fail on windows

Windows doesn't properly handle CRTL-C signals. Adjust server run to deal with it

google.protobuf.message.DecodeError: Truncated message.

Hello, I'm trying to run counter.py example.
when run 'tendermint node' ,there are some errors.


and The terminal display is as follows：