mathieulavoie/bitcluster

bitcluster's Issues

Fail to add multisig addresses

I noticed that there are only 116 Multisig addresses in the updated database, this can be confirmed by querying db.getCollection('addresses').find({_id : {$regex : "^3.*"}}); in the mongoDB.

Looking deeper I noticed that address_utils.py throws an error when seeing multisig addresses:

Pubkey not in recognized format
Transaction 6cbc0f757619a5535e1fd967487d33f75ed12153a06db8c661473a2813b17d26 Unable To Parse SigScript b'00483045022100d69...

I'm working on a fix for this and will send a pull request when it's ready and tested.

How to update from a dowloaded DB

Hi,

I am not too sure about the procedure to update the database after it has been build from the one available through torrent download.

What I tried to do :

• Get last block from mongo using this command
  db.transactions.find({}).sort({_id:-1}).limit(1);
  { "_id" : ObjectId("57c834d8e65b9d53a69b7dc1"), "amount" : 0.33435414, "amount_usd" : 225.30219676002, "destination_n_id" : 18722861, "block_id" : 421422, "trx_date" : "2016-07-19", "destination" : "12BQGj7T7aWdDBpKNbhKDyTm46ZdWc4kBC", "source_n_id" : 58723676, "source" : "1PzMToMuk2h5DnNKNekMBL59af1a1oWghW" }
• Build DB
  $ python build_cluster.py 421422
  (Takes forever to sync when the script says waiting but mongo shows activities in logs)
• Map DB
  $ python map_money.py 421422 427822 (Where the build "stoped")

*This step is failing with an http error BadStatusLine

Can you please tell how I should proceed to update ?

Is there a way to update using the DB available through torrent ? Is it up to date ?

Update database torrent

Last version is 443000, almost behind of 50000 blocks . Hosted instance (dev.bit-cluster.com) is up to date thought.

The torrent of the database does not have any peers with over 95%

As none of the Database peers have over 95% of the database files, the download can never complete. Please join the torrent with the full file set, so that it may be resumed. Thank you

Bug when looking up addresses

User has reported that some addresses including 1Bz6tMM3N9tzMsXFX.... could not be found using the tool.

Caught an error

Hello! I have a problem with
"Block 274900 crawled
Caught an error from Bitcoind RCP, Reconnecting and retrying...(1/10)
Block 275000 crawled" or

Caught an error from Bitcoind RCP, Reconnecting and retrying...(1/10)
Block 270300 crawled
Caught an error from Bitcoind RCP, Reconnecting and retrying...(1/10)
Caught an error from Bitcoind RCP, Reconnecting and retrying...(2/10)
Traceback (most recent call last):
File "build_cluster.py", line 54, in
start()
File "build_cluster.py", line 17, in start
while builder.crawl_block(block_id):
File "C:\Bitcluster\crawler\base_crawler.py", line 34, in crawl_block
block_hash = self.proxy.getblockhash(block_id)
File "C:\Users\alexm\AppData\Local\Programs\Python\Python35-32\lib\site-packages\bitcoin\rpc.py", line 381, in getblockhash
return lx(self._call('getblockhash', height))
File "C:\Users\alexm\AppData\Local\Programs\Python\Python35-32\lib\site-packages\bitcoin\rpc.py", line 154, in _call
response = self._get_response()
File "C:\Users\alexm\AppData\Local\Programs\Python\Python35-32\lib\site-packages\bitcoin\rpc.py", line 175, in _get_response
http_response = self.__conn.getresponse()
File "C:\Users\alexm\AppData\Local\Programs\Python\Python35-32\lib\http\client.py", line 1197, in getresponse
response.begin()
File "C:\Users\alexm\AppData\Local\Programs\Python\Python35-32\lib\http\client.py", line 297, in begin
version, status, reason = self._read_status()
File "C:\Users\alexm\AppData\Local\Programs\Python\Python35-32\lib\http\client.py", line 258, in _read_status
line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
File "C:\Users\alexm\AppData\Local\Programs\Python\Python35-32\lib\socket.py", line 575, in readinto
return self._sock.recv_into(b)

Creating the database from scratch
I have core i7 and 8 GB RAM

Sincerely,

Make an installation procedure.

Do an installation procedure to get Bitcluster run on your own machine.

Timeouts when running the Build script

Block 317777 crawled
Block 317778 crawled
Traceback (most recent call last):
File "build_cluster.py", line 52, in
start()
File "build_cluster.py", line 17, in start
while builder.crawl_block(block_id):
File "/opt/Bitcluster/crawler/base_crawler.py", line 27, in crawl_block
block = self.proxy.getblock(hash)
File "/home/apollo/.local/share/virtualenvs/Bitcluster/lib/python3.4/site-packages/bitcoin/rpc.py", line 382, in getblock
r = self._call('getblock', block_hash, False)
File "/home/apollo/.local/share/virtualenvs/Bitcluster/lib/python3.4/site-packages/bitcoin/rpc.py", line 186, in _call
response = self._get_response()
File "/home/apollo/.local/share/virtualenvs/Bitcluster/lib/python3.4/site-packages/bitcoin/rpc.py", line 207, in _get_response
http_response = self.__conn.getresponse()
File "/usr/lib/python3.4/http/client.py", line 1172, in getresponse
response.begin()
File "/usr/lib/python3.4/http/client.py", line 351, in begin
version, status, reason = self._read_status()
File "/usr/lib/python3.4/http/client.py", line 313, in _read_status
line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
File "/usr/lib/python3.4/socket.py", line 371, in readinto
return self._sock.recv_into(b)
socket.timeout: timed out

This is probably due to some kind of network connectivity issue. Is there a known way to improve reliability ? Like an auto retry or a longer wait time.

Can we as well restart the Build script up to the last block crawled or it needs to start from 1 every time ? Will that affect the mapping afterwards ?

Also can we get explanations on whats in a block exactly ? I suppose :

• Addresses
• Time stamps
• Wallet ID ?
  ? what else

Visual graph of bitcoin transfers

BitCluster should provide the users with a visual representation of bitcoin transfers. The user could select a node and see the TOP 10 incoming and outgoing nodes who have sent/received bitcoins in relation to the node. The amount of the transfers should be printed in the graph and the % that the TOP 10 represents should be indicated.

The users could navigate the graph by clicking on nodes. Only nodes that are relevant to the new node (maximum of 2 transfers away) should be kept on the graph.

There should be an option to toggle between nodeID and labels in the graph.

Labeling of nodes

The database should have one more collection that stores the nodeID and the label (name) of nodes. A scraper could be built to scrape the labels of wallet addresses off of reputable sources.

Caught an unhandled exception.

Any insights on the reason of this error? it happened while rebuilding the cluster database.

http.client.CannotSendRequest: Request-sent
Reconnecting and retrying...(1/10)
Caught an unhandled exception. See stacktrace:
ERROR:root:Request-sent
Traceback (most recent call last):
  File "/root/bitcluster/crawler/base_crawler.py", line 37, in crawl_block
    block_hash = self.proxy.getblockhash(block_id)
  File "/usr/local/lib/python3.5/dist-packages/bitcoin/rpc.py", line 429, in getblockhash
    return lx(self._call('getblockhash', height))
  File "/usr/local/lib/python3.5/dist-packages/bitcoin/rpc.py", line 199, in _call
    'Content-type': 'application/json'})
  File "/usr/lib/python3.5/http/client.py", line 1106, in request
    self._send_request(method, url, body, headers)
  File "/usr/lib/python3.5/http/client.py", line 1141, in _send_request
    self.putrequest(method, url, **skips)
  File "/usr/lib/python3.5/http/client.py", line 974, in putrequest
    raise CannotSendRequest(self.__state)
http.client.CannotSendRequest: Request-sent
Reconnecting and retrying...(1/10)

Caching and optimization

BitCluster should integrate caching and become more efficient. To do so, it would be useful to make it multi-thread.

Shadow addresses

The value of shadow addresses should be investigated.

Database download impossible

Hi,

Is this project still active?
The page http://dev.bit-cluster.com/magnet for downloading the database is not working (for me). Error message: ERR_CONNECTION_REFUSED comes up.

Does anybody knows what to do?

Thanks a lot!

No seed for the DB torrents

Hi,

We have been trying to download an updated DB from torrents but there no seeds. Is there a time we can schedule so we can download it ?

Sincerely,

Database download impossible

Hi,

Is this project still active?
The page http://dev.bit-cluster.com/magnet for downloading the database is not working (for me). Error message: ERR_CONNECTION_REFUSED comes up.

Does anybody knows what to do?

Thanks a lot!

Conversion from BTC to USD

A new collection should be created with the conversion rate between BTC and USD. This collection should auto-update as new blocks are integrated into BitCluster.

Auto-update

BitCluster should be able to auto-update itself.