trueblocks/trueblocks-core

The main repository for the TrueBlocks system

trueblocks-core's Issues

Separate monitored accounts into individual binary files and merge them on the fly

These data files must be hidden from the user. Also--if the user changes the list of accounts that goes into the hash cache, then the hash cache must either be re-written entirely (if there are new accounts) or cleaned up, if there are accounts that have been removed.

From https://github.com/Great-Hill-Corporation/ethslurp/issues/143

Chain reorgs

I’m not sure where to ask this. I’m running a node, but I’m not mining. Periodically, I retreive block and transaction data using RPC. It then save that data locally. I hear all this talk about re-orgs, so I would expect that once in a while, some of the data I pull would be “re-orged"

I should save data for X blocks. And then spin and re grab data fresh to check for re-orgs. This will be a big deal if the blocks are being re-orged, but I am not reflecting that.

From: https://github.com/Great-Hill-Corporation/ethslurp/issues/146

grabABI: should accept multiple addresses

and merge the resulting ABI together to create a single parseLib

utillib: JSON parser is slow

Search for // TODO: THIS PER DISPLAY LOOKUP IS SLOW

From https://github.com/Great-Hill-Corporation/ethslurp/issues/125

account cache for parsed data

The local cache of an account gets duplicate events if the program is stopped and started (and/or) the 'startDate' is forcing a duplicate record.

From https://github.com/Great-Hill-Corporation/ethslurp/issues/139

miniBlock database, if restarted, gets messed up with wrong indexes.

utillib: Remove all SFString code and depend only on the C++ string class plus utilities for ReverseReplace, etc.

Should handle default function and constructor better

I don't really handle the constructor or the default function correctly in the monitor code. Look for this code:

        if (!func.name.empty())
            functions[functions.getCount()] = func;

If you remove this, it will show a new line in the grabABI tests, but they are not correct.

getBlock --latest should pull from etherscan as well as the local node

Or better yet, --latest should be an option on getBlock. Further, it should report how many blocks it's behind. It should also report latest block from the hard drive cache and the miniBlock cache

From https://github.com/Great-Hill-Corporation/ethslurp/issues/148

transaction class in non etherscan mode does not have blockNum or cum

Or any other duplicated stuff. It also does not have pointers to pBlokc, nor does receipt have pointers to pTrans

This shows up as empty timestamp, empty cumultageGas, etc in toJSON 50004

See getBlock tests

From https://github.com/Great-Hill-Corporation/ethslurp/issues/122

utillib: --h (as opposed to -h) does not fail as it should

makeClass CToml1 file should be a CToml

This is the old 'pairFile' problem

lastBlock is incorrect

If lastBlock is first picked up from config.toml file do not add one. If lastBlock comes from the lastBlock file or the cacheDisplay function do add a one. Simple, but annoying. Maybe better to always just write lastBlock - 1 to the file or return it from the function

ethNames: should allow me to do this: ethslurp `ethname whatever -a`

-a should mean return address only

abis / sources

I should, given an address, pull its abi file, its source code (if found) and convert them both to something more easily readable as soon as I touch them as opposed to current which sort of just does this on an as-needed basis.

From https://github.com/Great-Hill-Corporation/ethslurp/issues/142

Transactions may fail without sending "out of gas". Internal calls does not use gas in parent (maybe)

An internal call in this transaction (0xd20a95a9f051d6961ca766a9b2d0f18b6afdf6d8465931a3f70f2c4b1c023688) ran out of gas, but this transaction did not run out of gas, so I don't identify them as inError. The gas!=gasUsed, therefore, I don't trace it.

Starting fullBlock scan from non-zero should use binary search

Look for this comment: // TODO: This should be a binary search not a scan. This is why it appears to wait

From: https://github.com/Great-Hill-Corporation/ethslurp/issues/136

ethslurp: folder is the wrong name

It should be $(HOME)/.quickBlocks

utillib: All assignments of "" to a string should be changed to EMPTY (but EMPTY should be renamed)

The EMPTY #define is just ugly. I was going to remove it, but I don't think I should as if I do and I use "" instead of EMPTY there will be that many more empty strings laying around. Probably better to have a single system-wide empty string.

Maybe change the name to emptyStr or something

From https://github.com/Great-Hill-Corporation/ethslurp/issues/131

grabABI: Autogenerated code with the same function name will overwrite other class definition files

In monitors especially if two smart contracts have similar function names.

https://github.com/Great-Hill-Corporation/ethslurp/issues/124

blockScrape and miniBlocks can both start twice. Bad idea. If either is running, neither should run

utillib: JSON parser is not recursive

The code for fromJSON is no recursive. It works for Ethereum because none of the classes are recursive, but it doesn't work in general.

From https://github.com/Great-Hill-Corporation/ethslurp/issues/126

whenBlock - show nBlocks / second in most recent X blocks

The reason whenBlocks exsits is so people can see how close they are to being synced. It would be interesting to show the speed of the most recent X blocks synced per second. When the hack blocks are occuring, it's super slow. When the regular sync is happening, it's much faster.

From https://github.com/Great-Hill-Corporation/ethslurp/issues/149

Bloom filters should report better

1. All bloom filters are 256 bytes long, big num do not report leading zeros
2. Start with '0x'

from https://github.com/Great-Hill-Corporation/ethslurp/issues/132

getBalance should run from --file to pull various accounts

Plus, it should default to latest block if none is presented

getBalance should allow for -e to report in ether and -d to report in dollars

ethslurp: In transactions.cpp parseParams needs to be removed

From https://github.com/Great-Hill-Corporation/ethslurp/issues/130

This code was a very early way to parse functions but it is totally wrong. It has all been subsumed by the monitors, but does have the benefit of not requiring a separate piece of software. In other words remove it, but don't lose the ability to parse a totally arbitrary input data the way ethslurp does. But also don't require a full-fledged monitor program.

In other words--input parsing should happen fully through the ABI if that is the only thing present, but through the parse library if that is what is needed for speed or flexibility. Note that monitors use this code to fill in the [{FUNCTION}] tag.

Some code (sha3 and getLatestBlock) depends on running node, but doesn't fail gracefully

There are many others probably, but do I want to check on every request?

From https://github.com/Great-Hill-Corporation/ethslurp/issues/129

Serialize in derived classes does not check parent's return value

For example, in CFund, which is derived from CTransaciton, the Serialize function ignores the return value (it's a void return anyway). If the parent does not serialize (because of writeDeleted == false, for example) then we do serialize the child. This may already be fixed.

From https://github.com/Great-Hill-Corporation/ethslurp/issues/145 in old ethslurp repo

grabABI: Locally stored ABI is invalid JSON

When I write the ABI file to ~/.ethslurp it isn't stored correctly. I think it's simply a matter of surrounding with '[' and ']', but make sure it's correct.

whenBlock does not work with block number zero (0)

I use '0' to determine if the input is okay, but that makes it not work with block '0'.

It would be interesting to have block '0' because we want to know what happened then.

makeClass should use a toml file for configuration.

miniBlock database does not lock the file

If the freshener is running and I try to run speedTest, speedTest hangs because it cannot find the end of file. Freshener should fail if the miniBlock database is in memory, which it is in teh case of loading it.

miniBlock read is not concurrency protected

grabABI: class files should be stored in a folder called classDefinitions

Plus the path in the config file should not be there. If the program is run in a folder, it should look for classDefinitions and if it can't find it, don't run, and then use the pwd for the folder in the config file

This is for makeClass

From https://github.com/Great-Hill-Corporation/ethslurp/issues/135

BLOCK_CACHE should not be hard coded

logBloom should show 0x0 if empty

not '0' which is what it does now for getBlock 50004. Plus it's both int he block and the receipt.t

From https://github.com/Great-Hill-Corporation/ethslurp/issues/123

Document minimum required setup for the node

At the very least, you cannot use warp, and you must have tracing = on. For 100% accuracy, you must also have pruning = archive.

Upper case / lower case in ABI files names

In ~/.ethslurp/abis there are .abi files and .json files. Some of the .json are uppercase/lowercase, all of the .abis are lowercase only filenames. Weird. Should fix it.

From https://github.com/Great-Hill-Corporation/ethslurp/issues/141

Contract creation is ignored in monitors

Contract creation is not noticed or recorded yet, but it should be pretty easy.

I simply have to look for contract address in the transaction's receipt in ofInterest function

From: https://github.com/Great-Hill-Corporation/ethslurp/issues/144

blockScrape --deep --check

Currently deep check only checks that each block has the correct list of transactions hashes. It should actually decend into the transactions and then into the receipts and then into the logs to check all data fields from QuickBlocks to the node.

From https://github.com/Great-Hill-Corporation/ethslurp/issues/140

makeClass config files

Storing the configuration files in a shared folder (~/.makeClass) doesn't work because classes conflict with each other with different implementations. Store the config files in the folders where the code arrives. Two reasons. Won't have to store the path in the file, and two generating source will be next to the code it generates. Right now, generating source is invisible.

makeClass, should be intereactive

makeClass, when it encounters a new class, should ask for a list of fields and a directory and create the file itself

Generated switch statements from makeClass

If a smart contract uses leading underbars (_), then all items in the switch statement check that first character. This is slower than it needs to be.

Accumultating caches is too slow

I need a much faster, and yet still decentrlaized way to accumulate per-contract caches

utillib: toJSON doesn't work at all for non-transaction blocks

Don't really know what this means.

From https://github.com/Great-Hill-Corporation/ethslurp/issues/121

On first time parsing an ABI

Store the parsed variable in the input data field. It's easy to generate the input data from the data itself and an ABI, it's hard to generate the parsed data because for every transaction, we have to search through all signatures. Also-- we should add counter to function and sort the function in the ABI by most frequent first thereby lessening search time.

From https://github.com/Great-Hill-Corporation/ethslurp/issues/127

Why is gasLimit part of the block data structure, but it's not written to binary?

From https://github.com/Great-Hill-Corporation/ethslurp/issues/133

ethslurp: Non-field display field tokens

Like [{NOW}], [{DATE:MONTH}] [{ADDR}] [{HEADER}][{RECORDS}] etc.

Also--these should be able to accept arbitrary internal text such as [Generated on {NOW}.] Perhaps this requires a new class called something like 'Generic'

Now that Slurp is a full fledged class it can take many additional fields for summary:

name
nTransactions
toValue
avgValue
largestValue
smallestValue
totalGasUsed
averageGasPrice
etc.

From https://github.com/Great-Hill-Corporation/ethslurp/issues/23

miniBlocks is not concurrency protected

If you run monitors watch.sh freshen x and then run testall, it will mess up the miniblocks mostly because ti takes so long to read miniblocks, and if it gets rewritten in between, it doesn't know where the end of the file is. Should protect against allowing write if miniblock is in the process of reading.