splunk/splunk-connect-for-ethereum

Splunk Connect for Ethereum

splunk-connect-for-ethereum's Issues

Enable calculation of block propagation latency

In order to calculate the latency from when a block was mined/validated and when it was received by ethlogger, we need to have two timestamps when the block was mined and when the block was received by the node or by the ethlogger.

Geth-specific metrics not collected

When connected to a geth or quorum node ethlogger continually reports the following error:

2020-06-10T21:34:23.624Z ethlogger:nodestats:error Failed to collect node metrics (35 consecutive failures): JsonRpcError: the method debug_metrics does not exist/is not available
    at Object.checkError (/ethlogger/lib/eth/jsonrpc.js:31:15)
    at Object.callback (/ethlogger/lib/eth/client.js:114:35)
    at executeBatchRequest (/ethlogger/lib/eth/client.js:24:23)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (internal/process/task_queues.js:97:5)
    at async Timeout.BatchedEthereumClient.processQueue (/ethlogger/lib/eth/client.js:97:20) {
  code: -32601,
  data: undefined
}

geth docs: https://geth.ethereum.org/docs/rpc/ns-debug

Add quorum example network

Is there any plan to upgrade splunk-connect-for-ethereum to Node.js v18.x or later?

Node.js v16.x has reached EOL.
https://github.com/nodejs/Release#readme

Metrics sent to splunk < 8 not working

--print-config shows multipleMetricFormatEnabled: false,

trace output

  ethlogger:eth:http:trace Sending JSON RPC request over HTTP: [ { jsonrpc: '2.0', id: 20, method: 'eth_blockNumber', params: [] } ] +4ms
  ethlogger:hec:debug Flushing HEC queue for time limit being reached +80ms
  ethlogger:hec:debug Flushing HEC queue with 1 messages +1ms
  ethlogger:hec:debug Flushing HEC queue for time limit being reached +5ms
  ethlogger:hec:debug Flushing HEC queue with 4 messages +1ms
  ethlogger:eth:http:trace Received JSON RPC response:
  ethlogger:eth:http:trace [ { jsonrpc: '2.0', result: '0x7df322', id: 20 } ] +9ms
  ethlogger:eth:http:debug Completed JSON RPC request in 9 ms +9ms
  ethlogger:blockwatcher:debug Received latest block number: 8254242 +74ms
  ethlogger:blockwatcher:debug Found 0 block ranges to process +0ms
  ethlogger:hec:trace Compressed batch of HEC messages from 828 bytes -> 453 bytes +0ms
  ethlogger:hec:trace Compressed batch of HEC messages from 1630 bytes -> 361 bytes +3ms
  ethlogger:hec:debug Successfully flushed 1 HEC messages in 1 attempts and 15 ms +9ms
  ethlogger:hec:debug Response from HEC: {"text":"Incorrect index","code":7,"invalid-event-number":1} +2ms
  ethlogger:hec:debug Failed to send batch to HEC (attempt 1) Error: HEC responded with status 400
    at HecClient.sendToHec (/ethlogger/lib/hec.js:361:27)
    at processTicksAndRejections (internal/process/task_queues.js:94:5) +0ms
  ethlogger:hec:error Failed to send batch to HEC (attempt 1): Error: HEC responded with status 400 +0ms

adding --no-collect-node-metrics to the docker run successfully sends events.

Splunk Enterprise
Version: 7.2.5.1
Build: 962d9a8e1586

Ethlogger resulting nulls

I have created a virtual image on Azure, and setup docker for splunk-connect-for-ethereum.

When I tried to see the network stats, I could not see any. I used search to review the transactions data, observed it is null.

Have you been in this situation, what I could do to get the data properly.

ABI signatures parser panics with certain nested tuple types

The following signature is an example that causes the ABI signature parser to panic:

((address,((uint256,address),(uint8,bytes32,bytes32)),(uint256,uint256,address,(uint8,bytes32,bytes32)),(uint256,uint256,address,(uint8,bytes32,bytes32)),(address,address,uint256,address,uint256,address,address,uint256,address,uint256,address,uint256,address,uint256,uint256,address,bytes32,uint256,uint256,(uint8,bytes32,bytes32),(uint8,bytes32,bytes32),(uint8,bytes32,bytes32))))

ERROR: for ethlogger Cannot create container for service ethlogger

Hi I got this error on windows 10 docker,

I am following this example: https://github.com/splunk/splunk-connect-for-ethereum/tree/master/examples/docker-compose-basic
with mainnet and infura_project_id

But unable to start ethlogger with below error.

docker-compose up -d
splunk is up-to-date
Creating ethlogger ... error

ERROR: for ethlogger Cannot create container for service ethlogger: mkdir C:\path_to_dir\splunk-connect-for-ethereum\examples\docker-compose-basic\abis: The system cannot find the file specified.

ERROR: Encountered errors while bringing up the project.

Any help is greatly appreciated

Capture peer information in Besu

Besu supports a number of RPC methods which are marked as geth only right now, notably admin_peers.

It would be interesting to add support for Besu to capture this type of information.

Unable to open splunk localhost:8000 on MacBook Pro M1 safari browser

I have installed Docker, and followed instructions from https://github.com/splunk/splunk-connect-for-ethereum/tree/master/examples/docker-compose-basic

When I tried to open http://localhost:8000, Safari(Mac) is not connecting.

Any help is greatly appreciated.

Proxy Contract Support

Proxy contracts are a common pattern used for different purposes, such as making smart contracts upgradable or reducing the amount of bytecode to deploy several copies of similar contracts. Instead of calling a contract directly, a proxy contract is called, which then in turn invokes the target contract via DELEGATECALL. This indirection renders contract fingerprinting useless.

Goal of this feature is to automatically detect common types of proxies, annotate Splunk events with some addtional information about the proxy (type, target) and to ABI decode based on the target contract.

Proxy standards to investigate/support:

• EIP-1967 Standard Proxy Storage Slots
• EIP-1167 Minimal Proxy Contract
• EIP-1822 Universal Upgradeable Proxy Standard

Program may exit on startup if Splunk is not started

Reproduction steps:

• Start the program, pointing at a Splunk endpoint that is not currently available.

Expected:

• The program is able to start and is marked unhealthy. Eventually it connects to Splunk when it becomes available.

Actual:

• The program crashes with the following:

2020-05-06T17:46:53.682Z ethlogger:introspect:info Introspecting target ethereum node at jsonprc+https://xxxxxxxxxxxxxxxxx
2020-05-06T17:46:53.902Z ethlogger:introspect:info Retrieved ethereum node version: Geth/v1.8.18-stable(quorum-v2.4.0)/linux-amd64/go1.10.8
2020-05-06T17:46:54.058Z ethlogger:cli:info Detected node platform='quorum:istanbul' protocol=64 chainId=594 networkId=594 chain=buildcoin network=null
2020-05-06T17:46:54.178Z ethlogger:hec:warn HEC service not (yet) available: HEC service https://splunk:8088/services/collector/event/1.0 not available: FetchError: request to https://splunk:8088/services/collector/health failed, reason: connect ECONNREFUSED 172.18.0.2:8088
2020-05-06T17:46:54.178Z ethlogger:hec:info Waiting for HEC service https://splunk:8088/services/collector/event/1.0 to become available (timeout 60000 ms)
2020-05-06T17:47:55.498Z ethlogger:cli:error FATAL: RetryTimeoutError: HEC service https://splunk:8088/services/collector/event/1.0 not available: FetchError: request to https://splunk:8088/services/collector/health failed, reason: connect ECONNREFUSED 172.18.0.2:8088
    at Object.retry (/ethlogger/lib/utils/retry.js:53:23)
    at processTicksAndRejections (internal/process/task_queues.js:97:5)
2020-05-06T17:47:55.500Z ethlogger:health:info Heath state monitor stopped
 ›   Error: HEC service https://splunk:8088/services/collector/event/1.0 not 
 ›   available: FetchError: request to 
 ›   https://splunk:8088/services/collector/health failed, reason: connect 
 ›   ECONNREFUSED 172.18.0.2:8088

The eth_protocolVersion API call is gone as it made no sense

I got this error

warn Ethereum node failed to respond to protocol version request JsonRpcError: the method eth_protocolVersion does not exist/is not available

https://blog.ethereum.org/2021/03/03/geth-v1-10-0

This method has been deprecated in geth v1.10

Enable decoding of ABIs without parameter names (anonymous decoding)

There are times when the exact ABI is unavailable for decoding. In those situations we should decode function calls without parameter names if there is a function signature match.

Build instructions for developing missing main build

The instructions for building for developing are missing the yarn build after yarn install.

BigInt overflow for "0x70ce7011fc4d400" - cannot convert to number

2020-07-08T17:49:27.461Z ethlogger:utils:retry:warn Task block range 10237208-10237232 failed: Error: BigInt overflow for "0x70ce7011fc4d400" - cannot convert to number

Compute internal transaction call graph

Ethereum blocks are made up of transactions, which are one of the things ethlogger currently tracks. However, richer information can be extracted from these. A transaction that targets or crates a smart contract can in turn spawn new so-called 'internal' transactions by executing the CALL, STATICCALL and DELEGATECALL opcodes.

Internal transactions are similar to regular ones, and have pretty much the same attributes: from, to, data, value, gas, gasprice (this value cannot be changed), and execution status (success, revert, invalid opcode, out of gas).

This issue is intended to gather ideas and feedback around why and how internal transactions might be tracked by ethlogger.

Why

Any transaction monitoring should include internal transactions: just looking at outer calls provides an incomplete view of on-chain activity. Many contracts and usage patterns revolve exclusively around internal transactions, including smart accounts and contracts intended to be used by other contracts. A prime example is the ERC1820 Registry Contract: at a glance it never had any activity, but looking at the internal transactions makes it clear that it's actually heavily used.

The call graph of all addresses involved in a transaction also provides information that cannot be otherwise obtained. It can be used to detect contract re-entrancy, learn about internal reverted calls, or even flag suspicious transactions that involve a large number of platforms.

As an example, the recent bZx hack took place in an apparently harmless transaction. Emitted events show lots of activity, but are not easy to follow. The internal call graph however shows a large number of calls with large Ether values to a number of DeFi systems, and is a tool much better suited to analyse what happened.

How

Construction of the call graph requires knowledge of the opcodes executed by the EVM in a transaction, along with environment information (e.g. status of the EVM stack). There's two big possible approaches that I know of that can be taken here.

The hardcore route would be taking an EVM implementation (such as ethereumjs-vm), and adding instrumentation on top of it to detect internal calls. Note that there is already official support for doing this, though some encapsulation around it would be beneficial.

Alternatively, it is possible to use traces over JSON-RPC to obtain this data. Notably, the debug_traceTransaction method provides detailed execution information that should be enough to obtain the required result. A challenge here is restricting what data is saved to achieve good performance, both in terms of speed, memory, and network usage. Follow the links for more details and suggestions on this topic.

Note that both approaches require a 'special' node that is either instrumented or provides access to debugging methods: this will likely translate into an infrastructure requirement for people using ethlogger to get this information. The introspection features would likely come into play here.

Broken Links In Documentation

There are some minor changes in the documentation that need updating due to broken links. The PR can be found here:
#72

As per the contributor documentation I have raised an Issue alongside the PR and completed the contributor agreement:
https://www.splunk.com/en_us/form/contributions/thanks.html