splunk/splunk-connect-for-ethereum

Splunk Connect for Ethereum

Splunk Connect for Ethereum

⚠️ Disclaimer : Post September 15, 2022 Merge Support ⚠️

Splunk Connect for Ethereum has not been fully tested with changes resulting from the move from Proof-of-Work (POW) to Proof-of-Stake (POS), more commonly known as the Merge, that occurred on September 15, 2022. There should be no impact to permissioned ethereum-based networks, but POS-based Ethereum mainnet has not been fully tested.

Splunk Connect for Ethereum (aka ethlogger) makes it easy to ingest data about Ethereum ledgers, node information, and node metrics into another system for analysis. Currently, it can log to the Splunk HTTP Event Collector and stdout. It can also deserialize transactions and events if given a contract ABI. Some benefits include:

• Support for any Ethereum 1.0 JSON-RPC compatible blockchain or sidechain, including Quorum 🦄.
• Reliably saves state between restarts so you never have missing or duplicated events 🧘‍.
• Highly performant batching and compression algorithm 🚄.
• Extracts a rich set of node information and metrics in order to gain deep insight into your node 🙉.
• Introspects your node platform (i.e. geth, parity, besu or quorum) in order to ensure maximum data extraction 🎂.
• Contract fingerprinting in order to match ABIs with function signatures (i.e. we can decode the parameter names too!) 🔎.
• Enables awesome dashboards (keepin' it 200) 💯💯. Check out the companion Splunk App for Ethereum for some starter searches and dashboards too! [Download the latest release[(https://github.com/splunk/splunk-app-for-ethereum/releases/latest)] and install it into your own Splunk instance to visulaize the data.

If you want to learn more about Splunk's efforts to make blockchains stable, secure, and scalable raise an issue here or email us at [email protected].

Prerequisites

Prior to running Splunk Connect for Ethereum you will need to ensure your Splunk server has been configured with an HTTP Event Collector (HEC) endpoint and token and that the token specified has the ability to write data to the indexes provided in your configuration settings.

You will need 2 separate indexes; 1 for events and 1 for metrics.

Usage

$ ethlogger [...options]

Details about ethlogger's command-line usage in the CLI docs

Configuration

Find out how to configure ethlogger in the configuration docs.

Docker

Splunk Connect for Ethereum can be run in a docker container. You can pull the image from the GitHub container registry.

Example:

$ docker run -it ghcr.io/splunkdlt/ethlogger:latest \
    --eth-rpc-url=https://rpc.gnosischain.com\
    --start-at-block=latest \
    --hec-url=https://mysplunkserver.com:8088 \
    --hec-token=123-123-123-123 \
    --hec-events-index=main \
    --hec-metrics-index=metrics

There is also an example on how to run ethlogger in docker-compose. This example will spin up a container for Splunk, install the Splunk App for Ethereum, and start a container for Splunk Connect for Ethereum. Ethereum data will start flowing into Splunk within a few minutes starting from the latest block. To get started follow these steps:

1. Download the docker-compose.yaml file.

$ curl -O https://raw.githubusercontent.com/splunk/splunk-connect-for-ethereum/main/examples/docker-compose-basic/docker-compose.yaml

2. Start the docker containers using docker-compose.

$ docker-compose up

3. Wait for all containers to start (typically just a few minutes). You can rely on the output of docker ps to see the state of services. To see the logs use docker logs ethlogger and docker logs splunk.

4. Login to Splunk and check out the Splunk App for Ethereum to explore the data ingested by ethlogger. Login using user admin and password changeme and start exploring! The Splunk App for Ethereum has a number of dashboards to help get you started quickly. Note: if you change the index name please update the ethereum_index macro so the dashboards will populate.

5. Create a new index named metrics that is a metric type of index. This will be used to gather monitoring metrics for ethlogger itself. You can create the index by choosing Settings and then Indexes from the menu in Splunk. Once created, check out the Node Health - SC4Ethereum dashboard under the Node Monitoring menu.

6. Shut it down and clean up for next time.

$ docker-compose down
$ rm checkpoints.json
$ docker volume prune

Troubleshooting

There's a lot of information available via RPC on Ethereum nodes, but they need to be enabled via the command line on startup for ethlogger to connect. See the relevant docs for:

• Geth and Quorum Docs or you can turn it all on: --http.api admin,db,eth,debug,miner,net,shh,txpool,personal,web3
• Quorum Specific Endpoints: --http.api quorum,istanbul,raft (note: only one of istanbul or raft is active depending on the consensus method)
• Geth and Quorum: ensure that the host ethlogger is running on is whitelisted in the --http.vhosts cli setting.
• OpenEthereum (Parity) Endpoints: note that the command line flag is --jsonrpc-apis APIs
• Besu Endpoints or you can turn on: --rpc-http-enabled --rpc-http-api admin,eth,debug,miner,net,txpool,priv,trace,web3

Contributing

Thank you for considering to contribute to Splunk Connect for Ethereum! Please read the contribution guidelines to get started.

splunk-connect-for-ethereum's Issues

Broken Links In Documentation

There are some minor changes in the documentation that need updating due to broken links. The PR can be found here:
#72

As per the contributor documentation I have raised an Issue alongside the PR and completed the contributor agreement:
https://www.splunk.com/en_us/form/contributions/thanks.html

Enable decoding of ABIs without parameter names (anonymous decoding)

There are times when the exact ABI is unavailable for decoding. In those situations we should decode function calls without parameter names if there is a function signature match.

Capture peer information in Besu

Besu supports a number of RPC methods which are marked as geth only right now, notably admin_peers.

It would be interesting to add support for Besu to capture this type of information.

ABI signatures parser panics with certain nested tuple types

The following signature is an example that causes the ABI signature parser to panic:

((address,((uint256,address),(uint8,bytes32,bytes32)),(uint256,uint256,address,(uint8,bytes32,bytes32)),(uint256,uint256,address,(uint8,bytes32,bytes32)),(address,address,uint256,address,uint256,address,address,uint256,address,uint256,address,uint256,address,uint256,uint256,address,bytes32,uint256,uint256,(uint8,bytes32,bytes32),(uint8,bytes32,bytes32),(uint8,bytes32,bytes32))))

Compute internal transaction call graph

Ethereum blocks are made up of transactions, which are one of the things ethlogger currently tracks. However, richer information can be extracted from these. A transaction that targets or crates a smart contract can in turn spawn new so-called 'internal' transactions by executing the CALL, STATICCALL and DELEGATECALL opcodes.

Internal transactions are similar to regular ones, and have pretty much the same attributes: from, to, data, value, gas, gasprice (this value cannot be changed), and execution status (success, revert, invalid opcode, out of gas).

This issue is intended to gather ideas and feedback around why and how internal transactions might be tracked by ethlogger.

Why

Any transaction monitoring should include internal transactions: just looking at outer calls provides an incomplete view of on-chain activity. Many contracts and usage patterns revolve exclusively around internal transactions, including smart accounts and contracts intended to be used by other contracts. A prime example is the ERC1820 Registry Contract: at a glance it never had any activity, but looking at the internal transactions makes it clear that it's actually heavily used.

The call graph of all addresses involved in a transaction also provides information that cannot be otherwise obtained. It can be used to detect contract re-entrancy, learn about internal reverted calls, or even flag suspicious transactions that involve a large number of platforms.

As an example, the recent bZx hack took place in an apparently harmless transaction. Emitted events show lots of activity, but are not easy to follow. The internal call graph however shows a large number of calls with large Ether values to a number of DeFi systems, and is a tool much better suited to analyse what happened.

How

Construction of the call graph requires knowledge of the opcodes executed by the EVM in a transaction, along with environment information (e.g. status of the EVM stack). There's two big possible approaches that I know of that can be taken here.

The hardcore route would be taking an EVM implementation (such as ethereumjs-vm), and adding instrumentation on top of it to detect internal calls. Note that there is already official support for doing this, though some encapsulation around it would be beneficial.

Alternatively, it is possible to use traces over JSON-RPC to obtain this data. Notably, the debug_traceTransaction method provides detailed execution information that should be enough to obtain the required result. A challenge here is restricting what data is saved to achieve good performance, both in terms of speed, memory, and network usage. Follow the links for more details and suggestions on this topic.

Note that both approaches require a 'special' node that is either instrumented or provides access to debugging methods: this will likely translate into an infrastructure requirement for people using ethlogger to get this information. The introspection features would likely come into play here.

Geth-specific metrics not collected

When connected to a geth or quorum node ethlogger continually reports the following error:

2020-06-10T21:34:23.624Z ethlogger:nodestats:error Failed to collect node metrics (35 consecutive failures): JsonRpcError: the method debug_metrics does not exist/is not available
    at Object.checkError (/ethlogger/lib/eth/jsonrpc.js:31:15)
    at Object.callback (/ethlogger/lib/eth/client.js:114:35)
    at executeBatchRequest (/ethlogger/lib/eth/client.js:24:23)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (internal/process/task_queues.js:97:5)
    at async Timeout.BatchedEthereumClient.processQueue (/ethlogger/lib/eth/client.js:97:20) {
  code: -32601,
  data: undefined
}

geth docs: https://geth.ethereum.org/docs/rpc/ns-debug

Enable calculation of block propagation latency

In order to calculate the latency from when a block was mined/validated and when it was received by ethlogger, we need to have two timestamps when the block was mined and when the block was received by the node or by the ethlogger.

BigInt overflow for "0x70ce7011fc4d400" - cannot convert to number

2020-07-08T17:49:27.461Z ethlogger:utils:retry:warn Task block range 10237208-10237232 failed: Error: BigInt overflow for "0x70ce7011fc4d400" - cannot convert to number

Is there any plan to upgrade splunk-connect-for-ethereum to Node.js v18.x or later?

Node.js v16.x has reached EOL.
https://github.com/nodejs/Release#readme

Metrics sent to splunk < 8 not working

--print-config shows multipleMetricFormatEnabled: false,

trace output

  ethlogger:eth:http:trace Sending JSON RPC request over HTTP: [ { jsonrpc: '2.0', id: 20, method: 'eth_blockNumber', params: [] } ] +4ms
  ethlogger:hec:debug Flushing HEC queue for time limit being reached +80ms
  ethlogger:hec:debug Flushing HEC queue with 1 messages +1ms
  ethlogger:hec:debug Flushing HEC queue for time limit being reached +5ms
  ethlogger:hec:debug Flushing HEC queue with 4 messages +1ms
  ethlogger:eth:http:trace Received JSON RPC response:
  ethlogger:eth:http:trace [ { jsonrpc: '2.0', result: '0x7df322', id: 20 } ] +9ms
  ethlogger:eth:http:debug Completed JSON RPC request in 9 ms +9ms
  ethlogger:blockwatcher:debug Received latest block number: 8254242 +74ms
  ethlogger:blockwatcher:debug Found 0 block ranges to process +0ms
  ethlogger:hec:trace Compressed batch of HEC messages from 828 bytes -> 453 bytes +0ms
  ethlogger:hec:trace Compressed batch of HEC messages from 1630 bytes -> 361 bytes +3ms
  ethlogger:hec:debug Successfully flushed 1 HEC messages in 1 attempts and 15 ms +9ms
  ethlogger:hec:debug Response from HEC: {"text":"Incorrect index","code":7,"invalid-event-number":1} +2ms
  ethlogger:hec:debug Failed to send batch to HEC (attempt 1) Error: HEC responded with status 400
    at HecClient.sendToHec (/ethlogger/lib/hec.js:361:27)
    at processTicksAndRejections (internal/process/task_queues.js:94:5) +0ms
  ethlogger:hec:error Failed to send batch to HEC (attempt 1): Error: HEC responded with status 400 +0ms

adding --no-collect-node-metrics to the docker run successfully sends events.

Splunk Enterprise
Version: 7.2.5.1
Build: 962d9a8e1586

Unable to open splunk localhost:8000 on MacBook Pro M1 safari browser

I have installed Docker, and followed instructions from https://github.com/splunk/splunk-connect-for-ethereum/tree/master/examples/docker-compose-basic

When I tried to open http://localhost:8000, Safari(Mac) is not connecting.

Any help is greatly appreciated.

Program may exit on startup if Splunk is not started

Reproduction steps:

• Start the program, pointing at a Splunk endpoint that is not currently available.

Expected:

• The program is able to start and is marked unhealthy. Eventually it connects to Splunk when it becomes available.

Actual:

• The program crashes with the following:

2020-05-06T17:46:53.682Z ethlogger:introspect:info Introspecting target ethereum node at jsonprc+https://xxxxxxxxxxxxxxxxx
2020-05-06T17:46:53.902Z ethlogger:introspect:info Retrieved ethereum node version: Geth/v1.8.18-stable(quorum-v2.4.0)/linux-amd64/go1.10.8
2020-05-06T17:46:54.058Z ethlogger:cli:info Detected node platform='quorum:istanbul' protocol=64 chainId=594 networkId=594 chain=buildcoin network=null
2020-05-06T17:46:54.178Z ethlogger:hec:warn HEC service not (yet) available: HEC service https://splunk:8088/services/collector/event/1.0 not available: FetchError: request to https://splunk:8088/services/collector/health failed, reason: connect ECONNREFUSED 172.18.0.2:8088
2020-05-06T17:46:54.178Z ethlogger:hec:info Waiting for HEC service https://splunk:8088/services/collector/event/1.0 to become available (timeout 60000 ms)
2020-05-06T17:47:55.498Z ethlogger:cli:error FATAL: RetryTimeoutError: HEC service https://splunk:8088/services/collector/event/1.0 not available: FetchError: request to https://splunk:8088/services/collector/health failed, reason: connect ECONNREFUSED 172.18.0.2:8088
    at Object.retry (/ethlogger/lib/utils/retry.js:53:23)
    at processTicksAndRejections (internal/process/task_queues.js:97:5)
2020-05-06T17:47:55.500Z ethlogger:health:info Heath state monitor stopped
 ›   Error: HEC service https://splunk:8088/services/collector/event/1.0 not 
 ›   available: FetchError: request to 
 ›   https://splunk:8088/services/collector/health failed, reason: connect 
 ›   ECONNREFUSED 172.18.0.2:8088

Proxy Contract Support

Proxy contracts are a common pattern used for different purposes, such as making smart contracts upgradable or reducing the amount of bytecode to deploy several copies of similar contracts. Instead of calling a contract directly, a proxy contract is called, which then in turn invokes the target contract via DELEGATECALL. This indirection renders contract fingerprinting useless.

Goal of this feature is to automatically detect common types of proxies, annotate Splunk events with some addtional information about the proxy (type, target) and to ABI decode based on the target contract.

Proxy standards to investigate/support:

• EIP-1967 Standard Proxy Storage Slots
• EIP-1167 Minimal Proxy Contract
• EIP-1822 Universal Upgradeable Proxy Standard

Add quorum example network

The eth_protocolVersion API call is gone as it made no sense

I got this error

warn Ethereum node failed to respond to protocol version request JsonRpcError: the method eth_protocolVersion does not exist/is not available

https://blog.ethereum.org/2021/03/03/geth-v1-10-0

This method has been deprecated in geth v1.10

ERROR: for ethlogger Cannot create container for service ethlogger

Hi I got this error on windows 10 docker,

I am following this example: https://github.com/splunk/splunk-connect-for-ethereum/tree/master/examples/docker-compose-basic
with mainnet and infura_project_id

But unable to start ethlogger with below error.

docker-compose up -d
splunk is up-to-date
Creating ethlogger ... error

ERROR: for ethlogger Cannot create container for service ethlogger: mkdir C:\path_to_dir\splunk-connect-for-ethereum\examples\docker-compose-basic\abis: The system cannot find the file specified.

ERROR: Encountered errors while bringing up the project.

Any help is greatly appreciated

Build instructions for developing missing main build

The instructions for building for developing are missing the yarn build after yarn install.

Ethlogger resulting nulls

I have created a virtual image on Azure, and setup docker for splunk-connect-for-ethereum.

When I tried to see the network stats, I could not see any. I used search to review the transactions data, observed it is null.

Have you been in this situation, what I could do to get the data properly.