blockchain-etl/iotex-etl

ETL (extract, transform and load) tools for ingesting IoTeX blockchain data to Google BigQuery and Pub/Sub

IoTeX ETL

Overview

IoTeX ETL allows you to setup an ETL pipeline in Google Cloud Platform for ingesting IoTeX blockchain data into BigQuery and Pub/Sub. It comes with CLI tools for exporting IoTeX data into JSON newline-delimited files partitioned by day.

Data is available for you to query right away in Google BigQuery.

Architecture

Google Slides version

1. The nodes are run in a Kubernetes cluster. Refer to IoTeX Node in Kubernetes for deployment instructions.

2. Airflow DAGs export and load IoTeX data to BigQuery daily. Refer to IoTeX ETL Airflow for deployment instructions.

3. IoTeX data is polled periodically from the nodes and pushed to Google Pub/Sub. Refer to IoTeX ETL Streaming for deployment instructions.

4. IoTeX data is pulled from Pub/Sub, transformed and streamed to BigQuery. Refer to IoTeX ETL Dataflow for deployment instructions.

Setting Up

1. Follow the instructions in IoTeX Node in Kubernetes to deploy an IoTeX node in GKE. Wait until it's fully synced. Make note of the Load Balancer IP from the node deployment, it will be used in Airflow and Streamer components below.

2. Follow the instructions in IoTeX ETL Airflow to deploy a Cloud Composer cluster for exporting and loading historical IoTeX data. It may take several hours for the export DAG to catch up. During this time "load" and "verify_streaming" DAGs will fail.

3. Follow the instructions in IoTeX ETL Streaming to deploy the Streamer component. For the value in last_synced_block.txt specify the last block number of the previous day. You can query it in BigQuery: SELECT height FROM crypto_iotex.blocks ORDER BY height DESC LIMIT 1.

4. Follow the instructions in IoTeX ETL Dataflow to deploy the Dataflow component. Monitor "verify_streaming" DAG in Airflow console, once the Dataflow job catches up the latest block, the DAG will succeed.

iotex-etl's Issues

Setup travis script for running Dataflow tests

Tests can be run with mvn test inside dataflow dir.

This should be possible to do in .travis.yml file https://stackoverflow.com/questions/27644586/how-to-set-up-travis-ci-with-multiple-languages

Move iotex-etl cli files to the cli subfolder

• All files and folders except .github, airflow, dataflow, streaming, .gitignore should be moved to cli subfolder.

• https://github.com/elgohr/Publish-Docker-Github-Action#workdir workdir may need to be updated in publish-to-dockerhub.yml.

• publish-to-pypi.yml may need to be updated.

Create Source Repository to hold Dataflow chainConfig.json file

Also add a section in README similar to https://github.com/blockchain-etl/iotex-etl/tree/master/airflow#creating-a-cloud-source-repository-for-airflow-variables.

Replace "iotex-etl-dev" with <your_project> in chainConfigIotexDev.json

---

For automation we need to do it in two steps: https://github.com/blockchain-etl/hedera-etl#running-on-gcp-dataflow.

First cloudbuild will upload the dataflow template to GCS bucket

Second creates a job using that tempate and parameters file

Add verify streaming DAG

Use this as an example https://github.com/blockchain-etl/ethereum-etl-airflow/blob/master/dags/ethereum_verify_streaming_dag.py

Add field descriptions to schema files

Cloud Build config for deploying Airflow variables

https://stackoverflow.com/questions/54237270/import-variables-using-json-file-in-google-cloud-composer

"gcloud composer environments storage data import" makes the json file available in the gcs bucket

Then "gcloud composer environments run ... variables loads the json to Airflow variables"

producer_pubkey should be rename to producer in blocks

Use iotex_utils.pubkey_to_address(header.producerPubkey) to convert to adress.

Change instance type from custom-2-4096 to n1-standard-1 in streaming

https://github.com/blockchain-etl/iotex-etl/tree/master/streaming

Update type of bytes fields in blocks to BYTES

tx_root, receipt_root, delta_state_digest, signature

Add task to verify_streaming DAG that checks that mainnet.errors table is empty

mainnet.errors is where Dataflow job errors are dumped.

Add Configuration section in README for streaming

Need to add a table with available configs in README https://github.com/blockchain-etl/iotex-etl/tree/master/streaming. Similar to this https://github.com/blockchain-etl/iotex-kubernetes#configuration.