pinax-network/substreams

Substreams built by Pinax

Pinax Substreams

Substreams built by Pinax

Development Status

🔨 = In progress. 🛠 = Feature complete. Additional testing required. ✅ = Production-ready.

Substream	Status	Description
common	✅	Antelope Common modules for all events.
eosmechanics	✅	Antelope Block Producer Benchmarks.
eosio.ibc	🛠	Antelope eosio.ibc Lightproof block details.
accounts	🛠	Antelope accounts
eosio.token	✅	Antelope eosio.token token transfers
eosio.evm	🔨	Antelope eosio.evm contract events
antelope.oracles	🔨	Antelope Oracles prices from delphioracle and oracle.defi
atomicmarket	🔨	Metrics for AtomicHub Market
pomelo	🔨	Pomelo Grants events
pomelo.bounties	🔨	Pomelo Bounties events
ERC721	🔨	ERC-721: Non-Fungible Token Standard
ERC20	🔨	ERC-20: Token Standard

Further resources

• Substreams documentation

substreams's Issues

[Substream Proposal] `atomicassets`

Substream

• filter by contract = atomicassets

Protobuf

message Actions {
  repeated Action actions = 1;
}

message Action {
  uint32 block_num = 1;
  google.protobuf.Timestamp timestamp = 2;
  string transaction_id = 3;
  string account = 4;
  string name = 5;
  string json_data = 6;
}

Modules

• map_actions
• map_transfers
  • transfer
• map_logs
  • logtransfer
  • lognewoffer
  • lognewtempl
  • logmint
  • logsetdata
  • logbackasset
  • logburnasset

Headers

https://github.com/pinknetworkx/atomicassets-contract/blob/master/include/atomicassets.hpp

Actions

• init
• admincoledit
• setversion
• addconftoken
• transfer
• createcol
• setcoldata
• addcolauth
• remcolauth
• addnotifyacc
• remnotifyacc
• setmarketfee
• forbidnotify
• createschema
• extendschema
• createtempl
• locktemplate
• mintasset
• setassetdata
• announcedepo
• withdraw
• backasset
• burnasset
• createoffer
• canceloffer
• acceptoffer
• declineoffer
• payofferram
• logtransfer
• lognewoffer
• lognewtempl
• logmint
• logsetdata
• logbackasset
• logburnasset

Add some extra IPFS handling support

• Gets IPFS hash from reading the Substream's Binary package (from local filesystem or download)

Cache IPFS when downloading from Node.js consumer

• Cache IPFS when downloading with Node.js consumer

Accounts substream: missing accounts

luismarin123
cryptozhenek
terraeos2535
wangfeng5.tp
langchao1.tp
maicoolwhale
huanghe11.tp
yle.tag
babaoshu1.tp
lufei3131.tp
myosotis1.tp
m2xoelqsfo4v
wk.ldq123.tp
mjyqdslm.tag
keyifannao13

Add a --prefix flag to append a custom prefix to all keys

When we are overlapping keys from both eth and antelope in the redis database. We need to add a --prefix flag to append a custom prefix to all keys. That way, we can run the following command: substreams-sink-redis run --prefix eos and we would get a key like this :eos:uaw:

. This is a feature in substreams-sink-redis

eosio.token: OR params?

Let's say I want to track all transfers TO contract1 and all transfers FROM contract2.
How do we capture this with our params schema @DenisCarriere ?

[Substream Proposal] List of token transfers for accounts

Goal

Retrieve a list of incoming / outgoing token transfers for a certain account in a certain period.

Models

Briefly describe the models that are used as input / output for this Substream

Maps

Briefly list the maps including their input / output for this Substream

Stores

Briefly list the stores used in this Substream

Consumers

Briefly explain how to consume this substream and whether this is a custom consumer or a default sink

Validate accounts substreams data

• hyperion
• chain

Add shared code folder for substreams

Create a lib folder that will contain shared code among all substreams. Document it

[Substream Proposal] Token balance for accounts

Goal

Retrieve a list of token balances per account and a list of accounts that hold balances of a certain token.

Models

Briefly describe the models that are used as input / output for this Substream

Maps

Briefly list the maps including their input / output for this Substream

Stores

Briefly list the stores used in this Substream

Consumers

Briefly explain how to consume this substream and whether this is a custom consumer or a default sink

[Substream Proposal] List of Tokens

Goal

Retrieve a list of tokens that are available on the chain.

Models

Briefly describe the models that are used as input / output for this Substream

Maps

Briefly list the maps including their input / output for this Substream

Stores

Briefly list the stores used in this Substream

Consumers

Briefly explain how to consume this substream and whether this is a custom consumer or a default sink

Accounts substream: missing premium accounts

Probably claimed differently?

ERC20 substreams

• add graph_out sink EntityChanges
  https://github.com/streamingfast/substreams-entity-change
• add prom_out sink `PrometheusOperations
  https://github.com/pinax-network/substreams-sink-prometheus.rs
• support all ERC20 smart contract events https://ethereum.org/en/developers/docs/standards/tokens/erc-20/
  • Transfer (map_transfers)
  • Approve (map_approves)
• Support params to filter by particular smart contract or all contracts (ex: https://github.com/pinax-network/substreams/tree/develop/eosio.token)

References

• https://github.com/messari/substreams/tree/master/erc20-holdings
• https://github.com/pinax-network/substreams/tree/develop/eosio.token

Adding support for Substreams JSON formatted responses

• Substreams to JSON formatted responses @fschoell (❗blocked by StreamingFast)
• Add support to Node.js consumer @DenisCarriere

There's very little reason for the Substream Client side to need to parse serialized protobuf

StreamingFast team is future, will save a lot of headaches.

This should remove for the Protobuf requirements, all the data would be served by the Substream .spkg package on the server-side.

Usage

const block = substreams.parseBlockData(response);
if (!block) return;

for ( const output of block.outputs ) {
    console.log(JSON.stringify(output));
}

Current data

{
  "name": "store_tokens",
  "data": {
    "oneofKind": "storeDeltas",
    "storeDeltas": {
      "deltas": [
        {
          "operation": 1,
          "ordinal": "1",
          "key": "EOS,[email protected]",
          "oldValue": {},
          "newValue": {
            "type": "Buffer",
            "data": [
              8,186,1,18,6,8,202,129,239,216,5,26,11,101,111,115,105,111,46,116,111,107,101,110,34,3,69,79,83,40,4,48,128,128,233,131,177,222,22,58,5,101,111,115,105,111
            ]
          }
        }
      ]
    }
  },
  "logs": [
    "storing new token EOS,[email protected] from block 186 at 2018-06-09T11:58:02Z"
  ],
  "logsTruncated": false
}

Add bounties substreams

Track statelog action
ref https://github.com/pomelo-io/pomelo-rest-api/issues/614

Add to Common for transaction events

To Add

• map_block
• map_all_transaction_traces

Metrics

• Producer
• Transaction CPU usage
• First Authorization

[Substream Proposal] Blocktivity

Goal

Blocktivity is a tool that calculates the actions / transactions on chain and buckets them into hourly intervals. The current implementation can be found here. It would make sense to re-implement this tool as a Substream.

Models

message Blocktivity {
  uint64 block_num = 1;                     # block number 
  google.protobuf.Timestamp timestamp = 2;  # block creation timestamp (UTC)
  uint64 trxCount = 3;                      # number of transactions in this block
  uint64 actCount = 4;                      # number of actions in this block
  uint64 cpuUsageUs = 5;                    # cpu_usage_us of this block
  uint64 netUsageWords = 6;                 # net_usage_words of this block
}

Maps

map_blocktivity - Transforms a full block into a Blocktivity object

Stores

Each store will get the Blocktivity objects, calculate the key (the timestamp of the hour of this block) and then accumulates the data in multiple StoreAddInt64 stores.

store_trx_count - Adds up the number of transactions per key (hour)
store_act_count - Adds up the number of actions per key (hour)
store_cpu_usage_us - Adds up the cpu_usage_us values per key (hour)
store_net_usage_words - Adds up the net_usage_words per key (hour)

Consumers

This would probably be consumed by a custom consumer (Golang/NodeJS) that reads in the buckets and pushes the data as transactions to chain whenever a bucket is full (the current hour has passed).

Implement `eosio.token` Substream

Parameters

• contract
• owner
• symbol.code

Tables

• stat (Supply)
• accounts (Account Balances)

Actions

• create
• issue
• retire
• transfer
• open
• close

Static

• get_supply
  • map: contract::supply.symbol.code
• get_balance
  • map: contract::owner::balance.symbol.code

https://github.com/eosnetworkfoundation/eos-system-contracts/blob/main/contracts/eosio.token/include/eosio.token/eosio.token.hpp

Actions - References

/**
 * Allows `issuer` account to create a token in supply of `maximum_supply`. If validation is successful a new entry in statstable for token symbol scope gets created.
 *
 * @param issuer - the account that creates the token,
 * @param maximum_supply - the maximum supply set for the token created.
 *
 * @pre Token symbol has to be valid,
 * @pre Token symbol must not be already created,
 * @pre maximum_supply has to be smaller than the maximum supply allowed by the system: 1^62 - 1.
 * @pre Maximum supply must be positive;
 */
[[eosio::action]]
void create( const name&   issuer,
            const asset&  maximum_supply);
/**
 *  This action issues to `to` account a `quantity` of tokens.
 *
 * @param to - the account to issue tokens to, it must be the same as the issuer,
 * @param quantity - the amount of tokens to be issued,
 * @memo - the memo string that accompanies the token issue transaction.
 */
[[eosio::action]]
void issue( const name& to, const asset& quantity, const string& memo );

/**
 * The opposite for create action, if all validations succeed,
 * it debits the statstable.supply amount.
 *
 * @param quantity - the quantity of tokens to retire,
 * @param memo - the memo string to accompany the transaction.
 */
[[eosio::action]]
void retire( const asset& quantity, const string& memo );

/**
 * Allows `from` account to transfer to `to` account the `quantity` tokens.
 * One account is debited and the other is credited with quantity tokens.
 *
 * @param from - the account to transfer from,
 * @param to - the account to be transferred to,
 * @param quantity - the quantity of tokens to be transferred,
 * @param memo - the memo string to accompany the transaction.
 */
[[eosio::action]]
void transfer( const name&    from,
            const name&    to,
            const asset&   quantity,
            const string&  memo );
/**
 * Allows `ram_payer` to create an account `owner` with zero balance for
 * token `symbol` at the expense of `ram_payer`.
 *
 * @param owner - the account to be created,
 * @param symbol - the token to be payed with by `ram_payer`,
 * @param ram_payer - the account that supports the cost of this action.
 *
 * More information can be read [here](https://github.com/EOSIO/eosio.contracts/issues/62)
 * and [here](https://github.com/EOSIO/eosio.contracts/issues/61).
 */
[[eosio::action]]
void open( const name& owner, const symbol& symbol, const name& ram_payer );

/**
 * This action is the opposite for open, it closes the account `owner`
 * for token `symbol`.
 *
 * @param owner - the owner account to execute the close action for,
 * @param symbol - the symbol of the token to execute the close action for.
 *
 * @pre The pair of owner plus symbol has to exist otherwise no action is executed,
 * @pre If the pair of owner plus symbol exists, the balance has to be zero.
 */
[[eosio::action]]
void close( const name& owner, const symbol& symbol );

Tables - References

struct [[eosio::table]] account {
    asset    balance;

    uint64_t primary_key()const { return balance.symbol.code().raw(); }
};

struct [[eosio::table]] currency_stats {
    asset    supply;
    asset    max_supply;
    name     issuer;

    uint64_t primary_key()const { return supply.symbol.code().raw(); }
};

Static - References

 static asset get_supply( const name& token_contract_account, const symbol_code& sym_code )
 {
    stats statstable( token_contract_account, sym_code.raw() );
    const auto& st = statstable.get( sym_code.raw(), "invalid supply symbol code" );
    return st.supply;
 }

 static asset get_balance( const name& token_contract_account, const name& owner, const symbol_code& sym_code )
 {
    accounts accountstable( token_contract_account, owner.value );
    const auto& ac = accountstable.get( sym_code.raw(), "no balance with specified symbol" );
    return ac.balance;
 }

pomelo: add params

I.e. params="app_contract=app.pomelo&claim_contract=claim.pomelo&login_contract=login.eosn"

antelope.trxstats compilation error

error[E0599]: no associated item named `Unexpected` found for struct `anyhow::Error` in the current scope
  --> antelope.trxstats/src/lib.rs:28:35
   |
28 | ...(|| Error::Unexpected(format!("Invalid parameter format: {}", param)))?;
   |               ^^^^^^^^^^ associated item not found in `Error`

error[E0599]: no associated item named `Unexpected` found for struct `anyhow::Error` in the current scope
  --> antelope.trxstats/src/lib.rs:31:33
   |
31 | ...|_| Error::Unexpected(format!("Invalid param value for key '{}'", key...
   |               ^^^^^^^^^^ associated item not found in `Error`

error[E0599]: no associated item named `Unexpected` found for struct `anyhow::Error` in the current scope
  --> antelope.trxstats/src/lib.rs:39:36
   |
39 | ...Err(Error::Unexpected(format!("Unknown parameter: '{}'", key))),
   |               ^^^^^^^^^^ associated item not found in `Error`

For more information about this error, try `rustc --explain E0599`.
error: could not compile `antelope_trxstats` due to 3 previous errors

Create a substream for AtomicAssets NFTs on EOS and WAX

pomelo: add claim events

Watch db ops on claim.pomelo contract

Port Messari ERC721 substream

As for making it work with Subgraphs, it's more about making it compatible with Sinks

• add graph_out sink EntityChanges
  https://github.com/streamingfast/substreams-entity-change
• add prom_out sink `PrometheusOperations
  https://github.com/pinax-network/substreams-sink-prometheus.rs
• support all ERC721 smart contract events https://eips.ethereum.org/EIPS/eip-721
  • Transfer
  • Approve
  • ApprovalForAll
• Support params to filter by particular smart contract or all contracts

eosio.evm bridge

• account table deltas
• outgoing transfer from eosio.evm
  • to
  • quantity
  • memo

Transaction example: https://eos.eosq.eosnation.io/tx/55a675cfe259b1306c3b8140ff14b87183ed13e9d1fbadc5ba3cb37dfb7e1ba5

Host Substream Protobufs on IPFS

• Host Protobufs on IPFS
• Integrate downloaded Protobuf in Node.js consumer

syntax = "proto3";

package antelope.tokens.v1;

import "google/protobuf/timestamp.proto";


message Tokens {
  repeated Token tokens = 1;
}

message Token {
  uint32 block_num = 1;
  google.protobuf.Timestamp timestamp = 2;
  string contract = 3;
  string symcode = 4;
  uint32 precision = 5;
  int64 amount = 6;
  string issuer = 7;
}

Add to Common for IBC lightproof DB

To Add

• map_blockroot_merkle
• map_header

Types

• BlockRootMerkle iterator
• BlockHeader iterator

Data required

• block.number
• block.id
• block.header
• block.blockrootMerkle

Code example consuming data

function processBlock(data){
    return rootDB.transaction(async () => {
      let block = JSON.parse(JSON.stringify(data.block, null, "  "));
      if (data.step === "STEP_IRREVERSIBLE") {
        lib = block.number;
        statusDB.put("lib", block.number);
      }
      else { //if STEP_NEW
        let date = (new Date(parseInt(block.header.timestamp.seconds)*1000)).toISOString().replace('Z', '');
        if (block.header.timestamp.nanos) date = date.replace('000', '500')
        statusDB.put("lastBlockTimestamp", date); 
      }
      const blockMerkle = block.blockrootMerkle;
      blockMerkle.activeNodes.forEach((node,index) => blockMerkle.activeNodes[index] = toHex(node) );
      var buffer = serialize(block.id, blockMerkle.activeNodes);
      blocksDB.put(block.number, asBinary(buffer));
    });
  }

eosio.token: add `amount_lt` and `amount_gt` filters

To filter transfers by amount, i.e. to=swap.defi&contract=eosio.token&amount_lt=10000 - only transfers > 1.0000 EOS to swap.defi account

substreams broken

seems like probably all Substreams in https://github.com/pinax-network/substreams are broken. They depend on an old version of substreams-antelope which seems to be missing the serde import:

cargo build --target wasm32-unknown-unknown --release
warning: profiles for the non root package will be ignored, specify profiles at the workspace root:
package:   /Users/work/RustRover/substreams/erc721/Cargo.toml
workspace: /Users/work/RustRover/substreams/Cargo.toml
    Updating crates.io index
   Compiling substreams-antelope v0.1.1
error[E0432]: unresolved import `serde`
 --> /Users/work/.cargo/registry/src/github.com-1ecc6299db9ec823/substreams-antelope-0.1.1/src/pb/sf.antelope.type.v1.rs:1:5
  |
1 | use serde::{Serialize, Deserialize};
  |     ^^^^^ use of undeclared crate or module `serde`

error: cannot determine resolution for the derive macro `Serialize`
   --> /Users/work/.cargo/registry/src/github.com-1ecc6299db9ec823/substreams-antelope-0.1.1/src/pb/sf.antelope.type.v1.rs:559:10
    |
559 | #[derive(Serialize)]
    |          ^^^^^^^^^
    |
    = note: import resolution is stuck, try simplifying macro imports

For more information about this error, try `rustc --explain E0432`.
error: could not compile `substreams-antelope` due to 2 previous errors
make: *** [build] Error 101

Updating substreams-antelope to 0.3.2 seems to be fixing this, but is then breaking the block imports as substreams_antelope::Block has been renamed to substreams_antelope::pb::Block.

Add `authorization` to Substream common