consensys/ethql

A GraphQL interface to Ethereum :fire:

ethql's Issues

Request-scoped caching of web3 responses

During the resolution of a single query, it's possible to incur in the same web3 request twice, which is wasteful. This could happen if the user selects the balances of all to accounts in all transactions from a block, where the same account could've been the recipient of two transactions or more.

Current thinking is to evaluate DataLoader's caching facility, as it appears to be local to single queries.

Transaction decoding: tokens - allow fetching symbol, totalSupply, balances and allowances

Consider the following selection of an ERC20 transaction:

{
  block(number: 1234) {
    transactions {
       decoded {
         ... on ERC20Transfer {
           symbol // new
           totalSupply // new
           from {
             address
             tokenBalance // new
             allowance(spender: "0x1923...") // new
           }
           to {
              address
           }
           value
         }
       }
    }
  }
}

Contract storage: Solidity nested data structures accessors

modularity: Separate resolvers from model

EthQL currently models data entities through a variety of classes like EthqlAccount, EthqlBlock, EthqlTransaction, etc. Some of these inherit from their web3 counterparts, overriding scalar fields with entity relations.

In this manner, an EthqlTransaction no longer refers to accounts by their string addresses, but by references to EthqlAccounts, essentially adding referentiality to data entities.

However, these entities also contain the GraphQL resolution logic, which makes them heavy and difficult to extend. In a modular EthQL, modules should be able to add, extend, modify and remove resolvers, which proves difficult if these are buried in Typescript classes.

Tasks:

• Move resolution logic out of entities into dedicated resolvers (one per entity), e.g. resolvers/account.ts, resolvers/block.ts, etc.
• Merge all resolvers into a single resolver map.

Contract storage: Storage value accessor

Transaction decoders

This epic groups transaction decoding components, of which ERC-20 was the first one to be implemented in ethql.

Populate Account->type field

Enum with values:

• CONTRACT => when the account has code.
• EXTERNALLY_OWNED => when the account has no code.

ethql should be embeddable in other node applications

ethql should be published as an npm module that other projects can import and add to their applications. This is a placeholder description that will be enhanced as the requirement is specified further.

General documentation

Add documentation under the Github wiki.

Documentation should include:

• Setup and deployment models
• Query handbook (example queries)
• How to contribute (running tests, opening issues & PRs, etc)
• FAQ

What else should be added?

Aggregate web3 requests into JSON-RPC batches

The goal is to aggregate all web3 requests that occur within a single tick of the Node.js runtime into a single JSON-RPC batch request. We can leverage Facebook's DataLoader for this.

Proposal:

• Batching is the default behaviour, but it can be turned off by a configuration option.
• Resolvers / model should be oblivious to whether batching is used or not. This can be achieved by ES6 proxyfing the web3.eth object. One way or another, the calling code would only see a Promise back.
• Maybe aggregate requests across concurrent ethql requests (e.g. two ethql requests A and B could have their web3 requests grouped into a single JSON-RPC batch) -- this may not be possible if a VM tick cannot span multiple incoming requests?

Improve test coverage of CLI flags

The goal is to create a test suite that starts ethql with different sets of CLI flags and checks that the effective configuration is as expected in each case.

Contract storage: Solidity mapping accessor

Where is meteor accounts-blockstack?`

Hello Raul :)

where is the public meteor accounts-blockstack
https://twitter.com/raulvk/status/938863422103539713

Contract storage: Solidity dynamically-sized array accessor

Top-level query context concept

We are currently assuming the latest block in all JSON-RPC queries that accept a quantity. Instead, allow the user to select the block to which queries will be applied to, for example, fetch balances and transaction counts. Possible values: before TX, after TX, latest.

Implement blockOffset top-level query

This query selects a block based on a hash or a number, and an offset. The offset can be positive, negative or zero.

Validation: either hash or number MUST be informed, but not both, and offset MUST be informed.

Behaviour:

• If hash provided: call getBlock with the hash, add the offset to the block number that comes back, fetch that block and return it.
• If number provided: we just add the offset to the number and return that block.

Tests:

• happy path: all permutations of positive, negative, zero offset; and hash, number.
• error scenarios: inexistent block, hash and number not informed, both hash and number informed.

Account->tokenHoldings field

Consider the following query:

{
  account(address: "0x1234...") {
    tokenHoldings {
      snt: tokenContract(address: "0x744d70fdbe2ba4cf95131626614a1763df805b9e")
    }
  }
}

In the future, ethql could allow the user to configure static addresses for tokens they know/are interested in.

Evaluate a two-level per-request + shared cache

ethql currently supports a request-scoped cache, but there is benefit to shared caching. In a production scenario, it is likely that the youngest blocks will be hottest, and that highly used smart contracts will garner higher ethql traffic.

However, there is also risk:

• Requests from client A can lead to evicting entries that are hot for client B, hence making the runtime cost of a query non-deterministic. Cache pinning is a possibility here, but how to manage the pinning is a different discussion.
• Misbehaved clients could thrash cache intentionally to degrade the performance of ethql.
• Chain reorgs can render cache entries dirty, so an eviction process is required to remove entries affected by a reorg (reorgs can be notified to us via WSS).
• In the above, if the WSS connection is dropped, we'd need to temporarily disable the cache, and start with a cleared cache once the connection is re-established.

Ultimately I see a two-level cache:

• L1 => request-scoped cache, as it's already implemented.
• L2 => shared LRU or LFU cache with different backend implementations (e.g. in memory, Redis, etc.)

Consider first-class support for Message Calls

Currently the Block->transactions field takes a TransactionFilter as an argument that can filter transactions with input data (i.e. messages). We can introduce first-class support for message calls, such that we allow retrieving the function signature and an array of parameters, only from those transactions that are indeed messages.

Just an idea:

block(number: 123) {
  messageCalls(matching: 'functionName(uint256,bytes32)') {
    param1: parameter(index: 0)
    param2: parameter(index: 1)
  }
}

Resolve ENS names into addresses

The goal is to extend the Address scalar type to accept and resolve .eth names, such that the resolvers only see the resolved address. The ethereum-ens library could be used to perform the resolution.

We'll need to introduce an extra configuration option to set the address of the ENS contract. We can also support a keyword identifying the network against a set of hardcoded values (e.g. MAINNET, ROPSTEN, KOVAN, etc.).

Notes:

• The parseValue and parseLiteral of the Address scalar type would perform the ENS name => address resolution. It is not clear to me if these methods can be async or can return a promise, and if graphql-js will await on it before invoking the resolvers.
• If not, we will have to deal with this manually, by having the parse* methods return a promise, and downstream resolvers would have to await on it. If the passed in address is an actual address (not a name), a resolved Promise can be used.

Decoded transactions should refer to classes/entities rather than standards

ethql should refer to classes or entities rather than specific standards. For example, all of ERC20, ERC223, ERC827 relate to tokens.

It is sometimes unfeasible (or hardly worthwhile) to distinguish exactly which standards a given contract implements (although it could be achieved if the contract implements ERC165, or by inspecting the bytecode). Users are generally concerned with the underlying entity.

In the future we might introduce "standards inference" as a functionality, but for now that's not the goal.

Tasks:

• Remove all schema references to ERC20 and replace them with Token.
• 'standard' as a field should disappear in favour of 'class' = 'token'.
• The TX decoder for ERC20 should be renamed to TokenTxDecoder.

In the future, we'll introduce a NonFungibleTokenTxDecoder, in #33.

Implement createdContract field in transaction

If a transaction creates a contract (i.e. to address is null and the transaction receipt includes a contractAddress field), the field createdContract on the Transaction type must be an Account entity for the newly created contract address. Else, the field must be null.

Logs and receipts schema

Design a schema to cover these entities.

Expose ERC165 supportsInterface operation in schema

The ERC165 standard allows us to interrogate if a contract supports a specific function. It also appears to be a prerequisite for univocally identifying ERC721 contracts (#33).

The current proposal is to add a new field on the Account type. Suggested definition:

supportsInterface(selector: String!): ERC165Result!

Where ERC165Result is an enum with values:

• SUPPORTED => if the contract has a supportsInterface(bytes4) function as mandated by the standard, and the result of the call is true.
• NOT_SUPPORTED => if the contract has a supportsInterface(bytes4) function as mandated by the standard, and the result of the call is false.
• NON_INTROSPECTABLE => if the contract doesn't expose a supportsInterface(bytes4) function.

The selector argument is the String representation of the function selector (e.g. transfer(address,uint)). More info here: http://solidity.readthedocs.io/en/v0.4.21/abi-spec.html#function-selector.

The resolver would need to convert the String representation into its corresponding binary representation as per the link above (there might be functions in the web3 lib to do this).

Query documentation

I'd love to use ethql for a project I'm working on where I would be analyzing a specific token. Except for a couple examples in the readme, I don't see any documentation on how to use this tool for querying Ethereum. What's the best way to approach this?

modularity: Introduce the notion of "capabilities"

In our transition towards a modular and extensible EthQL, we want users to be able to add "capabilities" to EthQL.

A capability is a collection of the following namespaced items, which together define a unit of functionality to be added to EthQL:

• GraphQL subschema, using the extend keyword to enrich the prior schema, e.g. add tokenContract to the root query object in the case of ERC20.
• entity classes to support the subschema and the resolvers.
• resolvers for the added functionality.
• optional decoders.
• services TBD.

Tasks:

• Abstract definition of a capability (i.e. Typescript interfaces or classes).
• Segregate the current functionality of EthQL into two capabilities: core and erc20.
• Basic capability loading logic.

Only fetch block transactions when necessary

Currently we are calling web3.eth.getBlock(..., true) when resolving a block, which means that we are fetching all transactions from JSON-RPC unconditionally.

It is possible that the inner query does not request any transactions, e.g.

block(number: 123) {
  parent {
    hash
  }
}

The above case it would've been superfluous to fetch transactions.

Goal:

Make ethql intelligent enough so that it calls getBlock(number, true) only when resolving the query would require fetching all transaction data. In all other cases, it would call getBlock(number, false).

Transactions should be fetched when the query requests these fields:

• transactions
• transactionsInvolving
• transactionsParticipants

NOT with:

• transactionAt: the resolution logic should change to call web3.eth.getTransactionFromBlock.

Design:

To decide whether to fetch transaction or not, we should use the 4th parameter the GraphQL resolvers, as it gives us access to the "GraphQL Resolve Info" object. See https://graphql.org/graphql-js/type/#graphqlobjecttype.

See library: https://github.com/jakepusateri/graphql-list-fields for a way to obtain a list of the inner fields queried.

[Upgrade] Replace Express-GraphQL with Apollo Server 2?

Lower the load on Infura too for https://ethql-alpha.infura.io/graphql
https://blog.apollographql.com/announcing-apollo-server-2-2b69fb4702ce?gi=c9f8d90c7e53

Transaction decoding: ERC-721 Non-fungible tokens

Add support for decoding ERC-721 transactions.

• Import the ABI into the abi directory. Make sure the JSON is formatted.
• That ABI does not come from an official source, so check it against the expected interface.
• Identify decodable transaction types, and start a discussion in this ticket. For each TX type, indicate the proposed fields and data types. Let's reach team consensus around our understanding.
• Implement the decoding of each transaction type, as well as associated queries (e.g. symbol, token supply, etc.)

Contract storage: Solidity fixed-size array accesor

Extract out web3 typescript declarations info its own package?

Mono repo with lerna and yarn workspaces?

Pagination concept

See: https://graphql.org/learn/pagination/

The edges->nodes indirection advised on that doc may be too heavyweight for us. A simpler aproach like making pageable entities implement a Pageable interface that adds an inner field cursor may be sufficient. See discussion below.

Unit conversions for balances and values

When requesting transaction values and account balances, the user should have the ability to specify the unit it wants them in:

{
  account("0x1234...") {
    balance(unit: "finney")
  }
}

ethql already has an enum type in the schema with all the units supported by web3, so it's a matter of using the field argument and performing the conversion using web3.utils.fromWei().

storage returning errors for transaction -> to accounts

When storage is called on an account generated by "transaction -> to" it returns an error message "Provided address "null" is invalid, the capitalization checksum test failed, or its an indrect IBAN address which can't be converted." and then the data itself.

Example query:
{ block(number: 5000000) { transactions { to { storage { value(at: 0) } } } } }

Transaction decoding: ERC-223

See specification here: ethereum/EIPs#223

Some function names == ERC-20, so we'll need a mechanism to discriminate between standards when a single function fingerprint is not enough.

Documentation of Block->gasLimit is inaccurate

Contract storage accessors

Caching and batching of web3 requests

More Transaction Filters

Block type edges: Parent (Block) and Miner (Account)

The Block->parent and Block->miner fields are modelled, but they are not being populated at the current time.

• The first one should call web3.eth.getBlock with the parent's hash and would return an EthqlBlock.
• The second should return an EthqlAccount initialised with the miner's address.

Plugin design for application-specific queries

When deploying ethql as part of a dApp stack, users may want to enhance the list of supported transaction decoders, or even add application-specific queries or entities.

Transaction decoding: Ethereum Name Service

See ABI here: https://github.com/web3j/web3j/blob/master/core/src/main/resources/solidity/ens/build/ENS.abi

ENS registrar contract: https://etherscan.io/address/0x6090a6e47849629b7245dfa1ca21d94cd15878ef

Log extraction

Explicit test coverage for scalar types

Follows on from #70. Existing tests don't use GraphQL variables and hence only test the parseLiteral variant of the parsing.

The idea is to make scalar testing explicit by testing all parsing variants in a scalars.test.ts suite.

See #70 for more context.

Request-scoped cache is actually being shared

I was under the impression that a singleton instance of the DataLoader automatically scoped cache reads/writes to the GraphQL request at hand, but it's not the case. Instead, each request needs its own DL instance, if we are to follow the request-scoped cache model.

Transaction decoding: remove references to standard numbers

Some standards build upon others, e.g. ERC20 vs ERC223, so identifying the underlying standard may prove difficult, and sometimes impossible, without inspecting the contract bytecode.

If we were to inspect the bytecode, we could look for EVM JUMPDEST and match those against function signature hashes, to try to infer the external interface of a contract against a known set of ABIs.

However, if both interfaces are equivalent, we are out of luck.

Hence, instead of referring to standard numbers, ethql could refer to the concept behind the standard, e.g. tokens (or fungibleTokens, to differentiate from ERC-721).

Implement Transaction->status field

This field maps to the status field in the response of getTransactionReceipt, which should be called with the transaction hash. Value 1 maps to SUCCESS and value 0 maps to FAILURE.

An empty status is also possible, because this field was introduced in Byzantium (after block 4,370,000). If that's the case, this field should be empty.

Three test cases:

• Status SUCCESS.
• Status FAILURE.
• No status (pre 4,370,000).

Transaction Filters: By inputData

It may be useful to filter transactions by inputData content. This would allow users to target transactions that invoke arbitrary contract methods. The filter might look something like: transactions(filter: { inputStartsWith: "0x12345678" }).

Tests: mock JSON-RPC where possible

We're seeing recurrent test failures due to intermittent Infura timeouts. It was OK to depend on an external party at the beginning, but now that we have 100+ tests it is brittle to do so.

The proposed solution consists of intercepting web3 requests via the Provider and responding to them from a filesystem cache which has been populated from mainnet data.

There would be three modes of operation:

• record => requests and responses are recorded in the filesystem. When writing tests during development, you want to use this mode to automate creating the test data.
• replay => use the local test data created through the record mode to respond to web3 calls, where possible. If a cache entry cannot be found, invoke the endpoint.
• passthrough => allow all requests to go through to the endpoint.

transactionsRoles with to and from fields not returning correct transactions

transactionsRoles with the 'to' and 'from' fields specified is returning transactions that satisfy either the 'to' address or the 'from' address or satisfy both. It should return just transactions that satisfy both the 'from' and the 'to' address.