hermeznetwork/price-updater-service

New service to Price Update feature from Hermez

Price Updater Service

Price Updater is a web service to consult token and currency prices used by Hermez Node.

How It Works

1. The system was built with Golang and designed to run through commands.
2. Given a specific time to update (the time can be set in through configuration, or by using a default value of 30s), the system updates the prices of currency and tokens.
3. The prices of tokens and currencies are returned through an endpoint.

Instalation

To install the system just clone and build it!

$ git clone [email protected]:hermeznetwork/price-updater-service.git
$ cd price-updater-service/
$ go build -o priceupdater # or other name that you want

Click here to see set up documentation.

Usage

The system has been designed to run through the commands listed above. The commands are listed to be executed in the order below.

Configuration

You can set up the configuration of the system by env file

• HTTP_HOST: Host to server
• HTTP_PORT: Port to server
• POSTGRES_USER (required): Username of hermez node database
• POSTGRES_PASSWORD (required): Password of hermez node database
• POSTGRES_HOST (required): Host address of hermez node database
• POSTGRES_PORT (default: 5432): Port of hermez node database
• POSTGRES_DATABASE (required): Database name of hermez node database
• POSTGRES_SSL_ENABLED:
• POSTGRES_MAX_ID_CONNS:
• POSTGRES_MAX_OPEN_CONNS:
• ETH_NETWORK (required by Uniswap provider): URL address to Ethereum network (mainnet, rinkeby, goerli)
• ETH_HEZ_ROLLUP (required by Uniswap provider): Address of Smart Contract (mainnet, rinkeby, goerli)
• ETH_USDT_ADDRESS (required by Uniswap provider): Address of USDT token on network (mainnet, rinkeby, goerli)
• BBOLT_LOCATION (default: /tmp/price_updater.db): Local database used to system. You must choice a better location 👍
• MAIN_TIME_TO_UPDATE_PRICES (default: 30s): Time between updates executed by system.

Provider Configuration

There are two commands related to providers:

1. [REQUIRED] $ priceupdater change-prirotiry --priority <provider-name-1>,<provider-name-2>,<provider-name-3>

   The next command allows configure the provider priority. The first provider will be chosen by default but if it fails during retrieving the price for a specific token, the service will try to get the price for this token using the next provider in the list.

   Show example

   $ priceupdater change-priority --priority <provider_1>,<provider_2>,<provider_3>
   

Note: <provider_1> has the highest priority and <provider_2> the lowest

2. [REQUIRED] update-config --provider <provider-name> --configFile <path-to-config.json>

   After running change-provider the first time, you should load the provider's configuration

   Show example

   $ priceupdater update-config --provider bitfinex --configFile assets/bitfinex.json

Note: This action must be done for each provider we want to configure.

Project Settings

There is two commands for project settings:

1. [OPTIONAL] $ priceupdater setup-origin --origins "items,comma,separated" This command will set up hostnames that the system will accept as incoming requests. The default value is *, which means it accepts all requests.

2. [REQUIRED] setup-apikey --apiKey "pr1c3upd4t3r" This command will set up an apiKey that the system will accept as incoming requests.

Server

There is just one command for running the server:

1. $ priceupdater server

   This command will start the Price Updater server. It's possible to pass some configurations to the system.

   Show example

   $  priceupdater server --help
   
   Usage:
     price-updater server [flags]
   
   Flags:
         --eth-network string   ethereum address
         --fiat-key string      fiat api key
     -h, --help                 help for server
         --pg-dbname string     postgresql database name
         --pg-host string       postgresql host
         --pg-pass string       postgresql password
         --pg-port int          postgresql port (default 5432)
         --pg-user string       postgresql username

Update Static Tokens

There is just one command for update static tokens:

1. $ priceupdater update-static-token --tokenID 1 --price 1.531 This command will update the price of given tokenID

Providers

There a list of implemented providers by Price Updater

• Bitfinex
• Coingecko
• Uniswap

Provider Config

We have created a pattern using a JSON file to put the necessary configuration for running the providers on the system.

{
    "provider": "providerName",
    "config": {
        "0": "0xETHEREUM MAINNET ADDRESS",
        "1": "0xETHEREUM MAINNET ADDRESS",
        "2": "SYMBOL"
    }
}

The config object means: The key is the tokenID from your hermez-node database (possibly in Goerli, Rinkeby or Mainnet), and the value can be the Address of the token on Mainnet Ethereum network or the symbol.

API

There is a list of endpoints (and examples) that can be called on the Price Update Service. You can see these examples in the Postman Collection file at the project's root folder. The API has a HEADER required called X-API-KEY. To execute any endpoint's from Price Updater, you should pass it.

• All examples have been run with Httpie

Health endpoint

• Endpoint: /v1/health
• HTTP Method: GET
• Details: This will expose if the server is running.

☁  ~  http https://priceupdater.hermez.io/v1/health X-API-KEY:"pr1c3upd4t3r"
HTTP/1.1 200 OK
CF-Cache-Status: DYNAMIC
CF-RAY: 678fdf2f28c825f9-GIG
Connection: keep-alive
Content-Length: 7
Content-Type: text/plain; charset=utf-8
Date: Tue, 03 Aug 2021 13:25:26 GMT
Expect-CT: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
Last-Modified: Tue, 03 Aug 2021 13:25:26 GMT
NEL: {"report_to":"cf-nel","max_age":604800}
Report-To: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=NgEZxHGZ2mqgZn%2Fcyk%2FSwwjCNEhKp23K2rI5fjAYT5IwufaNr37oS5ZeSVZxIGq1uVQGCtzCIj1j3NJqjZO6O025FRrzxSlDGsMr%2Fdvkia30teNuQY%2Fny0%2F5j1gDI%2Fd9q%2FBFPkEYG4xowtc5d5yPrwkQqnQ%3D"}],"group":"cf-nel","max_age":604800}
Server: cloudflare

Healthy

Tokens endpoint

• Endpoint: /v1/tokens
• HTTP Method: GET
• Details: Returns a list of tokens prices.

We omitted some tokens in the response to make it more readable

☁  ~  http https://priceupdater.hermez.io/v1/tokens X-API-KEY:"pr1c3upd4t3r"
HTTP/1.1 200 OK
CF-Cache-Status: DYNAMIC
CF-RAY: 678fe041ddd6f84b-GIG
Connection: keep-alive
Content-Encoding: gzip
Content-Type: application/json
Date: Tue, 03 Aug 2021 13:26:10 GMT
Expect-CT: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
Last-Modified: Tue, 03 Aug 2021 13:26:10 GMT
NEL: {"report_to":"cf-nel","max_age":604800}
Report-To: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=hsJzqKJPtCm%2B0SvhPgiLHry1X1PNnGwmaXYiL%2Bm9JJ9urh7OEs%2BlMBO0%2Fb5VY8YNJQnmbJDxXAigF9goLX%2FP37gZ92L8NUYPeMztczd%2FQUUlnB5jZ4YXmzaCD%2F5pLzQBGtQOVDYLu7z5mWlIoOUJhtIjo9Y%3D"}],"group":"cf-nel","max_age":604800}
Server: cloudflare
Transfer-Encoding: chunked

{
    "tokens": [
        {
            "USD": 21.044,
            "decimals": 18,
            "ethereumAddress": "0xba100000625a3754423978a60c9317c58a424e3D",
            "ethereumBlockNum": 12197953,
            "id": 11,
            "itemId": 12,
            "name": "Balancer",
            "symbol": "BAL",
            "usdUpdate": "2021-08-03T13:25:22.212488Z"
        },
        {
            "USD": 2508.8,
            "decimals": 18,
            "ethereumAddress": "0x0000000000000000000000000000000000000000",
            "ethereumBlockNum": 0,
            "id": 0,
            "itemId": 1,
            "name": "Ether",
            "symbol": "ETH",
            "usdUpdate": "2021-08-03T13:25:22.243868Z"
        },
        {
            "USD": 31737,
            "decimals": 18,
            "ethereumAddress": "0x0bc529c00C6401aEF6D220BE8C6Ea1667F6Ad93e",
            "ethereumBlockNum": 12198043,
            "id": 13,
            "itemId": 14,
            "name": "yearn.finance",
            "symbol": "YFI",
            "usdUpdate": "2021-08-03T13:25:22.360085Z"
        },
        {
            "USD": 21.362,
            "decimals": 18,
            "ethereumAddress": "0x1f9840a85d5aF5bf1D1762F925BDADdC4201F984",
            "ethereumBlockNum": 12197927,
            "id": 8,
            "itemId": 9,
            "name": "Uniswap",
            "symbol": "UNI",
            "usdUpdate": "2021-08-03T13:25:22.373092Z"
        }
    ]
}

Get specific Token

• Endpoint: /v1/tokens/<TOKEN_ID>
• HTTP Method: GET
• Details: Returns the price for a specific token.

☁  ~  http https://priceupdater.hermez.io/v1/tokens/17 X-API-KEY:"pr1c3upd4t3r"
HTTP/1.1 200 OK
CF-Cache-Status: DYNAMIC
CF-RAY: 678fe6ec6910275e-GIG
Connection: keep-alive
Content-Encoding: gzip
Content-Type: application/json
Date: Tue, 03 Aug 2021 13:30:43 GMT
Expect-CT: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
Last-Modified: Tue, 03 Aug 2021 13:30:43 GMT
NEL: {"report_to":"cf-nel","max_age":604800}
Report-To: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=T0ihq%2BtQhUqsVnXh9dTtB93Tm8S7MMYFCKOfa8tdN6wq9Sis8okHYcZhh06DgcSC3bS99qeHRCPR%2BOLyTXbX%2FZEQ%2F0PLtjS%2FQKTNq%2FjzBXNwN14CSf6Z2cXHujsribuD3Jaoln5RL%2BlpgAnLrj6GtqiDUIM%3D"}],"group":"cf-nel","max_age":604800}
Server: cloudflare
Transfer-Encoding: chunked

{
    "USD": 7.44,
    "decimals": 18,
    "ethereumAddress": "0xDe30da39c46104798bB5aA3fe8B9e0e1F348163F",
    "ethereumBlockNum": 12605976,
    "id": 17,
    "itemId": 18,
    "name": "Gitcoin",
    "symbol": "GTC",
    "usdUpdate": "2021-08-03T13:30:37.330874Z"
}

If you try to get a token that doesn't exists on Price Updater:

☁  ~  http https://priceupdater.hermez.io/v1/tokens/171 X-API-KEY:"pr1c3upd4t3r"
HTTP/1.1 404 Not Found
CF-Cache-Status: DYNAMIC
CF-RAY: 678fe7d70f2a2603-GIG
Connection: keep-alive
Content-Length: 29
Content-Type: application/json
Date: Tue, 03 Aug 2021 13:31:20 GMT
Expect-CT: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
NEL: {"report_to":"cf-nel","max_age":604800}
Report-To: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=W2oI3RMtIsukqqB%2FPZvpJBNijXlGsRNhivwtCfL8HIFwb2MBDNjtE%2BqGIevp08dbyhPAzIYmvMk4Wet5ag5XTmG85oIepqLk0uwsUcfV2XMsz9fZnBM4h88zC5URtZcfH7VqTvPywJgnB9n2FMf3xmnjP%2FM%3D"}],"group":"cf-nel","max_age":604800}
Server: cloudflare

{
    "message": "token not found"
}

Filter tokens

• Endpoint: /v1/tokens?id=1|5
• HTTP Method: GET
• Details: Returns the price for a filtered token.

☁  ~  http https://priceupdater.hermez.io/v1/tokens id=="1|5" X-API-KEY:"pr1c3upd4t3r"

HTTP/1.1 200 OK
CF-Cache-Status: DYNAMIC
CF-RAY: 678fe9531fa32599-GIG
Connection: keep-alive
Content-Encoding: gzip
Content-Type: application/json
Date: Tue, 03 Aug 2021 13:32:21 GMT
Expect-CT: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
Last-Modified: Tue, 03 Aug 2021 13:32:21 GMT
NEL: {"report_to":"cf-nel","max_age":604800}
Report-To: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=Z%2FkQ4PDo%2F6j7tMUzzdOsrS9qNpJlpuzalOy8zy%2BYzJwYplD0%2ByuCU8hFxJo1IuZq2TLPP2e7Q7CEUp8Extjjx3joGBCxKgpaYU5skIZSGxGmGCcpVodgVBi9yGEGKfdQUgA1aI2rjiwQL2OXcl8HQLi%2Byng%3D"}],"group":"cf-nel","max_age":604800}
Server: cloudflare
Transfer-Encoding: chunked

{
    "tokens": [
        {
            "USD": 0,
            "decimals": 18,
            "ethereumAddress": "0x55a1Db90A5753e6Ff50FD018d7E648d58A081486",
            "ethereumBlockNum": 4417287,
            "id": 1,
            "itemId": 2,
            "name": "Hermez Network Token",
            "symbol": "HEZ",
            "usdUpdate": "1970-01-01T00:00:00Z"
        },
        {
            "USD": 0,
            "decimals": 18,
            "ethereumAddress": "0x661E5805cb0b671F0D7ba3CFD7398F885195f866",
            "ethereumBlockNum": 4611744,
            "id": 5,
            "itemId": 6,
            "name": "GAS GAS GAAAS",
            "symbol": "GAS",
            "usdUpdate": "1970-01-01T00:00:00Z"
        }
    ]
}

Currencies

• Endpoint: /v1/currencies
• HTTP Method: GET
• Details: Returns a list of currencies prices.

☁  ~  http https://priceupdater.hermez.io/v1/currencies X-API-KEY:"pr1c3upd4t3r"
HTTP/1.1 200 OK
CF-Cache-Status: DYNAMIC
CF-RAY: 678fe9531fa32599-GIG
Connection: keep-alive
Content-Encoding: gzip
Content-Type: application/json
Date: Tue, 03 Aug 2021 13:32:21 GMT
Expect-CT: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
Last-Modified: Tue, 03 Aug 2021 13:32:21 GMT
NEL: {"report_to":"cf-nel","max_age":604800}
Report-To: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=Z%2FkQ4PDo%2F6j7tMUzzdOsrS9qNpJlpuzalOy8zy%2BYzJwYplD0%2ByuCU8hFxJo1IuZq2TLPP2e7Q7CEUp8Extjjx3joGBCxKgpaYU5skIZSGxGmGCcpVodgVBi9yGEGKfdQUgA1aI2rjiwQL2OXcl8HQLi%2Byng%3D"}],"group":"cf-nel","max_age":604800}
Server: cloudflare
Transfer-Encoding: chunked

{
    "currencies": [
        {
            "baseCurrency": "USD",
            "currency": "CNY",
            "lastUpdate": "2021-08-03T13:32:05.317295Z",
            "price": 6.464898
        },
        {
            "baseCurrency": "USD",
            "currency": "EUR",
            "lastUpdate": "2021-08-03T13:32:05.32069Z",
            "price": 0.842115
        },
        {
            "baseCurrency": "USD",
            "currency": "JPY",
            "lastUpdate": "2021-08-03T13:32:05.323926Z",
            "price": 108.989498
        },
        {
            "baseCurrency": "USD",
            "currency": "GBP",
            "lastUpdate": "2021-08-03T13:32:05.327223Z",
            "price": 0.718935
        }
    ]
}

Get specific Currency

• Endpoint: /v1/currencies/
• HTTP Method: GET
• Details: Returns the price for a specific currency.

☁  ~  http https://priceupdater.hermez.io/v1/currencies/EUR X-API-KEY:"pr1c3upd4t3r"
HTTP/1.1 200 OK
CF-Cache-Status: DYNAMIC
CF-RAY: 678fea8b9912275e-GIG
Connection: keep-alive
Content-Encoding: gzip
Content-Type: application/json
Date: Tue, 03 Aug 2021 13:33:11 GMT
Expect-CT: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
Last-Modified: Tue, 03 Aug 2021 13:33:11 GMT
NEL: {"report_to":"cf-nel","max_age":604800}
Report-To: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=NDfgP002wUZsZ6ijFLgrqJcJxIsiLzw7iyiiJhFxnOqeSFejn9cvgDkcEscUShMNH2shqTv1dEn2aLK%2FIQ%2BDyN7UGUT%2BBndt9jT2XzVPeMGX9Cz3U0ZAtq5dbRyAFAUm3j0ujqdaiPpw8C3t4hUNNzUHiK4%3D"}],"group":"cf-nel","max_age":604800}
Server: cloudflare
Transfer-Encoding: chunked

{
    "baseCurrency": "USD",
    "currency": "EUR",
    "lastUpdate": "2021-08-03T13:33:07.03667Z",
    "price": 0.842115
}

If you try to get a currency that doesn't exists on Price Updater:

☁  ~  http https://priceupdater.hermez.io/v1/currencies/AOE X-API-KEY:"pr1c3upd4t3r"
HTTP/1.1 404 Not Found
CF-Cache-Status: DYNAMIC
CF-RAY: 678feb8eec6b2616-GIG
Connection: keep-alive
Content-Length: 36
Content-Type: application/json
Date: Tue, 03 Aug 2021 13:33:52 GMT
Expect-CT: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
NEL: {"report_to":"cf-nel","max_age":604800}
Report-To: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=vxorZ11iIOlGNJVQVPSL%2Fs89fqiktVPHV6svVX4W96IxlrlqchVt%2BuOPcVp3BApiAd3lt4QGUMWKT%2FE75g0X8bpPyhnMuFBgLJ1Do4ENsWG9cMXxisTMUb%2Bp76mfp138bPA3O5t1rPEEOweZzQeY0lpD60Q%3D"}],"group":"cf-nel","max_age":604800}
Server: cloudflare

{
    "message": "currency AOE not found"
}

Filter currencies

• Endpoint: /v1/currencies?base=USD&symbols=EUR|JPY
• HTTP Method: GET
• Details: Returns the price for a filtered currency.

☁  ~  http https://priceupdater.hermez.io/v1/currencies symbols=="EUR|JPY" X-API-KEY:"pr1c3upd4t3r"
HTTP/1.1 200 OK
CF-Cache-Status: DYNAMIC
CF-RAY: 678fe9531fa32599-GIG
Connection: keep-alive
Content-Encoding: gzip
Content-Type: application/json
Date: Tue, 03 Aug 2021 13:32:21 GMT
Expect-CT: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
Last-Modified: Tue, 03 Aug 2021 13:32:21 GMT
NEL: {"report_to":"cf-nel","max_age":604800}
Report-To: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=Z%2FkQ4PDo%2F6j7tMUzzdOsrS9qNpJlpuzalOy8zy%2BYzJwYplD0%2ByuCU8hFxJo1IuZq2TLPP2e7Q7CEUp8Extjjx3joGBCxKgpaYU5skIZSGxGmGCcpVodgVBi9yGEGKfdQUgA1aI2rjiwQL2OXcl8HQLi%2Byng%3D"}],"group":"cf-nel","max_age":604800}
Server: cloudflare
Transfer-Encoding: chunked

{
    "currencies": [
        {
            "baseCurrency": "USD",
            "currency": "EUR",
            "lastUpdate": "2021-08-09T10:32:26.296803Z",
            "price": 0.850602
        },
        {
            "baseCurrency": "USD",
            "currency": "JPY",
            "lastUpdate": "2021-08-09T10:32:26.298462Z",
            "price": 110.168013
        }
    ]
}

price-updater-service's Issues

Create documentation to configFiles of providers

The price updater can load configurations from a json file to be used as configuration of provider. The struct is pretty simple:

Bitfinex example:

{
  "provider": "bitfinex",
   "config": {
       "0": "eth-add"
   }

The config object represent the relation of tokenID on hermez-node and ethereum address of token on mainnet.

Use only one price updater to update many hermez node

Use only one price updater service to update the prices in many hermez node synchronized in the same network (environment).
This will ovoid the need of having multiple price updater services that sync the same tokens

Create assets folder to default configuration of providers

related to #9

Improvements on cli logs

Filter tokens list returned

Could the actual endpoint: /v1/tokens accepts an optional value to filter the list to a specific tokens. Params could be as Filter currencies endpoint that symbols are separated by | so we can request id=0|1 and maybe with this improvement we could remove the other endpoint /v1/tokens/<TOKEN_ID> or replace it to this approach.
Returns format could be the same

Rename price_updater_collection.json to be clear it is for Postman

Maybe rename the file price_updater_collection.jsonto postman_price_updater_collection.json it going to help the users clearly identifies the files purposes.

Improve README with more information

As user, I want to know what commands are required or optional to run on set up.

As a developer, I want to know that all requests need of Origin header

Create documentation to setup the Price Updater

Add API Key for testing

As we test from Origin localhost, it won't work once we do apply Origin limits. A possible solution is to have an easy to change API key that we can then share on Notion for internal development.

Improvements on documentation

[ ] - Create a documentation to setup the price updater
[ ] - Improvements to README

• What's commands are required
• Add $ in a text command to users know that command is runnable by binary

Create Linux service file

It will help the user to have Price Updater as a Linux Service.
However, to make it opcional and easy to use I would suggest to do not include it in a make file, I would suggest to let it as priceupdater.service configuration file in project root folder and another specific priceupdaterserviceconfig.md markdown file giving instructions how to setup it.

Change logger

Use the hermez node logger because it is much better

[logs] - Use log level error just where is really needed

The context is: A lot error logs are sent to elastic service, even this log isn't related to an error because are using a wrong log level.

Send logs with FileBeat

Pass configuration file as argument of cli

Price updater using hermez-node database

We had a chat with @laisolizq and we have some questions regarding price-updater functionality xD

Price-updater service requires some environment variables which includes connection to postgres hermez database:

• does it means that price-updater write directly to the hermez-node database ?
• continuing the question before, does API responses reads from hermez postgres database or price-updater has its own database ?
• wonder if price-updater can run on an external machine a part from hermez-node and if the price-updater could run without setting those variables and be an external service for anyone that needs to fetch prices

thanks !!

use default info from db (Get info from Smart Contract)

If the token it is not defined in the json, get the information from the table "token" and try to retrieve the price using this symbol or contract address. This is necesary for new tokens that can be set from the smart contract any time

Unit tests missing

It is necessary to add unit tests for all functionalities and endpoints.

Use config-library

https://github.com/hermeznetwork/config-library

add uniswap config

Add uniswap file to allow modify the uniswap token addresses

Improvement log on server start up

If there some error not expected, we should logging the message correctly.

Add new hardcoded provider

It could be worth it have another provider wich would be "hardcoded", and then the hardcoded prices would be all in the same file

Use multiple providers

Add the ability to use other provider for a specific token If the price can't be retrieved from the first provider by any reason.
It would be nice if we could config a priority list of providers.

Add README

It is necessary to add a README that explains the usage service and environment variables.