diadata-org/diadata

DIAdata.org platform

diadata's Issues

Feature request: Include Gold live data

Write a scraper that includes gold price data for the DIA database.

exchange-scrapers: OKEx is broken

Describe the bug
seems OKEx is receiving wrong values and can't parse trades

To Reproduce

run
go run cmd/exchange-scrapers/collectorTextOnly/scraper.go -exchange OKEx

trades are not showing up and error message:
WARN[0001] reconnect the scraping to ws, invalid character 'M' looking for beginning of value:[]

High CPU usage in some Exchange Scrapers

Some exchange scraper like SimexScraper have two nested loops that can cause high CPU usage. For example

for true {
    for key, el := range s.pairScrapers {
...
   }
}

Will get stuck with high cpu usage if the pairScrapers map[string]*SimexPairScraper is empty.
A equivalent issue is present in QuoineScraper

Improve Documentation

Adding a scraper I found https://docs.diadata.org/exchange-scrapers.html to be lacking.

Additionally the README.md has no information on how to run the project, where to find documentation or what is expected from the scrapers.

Huobi API Scraping

We are looking for a Go developer that can provide scrapers of a websocket API. If our collaboration is productive we also offer tasks that include scraping of REST APIs. We reliably want to get trades in realtime from cryptocurrency exchanges. This trade information is then written into our databases using a specific format. We have an existing framework for this which allows us to parametrize the scraper. We work with Go and expect that the scrapers will work concurrently and efficiently.

We will provide the necessary documentation for the task and introduce you to our framework. https://github.com/huobiapi

Set-up instruction for the task can be found under: http://docs.diadata.org/scraper.html
Documentation

Simex API Scraping

We are looking for a Go developer that can provide scrapers of a websocket API. If our collaboration is productive we also offer tasks that include scraping of REST APIs. We reliably want to get trades in realtime from cryptocurrency exchanges. This trade information is then written into our databases using a specific format. We have an existing framework for this which allows us to parametrize the scraper. We work with Go and expect that the scrapers will work concurrently and efficiently.

We will provide the necessary documentation for the task and introduce you to our framework. https://simex.global/en/docs/introduction

Set-up instruction for the task can be found under: http://docs.diadata.org/scraper.html
Documentation

LBank API Scraping

We are looking for a Go developer that can provide scrapers of a websocket API.

We reliably want to get trades in realtime from cryptocurrency exchanges. This trade information is then written into our databases using a specific format. We have an existing framework for this which allows us to parametrize the scraper. We work with Go and expect that the scrapers will work concurrently and efficiently.

We will provide the necessary documentation for the task and introduce you to our framework. https://github.com/LBank-exchange/lbank-official-api-docs

Set-up instruction for the task can be found under: http://docs.diadata.org/scraper.html
Documentation

Tether Circulating Supply Scraper

Implement a Tether Blockchain scraper reporting the amount of circulating USDT coins using our Go framework. Detailed documentation and examples can be found at https://docs.diadata.org/blockchain-scrapers.html

Ripple Circulating Supply Scraper

Implement a Ripple Blockchain scraper reporting the amount of circulating XRP coins using our Go framework. Detailed documentation and examples can be found at https://docs.diadata.org/blockchain-scrapers.html

gate.io API Scraping

We are looking for a Go developer that can provide scrapers of a websocket API. If our collaboration is productive we also offer tasks that include scraping of REST APIs. We reliably want to get trades in realtime from cryptocurrency exchanges. This trade information is then written into our databases using a specific format. We have an existing framework for this which allows us to parametrize the scraper. We work with Go and expect that the scrapers will work concurrently and efficiently.

We will provide the necessary documentation for the task and introduce you to our framework.
https://www.gate.io/api2

Set-up instruction for the task can be found under: http://docs.diadata.org/scraper.html
Documentation

Add all pairs from exchanges

I noticed some exchanges are not scraping all the available pairs.

For example the most recent exchange gate.io (#47) only captures a small subset of the available pairs.

Instead of manually adding them to the exchange-scrapers.json file it's probably better to automate this using any available APIs such as the one linked above.

Bittrex REST API Scraper

We are looking for a Go developer that can provide a scraper for the bittrex REST API. We reliably want to get all trades in realtime from the Bittrex cryptocurrency exchange. This trade information is then written into our databases using a specific format. We have an existing framework for this which allows us to parametrize the scraper. We work with Go and expect that the scrapers will work concurrently and efficiently.

We will provide the necessary documentation for the task and introduce you to our framework.
The documentation for the bittrex API (and the endpoint /public/getmarkethistory) can be found here:
https://support.bittrex.com/hc/en-us/articles/115003723911-Developer-s-Guide-API

The scraper should work like the other ones in our repo, we are also happy to help with setting up.

Sync all blockchain scrapers

All blockchain scrapers should be synced in the backend. sshfs reconnect should be automatic.

Bitfinex API Scraping

We are looking for a Go developer that can provide scrapers of a websocket API. If our collaboration is productive we also offer tasks that include scraping of REST APIs. We reliably want to get trades in realtime from cryptocurrency exchanges. This trade information is then written into our databases using a specific format. We have an existing framework for this which allows us to parametrize the scraper. We work with Go and expect that the scrapers will work concurrently and efficiently.
We will provide the necessary documentation for the task and introduce you to our framework.

We will provide the necessary documentation for the task and introduce you to our framework. https://docs.bitfinex.com/docs

Set-up instruction for the task can be found under: http://docs.diadata.org/scraper.html
Documentation

ETH/EUR CME Future / Algorithm & Oracle

Develop ETH/EUR algorithm and oracle, based on availible DIA database, that is matching the CME future ETH price the closest https://www.cmegroup.com/trading/cryptocurrency-indices/cf-ethereum-reference-rate.html

• Use existing ECB reference data for EUR/USD rate on database (https://github.com/diadata-org/api-golang/blob/master/exchange-scrapers/scrapers/ECBScraper.go)
• Add Documentation
• Add Oracle, updating the price after ECB publishing (4pm CET) daily.

BitBay API scraper

We are looking for a Go developer that can provide scrapers for the BitBay exchange API.

We reliably want to get trades in realtime from the BitBay exchange. This trade information is then written into our databases using a specific format. We have an existing framework for this which allows us to parametrize the scraper. We work with Go and expect that the scrapers will work concurrently and efficiently.
The URL of the API can be found here: https://bitbay.net/de/public-api

The submitted work has to include the API scraper as well as a pull request with the respective documentation of the added code.

The DIA documentation on how to contribute can be found here: https://github.com/diadata-org/diadata

Dash Circulating Supply Scraper

Implement a Dash Blockchain scraper reporting the amount of circulating DSH coins using our Go framework. Detailed documentation and examples can be found at https://docs.diadata.org/blockchain-scrapers.html

Price Delta Discovery

Discovery for price differences of the same pair on different exchanges.

hitBTC API Scraping

We are looking for a Go developer that can provide scrapers of a websocket API. If our collaboration is productive we also offer tasks that include scraping of REST APIs. We reliably want to get trades in realtime from cryptocurrency exchanges. This trade information is then written into our databases using a specific format. We have an existing framework for this which allows us to parametrize the scraper. We work with Go and expect that the scrapers will work concurrently and efficiently.

We will provide the necessary documentation for the task and introduce you to our framework. https://api.hitbtc.com

Set-up instruction for the task can be found under: http://docs.diadata.org/scraper.html
Documentation

Cardano Circulating Supply Scraper

Implement a Cardano Blockchain scraper reporting the amount of circulating ADA coins using our Go framework. Detailed documentation and examples can be found at https://docs.diadata.org/blockchain-scrapers.html

EOS Circulating Supply Scraper

Implement a EOS Blockchain scraper reporting the amount of circulating EOS coins using our Go framework. Detailed documentation and examples can be found at https://docs.diadata.org/blockchain-scrapers.html

Stellar Circulating Supply Scraper

Implement a Stellar Blockchain scraper reporting the amount of circulating XLM coins using our Go framework. Detailed documentation and examples can be found at https://docs.diadata.org/blockchain-scrapers.html

Bitcoin Cash Supply data provider

Implement a Bitcoin Cash Blockchain scraper reporting the amount of mined BCC coins using our Go framework. Detailed documentation and examples can be found at http://docs.diadata.org/blockchain-scrapers.html

Coinhub: Design Fixes

For our site https://coinhub.diadata.org we are looking for some polishing. This task includes minor design changes in our vue.js site (can be found in frontend/) and the implementation of a dropdown menu, that selects different sources from our backend and displays them in our highstocks chart. A file with all changes layd out will be shared with the successful applicant for this bounty.
Ideally, we can cooperate by inviting you to our community slack channel.

www.zb.com api scrapping

We are looking for a Go developer that can provide a scraper for the zb.com exchange. Your contribution will be part of our open source project. We work with Go and expect that the scrapers will work concurrently and efficiently.

We would like you to use and evaluate this library in the process:
https://github.com/nntaoli-project/GoEx

We will provide the necessary documentation for the task and introduce you to our framework.
Set-up instruction for the task can be found under: http://docs.diadata.org/scraper.html

DIA Whitepaper | Critizise and review

Experts in data protection, tokenomics, crowd-sourcing and crowd-wisdom wanted.

We are looking for interested editors, freelancers, developers in one of these areas: crowd-wisdom, tokenomics, data protection (GDPR) in order to criticise and improve our Whitepaper.

Apply to this ticket with a short intro and why you are a good fit.

Whitepaper will be send after selection to 1 freelancer, we might to into subsequent rounds.

Tezos Blockchain scraper

Implement a Tezos Blockchain scraper reporting the amount of circulating XTZ coins using our Go framework. Detailed documentation and examples can be found at https://github.com/diadata-org/diadata/blob/master/documentation/methodology/SupplyNumbers.md

Reorganisation of database and collection backend

Finish reorganisation of the backend servers for the API services and the price and supply collection. Add documentation on docs.diadata.org about the architecture of the backend (e.g, which services use redis, how are they connected).

Pair Discovery

29.10: FG

Monero Circulating Supply Scraper

Implement a Monero Blockchain scraper reporting the amount of circulating XMR coins using our Go framework. Detailed documentation and examples can be found at https://docs.diadata.org/blockchain-scrapers.html

tansparency in plot project

get the transparency change made into the
https://github.com/diadata-org/plot
go upstream with a PR so we don't need to use a local version

Distribution Algos: Generalized hyperbolic; Normal inverse Gaussian; Generalized t; Laplace; Student t

1. Develop an algorithm that implements the described distributions in line with "A Statistical Analysis of Cryptocurrencies", https://www.mdpi.com/1911-8074/10/2/12

• P. 7 (Formulas)
• P.11 (Check for backtesting)

2. Integrate it with our filter service.

3. Backtest the documentation, submit respective documentation as pull request.

Kucoin API Scraper

We are looking for a Go developer that can provide scrapers of the Kucoin websocket API. If our collaboration is productive we also offer tasks that include scraping of REST APIs. We reliably want to get trades in realtime from cryptocurrency exchanges. This trade information is then written into our databases using a specific format. We have an existing framework for this which allows us to parametrize the scraper. We work with Go and expect that the scrapers will work concurrently and efficiently.

We will provide the necessary documentation for the task and introduce you to our framework. https://kucoinapidocs.docs.apiary.io/#introduction/websocket

Set-up instruction for the task can be found under: https://docs.diadata.org/blockchain-scrapers.html

coinhub/coin details spinner

When I am exploring plots of different currencyes whenever I change a parameter i.e exchange or base currency it takes 1-2 Second to update the plots. There is no feedback that the website is working so its very confusing. Adding a spinner or a please wait pop up should help

Listing exchanges API endpoint

API endpoint that lists all exchanges (with pairs that are watched) from where DIA is getting trade data. The endpoint should include price and volume data of the pair.

ETC Blockchain Scraper

Implement a Ethereum Classic Blockchain scraper reporting the amount of circulating ETC coins using our Go framework. Detailed documentation and examples can be found at https://github.com/diadata-org/diadata/blob/master/documentation/methodology/SupplyNumbers.md

API endpoint for 7d Graphs

Create an API endpoint for the 7d history of the price of every coin tracked in the backend. It should return a PNG (transparent background).

Implement Close() for all exchange scrapers

Not all Exchange scrapers properly implemented the Close() function.
Some exchanges do implement close but do not terminate the mainLoop() routine which cause performance issues on the pairDiscoveryService service

NEO Circulating Supply Scraper

Implement a NEO Blockchain scraper reporting the amount of circulating NEO coins using our Go framework. Detailed documentation and examples can be found at https://docs.diadata.org/blockchain-scrapers.html

FilterMA not ignoring Trade out of order

In func (s *FilterMA) compute(trade dia.Trade) if the trade timestamp is older than s.lastTrade should be ignored. In the following test the error message FilterMA: Ignoring Trade out of order is shown however the values are being used on the average computation which mean it is not ignored.

func TestFilterMaIgnore(t *testing.T) {

	filterParam := 10
	firstPrice := 50.0

	d := time.Date(2016, time.August, 15, 0, 0, 0, 0, time.UTC)
	f := NewFilterMA("XRP", "", d, filterParam)
	steps := filterParam
	p := firstPrice
	priceIncrements := 1.0
	for i := 0; i <= steps; i++ {
		f.compute(dia.Trade{EstimatedUSDPrice: p, Time: d})
		d = d.Add(-time.Second)
		p += priceIncrements
	}
	f.finalComputeEndOfBlock(d)
	v := f.filterPointForBlock()
	if v.Value != firstPrice {
		t.Errorf("error should be initial value:%f got:%f", firstPrice, v.Value)
	}
}

OKEx API Scraping

We are looking for a Go developer that can provide scrapers of a websocket API. If our collaboration is productive we also offer tasks that include scraping of REST APIs. We reliably want to get trades in realtime from cryptocurrency exchanges. This trade information is then written into our databases using a specific format. We have an existing framework for this which allows us to parametrize the scraper. We work with Go and expect that the scrapers will work concurrently and efficiently.

We will provide the necessary documentation for the task and introduce you to our framework. https://github.com/okcoin-okex/API-docs-OKEx.com

Set-up instruction for the task can be found under: http://docs.diadata.org/scraper.html
Documentation

CoinData page

Re-structure the page "CoinDetails" so that it shows the general information at the top, a history chart below, and the market details on bottom. In detail these are the general building blocks:

• The general information section will be discussed in detail for smaller design changes.
• Integrate live market data into the already existing template from this API endpoint: https://api.diadata.org/v1/symbol/BTC (of course there is one for every currency). There should be a table with a line for each exchange displaying the name of the exchange, price, and priceYesterdayvolumeYesterdayUSD from the API.
• When clicking on an exchange name, a list of the last 10 trades should show up. These are located at "LastTrades" for each exchange in the json the same API call from above returns. This list should include all information in the JSON at that location.
• The price history chart should be done using Highstocks, our API will provide a json object with historical data.

test

test

Realized Cap of Bitcoin (and, if possible, of other cryptocurrencies)

Is your feature request related to a problem? Please describe.
At the Honeybadger 2018 conference Nic Carter presented the realized cap defined as such (by David Puell):
Instead of counting all of the mined coins at equal, current price, the UTXOs are aggregated and assigned a price based on the BTCUSD market price at the time when said UTXOs last moved.

I tried to model this metric by examining the positive parts of the first derivative of the UTXO-Set (i.e. trying to look at the new UTXOs and not at all of them) and mutiplying it with a mean UTXO-value. After that I made a cumulative sum.

The result is similar to the realized cap but not the same.

Describe the solution you'd like
Similar to the market cap a metric where you can see the realized cap in a fast manner (eg. for calculating the MVRV)

Describe alternatives you've considered
Technically the realized cap is "just another metric". For instance the NVT-Ratio by Willy Woo and the MVRV-Ratio by David Puell behave quite similar, so there are alternatives.

The regular Market Cap as well as the Thermo Cap (cumulative Sum of the value of all Mining Rewards by the time they were paid out) are completely different metrics and more of upper/lower bounds to this value.

Additional context
Some interesting links for this issue:
https://blog.goodaudience.com/bitcoin-market-value-to-realized-value-mvrv-ratio-3ebc914dbaee
https://www.youtube.com/watch?v=D2WXxgZ8h-0&t=78m8s

TRON Circulating Supply Scraper

Implement a TRON Blockchain scraper reporting the amount of circulating TRX coins using our Go framework. Detailed documentation and examples can be found at https://docs.diadata.org/blockchain-scrapers.html

Price and Volume Oracles for coin data

Write an oracle smart contract and backend integration so that every coin's data gets updated periodically. Add documentation and a subpage on the functionality of the oracle with a link to Etherscan.

Issue Retrieving configFile in scraper.go

Hi there,

• I found an issue when i try to retrieve configFile , initialized inside scraper.go

• scraper.go, Row 55:

  • configFile := os.Getenv("GOPATH") + "/go/src/github.com/diadata-org/api-golang/exchange-scrapers/config/exchange-scrapers.json"

• There are two issues:

  • 1. os.Getenv("GOPATH") can return empty string, from go 1.8.tested in Mac OS.
  • It should be fixed with build.Default.GOPATH
  • see it -> https://stackoverflow.com/questions/32649770/how-to-get-current-gopath-from-code#32650077
  • 2. The hardcoded string starts with "/go", it is duplicate because it is returned also from "GOPATH"

• I tested the code fixing these issues and it works.

[EDIT]

• I exported GOPATH and it works now , also with os.Getenv("GOPATH"). It remains only the second issue.
• For the first one maybe it could be useful to add a check to set default path in case of empty string returned, exactly how the stackoverflow example above.

Use $GOPATH instead of hardcoded path

In exchange-scrapers/cmd/scraper.go#L55-L56 there is a hardcoded path which will not work for everyone, I modified it to use the $GOPATH environment variable instead.

configFile := os.Getenv("GOPATH") + "/src/github.com/diadata-org/api-golang/exchange-scrapers/config/exchange-scrapers.json"

Maybe it should be set by a flag if $GOPATH is not set which may be the future of Go as module support becomes more commonplace.

Frontend for Gitcoin Bounty Explorer

Write a frontend for the Gitcoin bounty API that shows open diadata-org bounties as "widget" (i.e. JS element) embedded in our wordpress site. The gitcoin API is documented here.

The widget should have a selector for showing "open" bounties and "all" bounties (so open, currently in work, and closed ones). These can be retrieved through the API.

Secondly, a form should be made where a new bounty can be submitted, that ideally pre-fills the form at gitcoin.co and submits it there. The objective is to send out bounties through our site but they are listed as normal gitcoin bounties.

We can give you access to a wordpress instance for deployment. Expect several feedback rounds to make everything work smoothly. We will invite you to our Slack channel for easier communication.

Pair Scraper for all supported exchanges

This issue evolved from #52.
We need a pair discovery service that fetches all available pairs for each exchange and store it in redis.
It should regularly check which pairs are tradable and update the database if necessary. Whenever a dedicated API endpoint is available, please use it. Document your solution in our docs/ directory.