Comments (14)
Sorry for the delayed response, am in a pretty hectic time period at the moment transitioning between work.
I only use the CMC API a few times throughout the crypto package when i'm required to get the listing of all coins/slugs or the current prices and these shouldn't be too hard to identify a solution for (crypto_prices, daily_market, global_market or crypto_list
).
All the historical data and exchange functions use the crypto_list
once on initiation to establish coin listing then are just reliant on the HTTP request limits for scraping historical data - I think the free version of their professional API would definitely support most usage, assuming that normal HTTP requests don't eat into the API credits..
I just wish there was a way to be able to identify how often this package gets used so I could work out how many calls get made through it to work out a more scalable solution for everyone when it comes to the historical/exchange data.
Interested in your thoughts
from crypto.
I can't imagine CMC will remove functionality from their site just because of the professional API.
I've been demoing some code that i'd be glad to get some feedback on around ways to serve the crypto_list function..
Please note its dependent that you're using Rstudio, and uses interactive prompt to add api key..
This effectively will replace these lines in the current crypto_list function which use the old API...
json <- "https://s2.coinmarketcap.com/generated/search/quick_search.json"
coins <- use_rate_limit(jsonlite::read_json(json, simplifyVector = TRUE))
At the minimum for the existing history information theres two options I see
-
- Using a solution similar to the below for allowing secure provision of coinmarketcap apikey using their free package.
-
- Create my own API endpoint or REST service to provide the coin slugs.
Let me know how the below goes and what your thoughts are..
## DEPENDENCIES
# deb: libsodium-dev (Debian, Ubuntu, etc)
# rpm: libsodium-devel (Fedora, EPEL)
# csw: libsodium_dev (Solaris)
# brew: libsodium (OSX)
# install.packages("rio")
# install.packages("httr")
library(tidyverse)
library(utils)
# Prompt to install keyring dependencies
install_keyring <- function() {
vers <- version$minor %>% as.numeric()
dependent_packages <- c("keyring","rstudioapi")
if(vers >= 5.1) {
message("Crypto highly recommends you install keyring to manage your API keys.", appendLF = TRUE)
prompt <- askYesNo("Install keyring?")
if (prompt == "TRUE") {
utils::install.packages(dependent_packages, dependencies = TRUE)
}
} else {
utils::install.packages(dependent_packages, dependencies = TRUE)
}
}
# Professional CMC API call
cmc_api <- function(url) {
jsonlite::fromJSON(
httr::content(
httr::GET(url, httr::add_headers(
"X-CMC_PRO_API_KEY" = rstudioapi::askForSecret("coinmarketcap-apikey")),
httr::accept_json()), "text"))
}
# Checks keyring installation status, installs and then makes API call to listings
get_updated_coinlist <- function() {
packages <- c(is.element("keyring", installed.packages()[,1]),
is.element("rstudioapi", installed.packages()[,1]))
if (all(packages)) {
library(keyring)
library(rstudioapi)
} else {
install_keyring()
library(keyring)
library(rstudioapi)
}
base <- "https://pro-api.coinmarketcap.com"
endpoint <- "/v1/cryptocurrency/listings/latest"
params <- "?start=1&limit=5000&convert=USD"
url <- paste0(base, endpoint, params)
listings <- cmc_api(url)$data %>%
as.data.frame() %>%
dplyr::select(id, name, symbol, slug, cmc_rank)
names(listings) <- c("id","name","symbol","slug","rank")
rio::export(listings, "coin_list.feather")
return(listings)
}
coin_list <- get_updated_coinlist()
from crypto.
Any update on this?
Happy to donate some $$ if Jesse can get this switched over to Gecko or Nomics before CMC screws us over (couple weeks I think).
CMC is going to be so expensive it’ll basically be useless.
If Jesse can’t do it does anyone know anyone that can?
from crypto.
I used the package daily, just the getCoins function. But my understanding is that won’t be under the free version. OHLC type data is on their midtier plan.
Plus they now have monthly limits. I’m not exactly sure how they define a call (or how it works through the package), but I may be grabbing 500 coins per day via getCoins
from crypto.
Yeah getCoins
or crypto_history
uses just 1 API call per call of the function just to populate the list and generate the links, and then it goes and scrapes through generic HTTP requests, so should be safe there, it just scrapes all the data out of the HTML tables from the historical data page available here (https://coinmarketcap.com/currencies/bitcoin/historical-data/)[https://coinmarketcap.com/currencies/bitcoin/historical-data/], then it cleans it up and transforms a bit.
i.e. if you grabbed 500 coins per day in one go that would only be 1-5 API calls based on what I can see (saw some note about being 1 API credit per 100 coins or something similar).
from crypto.
Sounds good.
What about the fact that CMC says OHLC and historical data are only on their paid plans?
from crypto.
You should not have rate limit issue by going with our http://coingecko.com/api
We will be rolling out OHLC soon as a new endpoint
from crypto.
from crypto.
$299 a month to get historical data. I'm just trying to learn R and market patterns. They have lost their minds. +1 for CoinGecko support.
I can't figure out how to grab Total Market Capitalization with CMC anyway. Bitcoin loosely substitutes, but it's proof that CMC is not the end all.
@JesseVent If you figure out a way to charge BAT for your program calls, I'm up for that hustle. Perhaps that's an RStudio improvement request though. :-)
from crypto.
Jesse I'm afraid I'm not much help on the coding front.
Given today is the day CMC migrates... what do we need to do? What functionality will we lose?
from crypto.
FYI crypto_history still seems to work as intended. Today is supposed to be the day for the migration
I didn't re-download the package or anything.
from crypto.
i've added an extra attribute called coin_list to crypto_history(coin_list=NULL)
so that there are now 3 options available when getting the historical data.
Valid options for crypto_history(coin_list=c("api,"static",NULL)
coin_list="api"
will get the latest coin list using the professional version of the CMC API and will prompt for an API key,coin_list="static"
will use internally sourced data to populate the coin list but this will only provide the current list as of today.coin_list=NULL
which is the default option, will continue to use the existing public API endpoint for backwards compatibility.
If errors occur then you should change to the
static
orapi
options.
devtools::install_github("jessevent/crypto")
will install these enhancements.
from crypto.
GetCoins is failing now with error message: Error in unserialize(socklist[[n]]) : error reading from connection
is that related to the above messages?
from crypto.
@MR1991 That error's due to the rate limiter - basically CMC implemented a limit of 30 http requests a minute so parallel processing across cores is now redundant. I just did a fresh install from CRAN and ran following df <- getCoins(limit=20, cpu_cores = 1)
and works fine still.
I have to do another release and strip out parallel processing, and strip it back before resubmitting to CRAN.
from crypto.
Related Issues (20)
- Bind_rows issue HOT 3
- No package called 'plyr' issue
- no package called 'plyr' issue
- No package called 'plyr' issue
- Inconsistent and incomplete results from getCoins() and crypto_history() HOT 14
- Error in get(as.character(FUN), mode = "function", envir = envir) : object 'bind_rows' of mode 'function' was not found HOT 2
- error when loading library(crypto), HOT 5
- Could not find function "daily_market" HOT 1
- Error in Sys.setlocale("LC_TIME", locale) HOT 8
- crypto_history skipping coins again HOT 3
- Include historic data as .rda file HOT 1
- Crypto History Error HOT 2
- Subscript out of bounds error HOT 24
- Coinmarketcap API deprecated, library no longer working HOT 1
- Some missing tokens
- The package is no longer available on CRAN HOT 1
- crypto_history not working HOT 16
- Still working? HOT 1
- Crypto_histoty function is not working, could you please update? Thanks HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from crypto.