CMC "paid" plan and API key

[jessevent]

CMC "paid" plan and API key

[JesseVent]

Sorry for the delayed response, am in a pretty hectic time period at the moment transitioning between work.

I only use the CMC API a few times throughout the crypto package when i'm required to get the listing of all coins/slugs or the current prices and these shouldn't be too hard to identify a solution for (crypto_prices, daily_market, global_market or crypto_list).

All the historical data and exchange functions use the crypto_list once on initiation to establish coin listing then are just reliant on the HTTP request limits for scraping historical data - I think the free version of their professional API would definitely support most usage, assuming that normal HTTP requests don't eat into the API credits..

I just wish there was a way to be able to identify how often this package gets used so I could work out how many calls get made through it to work out a more scalable solution for everyone when it comes to the historical/exchange data.

Interested in your thoughts

[JesseVent]

I can't imagine CMC will remove functionality from their site just because of the professional API.

I've been demoing some code that i'd be glad to get some feedback on around ways to serve the crypto_list function..

Please note its dependent that you're using Rstudio, and uses interactive prompt to add api key..

This effectively will replace these lines in the current crypto_list function which use the old API...

json   <- "https://s2.coinmarketcap.com/generated/search/quick_search.json"
coins  <- use_rate_limit(jsonlite::read_json(json, simplifyVector = TRUE))

At the minimum for the existing history information theres two options I see

1. • Using a solution similar to the below for allowing secure provision of coinmarketcap apikey using their free package.
2. • Create my own API endpoint or REST service to provide the coin slugs.

Let me know how the below goes and what your thoughts are..

## DEPENDENCIES
# deb: libsodium-dev (Debian, Ubuntu, etc)
# rpm: libsodium-devel (Fedora, EPEL)
# csw: libsodium_dev (Solaris)
# brew: libsodium (OSX)

# install.packages("rio")
# install.packages("httr")
library(tidyverse)
library(utils)

# Prompt to install keyring dependencies
install_keyring <- function() {
  vers <- version$minor %>% as.numeric()
  dependent_packages <- c("keyring","rstudioapi")
  if(vers >= 5.1) {
    message("Crypto highly recommends you install keyring to manage your API keys.", appendLF = TRUE)
    prompt <- askYesNo("Install keyring?")
    if (prompt == "TRUE") {
      utils::install.packages(dependent_packages, dependencies = TRUE)
    }    
  } else {
    utils::install.packages(dependent_packages, dependencies = TRUE)
  }
}

# Professional CMC API call
cmc_api  <- function(url) {
  jsonlite::fromJSON(
    httr::content(
      httr::GET(url, httr::add_headers(
        "X-CMC_PRO_API_KEY" = rstudioapi::askForSecret("coinmarketcap-apikey")),
        httr::accept_json()), "text"))
}

# Checks keyring installation status, installs and then makes API call to listings
get_updated_coinlist <- function() {
  packages <- c(is.element("keyring", installed.packages()[,1]),
                is.element("rstudioapi", installed.packages()[,1]))
  
  if (all(packages)) {
    library(keyring)
    library(rstudioapi)
  } else {
    install_keyring()
    library(keyring)
    library(rstudioapi)
  }
  
  base     <- "https://pro-api.coinmarketcap.com"
  endpoint <- "/v1/cryptocurrency/listings/latest"
  params   <- "?start=1&limit=5000&convert=USD"
  url      <- paste0(base, endpoint, params)
  
  listings <- cmc_api(url)$data %>%
    as.data.frame() %>%
    dplyr::select(id, name, symbol, slug, cmc_rank)
  names(listings) <- c("id","name","symbol","slug","rank")
  rio::export(listings, "coin_list.feather")
  return(listings)
}

coin_list <- get_updated_coinlist()

[thelmmortal1]

Any update on this?

Happy to donate some $$ if Jesse can get this switched over to Gecko or Nomics before CMC screws us over (couple weeks I think).

CMC is going to be so expensive it’ll basically be useless.

If Jesse can’t do it does anyone know anyone that can?

[thelmmortal1]

I used the package daily, just the getCoins function. But my understanding is that won’t be under the free version. OHLC type data is on their midtier plan.

Plus they now have monthly limits. I’m not exactly sure how they define a call (or how it works through the package), but I may be grabbing 500 coins per day via getCoins

[JesseVent]

Yeah getCoins or crypto_history uses just 1 API call per call of the function just to populate the list and generate the links, and then it goes and scrapes through generic HTTP requests, so should be safe there, it just scrapes all the data out of the HTML tables from the historical data page available here (https://coinmarketcap.com/currencies/bitcoin/historical-data/)[https://coinmarketcap.com/currencies/bitcoin/historical-data/], then it cleans it up and transforms a bit.

i.e. if you grabbed 500 coins per day in one go that would only be 1-5 API calls based on what I can see (saw some note about being 1 API credit per 100 coins or something similar).

[thelmmortal1]

Sounds good.

What about the fact that CMC says OHLC and historical data are only on their paid plans?

[tmlee]

You should not have rate limit issue by going with our http://coingecko.com/api
We will be rolling out OHLC soon as a new endpoint

[thelmmortal1]

Ok I’ll check it out later today I did email cmc and they said OHLC and historical will be on the paid plans. Seems hard to believe given how many free options there are

…

Sent from my iPhone

On Dec 1, 2018, at 6:01 AM, Jesse ***@***.***> wrote: I can't imagine CMC will remove functionality from their site just because of the professional API. I've been demoing some code that i'd be glad to get some feedback on around ways to serve the crypto_list function.. Please note its dependent that you're using Rstudio, and uses interactive prompt to add api key.. This effectively will replace these lines in the current crypto_list function which use the old API... json <- "https://s2.coinmarketcap.com/generated/search/quick_search.json" coins <- use_rate_limit(jsonlite::read_json(json, simplifyVector = TRUE)) At the minimum for the existing history information theres two options I see Using a solution similar to the below for allowing secure provision of coinmarketcap apikey using their free package. Create my own API endpoint or REST service to provide the coin slugs. Let me know how the below goes and what your thoughts are.. ## DEPENDENCIES # deb: libsodium-dev (Debian, Ubuntu, etc) # rpm: libsodium-devel (Fedora, EPEL) # csw: libsodium_dev (Solaris) # brew: libsodium (OSX) # install.packages("rio") # install.packages("httr") library(tidyverse) library(utils) # Prompt to install keyring dependencies install_keyring <- function() { vers <- version$minor %>% as.numeric() dependent_packages <- c("keyring","rstudioapi") if(vers >= 5.1) { message("Crypto highly recommends you install keyring to manage your API keys.", appendLF = TRUE) prompt <- askYesNo("Install keyring?") if (prompt == "TRUE") { utils::install.packages(dependent_packages, dependencies = TRUE) } } else { utils::install.packages(dependent_packages, dependencies = TRUE) } } # Professional CMC API call cmc_api <- function(url) { jsonlite::fromJSON( httr::content( httr::GET(url, httr::add_headers( "X-CMC_PRO_API_KEY" = rstudioapi::askForSecret("coinmarketcap-apikey")), httr::accept_json()), "text")) } # Checks keyring installation status, installs and then makes API call to listings get_updated_coinlist <- function() { packages <- c(is.element("keyring", installed.packages()[,1]), is.element("rstudioapi", installed.packages()[,1])) if (all(packages)) { library(keyring) library(rstudioapi) } else { install_keyring() library(keyring) library(rstudioapi) } base <- "https://pro-api.coinmarketcap.com" endpoint <- "/v1/cryptocurrency/listings/latest" params <- "?start=1&limit=5000&convert=USD" url <- paste0(base, endpoint, params) listings <- cmc_api(url)$data %>% as.data.frame() %>% dplyr::select(id, name, symbol, slug, cmc_rank) names(listings) <- c("id","name","symbol","slug","rank") rio::export(listings, "coin_list.feather") return(listings) } coin_list <- get_updated_coinlist() — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.

[woodydeck]

$299 a month to get historical data. I'm just trying to learn R and market patterns. They have lost their minds. +1 for CoinGecko support.

I can't figure out how to grab Total Market Capitalization with CMC anyway. Bitcoin loosely substitutes, but it's proof that CMC is not the end all.

@JesseVent If you figure out a way to charge BAT for your program calls, I'm up for that hustle. Perhaps that's an RStudio improvement request though. :-)

[thelmmortal1]

Jesse I'm afraid I'm not much help on the coding front.

Given today is the day CMC migrates... what do we need to do? What functionality will we lose?

[thelmmortal1]

FYI crypto_history still seems to work as intended. Today is supposed to be the day for the migration

I didn't re-download the package or anything.

[JesseVent]

i've added an extra attribute called coin_list to crypto_history(coin_list=NULL) so that there are now 3 options available when getting the historical data.

Valid options for crypto_history(coin_list=c("api,"static",NULL)

1. coin_list="api" will get the latest coin list using the professional version of the CMC API and will prompt for an API key,
2. coin_list="static" will use internally sourced data to populate the coin list but this will only provide the current list as of today.
3. coin_list=NULL which is the default option, will continue to use the existing public API endpoint for backwards compatibility.

If errors occur then you should change to the static or api options.

devtools::install_github("jessevent/crypto") will install these enhancements.

[MR1991]

GetCoins is failing now with error message: Error in unserialize(socklist[[n]]) : error reading from connection
is that related to the above messages?

[JesseVent]

@MR1991 That error's due to the rate limiter - basically CMC implemented a limit of 30 http requests a minute so parallel processing across cores is now redundant. I just did a fresh install from CRAN and ran following df <- getCoins(limit=20, cpu_cores = 1) and works fine still.

I have to do another release and strip out parallel processing, and strip it back before resubmitting to CRAN.