Cannot list datasets beyond page 500

[lbhm]

Cannot list datasets beyond page 500

[jplotts]

Hi @lbhm - it's true that there are limits on how far back you can search via the API. We may look into revising those in the future, but there's no definitive timeline at this point. One workaround could be using the Meta Kaggle dataset. Admittedly, this would only allow you to query titles, so it's a bit less than you'd get from kaggle.com, but perhaps it's sufficient for your purpose.

[lbhm]

Thank you for the quick response @jplotts.

Unfortunately, just having titles/IDs is not quite enough for us. In this research project, we are looking into dataset search techniques that go beyond basic keyword search, so I am especially interested in column statistics (e.g., ranges, number of unique values, null values, etc.) to explore new search techniques.

Please allow me two follow-up questions:

• In the web UI, you are already showing detailed column statistics, such as value distribution histograms. Are there any plans to make these statistics available via the API in the near or mid-term? A simple list representation of histograms (like in numpy for example) would already be really helpful for us.
• Is there an upper limit on how many datasets I am allowed to download to generate such column statistics myself? Given the list of all dataset IDs from the Meta Kaggle dataset, I would also be fine with downloading them and computing statistics on our own hardware.

[jplotts]

Hi @lbhm -

1. I agree that the column statistics would be nice, but unfortunately, that's not on our roadmap right now.
2. We have rate limits on our API endpoints to prevent abuse. We don't track the total number of downloads a user makes, so you can download an arbitrary amount as long as you spread the requests out. I can't provide specific rate limit numbers (they may change anyway), but we believe they are sufficient for most reasonable use cases.

[lbhm]

Alright, I'll download the datasets with an appropriate rate and recreate the column statistics myself then.

Thank you for your help!