curran/data

A collection of public data sets

A collection of public data sets for testing out visualization methods. These data sets are at various stages of preparation, some are just raw data, some are CSV files, and some are exposed as AMD modules. This collection is messy, but with some digging you may find hidden gems.

Interesting Datasets

Most recently added on the top.

• Gothenburg Quality of Government Expert Survey
• UN Human Development Index
• Vega Datasets
• Data sources by EfoxMaps
• RawGraphs Sample Datasets
• nomis - UK Census data from 2021 (via Twitter)
• Cantometrics Data, from The Global Jukebox, music database
• Observable Curated Datasets | Old Observable Curated Datasets
• MFRED Electricity Usage Dataset
• BBC Shared Data Unit
• Most Popular Operating Systems
• Remaking Figures from Semiology of Graphics
• Awesome Public Datasets
• FMA: A Dataset For Music Analysis
• Food Nutrition Data
• Historical Weather Warnings
• PM2.5 Air Quality by Country over Time
• US Energy Information Administration Data (see also Analysis & Projections)
• Climate.gov Datasets
• The Economist Graphic Detail data
• Dataset collection: SPORTS DATA SETS FOR DATA MODELING, VISUALIZATION, PREDICTIONS, MACHINE-LEARNING.
• Dataset collection: information is beautiful - Data
• Dataset collection: R for Data Science Tidy Tuesdays
• Stranger Things Ratings
• SIPRI Arms Transfers Database
• CWUR - World University Rankings 2019-2020
• TopoJSON Collection World countries and subdivisions
• Classic datasets from Petra Isenberg et. al.
• Soul of the Community (American Statistical Association)
• World Population Prospects (United Nations)
• Employment (Bureau of Labor Statistics)
• Healthy People (Centers for Disease Control)
• GapMinder Data
• NASA Satellite-Derived Environmental Indicators
• IMF Public Finances in Modern History Database
• Executions in the US by type over time
• Datasets used in the book, An Introduction to Categorical Data Analysis
• Energy Information Administration Open Data
• Data sets from Five Thirty Eight
• Data sets in the Infovis Wiki
• Data sets from Andy Kirk's Link Archive
• Makeover Monday Datasets
• SOCR Datasets
• UCI Machine Learning Repository Datasets
• BrightKite User Check-ins (57.2 MB)
• ACLED (Armed Conflict Location and Event Data Project) (35MB)
• Safecast (3.2GB)
• Statistical Computing Statistical Graphics Data expo Airline on-time performance (12GB)
• The GDELT Data Set (~100GB)
• The Indian Census 2011
• Best Buy Developer API

Leads

These are "leads" to find interesting datasets. They have teasers of cool data, but it will take some work to find the data behind them.

• Social Explorer

data's Issues

Scrape Asylum applications submitted in Europe 2010-2014 data

http://www.telegraph.co.uk/news/worldnews/europe/11822343/Mapped-Where-do-migrants-apply-for-asylum-in-Europe.html

Add India Census Dataset

http://web.archive.org/web/20040616075334/http://www.censusindia.net/results/town.php?stad=A&state5=999

Really cool! What is the LICENSE?

This is a really cool collection @curran. What is the LICENSE? I understand that the datasets themselves might be licensed differently, but what is this repository licensed under?

Import Coxcomb Plot Data

http://understandinguncertainty.org/node/214

vegaExamples flights

The flight JSON files in the vegaExamples directory are, I believe, subsets of the corssFilter data here: https://github.com/square/crossfilter/tree/gh-pages/ which is 230k rows. This itself a subset of the ASA Data Expo dataset.

The ASA dataset is very big, but you might consider adding your own large subset from the original source.

As for the vegaExamples JSON files they are formatted poorly. Would you consider a PR these PrettyPrints the JSON files?

• I'm using this data here: http://hypercubed.github.io/Project-Chi/#/examples/universe *

Establish Order

Currently this collection of data sets is very chaotic. However, there has reached a critical mass of individual examples such that it might now be possible to distill the commonalities between them and form a coherent general structure out of them.

There are two main things to deal with: directory structure and file structure.

Directory Structure

Ideally, the directory structure should be able to be traversed by some program in order to build an index of all the data sets. In order to make this straightforward, one solution might be to introduce a fixed hierarchical structure like this:

{{dataSource}}/{{dataCollection}}/{{dataSet}}

where

• dataSource represents the organization that originally published the data.
• dataCollection represents a category of data sets published
• dataSet represents an individual data table

File Structure

The dsv-dataset project provides a metadata specification for annotating data sets with column types so they can be automatically parsed. The file structure should leverage dsv-dataset.

Perhaps each data set could have two files, one with the CSV data, and one with the metadata, like this:

{{dataSource}}/{{dataCollection}}/{{dataSet}}/data.csv
{{dataSource}}/{{dataCollection}}/{{dataSet}}/metadata.json

The disadvantage of this is that folks who want to move the .csv file into a different context will need to spend time thinking about what to name it, or just leave it as data.csv, which is rather generic.

Alternatively, the files could take on the name of the data set, like this

{{dataSource}}/{{dataCollection}}/{{dataSet}}/{{dataSet}}.csv
{{dataSource}}/{{dataCollection}}/{{dataSet}}/{{dataSet}}.json

As yet another alternative, data sets and their metadata could be combined into a single JSON file, whose contents might look something like this:

{
  "dsvString":"sepal_length,sepal_width,petal_length,petal_width,class\n5.1,3.5,1.4,0.2,setosa\n4.9,3.0,1.4,0.2,setosa\n4.7,3.2,1.3,0.2,setosa\n5.7,2.8,4.1,1.3,versicolor\n6.3,3.3,6.0,2.5,virginica\n5.8,2.7,5.1,1.9,virginica\n7.1,3.0,5.9,2.1,virginica",
  "metadata":{
    "delimiter": ",",
    "columns": [
      { "name": "sepal_length", "type": "number" },
      { "name": "sepal_width",  "type": "number" },
      { "name": "petal_length", "type": "number" },
      { "name": "petal_width",  "type": "number" },
      { "name": "class",        "type": "string" }
    ]
  }
}

This could live in a single file:

{{dataSource}}/{{dataCollection}}/{{dataSet}}/{{dataSet}}.json

The disadvantage of this approach is that anyone who wants to just get the CSV data out will need to write some code, rather than just copy an existing CSV file.

Alternatively, the data sets could live in the data collection directory, like this:

{{dataSource}}/{{dataCollection}}/{{dataSet}}.csv
{{dataSource}}/{{dataCollection}}/{{dataSet}}.json

This would make the data collection directories kind of messy. Also, having a README.md for each level might be a good thing too, which would make if favorable to have each data set reside in its own directory, like this:

{{dataSource}}/README.md
{{dataSource}}/{{dataCollection}}/README.md
{{dataSource}}/{{dataCollection}}/{{dataSet}}/README.md

Also, it might be nice to have an index at each level, so programs can query for what is there. These files could simply contain arrays of strings. This would make the full file layout look something like this:

{{dataSource}}/README.md
{{dataSource}}/dataCollections.json
{{dataSource}}/{{dataCollection}}/README.md
{{dataSource}}/{{dataCollection}}/dataSets.json
{{dataSource}}/{{dataCollection}}/{{dataSet}}/README.md
{{dataSource}}/{{dataCollection}}/{{dataSet}}/data.csv
{{dataSource}}/{{dataCollection}}/{{dataSet}}/metadata.json

Trees

What about cases where a data cube is partitioned across files, where each file contains a portion of the fact table where a certain dimension equals a certain value? For example, a data set may be partitioned across many files, one file per year. Or the partitioning could use one file per geographic region. Perhaps the directory tree can be built in such a way that it is possible to have a directory full of CSV files within a given data set, and all of them can share the same metadata file.

Letter Frequency

Is the letter frequency data set in here somewhere? http://bl.ocks.org/mbostock/3885304