datasets/world-cities

List of major cities of the world as a datapackage

List of major cities in the world

Data

The data is extracted from geonames, a very exhaustive list of worldwide toponyms.

This datapackage only list cities above 15,000 inhabitants. Each city is associated with its country and subcountry to reduce the number of ambiguities. Subcountry can be the name of a state (eg in United Kingdom or the United States of America) or the major administrative section (eg ''region'' in France''). See admin1 field on geonames website for further info about subcountry.

Notice that :

• some cities like Vatican city or Singapore are a whole state so they don't belong to any subcountry. Therefore subcountry is N/A.
• There is no guaranty that a city has a unique name in a country and subcountry (At the time of writing, there are about 60 ambiguities). But for each city, the source data primary key geonameid is provided.

Preparation

Preparation

This repository uses dataflows to process and normalize the data.

You first need to install the dependencies:

pip install -r scripts/requirements.txt

Then run the script

python scripts/process.py

License

All data is licensed under the Creative Common Attribution License as is the original data from geonames. This means you have to credit geonames when using the data. And while no credit is formally required a link back or credit to Lexman and the Open Knowledge Foundation is much appreciated.

All source code is licensed under the MIT licence.

world-cities's Issues

What should the licence be ?

The recommandations (http://data.okfn.org/doc/publish-faq#-strong-license-strong-) is to licence the data under PDDL and to add a reminder of the original licence of the source.

But @tfmorris noticed that the original data is CC-BY source, which meansthe derivative should be CC-BY as well. I accepted his changes because I tend to agree with him, but does anyone (@rgrp? @pdehaye ? anyone else ?) have an idea of the right licence ?

Use Goodtables to continuous data validation

Suggestion to use Goodtables.io. See example of datasets-br/state-codes, that is using. You can run offline by goodtables-py#cli.

Native name

I'm reporting here a discution we had when creating the dataset (datasets/awesome-data#47)

• The native name for each city would be great
• But is there only one native name ? Think of Anwerpen / Anvers in Belgium or Jaffna / யாழ்ப்பாணம் / යාපනය in Sri Lanka : they have several native names.
• The orignal dataset has a "isPrefered" field wich is not reliable (see Bejing or Sukkothai)
• One way to achieve multiple native names would be to have several columns "native_name1", "native_language1", "native_name2", "native_language2", etc. And fill the columns for each official language of the country of the city. It's not very nice in terme of data conception but it does the job

If anyone has a better idea, or want me to implement this one, please share it here...

Ensure we can get the city population

Follow up from #5...

We should make sure it's possible to join world cicties with https://github.com/datasets/population-city

geonamesid not exist, use geonameid as primaryKey

See https://github.com/datasets/world-cities/blob/master/datapackage.json

is wrong

Include column for Wikidata identifier, suggestion

Wikipedia have stable pages for all cities, and Wikidata supply an ID for it.
Today Wikidata IDs are playing important role as "concept identifier", for Web Semantic in general and for open projects like OpenStreetMaps, etc.

Example: the instances of municipalities of Brazil have a simple query at Wikidata,

SELECT ?item ?itemLabel
WHERE { 
  ?item wdt:P31 wd:Q3184121 .
  SERVICE wikibase:label { bd:serviceParam wikibase:language "en". }    
} LIMIT 500

Improvements / clarifications on contents and fields (subcountry, geopoint etc)

Comments from openspending/cosmopolitan#25 (comment)

• At least should include standard name and the English variant
• I assume that subcountry equals what is "region" in geonames, but "subcountry" is confusing terminology (for me)
• We also take location (geopoint) and population from geonames, but they are not present here. I grant that there are likely better data sources, particularly for population.

@lexman any thoughts? I know we already have #3 re native name. What about second two points?

@pwalsh @lexman re city population note that we have https://github.com/datasets/population-city

Bad type in JSON Table Schema

When validating the JSON file directly against the tabular data package JSON Schema, I find a bad type on this line:

int: u'int' is not one of [u'string', u'number', u'integer', u'boolean', u'object', u'array', u'datetime', u'date', u'time', u'duration', u'geopoint', u'geojson', u'any']