wo-ist-markt/wo-ist-markt-berlin-update-reminder

Script which checks if Berlin updated its data

Wo ist Markt?

A small visualization of the weekly markets in different cities. Available at: http://wo-ist-markt.de

• Five minutes talk about the project at 33C3 (12/2016)
• Short project summary in a Konrad Adenauer Stiftung publication (02/2017) (last page, in German)

Project description

Wo ist Markt? aims at providing an easy to use website for market information. Visitors of the website should be able to intuitively get answers to the following questions:

• Where does a market take place?
• When does a market take place?

Therefore, the website shows the position of markets as markers on a map. Each marker can be clicked or tapped to reveal the detail information associated with this market - most important, the opening hours. This can been seen on the following screenshot.

Kinds of markets

The project is not limited to specific kinds of markets. Typical markets in cities are:

• Farmer markets (selling vegetable, fruits, fish, plants, ...)
• Flea markets (selling used stuff, antique goods, ...)

Market data

The actual information (title, location, opening hours, ...) about the markets can be provided by anyone. In the optimal case it is provided by a city as open data. If not, any volunteer can collect such data and contribute it to the project. Market data must be maintained individually. There is no central project team to take care of updating data of multiple cities. This is a distributed project. This is also the reason why cities are missing. It needs someone to retrieve and add the data. It is of your choice if you collect the data manually or if you automate it with a script. If you need inspiration on how to automate this have a look at the preprocessing folder.

Project history

The project has been started by @torfsen as part of the Open Knowledge Lab Karlsruhe in February 2015. To support more cities the code has been refactored by @johnjohndoe as part of the Open Knowledge Lab Berlin in February 2016.

Navigation

• The individual market data for supported cities is displayed when the city name is given in the address bar of the browser.

  Example: http://wo-ist-markt.de/#karlsruhe.

  By default or on error Karlsruhe is rendered.

Contributing

• You are very welcome to get involved in the project. Therefore, we prepared a contribution guide which explains how to add market data and how to start coding on the project.

Communication

• Please use the GitHub issue tracker to discuss new and existing ideas.
• Come join the converation about this project in our Slack channel. Write to tobias DOT preuss AT googlemail DOT com if you need an invitation for the Slack group.

Supported cities

City name	Data source
Bad Homburg	City of Bad Homburg
Baden-Baden	City of Baden-Baden
Basel	City of Basel
Berlin	City of Berlin
Bielefeld	City of Bielefeld
Bochum	City of Bochum
Bonn	City of Bonn
Bottrop	City of Bottrop
Braunschweig	City of Braunschweig
Bremen	City of Bremen
Bremerhaven	City of Bremerhaven
Bretten	City of Bretten
Bruchköbel	Stadtmarketing Bruchköbel GmbH
Brühl	Stadt Brühl
Chemnitz	City of Chemnitz
Cottbus	City of Cottbus
Cuxhaven	City of Cuxhaven
Dortmund	City of Dortmund
Dresden	City of Dresden
Duisburg	Duisburg Kontor GmbH
Düsseldorf	City of Düsseldorf
Erlangen	City of Erlangen
Essen	EVB
Frankfurt am Main	City of Frankfurt am Main
Freiburg	City of Freiburg
Friedrichsdorf	City of Friedrichsdorf
Fürth	City of Fürth (Bavaria)
Gelsenkirchen	Gelsendienste
Hagen	City of Hagen
Hamburg	City of Hamburg
Hamm	City of Hamm
Hanau	City of Hanau
Hannover	City of Hannover
Heidelberg	City of Heidelberg
Herne	City of Herne
Hilden	City of Hilden
Hofheim am Taunus	City of Hofheim
Kaiserslautern	City of Kaiserslautern
Karlsruhe	City of Karlsruhe
Kassel	City of Kassel
Kiel	City of Kiel
Kleve	City of Kleve
Köln	City of Köln
Konstanz	City of Konstanz
Langenfeld (Rhld.)	City of Langenfeld (Rhld.)
Laupheim	City of Laupheim
Leipzig	City of Leipzig
Loxstedt	City of Loxstedt
Lübeck	City of Lübeck
Ludwigsburg	City of Ludwigsburg
Mallorca	Island of Mallorca
Marl	City of Marl
Moers	City of Moers
Mülheim an der Ruhr	City of Mülheim an der Ruhr
München	City of München
Münster	City of Münster
Nersingen	City of Nersingen
Neu-Anspach	City of Neu-Anspach
Neu-Ulm	City of Neu-Ulm
Oberursel	City of Oberursel
Oldenburg	City of Oldenburg (Oldb)
Osnabrueck	City of Osnabrueck
Paderborn	City of Paderborn
Potsdam	City of Potsdam
Rostock	City of Rostock
Rothenburg ob der Tauber	City of Rothenburg ob der Tauber
Saarbrücken	Landeshauptstadt Saarbrücken
Salzkotten	City of Salzkotten
Schleswig	City of Schleswig
Schwerin	City of Schwerin
Siegen	City of Siegen
Solingen	City of Solingen
Stralsund	City of Stralsund
Stuttgart	Stuttgarter Wochenmärkte
Suhl	City of Suhl
Trier	City of Trier
Tübingen	Wochenmarkt - Universitätsstadt Tübingen
Ulm	City of Ulm
Usingen	City of Usingen
Wiesbaden	City of Wiesbaden
Witten	Stadtmarketing Witten
Wolfenbüttel	Stadt Wolfenbüttel
Wuppertal	City of Wuppertal
Zwickau	Kultour Z.

Development

Tests can be run using npm:

$ npm install  # Just once after you've cloned the repo

$ npm test     # Whenever you want to run the tests

Please see also CONTRIBUTING.md on how to start development.

Deployment

This website is hosted on Netlify.

When a commit is added to the master branch then Netlify automatically publishes this state as the new version of the website available under https://wo-ist-markt.de.

wo-ist-markt-berlin-update-reminder's Issues

Script doesn't parse date from website

The script seems to be broken. I don't understand why. Here is the tail of raw log of this Travis build.

travis_time:start:07f735d7
�[0K$ bash update_reminder.sh
Dates don't match!
daten.berlin.de: 03.05.2019
wo-ist-markt.de: 
travis_time:end:07f735d7:start=1569776346008707586,finish=1569776347412036133,duration=1403328547,event=script
�[0K�[31;1mThe command "bash update_reminder.sh" exited with 1.�[0m

Done. Your build exited with 1.

/cc @NWuensche @k-nut

Check presence of GeoJSON download

Context

The GeoJSON data format download has been disappearing for a few month in 2023. It is the source of the proprocessing step. Finally, the government could be convinced to reincarnate the format.

Proposal

The shell script checks the pure presence of the GeoJSON data format.

Check if GeoJSON can be downloaded

⚠️ Depends on #8

Context

Currently, the GeoJSON data format is listed on the website but cannot be downloaded. Visiting the link leads to an Asset not found website.

~ curl -I https://www.berlin.de/sen/web/service/maerkte-feste/wochen-troedelmaerkte//index/all.geojson\?q\=
HTTP/2 301 
date: Sun, 12 Nov 2023 11:58:42 GMT
content-type: text/html; charset=UTF-8
location: https://www.berlin.de/sen/web/service/maerkte-feste/wochen-troedelmaerkte/index/all.geojson
vary: Accept-Encoding
age: 57
referrer-policy: strict-origin-when-cross-origin
x-content-type-options: nosniff
x-frame-options: SAMEORIGIN
x-xss-protection: 1; mode=block
strict-transport-security: max-age=31536000
content-security-policy: upgrade-insecure-requests

Proposal

The shell script downloads the GeoJSON file and checks its file size being more than a certain threshold.

Related

Inspiration can be taken from: https://github.com/johnjohndoe/file_download

Check if data source URL changed

Context

Recently, the URL of the data source changed:

• Old: https://daten.berlin.de/datensaetze/berliner-wochen-und-tr%C3%B6delm%C3%A4rkte
• New: https://daten.berlin.de/datensaetze/berliner-und-brandenburger-wochen-und-tr%C3%B6delm%C3%A4rkte

Current state

The script does not spot the URL change but output the following error instead:

daten.berlin.de: GeoJSON-URL not available!
Dates don't match!
daten.berlin.de: 14.03.2024

Proposal

Check if the known URL is still available. Output an error if not. Continue with lower-ranked checks.

Related

• #7

Check HTTP status code

Context

A while ago the weekend markets URL in the open data portal changed. As a result the webserver redirected from https://daten.berlin.de/datensaetze/wochen-und-trödelmärkte to https://daten.berlin.de/datensaetze/wochen-und-tr%C3%B6delm%C3%A4rkte-0 when requesting the former. This was not noticed by the shell script because there are also dates shown. By coincidence the date matched with the date of the current deployment of https://wo-ist-markt.de/#berlin.

Proposal

To verify that the expected URL is still valid the shell script could check the HTTP status code returned by the webserver.
If the webserver returns something else than 200 an error can be raised.

Related

• Update market data URL