dominikboenisch/training-the-archive

Research project combining artificial intelligence and museum collection data through machine learning and object recognition.

Training the Archive

A research project based at the German Ludwig Forum for International Art Aachen and the Hartware MedienKunstVerein (HMKV) in Dortmund, Germany that combines artificial intelligence and museum collection data through machine learning and object recognition. The project is funded by the Digital Culture Programme of the German Federal Cultural Foundation (Kulturstiftung des Bundes).

Over the next four years (2020-2023), the research project "Training the Archive" will explore the possibilities and risks of AI in relation to the automated structuring of data to support curatorial practice and artistic production. Connected to this is the research question of whether AI can learn research processes so that patterns, connections and associations become recognizable that are not apparent to humans in this form. Together with international artists and curators, a procedure is to be developed that will help to make digital archives - such as the collection of the Ludwig Forum Aachen - accessible in a new way.

Blog: https://trainingthearchive.ludwigforum.de/

Prototype:

Clustering of images from a museum collection to identify interesting links. Subsequently, the human being is to be brought back into the loop, in which networks for image recognition are retrained with the knowledge of curators, for example, in order to make the clusters more dynamic and personalized.

Step by step guide:

1. Scraping of data from the Open Source Library of the National Gallery of Denmark (SMK).
   • Notebook
2. Extracting feature vectors from Keras Applications or Tensorflow Hub
   • Notebooks
3. Generating a dataset for the training using triplet loss
   • Notebook
4. Training of a network with our annotations, which artworks are related and which are not
   • Notebooks
5. Clustering of artworks using different neural networks and visualization of the results
   • Notebook
6. Nearest (or even farthest) neighbors and walk through the latent space
   • Notebook

Results of the visualization:

All Images © Dominik Bönisch, Ludwig Forum Aachen using data from the Open Source Library of the National Gallery of Denmark (SMK).

Scatterplot

Gridplot

Nearest/Farthest neighbors

Walkthrough the latent space

Walkthrough the latent space using a tube-shaped scatterplot with an arrowed path

Contribute:

You have suggestions or feedback? We are very happy about that. Feel free to write us a line.

Acknowledgment:

The first prototype was developed within the context of the so-called AI school in the LINK programme of the Lower Saxony Foundation in cooperation with the tutors Dr. rer. nat. Jan Sölter and Dr. rer. nat. Thomas Rost.

License:

This prototype is licensed under the GPL-3.0 License. See the license file for detail.