flickr-foundation/flickypedia

A tool to copy CC-licensed images from Flickr to Wikimedia Commons

Flickypedia is a bridge between photos on Flickr and files on Wikimedia Commons. It includes:

• A web app for copying openly-licensed photos from Flickr to Wikimedia Commons ("uploadr")
• A bot for improving the structured data for Flickr photos which are already on Wikimedia Commons ("backfillr")
• A tool for getting data and statistics about Flickr photos on Wikimedia Commons ("extractr")

Our goal is that it results in higher quality records on Wikimedia Commons, with better connected data, better descriptive information, and makes it easier for Flickr photographers to see how their photos are being used.

Flickypedia was built by the US 501(c)(3) Flickr Foundation in 2023 in partnership with the Culture and Heritage team at the Wikimedia Foundation.

Usage

If you want to copy some photos from Flickr to Wikimedia Commons, you can use the web app at https://www.flickr.org/tools/flickypedia/

If you want to use the backfillr or extractr tools, you need to clone the repo and install dependencies:

$ git clone https://github.com/Flickr-Foundation/flickypedia.git
$ cd flinumeratr
$ python3 -m venv .venv
$ source .venv/bin/activate
$ pip install -e .

Then run the flickypedia CLI, which has help text that will explain what to do next:

$ flickypedia --help

Architecture

The Flickypedia code lives in the src folder, which is split into four main components:

• External APIs (apis) – code for interacting with external services, including Flickr, SDC, and Wikimedia Commons. This is reasonably generic and not be too specific to Flickypedia. If you're looking for pieces to reuse, this is a good place to start looking.

• Uploadr – a web app for copying openly-licensed photos from Flickr to Wikimedia Commons. This is a Flask app.

  The app is organised into a series of screens ("get photos", "select photos", "prepare info", and so on) – all the files associated with a screen are named to match the URL. e.g. the "prepare info" screen is at /prepare_info, so the associated files are prepare_info.py, prepare_info.html, prepare_info.scss, and so on.

• Backfillr – a CLI for updating SDC for Flickr photos that have already been uploaded to Commons. You can either run on a single file or many files at once.

  It fetches the existing SDC for a photo, finds the Flickr ID, then calculates the "new" SDC for the photo (if it was uploaded with Flickypedia today). It compares the new and existing SDC, and writes any missing statements back into Commons.

• Extractr – a CLI for getting data about existing Flickr photos on Commons from SDC snapshots.

Development

You can set up a local development environment by cloning the repo and installing dependencies:

$ git clone https://github.com/Flickr-Foundation/flickypedia.git
$ cd flinumeratr
$ python3 -m venv .venv
$ source .venv/bin/activate
$ pip install -e .

If you want to run tests, install the dev dependencies and run coverage:

$ source .venv/bin/activate
$ pip install -r dev_requirements.txt
$ coverage run -m pytest tests
$ coverage report

flickypedia's Issues

Write some ADRs for what's happened so far

Possible topics:

• Using Python and Flask
• OAuth 2.0 and how tokens are stored
• Looking up users from Wikidata
• Looking up duplicates
• Background tasks with Celery (vs Flask-Executor)
• Flickr API response cache

Add some validation to the fields on the "prepare info" page

Write the "proper" implementation of the auth code

I've got a rough auth implementation working, but it's a bit ropey – good enough for prototyping, but not much else. I should tidy it up and make it something that we can use in the real app.

Still to do:

• Write proper tests for it
• Add support for refresh tokens

Enable strict mode for Jinja2 templates

Ensure we filter for photos in the initial selection

e.g. http://localhost:5000/select_photos?flickr_url=https://www.flickr.com/photos/comfiaccoo/albums/72177720312210559 seems to contain some videos which we don't want!

Add support for "No Known Copyright Restrictions" (maybe)

Right now it looks like we're not going to support NKCR – but in case we reverse that decision later, this is a tracking ticket for all the places we'll need to make changes to support it.

• Add support for the {{Flickr-no known copyright restrictions}} template when we render Wikitext
• Add support in the structured data
• Add it to the list of supported licenses in config
• Tweak the copy

Do we need to apply a filter for "safe" images?

Quoting from a brief description of Flickr2Commons:

Magnus' Flickr2Commons upload tool (instant copying; only "safe" files)

We might be getting this behaviour "for free" – I don't know whether Flickr restricts public searches to safe content by default, or whether we might be getting it if you put in the right URL.

We should:

1. Decide if we want to keep this rule, and if so
2. Check whether Flickypedia is enforcing it, and add that code if not

Increase the number of photos on the "choose photos" screen to 100

Implement duplicate detection for uploads

• Look for matching URLs in the structured data
• Looking at the SHA-1 hashes

Make sure you don't lose URLs if there aren't any at that URL

Steps to reproduce:

1. Go to http://localhost:5000/find_photos
2. Enter "https://www.flickr.com/help", click "Go"
3. Get taken to a page with an error "There are no photos to show at that URL"

Notice that the input field is now empty – it should preserve your URL.

Review all the error states

• Select photos: single photo URL, but doesn't resolve

Support a bookmarklet for sending to Flickypedia

Build a proper mechanism for managing background uploads

Add a User-Agent header to our API requests

Model the data connection between Flickr and SDC

This should follow the mapping described in Commons:Flickypedia/Data Modeling.

Consider options for internationalisation

e.g. moving all the user-facing copy into a "strings" file

Decide if there's an upper bound for how many photos you can upload at once

Ask the owner of the FlickrTags template about adding hyperlinks

So people can find similar images that they can bring across to Commons.

Use Large Square to render photos in the "choose photos" screen

Add a <title> element to all the pages

Currently every page has a <title>Flickypedia</title>, which is a bit annoying when you have lots of Flickypedia entries in your history and you don't know which is which!

Investigate supporting search URLs

Write a brief privacy policy

Test cases for 'pick from a list of photos'

• Single photo, license okay and not in WMC
• Single photo, license okay and in WMC
• Single photo, license not okay
• Multiple photos, all license okay and none in WMC
• Multiple photos, all license okay and some in WMC
• Multiple photos, all license okay and all in WMC
• Multiple photos, some license okay and none in WMC
• Multiple photos, some license okay and some in WMC
• Multiple photos, some license okay and all in WMC
• Multiple photos, no license okay

Add "Coordinates and captured with" to the SDC

See Slack discussion here: https://flickrfoundation.slack.com/archives/C05AVC1JYL9/p1697615021046079?thread_ts=1696947242.703349&cid=C05AVC1JYL9

• Add a structured data entry to P1259
• Add the {{Location}} template to the Wikitext
• Make sure globe coordinates are shown nicely in the SDC preview
• Add tests for coordinates that don't have positive lat/long on Flickr

Add Wikitext as well as SDC to uploaded images

Rename the 'pages' namespace to 'views'

Add support for the shiny new P12120 property

Upload a single image to Wikimedia Commons using our auth

Do validation of titles before the upload step

Show some info about the album/gallery/group etc

If somebody selects a single photo and it's okay to use, skip straight to Prepare Info

Handle refresh of OAuth tokens

Clarify in our copy that you can only see public photos using the tool

Rename 'find flickr photos' with 'get flickr photos'

Fetch the description from the Flickr API

Plumb Flickr tags into the Wikitext

Drop support for Python 3.7

At least for the very first prototype, we may end up deploying this into a Glitch environment to match flinumeratr – but Glitch only supports Python 3.7 (even though it's EOL): https://support.glitch.com/t/upgrade-python-version-from-3-7-which-is-now-eol-to-something-more-recent/63011

Either Glitch gets its act together and bumps the Python version, or we look elsewhere – whatever the case, we'll be dropping the 3.7 support eventually. This is a ticket to track all the stuff we'll do when that happens.

• Remove 3.7 from the testing matrix in GitHub Actions
• Remove the flake8 --ignore=E231 lint in GitHub Actions
• Relax some of the version constraints in requirements.in

Add URL validation to the "choose photos" form

Add our auto-populated categories

Set up some credentials with the beta cluster for uploading images

There is a testing environment for Wikimedia Commons, as described in Commons:Guide to batch uploading:

If you want to test uploading and safely experiment with using tools or new templates in a safe environment, you can set up an account on the beta cluster. This is a mirror of Wikimedia Commons where if things go wrong you will not cause any disruption to the live environment. See http://commons.wikimedia.beta.wmflabs.org/ and this explanation.

This would be very useful for our purposes!

If it's a single photo, check for duplicates before we go to the Flickr API

Do we need some sort of message if we go single photo -> prepare info screen?

Key info in the description on the 'select photos' screen should be bolded

Filter by license and duplicate in the "select photos" screen

Add categories to the Wikitext

Find out how long the session lasts

How long are you logged in for?

Investigate supporting paginated URLs

In the "Have a look" duplicate link, if it's a single photo, link to that rather than the media search

Don't duplicate SDC into Wikitext; we can use the Information template

Consider using the profileurl (if available) in the structured data