hatnote/montage

📷 Photo evaluation tool for and by Wiki Loves competitions

Montage

Photo evaluation tool for and by Wiki Loves competitions

Round-based photo evaluation is a crucial step in the "Wiki Loves" series of photography competitions. Montage provides a configurable workflow that adapts to the conventions of all groups.

• montage on Wikimedia Commons
• montage on Phabricator

Testing

pip install tox into your virtualenv, then tox.

Deployment

Ensure you have the necessary libraries by running pip install -r requirements-dev.txt in a virtualenv of your choosing.

1. Ensure all changes are committed
2. Run the tests (tox)
3. Deploy to the dev instance with fab deploy. Ensure the script runs successfully.
   1. Check that the the dev site came back up fine.
   2. Log in and test things out.
   3. Check the montage-dev sentry (devlabs)
4. Use the audit log to check that the production instance isn't in active use.
5. Deploy to the production instance with fab deploy:tool=montage
   1. Check that the the production site came back up fine.
   2. Log in and test things out.
   3. Check the montage sentry (prod)

Some tips (at the time of writing):

• fab deploy without arguments will deploy master to montage-dev.
  • To change this, use a command of this format: fab deploy:branch=other-branch,tool=montage-beta
  • It will fail with a helpful message if your branch is not sync'd with origin. Push those commits!
• If you see a huge golang stack trace from fab deploy, it's likely this bug. These seem to fix themselves, come back later if you can, otherwise check out the thread for details on using GOMAXPROCS.
• All this only deploys the backend for now.

montage's Issues

Exporting reviews for coordinators

There should be a way for coordinators to download all reviews for all entries from a round.

Do we want a similar feature for jurors to export their own reviews?

Internationalize the juror interface

We should allow people to translate all the strings in the juror interface.

Make jury directions visible during judging

Future feature request: Make the directions for the jury more embedded into Montage once judging begins, rather than having them only visible on the first page. For example, a mouseover or just text at the bottom of the page. Something that makes it possible to refer back to directions without having to exit the round and go back to the main page.

https://tools.wmflabs.org/montage-beta should redirect

I keep forgetting to add the /#/ :) (Sorry, not sure if you have a separate issues tracker for specifically that installation or not.)

Adding public endpoints for participants and other community members

Right now, the only public endpoints are for login purposes, with the majority of Montage being for jurors and coordinators.

One major goal of Montage has always been to provide more functionality and transparency for participants and other interested community members. We're making progress along these lines by building a final report feature which would build into a public archive of contest results.

Another endpoint I've been considering checks the status of a photo (or photos), across all contests it's been in. As a photographer and contest participant in the past, I've found myself wondering how particular favorites of mine are favoring. One can imagine campaign-level configuration around whether this functionality only works when the results are finalized, or whether participants can find out whether they're still in the running while the judging is ongoing.

What other endpoints would be useful? Maybe a list of active contests? A list of most prolific organizers? Add other ideas for community endpoints below!

Request to become coordinator

We should have a form for users who wish to start a campaign as an organizer. Currently, new coordinators need to submit a request on the Commons page and are added manually by me or Leila.

Provide default instructions

When creating a new round, coordinators should be provided with a default (modifiable) set of instructions based on the round type (yes/no, rating, and ranking).

Voting buttons vs shortcuts

A comment from the an italian juror on the voting interface:

• The accept/decline buttons change position based on the title length. It should be better if the title would resize while the buttons are in a fixed position.
• The keyboard shortcuts alternative should be made more evident.
• Perhaps the description could be made smaller wrt the voting area

Better description for the threshold map

The coordinator view should describe the threshold map in plain language. For example, instead of saying "average score of 0.5", the dialogue should explain that this means "__ ranked the entry at least __ stars."

Activate/Pause/Finalize buttons

The intended use of the Activate/Pause/Finalize buttons isn't clear (unless you've used the software before).

Since the Activate and Pause buttons are essentially toggles, it would be better to combine them into a single toggled button.

Finalize might be better renamed 'Close round' or something similar.

Line breaks in Round Directions

Thanks for this great tool! I've got a feature request:

Make line breaks visible in Round Directions
This would be really helpful for readability when writing out judging criteria for a ranked round. Currently line breaks don't appear; just a big block of text, which can be hard to digest especially when comprehending directions that include a list of how to interpret 1-5 stars, etc.

POST indicator spins forever

Every 3-4 'votes', the loading indicator will just spin continuously without resolution. This is despite the POST appearing to complete successfully (according to Firebug).

All Firebug reports is

TypeError: dbg is undefined
chrome://firebug/content/debugger/debuggerLib.js
Line 556

...which doesn't seem very helpful.

I'm using an update to date version of Firefox (beta in fact). Mobile Firefox was working for me earlier and I'll try in Chrome now.

Adjustable task reassignment

When a juror is added or removed, some of the tasks will need to be redistributed among the new jury. The current algorithm will give evenly-distributed tasks, regardless of how many tasks a juror has already completed. This means that a juror that does a lot of work early may end up getting assigned more than their fair share of the total voting. Coordinators should have an option to adjust task reassignment if needed.

Include entry description

The juror view should include the description from Wikimedia Commons when examining an entry.

Should we allow coordinators to optionally disable this for a round?

Undo / go back

Jurors should have an easy button to return to the last entry, or undo their last action.

Import additional entries into ongoing round

A coordinator should be able to add more entries to an ongoing round. A coordinator may want to import photos from a category at one point, let jurors start voting, and then later check the category for new photos to import. They may also want to use multiple categories or CSVs to create a round.

We've heard from a few WLM organizers that it would be helpful to allow them to give jurors a head start on their work.

For rating and yes/no rounds, this probably won't substantially affect fairness (although maybe some jurors want to skim all the photos before they start voting). It won't work for ranking rounds.

Standardize permissions

We should have the following permissions:

• Maintainer: has organizer and coordinator permissions for all campaigns
  • Uses the /maintainer endpoint
• Organizer: can create new campaigns and add coordinators
  • Uses the /admin/add_campaign endpoint
• Coordinator: can edit a campaign, create new rounds, and create jurors
  • Uses the /admin endpoint
• Juror: can vote in a round
  • Uses the /juror endpoint

There are a few places where maintainers do not have full coordinator permissions, which can make it tough to debug.

Import provenance

Montage should track provenance for sets of entries into a round. Entries can come from a category on Wikimedia Commons, a list of files, or another round.

Image size criteria

When voting for WLM Italy I encountered this photo which is smaller than the thumbnail, so the thumbnail generation gives a HTTP 400 answer.
https://commons.wikimedia.org/wiki/File:Biblioteca_Fortunata.jpg

(it's Image #87040)

Perhaps it would be nice to detect these edge cases setting an automated filter (there's a minimum resolution limit IIRC)

Show progress when voting as well

In the voting interface, some progress can be shown, currently is only on the main page.

Subtle idea: animate the border of the toolbar (mont-toolbar md-toolbar.toolbar), like it's done on some online forms and some long form articles (I don't remember exactly where I saw it) Example https://kimmobrunfeldt.github.io/progressbar.js/

Use case: I need to vote 3.5k photos, I would like to see where I am.
Additionally, it would be cool to see some stat on how I'm voting, if it's a Yes/No round I would like to see the percentage of Yes votes.

EDIT: I didn't notice the "XXX images remaining" under the name :-D

Split view for ranking

Jurors should have two sets of entries in a ranking round. In one part, there is the ranked list of entries. In the second part, there should be an unsorted pool of photos. The second part could be a split screen, or a row of entries along the bottom of the screen.

This would make it easier for jurors to compare each photo against the ranked list.

Full resolution view / zoom

Jurors should be able to zoom in to see a photo in full resolution. Jurors may want to inspect an image closely to see its quality.

Error: image is undefined

I get this on one image, I think Engine_Vein%2C_Alderley_Edge_Mines_2016_080.jpg

Error: image is undefined
getImageName@https://tools.wmflabs.org/montage-beta/dist/bundle.min.js?v=16.10.12:72:25846
anonymous/fn@https://tools.wmflabs.org/montage-beta/dist/bundle.min.js?v=16.10.12 line 13 > Function:2:395
inputsWatchDelegate/<@https://tools.wmflabs.org/montage-beta/dist/bundle.min.js?v=16.10.12:10:863
$RootScopeProvider/this.$get</Scope.prototype.$digest@https://tools.wmflabs.org/montage-beta/dist/bundle.min.js?v=16.10.12:10:16748
$RootScopeProvider/this.$get</Scope.prototype.$apply@https://tools.wmflabs.org/montage-beta/dist/bundle.min.js?v=16.10.12:10:19215
done@https://tools.wmflabs.org/montage-beta/dist/bundle.min.js?v=16.10.12:9:4355
completeRequest@https://tools.wmflabs.org/montage-beta/dist/bundle.min.js?v=16.10.12:9:8221
createHttpBackend/</xhr.onload@https://tools.wmflabs.org/montage-beta/dist/bundle.min.js?v=16.10.12:9:9042
https://tools.wmflabs.org/montage-beta/dist/bundle.min.js?v=16.10.12
Line 9

Dark juror background

The juror view should have a dark background (or an option to change the background color) when viewing photos.

Closing a campaign

Coordinators should be able to close a campaign

Increasing qourum

Coordinators should be able to change the quorum settings for a round, which will assign new tasks to jurors. For now, we can support a quorum that only increases so we don't have to figure out how to dispose of votes.

Adding a new coordinator to a campaign

A coordinator should be able to add another user as a coordinator to a campaign.

Adding new jurors and increasing Quorum

Our WLE 2017 Latvia judging round is in progress but we wanted to add more jurors to it.

When I add new juror, remaining tasks are distributed between new and existing jurors. There is no way to change Quorum value.

We want to go from 3 to 5 jurors where every jurors has to evaluate all images.

Category suggestions during import

When coordinators import new photos, there should be an option for selecting photos from standard categories connected to a campaign. Instead of providing Category:Images from Wiki Loves Monuments 2016 in Germany, a coordinator should be able to select: Wiki Loves Monuments / Germany / 2016.

Limited ranking

There should be an option to rank a limited number of entries in a ranking round. For example, select the top 10 photos out of 15. The unranked photos will be tied for last place.

If the ranking limit is too small compared to the total available images, it could result in a lot of variance in the results. We may want to add a limit/warning in this case.

Decreasing quorum value for WLE-France

WLE France needs your help. In their round 2, they have removed one out of five jurors and their quorum was set to 5. Can you help them by reducing the quorum to 4, please? This will help them close their round 2. Thanks! :)

Detect and resolve files deleted on Wikimedia Commons

There should be a system for coordinators to see if entries in a round were deleted on Wikimedia Commons. Jurors should not be shown deleted files.

In-progress report

Coordinators should be able to view a vote report for a campaign in progress. Optionally, we can let coordinators configure the round to be "sealed" for the sake of fairness, so no one can see specific votes until all voting is done.

Disqualification

Coordinators should be able to disqualify photos at the start of the round. They should be able to select the reasons for disqualification (image size, uploaded by juror, or uploaded outside of the competition’s dates), and then confirm the number of affected entries before hitting submit.

Cannot edit description, error about jurors

When I try to edit description on active round I get error "Cannot change judges on active round".
When I pause it and try again I get error that new judges should be different from old. Description is not saved.

''Quroum" typo

On the Add Round page, the word Quorum is miss-spelled as ''Quroum"

Import warnings

When importing photos to a round, there should be warnings for unusual results, such as an empty set or auto-disqualified entries.

Sometimes montage enter in a loop of photographs

Hi! First of all thank you for the tool. We, a group of users of Wikimedia Spain, are feeling a problem.

When we have skip 5-15 photographs it increase the possibility of enter in a loop of the same photographs and it makes impossible to progress. When it happens we have to refresh the page and skip again the photographs skipped, but after 2-4 photographs skipped the error persists and we have to refresh again. A loop very annoying. We have detect that it often happens when we have skipped 12 photographs, perhaps it could help you.

I don't know if its could help you to solve it, but I paste here the error that shows in the Firefox console:

"Error: image is undefined
getImageName@https://tools.wmflabs.org/montage-beta/dist/bundle.min.js?v=16.10.15:69:24459
anonymous/fn@https://tools.wmflabs.org/montage-beta/dist/bundle.min.js?v=16.10.15 line 13 > Function:2:395
inputsWatchDelegate/<@https://tools.wmflabs.org/montage-beta/dist/bundle.min.js?v=16.10.15:10:863
$RootScopeProvider/this.$get</Scope.prototype.$digest@https://tools.wmflabs.org/montage-beta/dist/bundle.min.js?v=16.10.15:10:16748
$RootScopeProvider/this.$get</Scope.prototype.$apply@https://tools.wmflabs.org/montage-beta/dist/bundle.min.js?v=16.10.15:10:19215
timeout/timeoutId<@https://tools.wmflabs.org/montage-beta/dist/bundle.min.js?v=16.10.15:11:59
completeOutstandingRequest@https://tools.wmflabs.org/montage-beta/dist/bundle.min.js?v=16.10.15:7:7039
Browser/self.defer/timeoutId<@https://tools.wmflabs.org/montage-beta/dist/bundle.min.js?v=16.10.15:7:10168
"

I think that error is the same or similar to another issue, Error: image is undefined #6.

Thanks in advance and good luck solving it!

Regards, Iván

Compute ranking winners

A coordinator should be able to close the final ranking round, before generating a report (#29) and closing a campaign (#26).

Campaign start and end date

The start and end date should be an adjustable setting in the disqualification panel, not necessarily always the same as the start and end dates for the campaign.

The dates and times should always be shown in UTC, as well.

Adding new entries during a capaign

A coordinator may want to start voting in the first round of a campaign, with an option to load in more photos as they are submitted. This feature could be limited to the first round and a non-ranking round for the sake of fairness.

Starting voting early is particularly helpful for campaigns with a large number of entries.

Separate page for non-active campaigns

The coordinator and juror panels should show active campaigns on the main page, and have another page for all campaigns. This will avoid having older campaigns cluttering up the main panel.

Before building this, we need a feature to close a campaign (#26).

Waiting versus no-action signals

@yarl in montage, while I click on "activate" or "create" options, sometimes I have to wait for something to happen but there are no visual clues that tell me something is happening and montage is processing my request. Please add a visual clue for when we have to wait for something to happen. Thanks! :)

File name sanitization

https://commons.wikimedia.org/wiki/File:Rocca_Sanvitale_in_B%26W.jpg

Gets loaded as https://commons.wikimedia.org/w/thumb.php?f=Rocca_Sanvitale_in_B&W.jpg&w=800

Resulting in a HTTP 400 answer

Internal Server Errror on stale cookie

When visiting montage-dev and logging in, I saw a 500 on complete_login, coming from a KeyError on cookie['request_token_key']. The cookie at fault was a non-http-only cookie set on the root path (/, not /montage-dev/). Not sure where this cookie came from, but its expiry was for the session, and it was clearly signed by us, just like the normal cookie. In the future we should clear such cookies and redirect home to retry log ins.

Downloading results from rating round without creating new round

Coordinators should be able to download rating results without creating a new round.

Currently, a rating round can only be finalized by advancing to another round. Some coordinators may choose to end with a rating round, or may want to see the outcome before configuring the next round. Perhaps there should be a config setting whether coordinators can view the voting results for in-progress rounds.

See a list of disqualified files

A coordinator should be able to see a list of all the files (and the reasons) for disqualification for each round.

Easy access voting instructions

Jurors should be able to easily find voting instructions from any juror view.

Final report for campaigns

Coordinators should be able to export a report with results from the full campaign.

Complete coordinator view

The coordinator view should be able to see all of the settings for a round and campaign. For example, currently a coordinator cannot see a quorum value once the round is created.