whym/commons-app-stats

Tool to show mobile contribution stats of Wikimedia Commons

This is the source code of a web service hosted by tools.wmflabs.org. It shows charts indicating the number of files upload using the Android app for Wikimedia Commons. It is entirely written in Python.

All the data is extracted from the central database of Wikimedia Commons using an access that allows retrieving publicly accessible data only. In other words, this just gives a graphical summary of publicly accessible data such as the chronological list of all mobile uploads -- if a photo has not been uploaded, that won't be counted here either.

Setup

Expand this package after logging into tools.wmflabs.org.

Make sure to have a working environment with pandas etc. (See requirements.txt for details.) Example commands for setting these up:

virtualenv venv --system-site-packages --python /usr/bin/python3
echo 'source $HOME/venv/bin/activate' >> ~/.bash_profile
source ~/.bash_profile
pip install pip -U
pip install -r requirements.txt

Verify the installations by:

python -c'import pandas; print(pandas.__version__)'

Run

See ./crontab in the top level of the repository.

Output

More charts are available at http://tools.wmflabs.org/commons-app-stats/.

commons-app-stats's Issues

No more data after 2023Q1

Both the chart and CSV seem to stop on March 20, 2023:

https://commons-app-stats.toolforge.org/latest_quarter.csv.gz

Maybe the tool stopped running?

Old quarterly histogram?

Hi @whym ,

The new quarterly histogram seems to be a bit buggy - edits aren't shown in 2017, and uploads in 2017 Q2 appears lower than I recall. Is there a reason why we switched to the new quarterly histogram? I admit to being partial to the old one which displayed data from the start of the app (in 2013 or something I think), as that gave a very good overall picture of stats. :)

Weekly stats stalled?

Hi @whym ,

I just glanced at this chart today and it seems that the weekly stats have stalled? The last bar was from 18 Feb 2018. Is it normal to have a month's lag time, or is something wrong with the stats?

Also, does this affect the quarterly count at 2018 Q1?

Thanks!

Swap colors around for deletes and edits?

Minor issue, but red is typically associated with "bad", aka deletions IMO. Strange to see edits in red and deletions in green. :)

Manually specify colors used in matplotlib's plots

It seems like the standard solution for specifying colors is

commons-app-stats/local/bin/plot_batch.py

Line 82 in ef920e8

#samples.plot.bar(stacked=True, ax=ax, ec=(0.1, 0.1, 0.1, 0.7), alpha=0.7, color=['blue', 'green', 'red', 'black'])

which works locally but not on the Tools server (aka Wikimedia Cloud) for me.

This was a byproduct of solving #5 by updating libraries.

Quarterly histogram broken

Hi @whym ,

It seems like the quarterly histogram is broken again?

Sorry for the trouble, haha. :)

Quarterly histogram and CSV broken

Hi @whym , the quarterly histogram appears to be broken. CSV file has lots of "NoName"s and "NoTitle"s too.

Sorry to bother but it's the stats time of the year again. :)

Marked drop in uploads for the past ~5 weeks?

I noticed that there was a marked drop in uploads for the past ~5 wks (from ~500/wk to <100/wk). Do you know if there might be an error in the data collection @whym ? If not (and therefore the drop is accurate), I will need to take a look at why the uploads have dropped so much.

Thanks!

Data collection issue

Hi @whym , the data collection appears to be broken - not just the histogram, but even the .csv file shows 'NoName' and 'NoTitle' for everything.

Sorry to bother you, but I am trying to start collecting stats for our final report, and 2 of our metrics of success rely on the data obtained from this repo. :)

Thanks so much!

Quarterly stats displaying only thin lines

Hi @whym , I think the quarterly stats might be broken - they only display lots of thin lines to me (instead of the histogram).

Research sprint: how many mobile-only users are there?

Context: I'm trying to make the stats tool more informative, and as a preparation, I'm creating a few datasets and graphs. Some of these may get into the next iteration of the stats tool if the information seems useful.

Question: We know the number of active installs is around 3400 (as of March 2018, probably a bit more than that now). How many of them are using the app as a primary tool to contribute to Commons?

Results: For this I made https://quarry.wmflabs.org/query/27068 (​​https://quarry.wmflabs.org/run/262407/output/0/json?download=true) to retrieve the list of all Commons contributors who made at least one upload. If I use a (admittedly arbitrary) threshold of mobile_app_uploads / editcount = 0.6, it seems to show that there are 1000-ish app-only Commons contributors vs 3000-ish of the app's users.

What's next:

• It's just a quickly made query and I'm not sure if I'm getting this number correct actually - is this consistent with the number of app installs? Anything I'm missing?
• Since we now have a list of mobile-only users, how is the segment of users different from other segments, in terms of style, quality, geography, etc?

Dataset

Viewing histogram data in number format

Hi @whym ,

I've been following these stats with interest and was wondering if there was a way of viewing the exact numbers that make up the histograms? (e.g. 2016 Q4 might have 4239 new uploads, 2 overwrites, 467 deletes, etc). I glanced at the raw csv data but that only seems to have the unaggregated data.

The reason I'm asking is that I was planning to use the stats aggregated by your tool to report on the measures of success for my IEG when it ends next month, if at all possible.

Thanks!

Transfer to commons-app organization?

Not that there is any problem with how things are right now, but have you considered trasnferring this project to the organization at https://github.com/commons-app?
It would be more visible to developers who roam around the Commons app looking for things to improve :-)
Of course if you prefer to keep it, it is perfectly fine.
Cheers!

Stats stuck on week of 16 Sep

@whym Seems like the last week recorded at https://tools.wmflabs.org/commons-app-stats/ is 16 Sep?

Collection stopped in early August?

Hi @whym , the last week of stats that I can see is from 5th August. Any idea what the issue is? :)

Thanks!

Exclude Suggested Edits edits (made via the Wikipedia app)

We need to exclude Suggested Edits edits from the stats. Wikipedia app now allows users to edit Wikimedia Commons (https://www.mediawiki.org/wiki/Wikimedia_Apps/Suggested_edits#Image_captions). Example: https://commons.wikimedia.org/w/index.php?title=File:Yukyuzan_Kouen_2019,05.jpg&diff=prev&oldid=388969580 . Those edits seem to be tagged with "mobile app edit" and "Suggested Edits edit" tags, while edits via Commons app have the former only - which I think can be used to separate the two.

Split from #14 (comment)