notconfusing/cocytus

produce a stream of citiation data coming off wikimedia

cocytus's Issues

Load troubleshooting

We are falling behind in queue processing on average.
Changed to burst worker mode to mitigate this somewhat.
[master a214d94] change to burst processing to better handle load

Long term we will need more workers. May need to move to a dedicated instance because of this.

Commons?

I haven't yet seen any mention of a Commons page in
http://events.labs.crossref.org/events/types/WikipediaCitation ,
not even during periods when OAMI has uploaded stuff, e.g.
https://commons.wikimedia.org/wiki/File:A-new-type-of-ant-decapitation-in-the-Phoridae-%28Insecta-Diptera%29-biodiversity_data_journal-3-e4299-g001.ogv .

heartbeat regularity

assert that heartbeats are being generated and sent at a constant intervals

Scrambled DOI

10.2/3'\n|-\n|  2015-03-03  Wikipedia Cocytus   

Action
    add
Page URL
    http://it.wikibooks.org/wiki/Disposizioni%20foniche%20di%20organi%20a%20canne/Europa/Italia/Veneto/Provincia%20di%20Padova/Rubano/Sarmeola%20-%20Chiesa%20dell%27Opera%20della%20Provvidenza%20di%20Sant%27%20Antonio
Timestamp
    1425385978
Title
    Disposizioni foniche di organi a canne/Europa/Italia/Veneto/Provincia di Padova/Rubano/Sarmeola - Chiesa dell'Opera della Provvidenza di Sant' Antonio
Wiki
    it.wikibooks.org 

Timestamps

Time stamps in Unix epoch time are not very human readable, nor do they clearly link to the actual version of the wiki page. So what about using the respective oldid and its human-readable timestamp (in UTC) instead or in addition?

WikiSym paper?

What about submitting a WikiSym paper along the lines of the Wikimania submissions?
The deadline for submission has been extended until next Monday (April 13):
http://www.wikisym.org/ .

track if a page is deleted

Redirects

The following entry suggests that the DOI 10.1080/02688690701447420 has been added to http://en.wikipedia.org/wiki/Blood%E2%80%93cerebrospinal%20barrier , but this is (and was) a redirect, and the DOI is actually used in the redirect's target, https://en.wikipedia.org/wiki/Choroid_plexus .

10.1080/02688690701447420 2015-02-21 Wikipedia Cocytus Action add Page URL http://en.wikipedia.org/wiki/Blood%E2%80%93cerebrospinal%20barrier Timestamp 1424486438

pushing link with diff format

https://en.wikipedia.org/w/index.php?title=Canarian_American&diff=651119337&oldid=651119306

line 12 of crossref_push: https://github.com/notconfusing/cocytus/blob/master/crossref_push.py#L12

instead of
article_url = url = "{server_url}/wiki/{safe_title}".format(server_url=server_url, safe_title=safe_title)
we need something like
https://en.wikipedia.org/wiki/index.php?title=Canarian_American&diff=651119337&oldid=651119306

diff = rcdict['revision']['new']
oldid = rcdict['revision']['old']
article_url = url = "{server_url}/w/index.php?title={safe_title}&diff={diff}&oldid={oldid}".format(server_url=server_url, safe_title=safe_title, diff=diff, oldid=oldid) ```

Migrate from socket.io to SSE / EventSource based EventStreams

Hiya! WMF is planning on decomissioning the socket.io based RCStream in July this year. We've built a new service called EventStreams that takes its place.

• https://blog.wikimedia.org/2017/03/20/eventstreams/
• https://wikitech.wikimedia.org/wiki/EventStreams

I think cocytus should be easy enough to switch over, but let me know if we can help in any way.

Thanks!

investigate consequences of nfs work

Ino in this Wikimedia issue about pending work they are doing to nfs, need to make sure we are not affected or plan accordingly if so.

https://phabricator.wikimedia.org/T87870

Trailing characters in listed DOIs

There are a number of cases where the extraction of the DOI is incomplete, leaving some trailing characters, e.g.

10.1111/tpj.12145/abstract 2015-02-20 Wikipedia Cocytus Action add Page URL http://en.wikipedia.org/wiki/Flagellum Timestamp 1424475490

10.1093/emboj/16.11.3219/full 2015-02-20 Wikipedia Cocytus Action remove Page URL http://en.wikipedia.org/wiki/User%3AOlaneli/sandbox Timestamp 1424472621

10.1099/ijs.0.025098-0\n 2015-02-20 Wikipedia Cocytus Action add Page URL http://en.wikipedia.org/wiki/Template%3ACite%20pmid/20639229 Timestamp 1424470143

10.1002/14356007.a04_011.pub2</ref> 2015-02-21 Wikipedia Cocytus Action add Page URL http://el.wikipedia.org/wiki/%CE%9F%CE%BE%CE%B5%CE%AF%CE%B4%CE%B9%CE%BF%20%CF%84%CE%BF%CF%85%20%CE%B2%CE%B7%CF%81%CF%85%CE%BB%CE%BB%CE%AF%CE%BF%CF%85 Timestamp 1424478430