sage-bionetworks/challengeutils

Synapse challenge utility functions

Sage Bionetworks Challenge Utilities

Warning

Starting early 2024, this package will no longer be maintained. We are in the process of creating a new-and-improved version of challengeutils - more information coming soon!

challengeutils is a set of tools and commands that provides an interface for managing crowd-sourced challenges administered on Synapse, including but not limited to, DREAM Challenges. Its main purpose is to ease the process of creating, monitoring, and ending a challenge, as well as provide useful functions for post-competition analysis. This package is being actively developed and maintained by DREAM and Informatics & Biocomputing (IBC), Computational Oncology Group at Sage Bionetworks.

Documentation

challengeutils functionality is documented, click here to check it out!

Installation

pip install challengeutils
challengeutils -v

This repository also uses pre-commit to autolint files according to Black's coding styles. To run the hook in your local dev environment, enter the following:

pip install pre-commit
pre-commit install

Now pre-commit will run automatically on git commit! For example:

$ git commit -m 'update readme' -a
Check Yaml...........................................(no files to check)Skipped
Fix End of Files.........................................................Passed
Trim Trailing Whitespace.................................................Passed
black................................................(no files to check)Skipped
[add-pre-commit 75b4393] update readme
 1 file changed, 23 insertions(+)

Contributing

Thinking about contributing to challengeutils? Get started by reading our Contributor Guide.

challengeutils's Issues

add a linter to test code

Beautify sphinx documentation

Simplify Scoring Harness

#73, #47

return invite if posted

return the invite object if it POSTs, otherwise None.

Add getting emails of participanting teams based on query

Prepare 1.0 release

#73 , #57

copy_thread should also copy replies

Currently the logic in copying threads and replies are in copy_forum. Update copy threads to have copy replies:

import synapseclient
syn = synapseclient.login
discussion_cls = challengeutils.discussion.DiscussionApi(syn)
thread_dict = discussion_cls.get_thread({threadid})
thread_bundle = challengeutils.discussion.copy_thread(syn, thread=thread_dict, project={projectid})
thread_replies = challengeutils.discussion.get_thread_replies(syn, {threadid})
for reply in thread_replies:
    challengeutils.discussion.copy_reply(syn, reply, thread_bundle['id'])

Update LIVE_PAGE_MARKDOWN

Center the preregister button and add <br>**Join them now!**

removal of members from a Team

Define what users see in __init__.py

Currently, there is nothing in __init__.py

Investigate using click over argparse

Add functions to interact with synapse discussions API

#25

Add in common challengestats functions as module

There are common functions that are used in gathering challenge data. These functions can be included in this repository.

Simplify command download submission

Remove

if filepath is not None:
            os.rename(filepath, 'submission-' + args.submissionid)
            filepath = 'submission-' + args.submissionid
        submission_dict['file_path'] = 'submission-' + args.submissionid
        with open(args.output, "w") as sub_out:
            json.dump(submission_dict, sub_out)
        logger.info(args.output)

Remove readthedocs

Once these two are pulled in and a release occurs, readthedocs can be removed.
#135
#136

move readme from subfolder to main

Linking project writeup submissions with main queue

https://github.com/Sage-Bionetworks/challengeutils/blob/master/archive-challenge-project-tool/challenge.py#L198

Move archive project tool to its own github

Link to Synapse documentation on Challenge Infrastructure

challengeutils createchallenge Does a lot of things, but what I do once it's done is not clear from the help message or the repository's README. We should link to the Challenge Infrastructure Synapse docs near the beginning of the README since it explains what to do with all the entities created by createchallenge https://docs.synapse.org/articles/challenge_administration.html

Separate DREAM challenge creation with minimal challenge creation

Currently our createchallenge creates a ton of components such as 2 projects and 4 teams that a user of the platform will most likely not appreciate. Convert this function into createdreamchallenge and re-scope the createchallenge function to create a more simple format.

use a release cycle

Should use semantic versioning with develop branch, etc. for managing updates. Can mirror what Synapse Python client does. @thomasyu888 we can discuss!

Add inline python functions from ChallengeWorkflowTemplate

per @andrewelamb and his work with the templates, the fact that toil requires the use of 2.7 doesn't mean that the tools used in the workflow should fall under this restriction.

• get docker configuration
• download submission

dev branch is behind master branch

I want to contribute a utility function (see https://sagebionetworks.jira.com/browse/CHALLENGE-86) but the dev branch is quite a few commits behind the master branch. It would be best if I could submit my PR to the dev branch and later merge dev with master.

I think a rebase should do the trick.

Get challenge participants

restGET("/challenge/{challengeid}/participant")

Wrap docker svg to a link to dockerhub

Fix notification flag in simple scoring harness

fails if destination has no root wiki

should create one if it needs to.

Use dataclasses for discussion, thread, reply objects

Send score email to entire team, rather than just the submitter

add move forum threads functionality

@thomasyu888 shared an awesome script with me for moving forum threads. Would be nice to have it here or in the Synapse Python client.

unregistering teams

Unregistering teams can only be done by a team manager of the team. Unsure if this would be helpful.

maintain order of wiki sub-pages when copying

https://docs.synapse.org/rest/GET/entity/ownerId/wiki2orderhint.html

Support synapseclient 2.0.0 in scoring harness

challengeutils query returns malformed csv

Because it sticks welcome information at the top, so the top two lines of stdout become:

Welcome, Robert Allaway!

Add tests to cli

---------- coverage: platform darwin, python 3.7.6-final-0 -----------
Name                                                 Stmts   Miss  Cover
------------------------------------------------------------------------
challengeutils/__init__.py                               3      0   100%
challengeutils/__main__.py                             176    176     0%
challengeutils/__version__.py                            1      0   100%
challengeutils/challenge.py                              6      6     0%
challengeutils/createchallenge.py                      100     14    86%
challengeutils/discussion.py                           127     28    78%
challengeutils/download_current_lead_submission.py      35      2    94%
challengeutils/evaluation_queue.py                      34      0   100%
challengeutils/helpers.py                               67     39    42%
challengeutils/mirrorwiki.py                            51     51     0%
challengeutils/permissions.py                           29      4    86%
challengeutils/utils.py                                202     56    72%
challengeutils/writeup_attacher.py                      18     18     0%
------------------------------------------------------------------------
TOTAL                                                  849    394    54%

Document command_download_submission

Currently the function is undocumented so the reason why the submission is renamed is unknown to the user.

Add helper functions for participating in challenge

• Createworkspace
• Upload submission + submit
  • File based
  • docker based
• Add documentation for using these tools for participating in challenge

mirror wiki fails

challengeutils -v 
1.6.0
 challengeutils mirrorwiki syn21849256 syn21849255
Downloading  [####################]100.00%   4.0kB/4.0kB (1.9MB/s) 52779013.md Done...
Downloading  [####################]100.00%   757.0bytes/757.0bytes (3.3MB/s) 51494118.md Done...
    INFO:challengeutils.mirrorwiki:Updating: 
Traceback (most recent call last):
  File "/Users/tschaffter/dev/challengeutils/env/bin/challengeutils", line 10, in <module>
    sys.exit(main())
  File "/Users/tschaffter/dev/challengeutils/env/lib/python3.7/site-packages/challengeutils/__main__.py", line 580, in main
    perform_main(syn, args)
  File "/Users/tschaffter/dev/challengeutils/env/lib/python3.7/site-packages/challengeutils/__main__.py", line 563, in perform_main
    args.func(syn, args)
  File "/Users/tschaffter/dev/challengeutils/env/lib/python3.7/site-packages/challengeutils/__main__.py", line 41, in command_mirrorwiki
    args.forceupdate)
  File "/Users/tschaffter/dev/challengeutils/env/lib/python3.7/site-packages/challengeutils/mirrorwiki.py", line 103, in mirrorwiki
    for filehandle in copied_filehandles['copyResults']]
TypeError: list indices must be integers or slices, not str
Collapse

Create tests

Reset docker submission tool (with quotas in mind)

{"key": "org.sagebionetworks.SynapseWorkflowHook.StatusDescription",
 "isPrivate": false,
 "value": "EXCEEDED TIME QUOTA"}
{"key": "org.sagebionetworks.SynapseWorkflowHook.TimeRemaining",
 "isPrivate": false,
 "value": 0}

Fix dead link to createChallenge.py in Synapse doc

The link behind "createChallenge.py" is dead in https://docs.synapse.org/articles/challenge_administration.html.

This issue was originally reported here: https://sagebionetworks.jira.com/browse/SWC-4729

Stop support for synapseclient < 2

test wiki names when mirroring

It chould notify if there are wiki pages without a matching name (either in the source and not the dest, or vice versa).

Refactor mirrorwiki

Add tests

use kwargs when you can, like in 'annotate_submission'

implement a 'dry run' for mirroring the wiki

It would be nice to have a 'test' run of mirroring to indicate what would happen without actually making any changes.

Submission status changer.

Closing #51, because the workflow is to fail the workflow job. But this status changer may still be helpful.

WorkflowHook feature -> Changing the status of a submission from ACCEPTED to INVALID, SCORED....

Expose invite member to team in command line

Per request of @mjrmason

Rename the command line functions calls

Calls like:

challengeutils setevaluationquota
challengeutils killdockeroverquota
challengeutils downloadsubmission

Should probably just be

challengeutils set-evaluation-quota
challengeutils kill-docker-over-quota
challengeutils download-submission

This will be a breaking change.

Show URL to resources created after creating challenge

Once the challenge and all its resources have been created by challengeutils createchallenge, it would be user friendly to display the URL to the resources (staging site, prod site, teams, etc.) just before returning. For example

...
Challenge name (prod site): URL
Challenge name (staging site): URL
Challenge name Admin (team): URL
Challenge name Participants (team): URL
...

These information can be retrieved from the current log. The suggested solution would save the user of challengeutils from scrolling through the logs.

change parameter 'to_public' to 'is_private' to match utils function parameter