Make sure that invalid submissions in UW queue are propogated to the main queue

[data2health]

Make sure that invalid submissions in UW queue are propogated to the main queue

[thomasyu888]

That's the second solution. It would just be a script that queries the two leaderboards, and sets respective submissions to REJECTED. I can implement once I'm back from the dream conference.

[thomasyu888]

@tschaffter. What do you mean by tiding?

[tschaffter]

@thomasyu888 I meant "Tidying up" instead of "Tiding", which is still a loose title for this ticket. Here I'm trying to capture work which I think you mentioned relative to submission IDs. Does that ring a bell?

[thomasyu888]

Ah. I changed the title. The workflow is

1. Submit model to main queue,
2. Validated models from 1 are submitted to UW internal queue
3. Ideally models in 2 would be SCORED, but.... if not, have to make sure that the submissions made in 1 are made INVALID, or else participants won't be able to submit more. (due to the submission quota)

[tschaffter]

@thomasyu888 Is there an update for this issue?

[thomasyu888]

There isn't. I'm manually rejecting submissions that pass the synthetic data set but fail on the main UW queue.

[tschaffter]

Is there a way to automatic this task?

[thomasyu888]

Hmm.. will have to look into this more. One possible solution is to add a step to check the status of the UW run, and if INVALID set the successful submission to the main queue as REJECTED. However, there are cases where the rundocker step fails. In this scenario, we would also need to set the submission to the main queue as REJECTED.

Another solution is to have a tool external to the workflows to look at submissions that are VALIDATED in the main queue and INVALID in the UW queue. When this is seen, the VALIDATED submission becomes REJECTED.

The 2nd solution is definitely easier to implement, but will have some sort of lag time (1-5 minutes). The 1st solution would require more thought.

[trberg]

I agree, I think the 2nd solution is an easier short term fix. I've also been using the challengeutils tool to change the status.

[thomasyu888]

Ya, I have been using challengeutils to update as well.

[trberg]

Would there be a way to use the challengeutils tool as a cronjob and just check submissions every minute or so?

[thomasyu888]

Closing for now, since I implemented the workaround.