Return the train-test splits from a task

[joaquinvanschoren]

Return the train-test splits from a task

[mfeurer]

Hm, ideally we would like an iterator regardless of the resampling strategy, right? Then we could do something like:

for train_indices, test_indices in task.iter_splits():
    ...

Additionaly to that, finding the number of folds and repeats seems like a good idea.

[joaquinvanschoren]

Yes, that sounds even better!

[mfeurer]

I just created a new feature branch and added this feature with commit 9329070

@PG-TUe can you please check whether this helps you? I can then merge it to develop and you can rebase your branch.

[PGijsbers]

Will try today/tomorrow (most likely tomorrow), and report back. Thanks.

[PGijsbers]

I've been looking at the code, and while the iteration over repeats and folds works well, there is one issue. As far as I can tell, there is still no way to find the number of repeats and folds from just the task object. This in turn gives issues, because using task.iterate_splits(), you don't know when the next split is actually a new fold of the same repeat, or a new fold of a new repeat. Of course, it is very important to know on which repeat/fold specifically your results were obtained for your experiments (and the predictions file).

So, the new code is fine, but additionally we would need a way to retrieve the number of repeats and folds from the task directly, before it is of any use.

Of course, I might have missed something, and perhaps it is possible to see the number of repeats and folds without accessing the split object. If so, please point me in the right direction ;)

[PGijsbers]

I got around to doing it myself (commit). Iterating over all folds and repeats together as one can still be done:

task.iterate_all_splits()

However, to more easily use repeats and folds, you can now also do:

for rep in task.iterate_repeats():
    for fold in rep:
        train, test = fold

Of course, you can instead use for train, test in rep directly as well.

[mfeurer]

Looks great. Can this be closed?

[PGijsbers]

In that case, yes. Issue closed.