Tasks creation progress bar

[toloka]

Tasks creation progress bar

[Losik]

Hello.

I am uploading multiple tasks to a pool using toloka_client.create_tasks(tasks) and would like to have an estimate on how long the upload would take. Is there any way to monitor the progress of task creation?

Unfortunately, there is no out of the box way to monitor progress at the moment =( Am I right to assume that your upload has taken segnificantly more time than you expected? If so, one insight is that currently, upload time is more affected by the number of tasks, not the data size.

I realise that I can wrap a tqdm around a loop with toloka_client.create_task(task), but that would switch the upload from asynchronous to synchronous? Is there a better way to do this?

If I understood correctly, you are asking if a code like this will work slower that simple toloka_client.create_task(task)

from tqdm import tqdm
from itertools import islice

def split_into_batches(sequence, size):
    """Splits a given sequesnce into a list of lists of consequtive elements"""
    iterator = iter(sequence)
    return list(iter(lambda: list(islice(iterator, size)), []))

...

for tasks_batch in tqdm(split_into_batches(tasks, 3)):
    toloka_client.create_tasks(tasks_batch)

If so, the answer is no, it won't be slower. async_mode does not stand for Python coroutines but for asynchronous type of communication with API. Let me explain:

• If async_mode is True (default) the method will upload your tasks in one HTTP request, get back an TasksCreateOperation instance and poll its status until uploaded tasks won't be actually created.
• If async_mode is False there will be a single HTTP request to API that will hang until both tasks upload and processing is finished. This is not a recommended approach, because this single request may hit a request timeout

So there is no parallelism involved and you should be able to use tqdm + split_into_batches without significant performance drop.

[arinaruck]

Thank you! The tqdm + batchification solve my issue, indeed!