toloka/toloka-kit

Toloka-Kit is a Python library for working with Toloka API.

toloka-kit's Issues

[DOCS]

Problem description

1. Create a pool via UI
2. Upload tasks with create_tasks via Kit
3. Pool can't be launched because there's no mixer and no suites are created

Documentation links

https://toloka.ai/en/docs/toloka-kit/reference/toloka.client.TolokaClient.create_tasks

Potential fix suggestion

Here mixer_config is mentioned: https://toloka.ai/en/docs/api/concepts/tasks
Maybe add smth like this

[Feature]: Add parameter 'infinite_overlap': True for control tasks

Hey, it'd be great to add infinite overlap for control tasks in this example: https://toloka.ai/en/docs/toloka-kit/reference/toloka.client.TolokaClient.create_tasks and somehow highlight that it's important

Improvement 6.streaming_pipelines example

Thanks for a very helpful example 6.streaming_pipelines. Using this example, I figured out how to work with streaming pipelines.

The only point that I think is worth improving:

The HandleSbS class has a line self.scores_by_image[image] = scores.sort_values(ascending=False), that stores the final markup responses. At the same time, after the end of the pipeline, the result of the final marking is not displayed anywhere. It would be great if the notebook had a final cell in which the markup result was displayed.

[BUG] type annotations for optional parameters

Observed behavior

Type annotations of optional parameters are not propagated to __init__ methods of the attrs generated classes, which leads to incrorect stubs and failing mypy.

Expected behavior

Every Optional[...] or Union[..., None] parameter should have a correct annotation in the __init__ method

Python Version

3.9

Toloka-Kit Version

1.0.2

Other Packages Versions

No response

Example code

toloka-kit/src/client/search_requests.pyi

Line 614 in 8d672fb

status: typing.Union[str, toloka.client.assignment.Assignment.Status, typing.List[typing.Union[str, toloka.client.assignment.Assignment.Status]]] = None,

Relevant log output

No response

Task from example project SQUAD2.0 is not built as expected (validation error in preview).

The task from example project for SQUAD2.0 notebook is not built as expected. Performers will not be able to mark "No" option because it would cause validation error:

Moreover it works fine in the same project with preview of template builder in web-version:

I guess it is enough to add required=False for 'answer' in output specs for project, because the check that the answer is not empty is built-in by default, but does the template builder work as intended in this case?

[BUG] TypeError: __init__() got an unexpected keyword argument 'verified'

Observed behavior

The exception occurs when trying to set verified=False to the languages filter with str value.

  File "/usr/lib/python3.8/code.py", line 90, in runcode
    exec(code, self.locals)
  File "<console>", line 1, in <module>
  File "/home/maya/Documents/cvat/env/lib/python3.8/site-packages/toloka/client/primitives/operators.py", line 109, in include
    return cls(InclusionOperator.IN, *args, **kwargs)
TypeError: __init__() got an unexpected keyword argument 'verified'

Expected behavior

Explicitly specifying the verified=False parameter should not affect the creation of the filter.

Python Version

3.8

Toloka-Kit Version

1.2.0.post1

Other Packages Versions

No response

Example code

toloka.filter.Languages.in_('EN', verified=False)

Relevant log output

No response

"UnboundLocalError: local variable 'start_soon' referenced before assignment" in pipeline.py

Hi!

I occasionally – not every time – encounter the following error when running a Toloka pipeline:

Traceback (most recent call last):
  File "demo.py", line 25, in <module>
    y.start()
  File "pipeline.py", line 93, in start
    loop.run_until_complete(self.pipeline.run())
  File "/usr/local/Cellar/[email protected]/3.9.7/Frameworks/Python.framework/Versions/3.9/lib/python3.9/asyncio/base_events.py", line 642, in run_until_complete
    return future.result()
  File "/lib/python3.9/site-packages/toloka/streaming/pipeline.py", line 270, in run
    sleep_time = (start_soon - datetime.now()).total_seconds()
UnboundLocalError: local variable 'start_soon' referenced before assignment

To give you some context, you can find the code for both demo.py and my pipeline.py below:

demo.py

# -*- coding: utf-8 -*-

from task_specs import ImageClassificationTask
from pipeline import TaskSequence
import json
import toloka.client as toloka

# Read the credentials
with open('creds.json') as cred_f:

    creds = json.loads(cred_f.read())
    tclient = toloka.TolokaClient(creds['token'], creds['mode'])

exam = ImageClassificationTask(configuration='text_exam.json', client=tclient)

main = ImageClassificationTask(configuration='text_detect.json', client=tclient)

outline = ImageClassificationTask(configuration='text_outline.json', client=tclient)

y = TaskSequence(sequence=[exam, main, outline], client=tclient)

y.start()

pipeline.py

# -*- coding: utf-8 -*-

import asyncio
from core_functions import *
from toloka.util.async_utils import AsyncMultithreadWrapper
from toloka.streaming import AssignmentsObserver, Pipeline, PoolStatusObserver
from wasabi import Printer

msg = Printer(pretty=True, timestamp=True, hide_animation=True)


class TaskSequence:
    def __init__(self, sequence, client):

        # Set up attributes
        self.complete = False       # Tracks if all tasks have been completed
        self.sequence = sequence    # A list of CrowdsourcingTask objects
        self.client = client        # A Toloka Client object
        self.pipeline = None        # Placeholder for a Toloka Pipeline object

        msg.info(f'Creating a task sequence')

        # Loop over the tasks to verify that they are connected properly
        for task in sequence:

            # Check if actions have been defined in the configuration
            if task.action_conf:

                # Check if the next task has been defined in the configuration
                if task.action_conf['next']:

                    # Fetch the name of the next task from the configuration
                    next_task = task.action_conf['next']

                    # Check that the next task exists in the task sequence
                    if next_task not in [task.name for task in sequence]:

                        raise_error(f'Cannot find a task named {next_task} in the task sequence. '
                                    f'Please check the name of the task under the key '
                                    f'"actions/next" in the configuration file.')

        msg.info(f'Printing tasks, inputs and outputs')

        # Set up headers and a placeholder for data
        header = ('Name', 'Input', 'Output', 'Pool ID')
        data = []

        # Loop over the tasks
        for task in sequence:

            # Collect input and output data from the configuration
            inputs = [f'{k} ({v})' for k, v in task.conf['data']['input'].items()]
            outputs = [f'{k} ({v})' for k, v in task.conf['data']['output'].items()]

            # Append data as a tuple to the list
            data.append((task.name, ', '.join(inputs), ', '.join(outputs), task.pool.id))

        # Print a table with inputs and outputs
        msg.table(data=data, header=header, divider=True)

        # Create the pipeline
        self.create_pipeline()

    def start(self):

        # Create an event loop
        loop = asyncio.get_event_loop()

        try:

            msg.info(f'Starting the task sequence')

            # Open all pools in the sequence that contain tasks. Note that pools without tasks cannot
            # be opened: they will be opened when tasks are added to them by these initial tasks.
            for task in self.sequence:

                if task.tasks is not None:

                    # Open main pool
                    self.client.open_pool(pool_id=task.pool.id)

                if task.training is not None:

                    # Open training pool
                    self.client.open_pool(pool_id=task.training.id)

            # Call the Toloka pipeline() method within the event loop
            loop.run_until_complete(self.pipeline.run())

        finally:

            # Finish the event loop
            loop.close()

            msg.good(f'Successfully completed the task sequence')

    def create_pipeline(self):

        # Create an asyncronous client
        async_client = AsyncMultithreadWrapper(self.client)

        # Loop over the tasks and create an AssignmentsObserver object for each task. Exam tasks
        # are excluded, because they do not require observers, as they do not generate further
        # tasks.
        a_observers = {task.name: AssignmentsObserver(async_client, task.pool.id)
                       for task in self.sequence if not task.exam}

        # Set up pool observers for monitoring all pool states
        p_observers = {task.name: PoolStatusObserver(async_client, task.pool.id)
                       for task in self.sequence}

        # Create a Toloka Pipeline object and register observers
        self.pipeline = Pipeline()

        # Register each assignment observer with the Pipeline object
        for name, a_observer in a_observers.items():

            self.pipeline.register(observer=a_observer)

            msg.info(f'Registered an assignments observer for task {name}')

        # Register each pool observer and actions
        for name, p_observer in p_observers.items():

            p_observer.on_closed(lambda pool: msg.info(f'Closed pool with ID {pool.id}'))
            p_observer.on_open(lambda pool: msg.info(f'Opened pool with ID {pool.id}'))

            self.pipeline.register(observer=p_observer)

            msg.info(f'Registered a pool observer for task {name}')

        # Create a dictionary of CrowdsourcingTasks keyed by their names
        task_objs = {task.name: task for task in self.sequence}

        # Loop over the observers and get the actions configuration to determine task flow
        for name, observer in a_observers.items():

            # Get the CrowdsourcingTask object from the TaskSequence by matching its name
            current_task = task_objs[name]

            # Check if actions have been configured
            if current_task.action_conf is not None:

                if 'on_accepted' in current_task.action_conf:

                    if type(current_task.action_conf['on_accepted']) == str:

                        # Attempt to register the action with the AssignmentObserver. If a task is
                        # accepted, it will be sent to the CrowdsourcingTask object defined in the
                        # configuration.
                        try:

                            observer.on_accepted(task_objs[current_task.action_conf['on_accepted']])

                            msg.info(f'Setting up a connection from {name} to '
                                     f'{task_objs[current_task.action_conf["on_accepted"]].name} '
                                     f'on acceptance ...')

                        except KeyError:

                            raise_error(f'Could not find a CrowdsourcingTask object named '
                                        f'{current_task.action_conf["on_accepted"]} in the '
                                        f'TaskSequence. Please check the configuration '
                                        f'under the key "actions"!')

                if 'on_submitted' in current_task.action_conf:

                    try:

                        # Register the action with the AssignmentObserver. If a task is submitted,
                        # it will be sent to the CrowdsourcingTask object defined in the configuration.
                        observer.on_submitted(task_objs[current_task.action_conf['on_submitted']])

                        msg.info(f'Setting up a connection from {name} to '
                                 f'{task_objs[current_task.action_conf["on_submitted"]].name} '
                                 f'on submission ...')

                    except KeyError:

                        raise_error(f'Could not find a CrowdsourcingTask object named '
                                    f'{current_task.action_conf["on_submitted"]} in the '
                                    f'TaskSequence. Please check the configuration '
                                    f'under the key "actions"!')

                if 'on_rejected' in current_task.action_conf:

                    try:

                        # Register the action with the AssignmentObserver. If a task is rejected,
                        # it will be sent to the CrowdsourcingTask object defined in the configuration.
                        observer.on_rejected(task_objs[current_task.action_conf['on_rejected']])

                        msg.info(f'Setting up a connection from {name} to '
                                 f'{task_objs[current_task.action_conf["on_rejected"]].name} '
                                 f'on rejection ...')

                    except KeyError:

                        raise_error(f'Could not find a CrowdsourcingTask object named '
                                    f'{current_task.action_conf["on_rejected"]} in the '
                                    f'TaskSequence. Please check the configuration '
                                    f'under the key "actions"!')

Let me know if you have any questions!

Connecting to Я.Задания

Problem description

Seems that toloka-kit can't be used with Я.задания out of box.

Feature description

Hi! Could you describe please how toloka-kit could be used with Я.Задания?

Potential alternatives

No response

Additional information

No response

dynamic answer processing feature

Hi! I think it would be great to have an opportunity to process annotation results on-the-fly, what do you think? Possible use cases are:

• if the annotator makes a lot of mistakes on golden sets we can show him feedback on what he is doing bad and where he has to focus his attention
• ban him immediately if he is still doing bad annotation and resend this task in pool

Make optional dependencies

Some of dependencies could be optional:

'pandas',
'plotly',
'ipyplot',
'jupyter-dash',

May be more.
Those are huge packages and not needed for everyone. You could wrap it with "extras_require" in setup and dummy imports in code like

try:
  import pandas as pd
except ImportError:
  pd = None

Infinite overlap

Hi,

How to make infinite overlap like in API?
https://yandex.ru/dev/toloka/doc/concepts/edit-overlap.html

[Question] Pool type

Hii, is it somehow possible to set a pool type in Toloka Kit?
Like in this example with exam pool: https://colab.research.google.com/github/oleg-cat/blood-test/blob/main/blood-test.ipynb ?
As far as I can see from the Docs - there can be only 2 types: regular and training. Though it's possible to set in the UI

Aggregation issue

I try to aggregate result from my pool with code:

    aggregation_operation = toloka_client.aggregate_solutions_by_pool(
        type=AggregatedSolutionType.DAWID_SKENE,
        pool_id=pool_id,
        fields=[toloka.aggregation.PoolAggregatedSolutionRequest.Field(name=field_name)]
    )
    aggregation_operation = toloka_client.wait_operation(aggregation_operation)
    print('Results aggregated')

    aggregation_result = toloka_client.find_aggregated_solutions(aggregation_operation.id, limit=100_000)
    verification_results = aggregation_result.items
    while aggregation_result.has_more:
        aggregation_result = toloka_client.find_aggregated_solutions(
            aggregation_operation.id,
            task_id_gt=aggregation_result.items[len(aggregation_result.items) - 1].task_id,
        )
        verification_results = verification_results + aggregation_result.items

But I get only 3660 task aggregation, but there are 5000 tasks, if I use online version, it works correct (on different pools the same story).

why does it happen? and how to fix it?
Maybe better use API without the KIT?
Thank you for your answer.

[FEATURE] Add support of cattr 22.2.0

Problem description

Currently when I'm trying to install package to Google Cloud Composer (managed Airflow cluster) I get the following error on recent version of Composer:

toloka-kit 1.0.2 has requirement cattrs<22.2.0,>=1.1.1, but you have cattrs 22.2.0.

I can see in setup.py a comment:

# cattr 22.2.0 breaks tests/primitives/test_operator.py
        'cattrs >= 1.1.1, < 22.2.0',

Feature description

Can you please consider adding support of cattr==22.2.0?

Potential alternatives

No response

Additional information

No response

[BUG] Impossible to get a pool with filter that contains a subfilter with null value

Observed behavior

If for some reason a pool contains a filter with null value (e.g. client.filter.DateOfBirth == None) then such pool cannot be received via get_pool. It is also a question of whether such filters should be allowed...

Expected behavior

No response

Python Version

3.8

Toloka-Kit Version

1.2.0.post1

Other Packages Versions

No response

Example code

pool = toloka_client.get_pool(pool_id)

Relevant log output

+ Exception Group Traceback (most recent call last):
  |   File "<stdin>", line 1, in <module>
  |   File "/home/maya/Documents/cvat/env/lib/python3.8/site-packages/toloka/util/_managing_headers.py", line 73, in wrapped
  |     return run_in_current_context(func, *args, **kwargs)
  |   File "/home/maya/Documents/cvat/env/lib/python3.8/site-packages/toloka/util/_managing_headers.py", line 100, in run_in_current_context
  |     result = func(*args, **kwargs)
  |   File "/home/maya/Documents/cvat/env/lib/python3.8/site-packages/toloka/client/__init__.py", line 1587, in get_pool
  |     >>> new_pool = toloka.client.Pool(
  |   File "/home/maya/Documents/cvat/env/lib/python3.8/site-packages/cattrs/converters.py", line 334, in structure
  |     return self._structure_func.dispatch(cl)(obj, cl)
  |   File "/home/maya/Documents/cvat/env/lib/python3.8/site-packages/toloka/client/_converter.py", line 21, in <lambda>
  |     lambda type_: hasattr(type_, 'structure'),
  |   File "/home/maya/Documents/cvat/env/lib/python3.8/site-packages/toloka/client/primitives/base.py", line 317, in structure
  |     continue
  |   File "/home/maya/Documents/cvat/env/lib/python3.8/site-packages/cattrs/converters.py", line 334, in structure
  |     return self._structure_func.dispatch(cl)(obj, cl)
  |   File "/home/maya/Documents/cvat/env/lib/python3.8/site-packages/cattrs/converters.py", line 646, in _structure_optional
  |     return self._structure_func.dispatch(other)(obj, other)
  |   File "/home/maya/Documents/cvat/env/lib/python3.8/site-packages/toloka/client/_converter.py", line 21, in <lambda>
  |     lambda type_: hasattr(type_, 'structure'),
  |   File "/home/maya/Documents/cvat/env/lib/python3.8/site-packages/toloka/client/filter.py", line 88, in structure
  |     return FilterAnd.structure(data)
  |   File "/home/maya/Documents/cvat/env/lib/python3.8/site-packages/toloka/client/filter.py", line 144, in structure
  |     return super(FilterCondition, cls).structure(data)
  |   File "/home/maya/Documents/cvat/env/lib/python3.8/site-packages/toloka/client/primitives/base.py", line 317, in structure
  |     continue
  |   File "/home/maya/Documents/cvat/env/lib/python3.8/site-packages/cattrs/converters.py", line 334, in structure
  |     return self._structure_func.dispatch(cl)(obj, cl)
  |   File "/home/maya/Documents/cvat/env/lib/python3.8/site-packages/cattrs/converters.py", line 545, in _structure_list
  |     raise IterableValidationError(
  | cattrs.errors.IterableValidationError: While structuring typing.List[toloka.client.filter.FilterCondition] (1 sub-exception)
  +-+---------------- 1 ----------------
    | Exception Group Traceback (most recent call last):
    |   File "/home/maya/Documents/cvat/env/lib/python3.8/site-packages/cattrs/converters.py", line 535, in _structure_list
    |     res.append(handler(e, elem_type))
    |   File "/home/maya/Documents/cvat/env/lib/python3.8/site-packages/toloka/client/_converter.py", line 21, in <lambda>
    |     lambda type_: hasattr(type_, 'structure'),
    |   File "/home/maya/Documents/cvat/env/lib/python3.8/site-packages/toloka/client/filter.py", line 86, in structure
    |     return FilterOr.structure(data)
    |   File "/home/maya/Documents/cvat/env/lib/python3.8/site-packages/toloka/client/filter.py", line 117, in structure
    |     return super(FilterCondition, cls).structure(data)
    |   File "/home/maya/Documents/cvat/env/lib/python3.8/site-packages/toloka/client/primitives/base.py", line 317, in structure
    |     continue
    |   File "/home/maya/Documents/cvat/env/lib/python3.8/site-packages/cattrs/converters.py", line 334, in structure
    |     return self._structure_func.dispatch(cl)(obj, cl)
    |   File "/home/maya/Documents/cvat/env/lib/python3.8/site-packages/cattrs/converters.py", line 545, in _structure_list
    |     raise IterableValidationError(
    | cattrs.errors.IterableValidationError: While structuring typing.List[toloka.client.filter.FilterCondition] (1 sub-exception)
    | Structuring typing.List[toloka.client.filter.FilterCondition] @ index 0
    +-+---------------- 1 ----------------
      | Traceback (most recent call last):
      |   File "/home/maya/Documents/cvat/env/lib/python3.8/site-packages/cattrs/converters.py", line 535, in _structure_list
      |     res.append(handler(e, elem_type))
      |   File "/home/maya/Documents/cvat/env/lib/python3.8/site-packages/toloka/client/_converter.py", line 21, in <lambda>
      |     lambda type_: hasattr(type_, 'structure'),
      |   File "/home/maya/Documents/cvat/env/lib/python3.8/site-packages/toloka/client/filter.py", line 90, in structure
      |     return Condition.structure(data)
      |   File "/home/maya/Documents/cvat/env/lib/python3.8/site-packages/toloka/client/filter.py", line 175, in structure
      |     return super(FilterCondition, cls).structure(data)
      |   File "/home/maya/Documents/cvat/env/lib/python3.8/site-packages/toloka/client/primitives/base.py", line 305, in structure
      |     spec_class = cls._variant_registry.generate_subtype(cls, spec_value)
      |   File "/home/maya/Documents/cvat/env/lib/python3.8/site-packages/toloka/client/filter.py", line 175, in structure
      |     return super(FilterCondition, cls).structure(data)
      |   File "/home/maya/Documents/cvat/env/lib/python3.8/site-packages/toloka/client/primitives/base.py", line 305, in structure
      |     spec_class = cls._variant_registry.generate_subtype(cls, spec_value)
      |   File "/home/maya/Documents/cvat/env/lib/python3.8/site-packages/toloka/client/filter.py", line 175, in structure
      |     return super(FilterCondition, cls).structure(data)
      |   File "/home/maya/Documents/cvat/env/lib/python3.8/site-packages/toloka/client/primitives/base.py", line 317, in structure
      |     continue
      |   File "/home/maya/Documents/cvat/env/lib/python3.8/site-packages/cattrs/converters.py", line 334, in structure
      |     return self._structure_func.dispatch(cl)(obj, cl)
      |   File "/home/maya/Documents/cvat/env/lib/python3.8/site-packages/cattrs/converters.py", line 440, in _structure_call
      |     return cl(obj)
      | TypeError: int() argument must be a string, a bytes-like object or a number, not 'NoneType'
      | Structuring typing.List[toloka.client.filter.FilterCondition] @ index 0
      +------------------------------------

404 from API is not eloquent enough

Toloka API returns just 404 in case of inability to find a valid entity. But if I am making a request that uses few entities (like I am trying to set skills for workers -- I am using both worker and skill entities), there is no easy way to distinguish whether 404 is due because skill does not exist, or worker.

[BUG] Error with certificates on the new version

Observed behavior

I have all the necessary certificates
When creating TolokaClient on version 0.1.22, everything works, but an error occurs on version 1.2.2. On version 1.2.2, you have to specify an explicit path to the certificate in the Toloka Client constructor

Expected behavior

No response

Python Version

3.8

Toloka-Kit Version

1.2.2

Other Packages Versions

No response

Example code

// working example on 0.1.22
toloka_client = TolokaClient(
    os.environ['TOLOKA_TOKEN'], url=args.toloka_api_url,
)

// The example above does not work on version 1.2.2
// working example on 1.2.2
toloka_client = TolokaClient(
    os.environ['TOLOKA_TOKEN'], url=args.toloka_api_url,
    verify=args.path_to_sert,
)

Relevant log output

Error accessing the Toloka API:

Exception: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1131)

Image Segmentation example is in fact Object Detection

This example says that it's an image segmentation project. However, image segmentation means that you can select an arbitrary area of an image. In fact, in this project, performers only draw bounding boxes, so it's an object detection task.

Please fix this in the title and the notebook.

KeyError while using toloka.client.TolokaClient.find_pools/get_pools

I use toloka.client.TolokaClient.find_pools, today I get KeyError:

---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
~/venv/jupyter_3.9/lib/python3.9/site-packages/toloka/client/primitives/base.py in structure(cls, data)
    264                 spec_value = cls._variant_registry.enum(data_field)
--> 265                 spec_class = cls._variant_registry.registered_classes[spec_value]
    266             except Exception:

KeyError: <Key.verified: 'verified'>

During handling of the above exception, another exception occurred:

SpecClassIdentificationError              Traceback (most recent call last)
/tmp/ipykernel_26997/4196115752.py in <module>
----> 1 for pool in client.find_pools(PoolSearchRequest(status=[toloka.pool.Pool.Status.OPEN])):
               pass

~/venv/jupyter_3.9/lib/python3.9/site-packages/toloka/util/_codegen.py in wrapped(*args, **kwargs)
    222                     fit, func_problem = _check_arg_type_compatibility(func_sig, arg_name, arg_type, bound)
    223                     if fit:
--> 224                         return func(*args, **kwargs)
    225                 else:
    226                     return func(*args, **kwargs)

~/venv/jupyter_3.9/lib/python3.9/site-packages/toloka/util/_managing_headers.py in wrapped(*args, **kwargs)
     53                     stack.enter_context(SetVariable(top_level_method_var, func.__name__))
     54 
---> 55                 return func(*args, **kwargs)
     56 
     57         return wrapped

~/venv/jupyter_3.9/lib/python3.9/site-packages/toloka/client/__init__.py in find_pools(self, request, sort, limit)
   1466         sort = None if sort is None else structure(sort, search_requests.PoolSortItems)
   1467         response = self._search_request('get', '/v1/pools', request, sort, limit)
-> 1468         return structure(response, search_results.PoolSearchResult)
   1469 
   1470     @add_headers('client')

~/venv/jupyter_3.9/lib/python3.9/site-packages/cattr/converters.py in structure(self, obj, cl)
    292         """Convert unstructured Python data structures to structured data."""
    293 
--> 294         return self._structure_func.dispatch(cl)(obj, cl)
    295 
    296     # Classes to Python primitives.

~/venv/jupyter_3.9/lib/python3.9/site-packages/toloka/client/_converter.py in <lambda>(data, type_)
     15 converter.register_structure_hook_func(
     16     lambda type_: hasattr(type_, 'structure'),
---> 17     lambda data, type_: type_.structure(data)  # type: ignore
     18 )
     19 converter.register_unstructure_hook_func(  # type: ignore

~/venv/jupyter_3.9/lib/python3.9/site-packages/toloka/client/primitives/base.py in structure(cls, data)
    279             value = data.pop(key)
    280             if field.type is not None:
--> 281                 value = converter.structure(value, field.type)
    282 
    283             kwargs[field.name] = value

~/venv/jupyter_3.9/lib/python3.9/site-packages/cattr/converters.py in structure(self, obj, cl)
    292         """Convert unstructured Python data structures to structured data."""
    293 
--> 294         return self._structure_func.dispatch(cl)(obj, cl)
    295 
    296     # Classes to Python primitives.

~/venv/jupyter_3.9/lib/python3.9/site-packages/cattr/converters.py in _structure_optional(self, obj, union)
    523         )
    524         # We can't actually have a Union of a Union, so this is safe.
--> 525         return self._structure_func.dispatch(other)(obj, other)
    526 
    527     def _structure_union(self, obj, union):

~/venv/jupyter_3.9/lib/python3.9/site-packages/cattr/converters.py in _structure_list(self, obj, cl)
    470         else:
    471             elem_type = cl.__args__[0]
--> 472             return [
    473                 self._structure_func.dispatch(elem_type)(e, elem_type)
    474                 for e in obj

~/venv/jupyter_3.9/lib/python3.9/site-packages/cattr/converters.py in <listcomp>(.0)
    471             elem_type = cl.__args__[0]
    472             return [
--> 473                 self._structure_func.dispatch(elem_type)(e, elem_type)
    474                 for e in obj
    475             ]

~/venv/jupyter_3.9/lib/python3.9/site-packages/toloka/client/_converter.py in <lambda>(data, type_)
     15 converter.register_structure_hook_func(
     16     lambda type_: hasattr(type_, 'structure'),
---> 17     lambda data, type_: type_.structure(data)  # type: ignore
     18 )
     19 converter.register_unstructure_hook_func(  # type: ignore

~/venv/jupyter_3.9/lib/python3.9/site-packages/toloka/client/primitives/base.py in structure(cls, data)
    279             value = data.pop(key)
    280             if field.type is not None:
--> 281                 value = converter.structure(value, field.type)
    282 
    283             kwargs[field.name] = value

~/venv/jupyter_3.9/lib/python3.9/site-packages/cattr/converters.py in structure(self, obj, cl)
    292         """Convert unstructured Python data structures to structured data."""
    293 
--> 294         return self._structure_func.dispatch(cl)(obj, cl)
    295 
    296     # Classes to Python primitives.

~/venv/jupyter_3.9/lib/python3.9/site-packages/cattr/converters.py in _structure_optional(self, obj, union)
    523         )
    524         # We can't actually have a Union of a Union, so this is safe.
--> 525         return self._structure_func.dispatch(other)(obj, other)
    526 
    527     def _structure_union(self, obj, union):

~/venv/jupyter_3.9/lib/python3.9/site-packages/toloka/client/_converter.py in <lambda>(data, type_)
     15 converter.register_structure_hook_func(
     16     lambda type_: hasattr(type_, 'structure'),
---> 17     lambda data, type_: type_.structure(data)  # type: ignore
     18 )
     19 converter.register_unstructure_hook_func(  # type: ignore

~/venv/jupyter_3.9/lib/python3.9/site-packages/toloka/client/filter.py in structure(cls, data)
     80             return FilterOr.structure(data)
     81         if 'and' in data:
---> 82             return FilterAnd.structure(data)
     83         else:
     84             return Condition.structure(data)

~/venv/jupyter_3.9/lib/python3.9/site-packages/toloka/client/filter.py in structure(cls, data)
    130     @classmethod
    131     def structure(cls, data):
--> 132         return super(FilterCondition, cls).structure(data)
    133 
    134 

~/venv/jupyter_3.9/lib/python3.9/site-packages/toloka/client/primitives/base.py in structure(cls, data)
    279             value = data.pop(key)
    280             if field.type is not None:
--> 281                 value = converter.structure(value, field.type)
    282 
    283             kwargs[field.name] = value

~/venv/jupyter_3.9/lib/python3.9/site-packages/cattr/converters.py in structure(self, obj, cl)
    292         """Convert unstructured Python data structures to structured data."""
    293 
--> 294         return self._structure_func.dispatch(cl)(obj, cl)
    295 
    296     # Classes to Python primitives.

~/venv/jupyter_3.9/lib/python3.9/site-packages/cattr/converters.py in _structure_list(self, obj, cl)
    470         else:
    471             elem_type = cl.__args__[0]
--> 472             return [
    473                 self._structure_func.dispatch(elem_type)(e, elem_type)
    474                 for e in obj

~/venv/jupyter_3.9/lib/python3.9/site-packages/cattr/converters.py in <listcomp>(.0)
    471             elem_type = cl.__args__[0]
    472             return [
--> 473                 self._structure_func.dispatch(elem_type)(e, elem_type)
    474                 for e in obj
    475             ]

~/venv/jupyter_3.9/lib/python3.9/site-packages/toloka/client/_converter.py in <lambda>(data, type_)
     15 converter.register_structure_hook_func(
     16     lambda type_: hasattr(type_, 'structure'),
---> 17     lambda data, type_: type_.structure(data)  # type: ignore
     18 )
     19 converter.register_unstructure_hook_func(  # type: ignore

~/venv/jupyter_3.9/lib/python3.9/site-packages/toloka/client/filter.py in structure(cls, data)
     78     def structure(cls, data: dict):
     79         if 'or' in data:
---> 80             return FilterOr.structure(data)
     81         if 'and' in data:
     82             return FilterAnd.structure(data)

~/venv/jupyter_3.9/lib/python3.9/site-packages/toloka/client/filter.py in structure(cls, data)
    106     @classmethod
    107     def structure(cls, data):
--> 108         return super(FilterCondition, cls).structure(data)
    109 
    110 

~/venv/jupyter_3.9/lib/python3.9/site-packages/toloka/client/primitives/base.py in structure(cls, data)
    279             value = data.pop(key)
    280             if field.type is not None:
--> 281                 value = converter.structure(value, field.type)
    282 
    283             kwargs[field.name] = value

~/venv/jupyter_3.9/lib/python3.9/site-packages/cattr/converters.py in structure(self, obj, cl)
    292         """Convert unstructured Python data structures to structured data."""
    293 
--> 294         return self._structure_func.dispatch(cl)(obj, cl)
    295 
    296     # Classes to Python primitives.

~/venv/jupyter_3.9/lib/python3.9/site-packages/cattr/converters.py in _structure_list(self, obj, cl)
    470         else:
    471             elem_type = cl.__args__[0]
--> 472             return [
    473                 self._structure_func.dispatch(elem_type)(e, elem_type)
    474                 for e in obj

~/venv/jupyter_3.9/lib/python3.9/site-packages/cattr/converters.py in <listcomp>(.0)
    471             elem_type = cl.__args__[0]
    472             return [
--> 473                 self._structure_func.dispatch(elem_type)(e, elem_type)
    474                 for e in obj
    475             ]

~/venv/jupyter_3.9/lib/python3.9/site-packages/toloka/client/_converter.py in <lambda>(data, type_)
     15 converter.register_structure_hook_func(
     16     lambda type_: hasattr(type_, 'structure'),
---> 17     lambda data, type_: type_.structure(data)  # type: ignore
     18 )
     19 converter.register_unstructure_hook_func(  # type: ignore

~/venv/jupyter_3.9/lib/python3.9/site-packages/toloka/client/filter.py in structure(cls, data)
     82             return FilterAnd.structure(data)
     83         else:
---> 84             return Condition.structure(data)
     85 
     86 

~/venv/jupyter_3.9/lib/python3.9/site-packages/toloka/client/filter.py in structure(cls, data)
    155     @classmethod
    156     def structure(cls, data):
--> 157         return super(FilterCondition, cls).structure(data)
    158 
    159 

~/venv/jupyter_3.9/lib/python3.9/site-packages/toloka/client/primitives/base.py in structure(cls, data)
    267                 raise SpecClassIdentificationError(spec_field=spec_field,
    268                                                    spec_enum=cls._variant_registry.enum.__name__)
--> 269             return spec_class.structure(data)
    270 
    271         data = copy(data)

~/venv/jupyter_3.9/lib/python3.9/site-packages/toloka/client/filter.py in structure(cls, data)
    155     @classmethod
    156     def structure(cls, data):
--> 157         return super(FilterCondition, cls).structure(data)
    158 
    159 

~/venv/jupyter_3.9/lib/python3.9/site-packages/toloka/client/primitives/base.py in structure(cls, data)
    265                 spec_class = cls._variant_registry.registered_classes[spec_value]
    266             except Exception:
--> 267                 raise SpecClassIdentificationError(spec_field=spec_field,
    268                                                    spec_enum=cls._variant_registry.enum.__name__)
    269             return spec_class.structure(data)

SpecClassIdentificationError: SpecClassIdentificationError(spec_field='key', spec_enum='Key')

A week ago everything was OK. Is this the Toloka backend problem?

Attrs dependency requirement conflicts with Airflow

Hi, thank you for the helpful library

I am trying to use toloka-kit in Airflow environment. However, when I am installing both apache-airflow and toloka-kit packages it fails with dependency conflict error. Something like this:

apache-airflow 2.1.2+composer has requirement attrs<21.0,>=20.0, but you have attrs 21.2.0.

I tried different recent versions of Airflow (both v1 and v2 and master branch) and all are failing with a similar error

And it looks like that toloka-kit relies on API of new 21 version of attrs. Because I ran toloka-kit tests in virtual env with attrs==20.3.0 and 26 tests failed.

I understand that it would be unpleasant to downgrade attrs package version requirement here. However, Airflow is a very popular solution and I assume that I am not the only one who manages or will manage Toloka projects via Airflow. Moreover, I doubt that Airflow will update the dependency version soon. Currently this is a blocker for me and maybe to somebody else to migrate from bare API calls to toloka-kit. For example, I am not able to create a hook, PythonOperator or PythonSensor for crowdsourcing purposes via toloka-kit

What do you think?

PoolCompletedPercentage metric raises an error for closed pools

I'm setting up metrics for a Toloka Pipeline object, which consists of multiple Pools. Some Pools are initially closed, and are opened only when Tasks are forwarded from another Pool.

To monitor progress, I use the following code to set up the metrics for observing the PoolCompletedPercentage metric:

p_metrics = []

for task in task_sequence:

    p_metric = metrics.pool_metrics.PoolCompletedPercentage(pool_id=task.pool.id,
                                                            percents_name=f'{task.name}-pct',
                                                            toloka_client=task_sequence.client)

    p_metrics.append(p_metric)

p_metrics = MetricCollector(p_metrics, process_metrics)

I then run the Pipeline and the MetricCollector:

loop = asyncio.get_event_loop()
loop.run_until_complete(asyncio.gather(collector.run(), self.pipeline.run()))

This, however, raises the following error. I suspect that this results from some of the Pools being closed.

Got error in metric:
Traceback for <Task finished name='Task-3' coro=<BaseMetric.get_lines() done, defined at /lib/python3.9/site-packages/toloka/metrics/metrics.py:79> exception=TypeError("'NoneType' object is not subscriptable")> (most recent call last):
  File "/lib/python3.9/site-packages/toloka/metrics/metrics.py", line 98, in get_lines
    return result
  File "/lib/python3.9/site-packages/toloka/metrics/pool_metrics.py", line 190, in _get_lines_impl
    result[self._percents_name] = [(structure(response['finished'], datetime.datetime), response['result']['value'])]
TypeError: 'NoneType' object is not subscriptable

Tasks creation progress bar

Hi!
I am uploading multiple tasks to a pool using toloka_client.create_tasks(tasks) and would like to have an estimate on how long the upload would take. Is there any way to monitor the progress of task creation?
I realise that I can wrap a tqdm around a loop with toloka_client.create_task(task), but that would switch the upload from asynchronous to synchronous? Is there a better way to do this?

Thank you!

[BUG] ValidationApiError in collab example for SANDBOX

Observed behavior

I got error while run tutorial in collab https://colab.research.google.com/github/Toloka/toloka-kit/blob/main/examples/3.audio_analysis/audio_collection/audio_collection.ipynb
I choice "SANDBOX" and click all cells. All cells are success before uploading tasks:

tasks = [
toloka.Task(input_values={'text': text}, pool_id=new_pool.id)
for text in dataset['text'].values[:20]
]

toloka_client.create_tasks(tasks, allow_defaults=True)

new_pool = toloka_client.open_pool(new_pool.id)

Expected behavior

22.08.23 all works and all tasks were uploaded to pool

Python Version

3.8

Toloka-Kit Version

0.1.26

Other Packages Versions

No response

Example code

https://colab.research.google.com/github/Toloka/toloka-kit/blob/main/examples/3.audio_analysis/audio_collection/audio_collection.ipynb

SANDBOX is used

Relevant log output

ValidationApiError                        Traceback (most recent call last)
<ipython-input-24-3500c0c72b11> in <cell line: 6>()
      4 ]
      5 
----> 6 toloka_client.create_tasks(tasks, allow_defaults=True)
      7 
      8 new_pool = toloka_client.open_pool(new_pool.id)

3 frames
create_tasks_expanded_by_parameters_b78c00ecf9034a73a33a8233fe58069a in create_tasks_expanded_by_parameters(self, tasks, allow_defaults, open_pool, skip_invalid_items, operation_id, async_mode)
      2 
      3     parameters = CreateTasksParameters(allow_defaults=allow_defaults, open_pool=open_pool, skip_invalid_items=skip_invalid_items, operation_id=operation_id, async_mode=async_mode)
----> 4     return func(self, tasks, parameters)

/usr/local/lib/python3.10/dist-packages/toloka/client/__init__.py in _sync_via_async(self, objects, parameters, url, result_type, operation_type, output_id_field, get_method)
    374         # but no objects are created
    375         if not pools:
--> 376             raise ValidationApiError(
    377                 code='VALIDATION_ERROR',
    378                 message='Validation failed',

ValidationApiError: ValidationApiError(status_code=None, request_id=None, code='VALIDATION_ERROR', message='Validation failed', payload={})

[BUG] Can't get pool which has specified 'experimental group' parameter in audience filter settings

Observed behavior

Hello!
I'm trying to get pool info using get_pool() method (https://toloka.ai/en/docs/toloka-kit/reference/toloka.client.TolokaClient.get_pool).
Script crashes with an error:
IterableValidationError: While structuring typing.List[toloka.client.filter.FilterCondition] (1 sub-exception).

It happens in case when pool has specified "experimental group" parameter in audience filter settings:

If I don't specify "experimental group" parameter in pool settings then get_pool() method works properly without any crashes.

Expected behavior

get_pool() works properly without any crashes, and returns Pool object.

Python Version

3.8

Toloka-Kit Version

1.1.1

Other Packages Versions

absl-py
aiohttp
aiosignal
anyio
appdirs==1.4.4
argcomplete
argon2-cffi
argon2-cffi-bindings
astor==0.8.1
astroid
asttokens
astunparse==1.6.3
async-timeout
attrs
audioread
autopep8
Babel
backcall
bayesian-optimization
beautifulsoup4
bleach
blinker==1.4
blis
bokeh
boto==2.49.0
brotlipy==0.7.0
cachetools
catalogue
catboost
category-encoders
cattrs==22.1.0
certifi
cffi
chardet
charset-normalizer
click
cloudpickle
colorama
conda==22.9.0
conda-content-trust
conda-package-handling
conda-token
crcmod
cryptography
cssselect==1.1.0
cssutils
cudf==0.19.2
cupy
cx-Oracle
cycler
cymem
Cython
cytoolz==0.11.0
dask
dask-cuda==21.8.0
dask-cudf==0.19.2
dask-glm==0.2.0
dask-labextension==5.3.0
dask-ml
debugpy
decorator
defusedxml
dill
distributed
docstring-parser==0.15
emails
en-core-web-sm
entrypoints
exceptiongroup==1.1.0
executing
fastavro
fasteners
fastjsonschema
fastrlock==0.6
fasttext
filelock
findspark
flake8
flatbuffers
frozenlist
fsspec
funcy
future==0.18.2
gast
gcs-oauth2-boto-plugin
gensim
gitdb==4.0.9
GitPython==3.1.29
google-api-core
google-apitools
google-auth
google-auth-oauthlib
google-cloud-bigquery==1.22.0
google-cloud-core
google-cloud-storage
google-pasta
google-reauth
google-resumable-media
googleapis-common-protos
grpcio
gsutil
h11==0.14.0
h5py
HeapDict
httpcore==0.16.3
httplib2
httpx==0.23.3
huggingface-hub
hyperopt
idna
imagecodecs
imageio
implicit
importlib-metadata
importlib-resources
ipykernel
ipyparallel
ipython
ipython-genutils
isort
jedi==0.17.0
Jinja2
joblib
json5
jsonschema
jupyter-client
jupyter-contrib-core
jupyter-contrib-nbextensions
jupyter-core
jupyter-highlight-selected-word
jupyter-kernel-gateway==2.5.1
jupyter-latex-envs
jupyter-lsp
jupyter-nbextensions-configurator
jupyter-resource-usage
jupyter-server
jupyter-server-mathjax==0.2.6
jupyter-server-proxy
jupyterlab
jupyterlab-execute-time
jupyterlab-git==0.39.3
jupyterlab-lsp
jupyterlab-pygments
jupyterlab-server
jupyterlab-system-monitor
jupyterlab-templates==0.3.1
jupyterlab-topbar
Keras
Keras-Preprocessing
kiwisolver
langcodes
lazy-object-proxy
lckr-jupyterlab-variableinspector==3.0.9
librosa
lightfm
line-profiler==3.5.1
llvmlite==0.38.0
locket
lxml==4.5.1
Markdown
MarkupSafe
matplotlib
matplotlib-inline
mccabe==0.6.1
metakernel
mistune==0.8.4
mkl-fft==1.3.1
mkl-random
mkl-service==2.4.0
mock
monotonic==1.5
msgpack
multidict
multipledispatch==0.6.0
murmurhash
nbclassic
nbclient
nbconvert
nbdime==3.1.1
nbformat
nest-asyncio
networkx
nltk
notebook
numba
numexpr
numpy
nvtx
oauth2client==4.1.3
oauthlib
opt-einsum
packaging
pandas==1.2.4
pandocfilters
parso
partd
pathy
patsy==0.5.2
pbr
pexpect
pickleshare
Pillow==9.2.0
pkgutil-resolve-name
platformdirs
plotly
pluggy
pooch
portalocker
premailer
preshed
prometheus-client
prompt-toolkit
protobuf==3.14.0
psutil
ptyprocess
pure-eval
py4j
pyarrow==1.0.1
pyasn1
pyasn1-modules==0.2.8
pybind11==2.9.2
pycodestyle
pycosat==0.6.3
pycparser
pydantic==1.8.2
pydocstyle
pyflakes
Pygments
PyJWT
pykerberos==1.2.1
pyLDAvis
pylint
pymongo==3.12.0
PyMySQL
pyngrok
pynndescent
pynvml
pyOpenSSL
pyparsing
pyrsistent
PySocks
pyspark
python-dateutil
python-jsonrpc-server
python-language-server
pytorch-lightning
pytz
pyu2f
PyWavelets
PyYAML==6.0
pyzmq
redis
regex
requests
requests-oauthlib==1.3.0
resampy
retry-decorator
rfc3986==1.5.0
rmm==0.19.0
rope
rsa
ruamel-yaml-conda
sacremoses
sasl==0.2.1
scikit-image
scikit-learn
scipy
seaborn
selenium
Send2Trash
sentence-transformers
sentencepiece==0.1.95
shap
shellingham
simpervisor
simplejson==3.18.1
six
slicer
smart-open
smmap==5.0.0
sniffio
snowballstemmer
sortedcontainers
soundfile
soupsieve
spacy
spacy-legacy
spacy-loggers
spylon==0.3.0
spylon-kernel==0.4.1
srsly
stack-data
statsmodels
tblib
tenacity
tensorboard
tensorboard-data-server
tensorboard-plugin-wit
tensorflow
tensorflow-estimator
tensorflow-hub==0.8.0
termcolor==1.1.0
terminado
testpath
thinc
threadpoolctl
thrift==0.15.0
thrift-sasl==0.4.2
tifffile
tokenizers
toloka-kit==1.1.1
toml
tomli
tomlkit
toolz
torch==1.8.1
torchvision==0.2.2
tornado
tqdm
traitlets
transformers
typer
typing-extensions
ujson
umap-learn
urllib3
wasabi
wcwidth
webencodings==0.5.1
websocket-client
Werkzeug
wordcloud
wrapt
xgboost
xlrd
yadisk==1.2.17
yapf
yarl
zict==2.1.0
zipp

Example code

import toloka.client as toloka

toloka_token_sandbox = 'token'
toloka_env_sandbox = 'SANDBOX'

toloka_client_sandbox = toloka.TolokaClient(toloka_token_sandbox, toloka_env_sandbox)

toloka_client_sandbox.get_pool(pool_id='1451087')

Relevant log output

---------------------------------------------------------------------------
IterableValidationError                   Traceback (most recent call last)
Input In [9], in <cell line: 2>()
      1 # project_id = 133289
----> 2 toloka_client_sandbox.get_pool(pool_id='1451087')

File ~/.local/lib/python3.8/site-packages/toloka/util/_managing_headers.py:72, in add_headers.<locals>.wrapper.<locals>.wrapped(*args, **kwargs)
     69 if top_level_method_var not in ctx:
     70     stack.enter_context(set_variable(top_level_method_var, func.__name__))
---> 72 return run_in_current_context(func, *args, **kwargs)

File ~/.local/lib/python3.8/site-packages/toloka/util/_managing_headers.py:99, in run_in_current_context(func, *args, **kwargs)
     93 def run_in_current_context(func, *args, **kwargs):
     94     """Runs the function using context state from the moment of calling run_in_current_context function.
     95 
     96     Unlike Context.run supports generators, async generators and functions that return an awaitable (e.g. coroutines).
     97     """
---> 99     result = func(*args, **kwargs)
    100     if inspect.isawaitable(result):
    101         # capture context by running inside task
    102         loop = asyncio.get_event_loop()

File ~/.local/lib/python3.8/site-packages/toloka/client/__init__.py:1558, in TolokaClient.get_pool(self, pool_id)
   1545 """Gets pool data from Toloka.
   1546 
   1547 Args:
   (...)
   1555     ...
   1556 """
   1557 response = self._request('get', f'/v1/pools/{pool_id}')
-> 1558 return structure(response, Pool)

File ~/.local/lib/python3.8/site-packages/cattrs/converters.py:281, in Converter.structure(self, obj, cl)
    278 def structure(self, obj: Any, cl: Type[T]) -> T:
    279     """Convert unstructured Python data structures to structured data."""
--> 281     return self._structure_func.dispatch(cl)(obj, cl)

File ~/.local/lib/python3.8/site-packages/toloka/client/_converter.py:17, in <lambda>(data, type_)
     10 from ..util._extendable_enum import ExtendableStrEnum
     13 converter = cattr.Converter()
     15 converter.register_structure_hook_func(
     16     lambda type_: hasattr(type_, 'structure'),
---> 17     lambda data, type_: type_.structure(data)  # type: ignore
     18 )
     19 converter.register_unstructure_hook_func(  # type: ignore
     20     lambda obj: hasattr(obj, 'unstructure'),
     21     lambda obj: obj.unstructure()  # type: ignore
     22 )
     25 converter.register_structure_hook(uuid.UUID, lambda data, type_: type_(data))  # type: ignore

File ~/.local/lib/python3.8/site-packages/toloka/client/primitives/base.py:284, in BaseTolokaObject.structure(cls, data)
    282     value = data.pop(key)
    283     if field.type is not None:
--> 284         value = converter.structure(value, field.type)
    286     kwargs[field.name] = value
    288 obj = cls(**kwargs)

File ~/.local/lib/python3.8/site-packages/cattrs/converters.py:281, in Converter.structure(self, obj, cl)
    278 def structure(self, obj: Any, cl: Type[T]) -> T:
    279     """Convert unstructured Python data structures to structured data."""
--> 281     return self._structure_func.dispatch(cl)(obj, cl)

File ~/.local/lib/python3.8/site-packages/cattrs/converters.py:531, in Converter._structure_optional(self, obj, union)
    529 other = union_params[0] if union_params[1] is NoneType else union_params[1]
    530 # We can't actually have a Union of a Union, so this is safe.
--> 531 return self._structure_func.dispatch(other)(obj, other)

File ~/.local/lib/python3.8/site-packages/toloka/client/_converter.py:17, in <lambda>(data, type_)
     10 from ..util._extendable_enum import ExtendableStrEnum
     13 converter = cattr.Converter()
     15 converter.register_structure_hook_func(
     16     lambda type_: hasattr(type_, 'structure'),
---> 17     lambda data, type_: type_.structure(data)  # type: ignore
     18 )
     19 converter.register_unstructure_hook_func(  # type: ignore
     20     lambda obj: hasattr(obj, 'unstructure'),
     21     lambda obj: obj.unstructure()  # type: ignore
     22 )
     25 converter.register_structure_hook(uuid.UUID, lambda data, type_: type_(data))  # type: ignore

File ~/.local/lib/python3.8/site-packages/toloka/client/filter.py:88, in FilterCondition.structure(cls, data)
     86     return FilterOr.structure(data)
     87 if 'and' in data:
---> 88     return FilterAnd.structure(data)
     89 else:
     90     return Condition.structure(data)

File ~/.local/lib/python3.8/site-packages/toloka/client/filter.py:144, in FilterAnd.structure(cls, data)
    142 @classmethod
    143 def structure(cls, data):
--> 144     return super(FilterCondition, cls).structure(data)

File ~/.local/lib/python3.8/site-packages/toloka/client/primitives/base.py:284, in BaseTolokaObject.structure(cls, data)
    282     value = data.pop(key)
    283     if field.type is not None:
--> 284         value = converter.structure(value, field.type)
    286     kwargs[field.name] = value
    288 obj = cls(**kwargs)

File ~/.local/lib/python3.8/site-packages/cattrs/converters.py:281, in Converter.structure(self, obj, cl)
    278 def structure(self, obj: Any, cl: Type[T]) -> T:
    279     """Convert unstructured Python data structures to structured data."""
--> 281     return self._structure_func.dispatch(cl)(obj, cl)

File ~/.local/lib/python3.8/site-packages/cattrs/converters.py:470, in Converter._structure_list(self, obj, cl)
    468             ix += 1
    469     if errors:
--> 470         raise IterableValidationError(
    471             f"While structuring {cl!r}", errors, cl
    472         )
    473 else:
    474     res = [handler(e, elem_type) for e in obj]

IterableValidationError: While structuring typing.List[toloka.client.filter.FilterCondition] (1 sub-exception)

[FEATURE] pipeline support on windows

Problem description

toloka-kit in windows lacks usability due absence of key feature - real-time processing of assignments with pipelines

Feature description

Add pipeline implementation for windows

Potential alternatives

Make incompatibility explicit: for example, more descriptive error messages when one tries to initialize pipeline in Windows

Additional information

No response

Link to the documentation in README.md is broken.

The link https://yandex.ru/dev/toloka/doc/concepts/about.html?lang=en leads to 404 page.

Hook for automatic stubs and documentation re-generation

Currently we automatically create stubs and markdowns from docstrings only for commits made in our internal repository that then synchronizes with GitHub. For external contributors, however, there are three places that have to be modified in order to produce a fix to produce a fix to our documentation.

We need to automate this process for external contributors as well

TypeError: __init__() got an unexpected keyword argument 'allowed_methods' in v 0.1.23

toloka-kit==0.1.23
calling client.get_pool(pool_id=settings.pool_id) gives error
TypeError: __init__() got an unexpected keyword argument 'allowed_methods'

I installed the previous version 0.1.22 and get_pool works as expected!

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
[<ipython-input-7-e4710c779596>](https://localhost:8080/#) in <module>()
     20 
     21 client = TolokaClient(settings.token, 'PRODUCTION')
---> 22 ongoing_pool = client.get_pool(pool_id=settings.pool_id)
     23 if not ongoing_pool.is_closed():
     24     client.close_pool(settings.pool_id)

7 frames
[/usr/local/lib/python3.7/dist-packages/toloka/client/primitives/retry.py](https://localhost:8080/#) in __init__(self, retry_quotas, *args, **kwargs)
     45 
     46         self._last_response = kwargs.pop('last_response', None)
---> 47         super(TolokaRetry, self).__init__(*args, **kwargs)
     48 
     49     def new(self, **kwargs):

Is it possible to delete task from pool?

Hi,

Since there is no way to set overlap to 0 is it possible to delete task from pool via api as it is allowed in web version of Toloka?