Make Celery pluggable

[selinon]

Make Celery pluggable

[geetikabatra]

@fridex, is someone working on this issue?

[fridex]

@geetikabatra I don't think so. I planned it for the eventual release of 2.0 (I didn't expect to come this in before 1.0 release). Do you have a use case for it?

[geetikabatra]

@fridex , There is no specific use case. But I would like to work on it.

[fridex]

@geetikabatra cool! If you need any guidance or a sync, feel free to ask :)

[geetikabatra]

@fridex, Awesome. I can spend some time reading the documentation and the code, and we can sync up around next week. Please share any other link, if there is, that might be useful to me. :)

[fridex]

@fridex, Awesome. I can spend some time reading the documentation and the code, and we can sync up around next week. Please share any other link, if there is, that might be useful to me. :)

At least basic notion about Celery tasks - notably how to schedule task (the apply_async method on Celery's task instance) and how to retrieve states of tasks. Celery's docs can be helpful http://docs.celeryproject.org/en/latest/userguide/tasks.html

A basic knowledge of Selinon will be required - see official docs at http://selinon.readthedocs.io/en/latest/internals.html

Basically, there is a need to abstract apply_async and state checking into an interface, so there will be a need to create a class:

import enum
from abc import ABCMeta

class TaskState(enum.Enum):
      FINISHED
      PENDING
      FAILED
      UNKNOWN  # This will be an issue for Celery as it will always return pending state AFAIK

class SelinonBackend(metaclass=ABCMeta):
    @abstractmethod
    def schedule_task(self, task_name, node_args):
        """Schedule a task.

        @param task_name: name of task to be scheduled
        @param node_args: arguments supplied to the task
        @return: task id
        """

    @abstractmethod
    def check_task_state(self, task_id):
        """Check state of a task.

        @param task_id: id of task which state should be checked
        @return: task state
        """

And check for all occurrences of async apply and state checks. The constructor of SelinonBackend should be configurable via yaml files.

Celery or Dramatiq backend will simply implement SelinonBackend interface. If there is anything not clear, feel to ask.

[geetikabatra]

@fridex, Thanks for the explanation. I got most of it. Will let you know if I get stuck anywhere.

[weizhongjia]

is there any progress on this issue

[fridex]

is there any progress on this issue

Not any I would be aware of.