Asynchronous data source import

[re-data]

Asynchronous data source import

[mateuszklimek]

@hishoss Good point. One thing regarding celery, there actually already is asynchronous mechanism for running those checks in Redata - airflow. So I think using it maybe best, just one thing is we would need to find a way to trigger airflow job when datasource it added (it seems like quite standard operation, but I wasn't sure how to do it properly in airflow).

And actually solving those (triggering airflow job) would be also useful when doing custom scan, it would enable us to remove pulling approach (every minute) currently used.. :)

[hishoss]

Good point @mateuszklimek. Running a data source import as an airflow job looks like the best way to go. I'll dig in the code to understand the minute jobs then we can plan on how best to refactor that process.

[mateuszklimek]

👍

[mateuszklimek]

After switch to dbt based packge it's no longer actual