cloudcheflabs/dataroaster

dataroaster's Issues

Add API for redash, jupyterhub and kafka.

Add functionality to trino controller to detect exhausted trino clusters automatically.

Trino operator provides jmx apis to get a lot of mbean metrics exposed by trino coordinator and workers.
Probably, using this api, trino controller can detect exhausted trino cluster automatically.

Add REST API for dataroaster operator to create hive metastore automatically.

To create hive metastore, mysql server needs to be installed before installing hive metastore.
Add rest api for dataroaster operator to create mysql server and hive metastore automatically.

Add Trino Operator

Trino operator will be used to create / update / delete trino clusters.
In addition to that, It should also monitor trino clusters and sometimes replace the exhausted trino workers with the new ones if detected.
We can also consider support of graceful shutdown when there are existing queries being executed which have to be executed before trino cluster shutdown.

Add Airflow Workflow

Airflow is one of the popular workflows for data lakehouse.
Airflow should support the following list:

• dynamical dag workflow deployment using git-sync sidecar container.
• support remote logging to s3.
• KEDA based worker autoscaling.

Update prometheus jobs when trino configurations updated.

Everytime trino configuration updated, trino coordinator and workers in the dedicated trino cluster will be rollout restarted.
At that time, all the endpoints of trino coordinator and workers will get new addresses for their pods, so these changed endpoints addresses of trino coordinator and workers will be updated as prometheus jobs to prometheus configmap.

Add custom resources of data platform components.

Custom resources of data platform components like hive metastore, redash, etc needs to be added.

Add REST API to trino operator to get trino coordinator and workers mbean values like active executed query count, heap memory size.

BUG: fix errros in trino gateway when python trino client connect to it.

Add API to scale trino workers.

Get active executing queued query count from trino coordinator in every second and update it to redis cache.

Add specific apis for data platform components.

It is needed that specific apis for data platform components should be created to create these components automatically.
for instance,

• /v1/spark_thrift_server for spark thrift server
• /v1/trino for trino
• /v1/hive_metastore for hive metastore
• etc.

Add trino controller to control all the trino ecosystems.

Trino controller needs to control all the trino ecosystems.

• create trino operator
• create helm operator.
• create nginx ingress controller.
• create cert manager.
• create trino gateway.
• create trino cluster.
• create prometheus.
• create grafana.
• scale out trino cluster.
• register trino cluster to trino gateway.
• deregister trino cluster from trino gateway.
• detect exhausted trino cluster.

Add DataRoaster Operator

DataRoaster Operator is used to controll all the data platform components running on kubernetes.
It consists of mysql server, helm operator and dataroaster server which provides rest api to the client.

Add helm operator

Most of components provided by dataroaster is based on helm chart. With helm operator, these components can be installed on kubernetes easily.

Add REST API for dataroaster to create spark thrift server automatically.

To create spark thrift server, there are several things to be created beforehand.
dataroaster operator needs to create nfs server, rbac, s3 secret, and pvc automatically before spark thrift server created.

Bug: Watcher close error in trino operator.

if there is no action notified for a long time, watcher will be closed with the message like this.

com.cloudcheflabs.dataroaster.operators.trino.handler.TrinoClusterWatcher: close watcher

where is spark-thrift-server-4.8.0-SNAPSHOT-spark-job.jar

hello，I want to try tun the spark demo，but cannot download spark-thrift-server-4.8.0-SNAPSHOT-spark-job.jar？ where is can be download ，thanks？

Add Trino Gateway to proxy trino clusters.

Trino can be used for the cases of ETL, interactive(adhoc) and scheduled, etc.
We can create trino clusters for these usages individually. In order to load-balance trino clusters, and to support trino coordinator HA, trino gateway should be used. We can extend this concept to trino ecosystem.

Trino ecosystem consists of trino gateway, authenticator, admin and trino clusters, etc.
We can think of the following scenario which can happen in trino ecosystem.

• clients(applications, redash, tableau, cli, etc) will request queries to trino gateway.
• trino gateway will filter parameters of request to authenticate / authorize user with authenticator.
• the queries requested will be routed dynamically to the downstream trino cluster for the individual usages of ETL, interactive, scheduled.
• admin will create / update / delete trino clusters on kubernetes using trino operator.
• admin will register / deregister / activate / deactivate trino cluster to trino gateway.

Monitor trino clusters in trino operator

Add monitoring of trino clusters in trino operator.
To monitor trino clusters in trino operator, the following functionalities can be added to trino operator(#1):

Update rest api provided by trino controller to install trino with different version.

To install trino with different version, trino docker image tag needs to be parameterized in rest api currently provided by trino controller.