webankfintech / schedulis Goto Github PK

Schedulis is a high performance workflow task scheduling system that supports high availability and multi-tenant financial level features, Linkis computing middleware, and has been integrated into data application development portal DataSphere Studio

Home Page: https://github.com/WeBankFinTech/Schedulis.git

License: Apache License 2.0

Java 73.07% Shell 0.72% HTML 0.05% CSS 1.04% JavaScript 24.58% C 0.08% Less 0.46%

datasphere-studio azkaban workflow linkis schedulis scheduler qualitis wedatasphere

schedulis's People

Contributors

Stargazers

Watchers

Forkers

yangzhiyue peacewong xj2jx bleachzk buoylee zhengfan199525 wbjwwang sargentti alexzywu dennyzhou-zdx fysoft2006 junneyang bobqiu labchy zhch158 jgteng davidmr001 czone-league tgh-621 feloxx selfimpr001 lzk416 liulingjie huanzui yudandu medier ljj-0908 yayunde woodshope askmetoo deepbluethinker ambition119 chis123 qiangxue1206 fightingsz sevncz wangqianglong maxpda mu-l xiejiajun thinker1017 robinzyx xiaojinlu timgrk turbotree win-2019 nodder aiaudit huangweiboy2 jangocheng rk19016 dlimeng dystudio javinjunfeng guomingrui scutshuxue wyoceancn minda123456 quequan chrisc-cc frankfreedom yueyunyue jangocity liukunyuan wolfkiki newtaul castiel-chenxian zoneyeah corner4world xiaohelong akin-azure jfatty brianfdc dut3062796s wjandy iwasjoker jamesbonddu zhaoyansheng163 duhanmin liaobing 521hellogithub 1024hertz-hodgepodge robortl zhuliuw kabimon jimzhu0804 yubobo seleven binaryworld ztwu labulabuduo junhanqiao james23wang hello-colin ccweixiao jusescn 85315518 handong890 huahuashine wedatasphere

schedulis's Issues

【schedulis-release-0.6.2】登录成功后点击其他菜单或功能按钮，就跳转到登录页面

返回：{"error":"session"}

使用DSS发布工程失败

DSS 0.9.0
Linkis 0.9.4
Schedulis 0.5.0
当我在DSS发布任务的时候，发布任务失败，报错如下：

dss-server报错如下：

ui上面param,system两个大菜单功能无法使用

查看菜单的跳转地址是：/$viewers[0].pluginPath

jetty https功能修复

schedulis用户【bigdata】不能拿到/etc/profile环境变量，而且也拿不到.bashrc里面的变量

每次以不用用户调度schedulis，都要加载当前用户的相关环境变量，非常的麻烦

服务启动脚本优化

请问是否支持azkaban中面用yml配置的作业

如题。

失败任务隔几天去运行【准备执行】，拿到的run_date永远都是当前日期-1

比如说今天是20210120，之前run_date=20210115的任务失败了，隔了好几天发现问题然后对失败任务点击执行，发现拿到的run_date=20210119,，也就是当前日期-1. 这表明任务失败后，隔天接着失败运行，拿到的run_date和当时日期不一致

执行历史记录，慢查询优化

Schedulis登陆卡死

当我在wtss-web的配置文件azkaban.properties配置了如下内容时，下面的ip和端口都是正常启动的，前端用户登陆的时候会出现卡死的状态，当把下面的配置填写成一个不通的ip和端口，问题解决了
ladp.ip=localhost
ladp.port=9001

安装Schedulis0.6.2版本并成功启动，使用默认账号登录后，点击任意连接直接返回登录页面

默认登录账号信息如下：
username : superadmin
pwd : Abcd1234

单击任意顶部菜单链接，直接返回到登录页面，详见如下视频

WTSS.Web.Client.-.Google.Chrome.2022-04-15.11-13-22.mp4

schedulis 未实时统计数据

如截图所示，已经执行过失败和成功的项目。而首页显示的数据均为0.

版本：
schedulis_0.5.0

部署后登陆失败，麻烦修复下吧

org.eclipse.jetty.server.HttpChannel 600 handleException - /checkin java.lang.NoClassDefFoundError: Could not initialize class org.forgerock.opendj.ldap.LDAPConnectionFactory
请查看截图。
是否可以不依赖LDAP，看上去去掉LDAP依赖后也能解决问题

Schedulis单独部署执行工作流报错

06-01-2022 10:02:34 CST sql_3819 INFO - Starting job sql_3819 at 1641434554021
06-01-2022 10:02:34 CST sql_3819 INFO - job JVM args: -Dazkaban.flowid=Scheduler -Dazkaban.execid=79 -Dazkaban.jobid=sql_3819
06-01-2022 10:02:34 CST sql_3819 INFO - Building linkis job executor.
06-01-2022 10:02:34 CST sql_3819 INFO - Start to execute job
06-01-2022 10:02:34 CST sql_3819 INFO - ****** Job properties ******
06-01-2022 10:02:34 CST sql_3819 INFO - - Note : value is masked if property name ends with '_X'.
06-01-2022 10:02:34 CST sql_3819 INFO - azkaban.url=http://localhost:8081
06-01-2022 10:02:34 CST sql_3819 INFO - azkaban.job.id=sql_3819
06-01-2022 10:02:34 CST sql_3819 INFO - azkaban.job.attempt=0
06-01-2022 10:02:34 CST sql_3819 INFO - azkaban.flow.start.year=2022
06-01-2022 10:02:34 CST sql_3819 INFO - azkaban.link.execution.url=http://localhost:8081/executor?execid=79
06-01-2022 10:02:34 CST sql_3819 INFO - linkis.version=1.0.0
06-01-2022 10:02:34 CST sql_3819 INFO - type=linkis
06-01-2022 10:02:34 CST sql_3819 INFO - job.datachecker.jdo.option.name="job"
06-01-2022 10:02:34 CST sql_3819 INFO - bdp.datachecker.jdo.option.name="bdp"
06-01-2022 10:02:34 CST sql_3819 INFO - azkaban.flow.projectversion=1
06-01-2022 10:02:34 CST sql_3819 INFO - azkaban.flow.start.minute=02
06-01-2022 10:02:34 CST sql_3819 INFO - plugin.dir=/appcom/Install/AzkabanInstall/schedulis_0.6.1_exec/plugins/jobtypes/linkis
06-01-2022 10:02:34 CST sql_3819 INFO - azkaban.flow.execid=79
06-01-2022 10:02:34 CST sql_3819 INFO - azkaban.link.workflow.url=http://localhost:8081/manager?project=Scheduler&flow=Scheduler
06-01-2022 10:02:34 CST sql_3819 INFO - azkaban.job.attachment.file=/appcom/Install/AzkabanInstall/schedulis_0.6.1_exec/executions/79/Scheduler/Scheduler/jobs/sql_3819/_job.79.sql_3819.attach
06-01-2022 10:02:34 CST sql_3819 INFO - msg.eventchecker.jdo.option.name="msg"
06-01-2022 10:02:34 CST sql_3819 INFO - jvm.args=-Dazkaban.flowid=Scheduler -Dazkaban.execid=79 -Dazkaban.jobid=sql_3819
06-01-2022 10:02:34 CST sql_3819 INFO - azkaban.flow.uuid=551d3c1b-63b9-4755-8ca0-ea962a6524da
06-01-2022 10:02:34 CST sql_3819 INFO - azkaban.link.jobexec.url=http://localhost:8081/executor?execid=79&job=sql_3819
06-01-2022 10:02:34 CST sql_3819 INFO - job.datachecker.jdo.option.username=root
06-01-2022 10:02:34 CST sql_3819 INFO - azkaban.memory.check=true
06-01-2022 10:02:34 CST sql_3819 INFO - msg.eventchecker.jdo.option.password=cm9vdA==
06-01-2022 10:02:34 CST sql_3819 INFO - azkaban.flow.start.timestamp=2022-01-06T10:02:33.993+08:00
06-01-2022 10:02:34 CST sql_3819 INFO - linkistype=linkis.spark.sql
06-01-2022 10:02:34 CST sql_3819 INFO - azkaban.link.attempt.url=http://localhost:8081/executor?execid=79&job=sql_3819&attempt=0
06-01-2022 10:02:34 CST sql_3819 INFO - azkaban.flow.start.milliseconds=993
06-01-2022 10:02:34 CST sql_3819 INFO - wds.linkis.flow.contextID={"type":"HAWorkFlowContextID","value":"{/"instance/":null,/"backupInstance/":null,/"user/":/"dss/",/"workspace/":/"bdapWorkspace/",/"project/":/"Scheduler/",/"flow/":/"Scheduler/",/"contextId/":/"8-8--cs_1_devcs_1_dev31/",/"version/":/"v000001/",/"env/":/"BDAP_DEV/"}"}
06-01-2022 10:02:34 CST sql_3819 INFO - flow.dir=/appcom/Install/AzkabanInstall/schedulis_0.6.1_exec/executions/79
06-01-2022 10:02:34 CST sql_3819 INFO - azkaban.flow.submituser=superadmin
06-01-2022 10:02:34 CST sql_3819 INFO - labels={"route":"dev"}
06-01-2022 10:02:34 CST sql_3819 INFO - job.datachecker.jdo.option.url=jdbc:mysql://192.168.217.101:3306/hive?useUnicode=true&characterEncoding=UTF-8
06-01-2022 10:02:34 CST sql_3819 INFO - azkaban.flow.projectname=Scheduler
06-01-2022 10:02:34 CST sql_3819 INFO - wds.linkis.gateway.url=http://192.168.217.100:9001
06-01-2022 10:02:34 CST sql_3819 INFO - msg.eventchecker.jdo.option.url=jdbc:mysql://192.168.217.101:3306/schedulis?useUnicode=true&characterEncoding=UTF-8
06-01-2022 10:02:34 CST sql_3819 INFO - azkaban.flow.start.month=01
06-01-2022 10:02:34 CST sql_3819 INFO - azkaban.native.lib=/appcom/Install/AzkabanInstall/schedulis_0.6.1_exec/lib
06-01-2022 10:02:34 CST sql_3819 INFO - bdp.datachecker.jdo.option.username=[#DATACHECKER_BDP_USER]
06-01-2022 10:02:34 CST sql_3819 INFO - msg.eventchecker.jdo.option.username=root
06-01-2022 10:02:34 CST sql_3819 INFO - spark.home=/usr/hdp/2.6.2.14-5/spark2
06-01-2022 10:02:34 CST sql_3819 INFO - hive.home=/usr/hdp/2.6.2.14-5/hive
06-01-2022 10:02:34 CST sql_3819 INFO - wds.linkis.flow.job.creator=scheduler
06-01-2022 10:02:34 CST sql_3819 INFO - hadoop.home=/usr/hdp/2.6.2.14-5/hadoop
06-01-2022 10:02:34 CST sql_3819 INFO - flow.variable.user.to.proxy=dss
06-01-2022 10:02:34 CST sql_3819 INFO - bdp.datachecker.jdo.option.password=[#DATACHECKER_BDP_PWD]
06-01-2022 10:02:34 CST sql_3819 INFO - azkaban.flow.start.hour=10
06-01-2022 10:02:34 CST sql_3819 INFO - azkaban.job.log.file=/appcom/Install/AzkabanInstall/schedulis_0.6.1_exec/executions/79/Scheduler/Scheduler/jobs/sql_3819/_job.79.sql_3819.log
06-01-2022 10:02:34 CST sql_3819 INFO - user.to.proxy=dss
06-01-2022 10:02:34 CST sql_3819 INFO - azkaban.flow.flowid=Scheduler
06-01-2022 10:02:34 CST sql_3819 INFO - azkaban.flow.start.day=06
06-01-2022 10:02:34 CST sql_3819 INFO - azkaban.job.outnodes=Scheduler
06-01-2022 10:02:34 CST sql_3819 INFO - azkaban.link.job.url=http://localhost:8081/manager?project=Scheduler&flow=Scheduler&job=sql_3819
06-01-2022 10:02:34 CST sql_3819 INFO - azkaban.flow.projectlastchangedby=superadmin
06-01-2022 10:02:34 CST sql_3819 INFO - azkaban.job.innodes=
06-01-2022 10:02:34 CST sql_3819 INFO - azkaban.flow.start.timezone=Asia/Shanghai
06-01-2022 10:02:34 CST sql_3819 INFO - resources=[{"fileName":"1e5719d4-8822-4d2b-86b0-76c01fdf8907.sql","resourceId":"09267a9a-351e-4bfb-a545-affce313dee5","version":"v000001"}]
06-01-2022 10:02:34 CST sql_3819 INFO - execute.as.user=true
06-01-2022 10:02:34 CST sql_3819 INFO - azkaban.flow.projectid=5
06-01-2022 10:02:34 CST sql_3819 INFO - azkaban.flow.start.second=33
06-01-2022 10:02:34 CST sql_3819 INFO - command={"script":"1e5719d4-8822-4d2b-86b0-76c01fdf8907.sql"}
06-01-2022 10:02:34 CST sql_3819 INFO - working.dir=/appcom/Install/AzkabanInstall/schedulis_0.6.1_exec/executions/79/Scheduler/Scheduler/jobs/sql_3819
06-01-2022 10:02:34 CST sql_3819 INFO - job.datachecker.jdo.option.password=cm9vdA==
06-01-2022 10:02:34 CST sql_3819 INFO - wds.linkis.client.flow.author.user.token=WS-AUTH
06-01-2022 10:02:34 CST sql_3819 INFO - azkaban.job.metadata.file=_job.79.sql_3819.meta
06-01-2022 10:02:34 CST sql_3819 INFO - azkaban.flow.projectlastchangeddate=1641434057442
06-01-2022 10:02:34 CST sql_3819 INFO - bdp.datachecker.jdo.option.url=jdbc:mysql://[#DATACHECKER_BDP_IP]:[#DATACHECKER_BDP_PORT]/[#DATACHECKER_BDP_DBNAME]?useUnicode=true&characterEncoding=UTF-8
06-01-2022 10:02:34 CST sql_3819 INFO - ****** End Job properties ******
06-01-2022 10:02:34 CST sql_3819 ERROR - Job run failed!
java.lang.NoClassDefFoundError: Could not initialize class com.webank.wedatasphere.dss.plugins.azkaban.linkis.jobtype.job.JobBuilder
at com.webank.wedatasphere.dss.plugins.azkaban.linkis.jobtype.AzkabanDssJobType.run(AzkabanDssJobType.java:149) ~[?:?]
at azkaban.execapp.JobRunner.runJob(JobRunner.java:836) [azkaban-exec-server-0.6.1.jar:?]
at azkaban.execapp.JobRunner.jobRunHandle(JobRunner.java:1037) [azkaban-exec-server-0.6.1.jar:?]
at azkaban.execapp.JobRunner.doRun(JobRunner.java:591) [azkaban-exec-server-0.6.1.jar:?]
at azkaban.execapp.JobRunner.run(JobRunner.java:548) [azkaban-exec-server-0.6.1.jar:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:?]
at java.lang.Thread.run(Thread.java:748) [?:?]
06-01-2022 10:02:34 CST sql_3819 ERROR - Could not initialize class com.webank.wedatasphere.dss.plugins.azkaban.linkis.jobtype.job.JobBuilder cause: null
06-01-2022 10:02:34 CST sql_3819 INFO - Finishing job sql_3819 at 1641434554039 with status FAILED

定时调度删除后还是会调度执行

环境：0.5.0
模式：HA
现象：定时调度任务删除后还是会调度执行，重启web服务后问题解决

支持工作流级别的增量发布，修改哪里发布哪里

Schedulis 项目增多时，首页、定时调度页面访问缓慢

编译 schedulis jar包无法下载 mvn clean install `-Dmaven.test.skip=true`

系统静态变量--properties支持${run_date}

系统静态变量--properties支持${run_date}.
如abc=${run_date}

只读用户不能修改定时调度的参数设置

log4j升级为log4j2

schedulis 无法登陆WTSS Web Client

版本：
schedulis_0.5.0_exec
schedulis_0.5.0_web

现象：
页面点击输入默认用户及密码无反应。
wtss-webserver.log 有如下异常信息：
java.lang.NoClassDefFoundError: org/forgerock/opendj/ldap/LdapException
at com.webank.wedatasphere.schedulis.common.user.SystemUserManager.getUser(SystemUserManager.java:88)
at azkaban.webapp.servlet.LoginAbstractAzkabanServlet.createSession(LoginAbstractAzkabanServlet.java:541)
at azkaban.webapp.servlet.LoginAbstractAzkabanServlet.createSession(LoginAbstractAzkabanServlet.java:520)
at azkaban.webapp.servlet.LoginAbstractAzkabanServlet.handleAjaxLoginAction(LoginAbstractAzkabanServlet.java:597)
at azkaban.webapp.servlet.LoginAbstractAzkabanServlet.doPost(LoginAbstractAzkabanServlet.java:448)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:760)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1617)
at com.webank.wedatasphere.schedulis.web.webapp.LocaleFilter.doFilter(LocaleFilter.java:97)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1604)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:545)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1607)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1297)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:485)

执行 linkis 任务的时候报错 java.lang.NoSuchMethodError: com.webank.wedatasphere.dss.linkis.node.execution.parser.JobParamsParser.setSignalKeyCreator(Lcom/webank/wedatasphere/dss/linkis/node/execution/job/JobSignalKeyCreator;)V

具体报错信息

01-12-2021 17:42:14 CST shell_6653 INFO - wds.linkis.client.flow.author.user.token=WS-AUTH
01-12-2021 17:42:14 CST shell_6653 INFO - azkaban.job.metadata.file=_job.34.shell_6653.meta
01-12-2021 17:42:14 CST shell_6653 INFO - azkaban.flow.projectlastchangeddate=1638324848717
01-12-2021 17:42:14 CST shell_6653 INFO - ****** End Job properties  ******
01-12-2021 17:42:14 CST shell_6653 ERROR - Job run failed!
java.lang.NoSuchMethodError: com.webank.wedatasphere.dss.linkis.node.execution.parser.JobParamsParser.setSignalKeyCreator(Lcom/webank/wedatasphere/dss/linkis/node/execution/job/JobSignalKeyCreator;)V
	at com.webank.wedatasphere.dss.plugins.azkaban.linkis.jobtype.job.AzkanbanBuilder.init(AzkanbanBuilder.java:74) ~[?:?]
	at com.webank.wedatasphere.dss.plugins.azkaban.linkis.jobtype.job.AzkanbanBuilder.<init>(AzkanbanBuilder.java:69) ~[?:?]
	at com.webank.wedatasphere.dss.plugins.azkaban.linkis.jobtype.job.JobBuilder.<clinit>(JobBuilder.java:25) ~[?:?]
	at com.webank.wedatasphere.dss.plugins.azkaban.linkis.jobtype.AzkabanDssJobType.run(AzkabanDssJobType.java:149) ~[?:?]
	at azkaban.execapp.JobRunner.runJob(JobRunner.java:836) [azkaban-exec-server-0.6.1.jar:?]
	at azkaban.execapp.JobRunner.jobRunHandle(JobRunner.java:1037) [azkaban-exec-server-0.6.1.jar:?]
	at azkaban.execapp.JobRunner.doRun(JobRunner.java:591) [azkaban-exec-server-0.6.1.jar:?]
	at azkaban.execapp.JobRunner.run(JobRunner.java:548) [azkaban-exec-server-0.6.1.jar:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_231]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_231]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_231]
01-12-2021 17:42:14 CST shell_6653 ERROR - com.webank.wedatasphere.dss.linkis.node.execution.parser.JobParamsParser.setSignalKeyCreator(Lcom/webank/wedatasphere/dss/linkis/node/execution/job/JobSignalKeyCreator;)V cause: null
01-12-2021 17:42:14 CST shell_6653 INFO - Finishing job shell_6653 at 1638351734619 with status FAILED

实在是定位不到问题是什么了，是缺包还是配置不正确？

使用gson替换fastjson

schedulers集成到dss运行时报空指针异常

schedulers集成到dss运行报错：
Caused by: java.lang.NullPointerException
at azkaban.jobtype.JobTypeManager.loadJobTypeClassLoader(JobTypeManager.java:281) ~[azkaban-common-0.5.0.jar:?]
at azkaban.jobtype.JobTypeManager.loadJobTypes(JobTypeManager.java:213) ~[azkaban-common-0.5.0.jar:?]
at azkaban.jobtype.JobTypeManager.loadPluginJobTypes(JobTypeManager.java:157) ~[azkaban-common-0.5.0.jar:?]
... 28 more
在源码中可以看到是因为jobtype.lib.dir参数获取不到报出的空指针异常

schedulis 执行工作流失败

1.版本：
schedulis_0.5.0

现象：
上传zip包到schedulis后执行工作流失败
==>job 文件内容
type=command
command=echo ${greet}
==>properties 文件内容
greet=HELLO schedulis!

3.exec节点报错日志如下：

创建dss工程和发布流程失败

dss 版本：
dss-0.7.0
linkis-0.9.3
schedulis-0.5.0

现象：
界面无法创建dss 工程

WeBankFinTech/DataSphereStudio#138

用户都是按照hadoop用户来做的。
还需要再哪里做什么配置吗？

任务不能正常执行

任务状态一直是 Preparing

报错的日志为：

配置是正常
/appcom/Install/AzkabanInstall/schedulis_0.5.0_exec/conf/azkaban.properties
executor.server.id=1

/appcom/config/wtss-config/host.properties
executor1=1

hostname为：executor1

不需要密码就可以登陆

我发现不需要使用密码就可以登陆schedulis，只有我出现这种情况吗？

定时调度功能中调度配置相关权限问题

用户项目权限开关关闭状态下，定时调度功能中调度配置相关权限存在问题，用户配置只读权限，但是操作调度配置提示报错。

系统模块--用户session过期报错

Error injecting constructor, java.io.FileNotFoundException: conf/global.properties (No such file or directory) 启动不了

2022-02-07 15:10:07.606 ERROR [main] azkaban.utils.StdOutErrRedirect 65 write - com.google.inject.ProvisionException: Unable to provision, see the following errors:

Error injecting constructor, java.io.FileNotFoundException: conf/global.properties (No such file or directory)
at azkaban.execapp.FlowRunnerManager.(FlowRunnerManager.java:175)
at azkaban.execapp.FlowRunnerManager.class(FlowRunnerManager.java:108)
while locating azkaban.execapp.FlowRunnerManager
for the 3rd parameter of azkaban.execapp.AzkabanExecutorServer.(AzkabanExecutorServer.java:126)
at azkaban.execapp.AzkabanExecutorServer.class(AzkabanExecutorServer.java:97)
while locating azkaban.execapp.AzkabanExecutorServer

1 error
2022-02-07 15:10:07.606 ERROR [main] azkaban.utils.StdOutErrRedirect 65 write - at com.google.inject.internal.InjectorImpl$2.get(InjectorImpl.java:1028)
2022-02-07 15:10:07.606 ERROR [main] azkaban.utils.StdOutErrRedirect 65 write - at com.google.inject.internal.InjectorImpl.getInstance(InjectorImpl.java:1054)
2022-02-07 15:10:07.606 ERROR [main] azkaban.utils.StdOutErrRedirect 65 write - at azkaban.execapp.AzkabanExecutorServer.main(AzkabanExecutorServer.java:178)
2022-02-07 15:10:07.606 ERROR [main] azkaban.utils.StdOutErrRedirect 65 write - Caused by: java.io.FileNotFoundException: conf/global.properties (No such file or directory)
2022-02-07 15:10:07.606 ERROR [main] azkaban.utils.StdOutErrRedirect 65 write - at java.io.FileInputStream.open0(Native Method)
2022-02-07 15:10:07.607 ERROR [main] azkaban.utils.StdOutErrRedirect 65 write - at java.io.FileInputStream.open(FileInputStream.java:195)
2022-02-07 15:10:07.607 ERROR [main] azkaban.utils.StdOutErrRedirect 65 write - at java.io.FileInputStream.(FileInputStream.java:138)
2022-02-07 15:10:07.607 ERROR [main] azkaban.utils.StdOutErrRedirect 65 write - at azkaban.utils.Props.(Props.java:84)
2022-02-07 15:10:07.607 ERROR [main] azkaban.utils.StdOutErrRedirect 65 write - at azkaban.utils.Props.(Props.java:74)
2022-02-07 15:10:07.607 ERROR [main] azkaban.utils.StdOutErrRedirect 65 write - at azkaban.execapp.FlowRunnerManager.(FlowRunnerManager.java:212)
2022-02-07 15:10:07.607 ERROR [main] azkaban.utils.StdOutErrRedirect 65 write - at azkaban.execapp.FlowRunnerManager$$FastClassByGuice$$f5329b23.newInstance()
2022-02-07 15:10:07.607 ERROR [main] azkaban.utils.StdOutErrRedirect 65 write - at com.google.inject.internal.DefaultConstructionProxyFactory$FastClassProxy.newInstance(DefaultConstructionProxyFactory.java:89)

安装jobtype插件（linkis-jobtype-1.0.0.zip）后启动exec失败

2022-03-24 15:01:13.108 ERROR [main] azkaban.utils.StdOutErrRedirect 65 write - Caused by: azkaban.jobtype.JobTypeManagerException: azkaban.jobtype.JobTypeManagerException: azkaban.jobExecutor.utils.JobExecutionException: java.lang.IllegalStateException: java.lang.NoSuchFieldError: JOB_DEFAULT_TYPE

失败暂停功能优化

优化作业流超时终止时间，和失败暂停功能描述

Workflow repeated calls in HA mode

dss-linkis-node-execution 0.7.0 eventreceiver 下游任务会报错

报错如图:

手动替换0.9.1版本运行正常,**仓库没有0.9.1版本,建议增加
https://mvnrepository.com/artifact/com.webank.wedatasphere.dss/dss-linkis-node-execution

java.lang.NullPointerException

2022-06-08 09:20:25.124 INFO [main] azkaban.jobtype.JobTypeManager 77 loadPlugins - Plugin jobtypes failed to load. java.lang.NullPointerException azkaban.jobtype.JobTypeManagerException: java.lang.NullPointerException
at azkaban.jobtype.JobTypeManager.loadPluginJobTypes(JobTypeManager.java:161) ~[azkaban-common-0.6.2.jar:?]
at azkaban.jobtype.JobTypeManager.loadPlugins(JobTypeManager.java:75) [azkaban-common-0.6.2.jar:?]
at azkaban.jobtype.JobTypeManager.(JobTypeManager.java:61) [azkaban-common-0.6.2.jar:?]
at azkaban.execapp.FlowRunnerManager.(FlowRunnerManager.java:219) [azkaban-exec-server-0.6.2.jar:?]
at azkaban.execapp.FlowRunnerManager$$FastClassByGuice$$f5329b23.newInstance() [azkaban-exec-server-0.6.2.jar:?]
at com.google.inject.internal.DefaultConstructionProxyFactory$FastClassProxy.newInstance(DefaultConstructionProxyFactory.java:89) [guice-4.1.0.jar:?]
at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:111) [guice-4.1.0.jar:?]
at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:90) [guice-4.1.0.jar:?]
at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:268) [guice-4.1.0.jar:?]
at com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46) [guice-4.1.0.jar:?]
at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1092) [guice-4.1.0.jar:?]
at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40) [guice-4.1.0.jar:?]
at com.google.inject.internal.SingletonScope$1.get(SingletonScope.java:194) [guice-4.1.0.jar:?]
at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:41) [guice-4.1.0.jar:?]
at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:38) [guice-4.1.0.jar:?]
at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:62) [guice-4.1.0.jar:?]
at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:110) [guice-4.1.0.jar:?]
at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:90) [guice-4.1.0.jar:?]
at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:268) [guice-4.1.0.jar:?]
at com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46) [guice-4.1.0.jar:?]
at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1092) [guice-4.1.0.jar:?]
at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40) [guice-4.1.0.jar:?]
at com.google.inject.internal.SingletonScope$1.get(SingletonScope.java:194) [guice-4.1.0.jar:?]
at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:41) [guice-4.1.0.jar:?]
at com.google.inject.internal.InjectorImpl$2$1.call(InjectorImpl.java:1019) [guice-4.1.0.jar:?]
at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1085) [guice-4.1.0.jar:?]
at com.google.inject.internal.InjectorImpl$2.get(InjectorImpl.java:1015) [guice-4.1.0.jar:?]
at com.google.inject.internal.InjectorImpl.getInstance(InjectorImpl.java:1054) [guice-4.1.0.jar:?]
at azkaban.execapp.AzkabanExecutorServer.main(AzkabanExecutorServer.java:178) [azkaban-exec-server-0.6.2.jar:?]
Caused by: java.lang.NullPointerException
at azkaban.jobtype.JobTypeManager.loadJobTypeClassLoader(JobTypeManager.java:281) ~[azkaban-common-0.6.2.jar:?]
at azkaban.jobtype.JobTypeManager.loadJobTypes(JobTypeManager.java:213) ~[azkaban-common-0.6.2.jar:?]
at azkaban.jobtype.JobTypeManager.loadPluginJobTypes(JobTypeManager.java:157) ~[azkaban-common-0.6.2.jar:?]
... 28 more

项目管理--项目上传错误提示优化

优化项目上传失败，提示不明显

Executor安全加固

新增token认证，白名单

定时调度--权限校验

定时调度设置时，需要校验用户权限

首页详情--点击错误工作流

首页详情--点击错误工作流，跳转到执行历史页面

修复原生email告警功能

Schedulis是否支持job分布式执行

原生azkaban只支持flow级的分布式，但并不支持job级的，看手册并没有提到同一个flow中的多个job是否可以在多个execture上执行，请问是否支持job粒度的分布式任务执行呢？

项目已被删除，不能上传

2022-06-20 16:57:37.280 ERROR [Convert-Orchestration-Thread-5] com.webank.wedatasphere.dss.orchestrator.publish.job.OrchestratorConversionJob 99 run - Convert for project DSSProject{id=313, name='ESGDataEngine', description='ESG 数据引擎', createBy='esg', username='esg', createTime=Sat Jun 18 23:28:49 CST 2022, updateTime=Sat Jun 18 23:28:49 CST 2022, product='', applicationArea=0, business='', workspaceId=null, workspaceName='null', viewUsers=null, editUsers=null, executeUsers=null} failed. com.webank.wedatasphere.dss.common.exception.DSSErrorException: errCode: 50000 ,desc: errCode: 90012 ,desc: ErrorException: errCode: 90014 ,desc: errCode: 90013 ,desc: release project failed, {
"error" : "项目已被删除，不能上传."
} ,ip: esg001 ,port: 9005 ,serviceKind: dss-workflow-server-dev ,ip: esg001 ,port: 9005 ,serviceKind: dss-workflow-server-dev ,ip: esg001 ,port: 9005 ,serviceKind: dss-workflow-server-dev ,ip: esg001 ,port: 9003 ,serviceKind: dss-framework-orchestrator-server-dev
at com.webank.wedatasphere.dss.orchestrator.publish.job.OrchestratorConversionJob.run(OrchestratorConversionJob.java:93) [dss-framework-orchestrator-publish-1.0.1.jar:?]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_212]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_212]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_212]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_212]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_212]

请问是否可以指定任务在某一个executor节点运行？

azkaban可以通过在flow中添加useExecutor=executor_id指定任务具体在哪个节点上运行，但是schedulis中，我测试却无法成功，请问这个可以设置吗？

定时调度支持状态开启关闭

工作流级别的定时调度支持状态开启关闭。定时调度不删除，只是临时关闭执行，保留调度配置信息，需要执行时再次开启执行调度。

JobBuilder/azkanbanBuilder.build 线程不安全

azkanbanBuilder 是单例模式，在run的时候获取了jobPropsMap，传递给了成员变量

this.job = JobBuilder.getAzkanbanBuilder().setJobProps(this.jobPropsMap).build();

在build中，又通过azkanbanBuilder.jobPropsMap对job进行构造
fillJobInfo(job); fillLinkisJobInfo(job);

protected void fillJobInfo(Job job) { job.setCode(jobProps.get(LinkisJobTypeConf.COMMAND)); job.setParams(new HashMap<String, Object>()); job.setRuntimeParams(new HashMap<String, Object>()); }
高并发情况下，jobPropsMap线程不安全，在构造job时，导致了结果与期望不一致

执行工作调度时失败

17-06-2020 09:14:34 CST start INFO - Starting job start at 1592356474832
17-06-2020 09:14:34 CST start INFO - job JVM args: '-Dazkaban.flowid=D' '-Dazkaban.execid=6770' '-Dazkaban.jobid=start'
17-06-2020 09:14:34 CST start INFO - user.to.proxy property was not set, defaulting to submit user hadoop
17-06-2020 09:14:34 CST start INFO - Building command job executor.
17-06-2020 09:14:34 CST start INFO - Memory granted for job start
17-06-2020 09:14:34 CST start INFO - 1 commands to execute.
17-06-2020 09:14:34 CST start INFO - cwd=/home/DataSphereStudio/AzkabanInstall/azkaban-exec-server-0.1.0-SNAPSHOT/bin/executions/6770
17-06-2020 09:14:34 CST start INFO - effective user is: hadoop
17-06-2020 09:14:34 CST start ERROR - Job run failed!
azkaban.utils.UndefinedPropertyException: Missing required property 'azkaban.native.lib'
at azkaban.utils.Props.getString(Props.java:450)
at azkaban.jobExecutor.ProcessJob.run(ProcessJob.java:235)
at azkaban.execapp.JobRunner.runJob(JobRunner.java:813)
at azkaban.execapp.JobRunner.doRun(JobRunner.java:602)
at azkaban.execapp.JobRunner.run(JobRunner.java:563)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
17-06-2020 09:14:34 CST start ERROR - Missing required property 'azkaban.native.lib' cause: null
17-06-2020 09:14:34 CST start INFO - Finishing job start at 1592356474838 with status FAILED

作业流暂停功能

功能描述
1、用户可以在作业流running、failed_finishing过程中，暂停执行作业流。作业流暂停执行后，已经运行的任务（running、queue），还会继续执行，未开始执行的任务（readay），将不会提交执行。
2、作业流暂停执行后，用户可以对任务进行右键关闭执行，右键跳过和重试failed _waiting状态任务；
3、作业流暂停执行后，如果超过1小时没有恢复运行作业流，作业流会自动取消暂停恢复运行，超时时间可以在executor azkaban.properties中配置flow.paused.max.time.ms=120000，单位毫秒；

部署方式太复杂

比起exchangis来说，schedulis部署方式太麻烦了