apache/dolphinscheduler

Apache DolphinScheduler is the modern data orchestration platform. Agile to create high performance workflow with low-code

Apache Dolphinscheduler

About

Apache DolphinScheduler is the modern data orchestration platform. Agile to create high performance workflow with low-code. It is also provided powerful user interface, dedicated to solving complex task dependencies in the data pipeline and providing various types of jobs available out of the box

The key features for DolphinScheduler are as follows:

• Easy to deploy, provide four ways to deploy which including Standalone, Cluster, Docker and Kubernetes.
• Easy to use, workflow can be created and managed by four ways, which including Web UI, Python SDK, Yaml file and Open API
• Highly reliable and high availability, decentralized architecture with multi-master and multi-worker, native supports horizontal scaling.
• High performance, its performance is N times faster than other orchestration platform and it can support tens of millions of tasks per day
• Cloud Native, DolphinScheduler supports orchestrating multi-cloud/data center workflow, and supports custom task type
• Versioning both workflow and workflow instance(including tasks)
• Various state control of workflow and task, support pause/stop/recover them in any time
• Multi-tenancy support
• Others like backfill support(Web UI native), permission control including project, resource and data source

QuickStart

• For quick experience
  • Want to start with standalone
  • Want to start with Docker
• For Kubernetes
  • Start with Kubernetes
• For Terraform
  • Start with Terraform

User Interface Screenshots

• Homepage: Project and workflow overview, including the latest workflow instance and task instance status statistics.

• Workflow Definition: Create and manage workflow by drag and drop, easy to build and maintain complex workflow, support bulk of tasks out of box.

• Workflow Tree View: Abstract tree structure could clearer understanding of the relationship between tasks

• Data source: Manage support multiple external data sources, provide unified data access capabilities for such as MySQL, PostgreSQL, Hive, Trino, etc.

• Monitor: View the status of the master, worker and database in real time, including server resource usage and load, do quick health check without logging in to the server.

Suggestions & Bug Reports

Follow this guide to report your suggestions or bugs.

Contributing

The community welcomes everyone to contribute, please refer to this page to find out more: How to contribute, find the good first issue in here if you are new to DolphinScheduler.

Community

Welcome to join the Apache DolphinScheduler community by:

• Join the DolphinScheduler Slack to keep in touch with the community
• Follow the DolphinScheduler Twitter and get the latest news
• Subscribe DolphinScheduler mail list, [email protected] for user and [email protected] for developer

Landscapes

  

DolphinScheduler enriches the CNCF CLOUD NATIVE Landscape.

dolphinscheduler's Issues

Run the escheduler-server directly after importing the item and report the error directly.（导入项后运行escheduler-server直接报错）

Indicates that there is no class such as LogParameter in cn.escheduler.rpc

---

指示cn.escheduler.rpc中没有LogParameter等类

How to support HADOOP cluster with kerberos enabled?（如何支持启用了kerberos的HADOOP集群？）

I see the manuals for spark and hivesql without instructions.

---

我看使用手册spark和hivesql没有相关说明

Python task, can specify the python version

Is there a problem with the front-end UI deployment documentation?（前端UI部署文档是不是有问题？）

https://analysys.github.io/easyscheduler_docs_cn/%E5%89%8D%E7%AB%AF%E9%83%A8%E7%BD%B2%E6%96%87%E6%A1%A3.html#1%E3%80%81%E5%87%86%E5%A4%87%E5%B7%A5%E4%BD%9C

Wanted: Who is using DolphinScheduler

Who is using DolphinScheduler

Sincerely thank everyone who constantly keeps on using and supporting DolphinScheduler. We will try our best to make DolphinScheduler better and make the DolphinScheduler community and ecology more prosperous.

The original intention of this issue

1. We’d like to listen to the community to make DolphinScheduler better.
2. We want to attract more partners to contribute to DolphinScheduler.
3. Learn more about the practical use scenarios of DolphinScheduler to facilitate the next step of planning.

What we expect from you

Please submit a comment in this issue to include the following infomation:

your company, school or organization, and homepage
your city and country.
your contact info: blog, email, twitter (at least one).
for what business scenario do you use DolphinScheduler.

You can refer to the following sample answer:

* Organization: xueqiu
* Location: Beijing, China
* Contact: [email protected]
* Scenario: Data task and ETL scheduling platform
* WeChat/Slack:
* Logo Url:

Thanks again for your participation ! Your support is the motivation for us to move forward.
DolphinScheduler Community

Error when starting api server（启动api server时报错 ）

When starting the api server, the error class "javax.servlet.http.HttpSessionIdListener"'s signer information does not match signer information of other classes in the same package , but check the lib directory without repeated HttpSessionIdListener.class, why?

---

启动api server时报错 class "javax.servlet.http.HttpSessionIdListener"'s signer information does not match signer information of other classes in the same package ， 但是检查lib目录下又没有重复的HttpSessionIdListener.class，何解？

Recommendation: Resource file upload and use, can support batch and directory structure (建议：资源文件上传和使用，可以支持批量和目录结构 )

Such as the title

Does the scheduling job support calling the http interface and the JAVA package?（调度作业支持调用http接口和JAVA程序包吗？）

The link address is missing(链接地址缺少)

找不到tar.gz

修改tar目录:
tar -zxvf $workDir/../../escheduler-1.0.0-SNAPSHOT.tar.gz -C $installPath

sudo unknown user :test

Write a simple shell task. Execute the following error

How did this test come out?

---

写了一个简单的shell任务。执行时报下面的错误

这个test是怎么出来的？

Esceduler-daemon.sh does not recognize JAVA_HOME and causes installation to fail（ escheduler-daemon.sh无法识别JAVA_HOME导致安装失败）

Error message:

Cat escheduler-api-server-host1.out
Nohup: Unable to run command "/bin/java": no such file or directory

JAVA_HOME is available in the shell environment.

Echo $JAVA_HOME
/usr/java/jdk1.8.0

Temporary solution:
** In the file escheduler-daemon.sh write dead export JAVA_HOME=xxx**

I hope to have a more elegant way to solve this problem, such as:
The **.escheduler_env.sh file has been manually specified with environment variable information, and escheduler-daemon.sh can be reused directly. **

---

错误信息：

cat escheduler-api-server-host1.out
nohup: 无法运行命令"/bin/java": 没有那个文件或目录

JAVA_HOME在shell环境中可以获取到。

echo $JAVA_HOME
/usr/java/jdk1.8.0

临时解决方案：
在文件 escheduler-daemon.sh 写死export JAVA_HOME=xxx

期望能有个比较优雅的方式解决这个问题，比如：
.escheduler_env.sh文件已手动指定环境变量信息，escheduler-daemon.sh可直接复用。

One-click deployment optimization, use the boot, target 30 minutes to get started （一键部署优化，使用上引导，目标30分钟上手）

One-click deployment optimization, use the boot, target 30 minutes to get started

---

一键部署优化，使用上引导，目标30分钟上手

Missing LogParameter and RetStrInfo（缺少LogParameter和RetStrInfo）

LogViewServiceGrpc cannot find LogParameter and RetStrInfo below

---

LogViewServiceGrpc 下面找不到LogParameter和RetStrInfo

Different rent users are forbidden from uploading the same resource file

Different rent users are forbidden from uploading the same resource file, I think this is only a functional enhancement issue, so track it here. Thx.

There is a subtle spelling mistake in ProcessDao(ProcessDao里有一处细微的拼写错误)

List needFailoverProcessInstanceList = processDao.queryNeddFailoverProcessInstances(masterHost);

Escheduler backend service startup error （escheduler 后台启动报错）

[root@bogon escheduler-backend-1.0.1]# sh install.sh
1, replace the file
2, create a directory
[email protected]'s password:
[email protected]'s password:
[email protected]'s password:
[email protected]'s password:
3, stop the service
Ssh: Could not resolve hostname cd /; sh bin/escheduler-daemon.sh stop alert-server;: Name or service not known
4, delete the zk node
Deleted success
5, scp resources
Scp copy completed
6, start
Ssh: Could not resolve hostname cd /; sh bin/escheduler-daemon.sh start alert-server;: Name or service not known

Attached file configuration:

conf/config/install_config.confConfiguration

Install path, do not use the same path (pwd)

installPath="/usr/local/escheduler-backend-1.0.1"

Esceduler path:
[root@bogon bin]# pwd
/usr/local/escheduler-backend-1.0.1/bin
[root@bogon bin]# ls
Esceduler-daemon.sh
[root@bogon bin]#

---

[root@bogon escheduler-backend-1.0.1]# sh install.sh
1,替换文件
2,创建目录
[email protected]'s password:
[email protected]'s password:
[email protected]'s password:
[email protected]'s password:
3,停止服务
ssh: Could not resolve hostname cd /; sh bin/escheduler-daemon.sh stop alert-server;: Name or service not known
4,删除zk节点
deleted success
5,scp资源
scp拷贝完成
6,启动
ssh: Could not resolve hostname cd /; sh bin/escheduler-daemon.sh start alert-server;: Name or service not known

附文件配置：

conf/config/install_config.conf配置

安装路径,不要当前路径(pwd)一样

installPath="/usr/local/escheduler-backend-1.0.1"

escheduler路径：
[root@bogon bin]# pwd
/usr/local/escheduler-backend-1.0.1/bin
[root@bogon bin]# ls
escheduler-daemon.sh
[root@bogon bin]#

Suggestion: When the process definition is running, the pop-up box can manually set the global parameters. （建议: 点击流程运行的时, 弹出框可以手工设置全局参数）

call datax failure （调用datax失败）

[INFO] 2019-04-23 17:06:58.862 cn.escheduler.server.worker.log.TaskLogger:[178] - [taskAppId=TASK_42_159_699] ->
DataX (DATAX-OPENSOURCE-3.0), From Alibaba !
Copyright (C) 2010-2017, Alibaba Group. All Rights Reserved.

/bin/sh: java: command not found

hive run schedule setting run parameters cannot be set（hive运行调度设置运行参数无法设置）

Here hive needs to set the running parameters, found that writing here is invalid, how to set the running parameters of hive?

---

这里hive需要设置运行参数，发现写在这里是无效的，请问怎么设置hive的运行参数呢

outlook 邮件发送不了邮件

修改:

hive连接不上

修改pom.xml文件

Job definition - the mail task node can independently select the CC mail user（工作定义—邮件任务节点可以独立选取抄送邮件用户）

Feature Request: Add Pre/Post Statement support in SQL Task

For SQL (JDBC) Tasks, some times, more customization is needed. This is usually provided by some "pre-statement" to execute before execute the main SQL. So, I propose to add PreStatement & PostStatement support to SQL task.

This is an optional feature, so by default, if user doesn't add any Pre/Post statement, it won't affect any thing.

This feature can be useful in some situation, like:

1. before the main SQL, need to customize datetime format, timezone information, etc.
2. before the main SQL to insert data into yesterday's partition, need to check if that partition exists, if it don't exist, create new partition.

The mobile phone number in the user account cannot be recognized. The latest number of Unicom number 166 begins.（用户账号中手机号无法识别联通最新号码166开头）

As the title, the format will be incorrect.

---

如题，会提示格式不对。

In tenant management, special characters such as underscores in the tenant code cannot pass the check.（租户管理中，租户编码带下划线等特殊字符无法通过校验）

According to the setting, the tenant code = linux user name. When the linux user name is underlined and other special characters, the tenant will be prompted [Please enter the English tenant code].

Temporary solution: directly modify the contents of the corresponding database table t_escheduler_tenant.

Expectations: Front-end validation logic can be optimized to allow for special characters.

---

按照设定，租户编码=linux用户名，当linux用户名带下划线等特殊字符时，创建租户会提示【请输入英文租户编码】。

临时解决方案：直接修改对应数据库表t_escheduler_tenant的内容。

期望：前端校验逻辑可以优化下，允许特殊字符的情况。

Add task to disable run option on task node（在任务节点 增加任务禁止运行选项）

Task type support Oracle、CLICKHOUSE、SQLSERVER （任务类型支持 Oracle、CLICKHOUSE、SQLSERVER）

Provide external API, open task scheduling trigger interface （对外提供API，开放系统任务调度触发接口）

Data flow architecture diagram (数据流向架构图)

请大佬 提供一个 模块之间的数据流向架构图，可以使新手读代码能更准确和更方便，谢谢

More data source support（更多的数据源支持）

Can support oracle, I hope to support the mainstream database. At the same time, I hope to support more databases through jdbc. As for the stored procedure, you can use the jdbc standard stored procedure call method. thank.

_{Sent with GitHawk}

---

能否支持oracle，希望支持主流数据库。同时希望通过jdbc的方式，支持更多的数据库。至于存储过程这块，可以使用jdbc标准的存储过程的调用方法。感谢。

_{Sent with GitHawk}

Is there a problem with the front-end UI deployment documentation?（前端UI部署文档是不是有问题？）

https://analysys.github.io/easyscheduler_docs_cn/%E5%89%8D%E7%AB%AF%E9%83%A8%E7%BD%B2%E6%96%87%E6%A1%A3.html #1%E3%80%81%E5%87%86%E5%A4%87%E5%B7%A5%E4%BD%9C

The purpose of this manual is to install an Nginx, configure the nginx foreground access port 8888, the background proxy port 12345, and the directory to point to dist.
1,
According to this idea, I downloaded and installed Nginx and configured it strictly according to the deployment manual.
When visiting http://10**:8888, an error will automatically access the following URL, 404 error
Http://10***:8888/escheduler/users/get-user-info?_t=0.06944380999772859
404 Not Found

2,
  Background proxy port 12345 This is the port that is agreed to interact with the background or?
Because when the eschedule background is installed, one-click startup does not involve the proxy port.

---

https://analysys.github.io/easyscheduler_docs_cn/%E5%89%8D%E7%AB%AF%E9%83%A8%E7%BD%B2%E6%96%87%E6%A1%A3.html#1%E3%80%81%E5%87%86%E5%A4%87%E5%B7%A5%E4%BD%9C

这个手册，目的就是安装一个Nginx , 配置nginx 前台访问端口8888， 后台代理端口12345，目录指向dist。
1、
按照这个**，我下载安装了Nginx ，严格按照部署手册配置，
访问http://10**:8888 时候 报错，会自动访问如下这个URL，404错误
http://10***:8888/escheduler/users/get-user-info?_t=0.06944380999772859
404 Not Found

2、
后台代理端口12345 这个是约定跟后台交互的端口还是怎样？
因为在安装 eschedule后台的时候，一键启动，并没有涉及代理端口

What is the version used by node?（node 使用版本是多少）

1.0.1 version starts api server error java.lang.NoSuchFieldError: IS_SECURITY_ENABLED（1.0.1版本启动api server报错java.lang.NoSuchFieldError: IS_SECURITY_ENABLED）

As shown below, api-log prints as follows:
Error starting ApplicationContext. To display the auto-configuration report re-run your application with 'debug' enabled.
[ERROR] 2019-04-19 00:07:28.550 org.springframework.boot.SpringApplication:[839] - Application startup failed
java.lang.NoSuchFieldError: IS_SECURITY_ENABLED
         At org.apache.jasper.compiler.JspRuntimeContext.(JspRuntimeContext.java:197)
         At org.apache.jasper.servlet.JspServlet.init(JspServlet.java:150)
         At org.eclipse.jetty.servlet.ServletHolder.initServlet(ServletHolder.java:643)
         At org.eclipse.jetty.servlet.ServletHolder.initialize(ServletHolder.java:422)
         At org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:892)

After analysis, remove jasper-runtime-5.5.23.jar from lib, api server can be started, it should be a jar problem, I do not know what is dependent on this jar

---

如下所示，api-log打印如下：
Error starting ApplicationContext. To display the auto-configuration report re-run your application with 'debug' enabled.
[ERROR] 2019-04-19 00:07:28.550 org.springframework.boot.SpringApplication:[839] - Application startup failed
java.lang.NoSuchFieldError: IS_SECURITY_ENABLED
at org.apache.jasper.compiler.JspRuntimeContext.(JspRuntimeContext.java:197)
at org.apache.jasper.servlet.JspServlet.init(JspServlet.java:150)
at org.eclipse.jetty.servlet.ServletHolder.initServlet(ServletHolder.java:643)
at org.eclipse.jetty.servlet.ServletHolder.initialize(ServletHolder.java:422)
at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:892)

经过分析，把jasper-runtime-5.5.23.jar从lib中移除，api server则可以启动，应该是jar的问题，不知是什么依赖引用了此jar

EasyScheduler service can't find java_home path

修改 /usr/local/src/EasyScheduler/target/escheduler-1.0.0-SNAPSHOT/bin/escheduler-daemon.sh

/usr/local/src是自己目录

Support Oracle,CLICKHOUSE

Hope that support can call oracle stored procedures, SQL. Because in many traditional industries. Oracle database is still a lot.

Support clickhouse sql.

---

希望支持可以调用oracle存储过程,SQL. 因为在很多传统行业。oracle数据库还是很多。

支持clickhouse sql.

If you have a bug or an idea. Issues labeled help wanted or good first issue can be good first contributions

If you have a bug or an idea, read the contributing guidelines before opening an issue.

Issues labeled help wanted or good first issue can be good first contributions.

Does it support the kettle task?(是否支持kettle任务呢）

Can EasyScheduler support sub workflow task?

We may need to combine workflows into one sometimes.Does/will easyscheduler support this feature?

hadoop

Cannot support custom data transfer between nodes

The DAG graph can draw a ring diagram in some cases.（DAG图某些情况下可以画出环状图）

The method of recurring, taking the SHELL node as an example:

1. Drag 4 shell nodes, the names are s1, s2, s3, s4
2. Connection method: s1->s3; s2->s3; s3->s4; s4->s2; s4->s1

---

复现方法，以SHELL节点为例：
1.拖4个shell节点，名称分别为s1,s2,s3,s4
2.连接方法：s1->s3 ; s2->s3; s3->s4 ;s4->s2 ; s4->s1

The cluster python version is 3, and the EashScheduler python script print version is 2. What is the solution?（集群python版本是3，EashScheduler python脚本打印的版本是2，何解？）

Easy to schedule cluster installation, cluster python environment is 3, using the python node script of the scheduling platform to print python version is 2?

---

易调度集群安装，集群python 环境是3，利用调度平台的python节点脚本打印python 版本是2？

Does the API server SchedulerService interact with the master without passing rpc?（API server SchedulerService 与master交互时候没有经过rpc？）

Recently, when I saw the implementation of the SchedulerService, I had a question. SetScheduleState is used to set the schedule online.
Instead of calling the master's rpc, it is called directly: setSchedule. The setSchedule implementation is a directly modified QuartzExecutors

I feel that there is a problem in the distributed deployment of api and master. I don't know if I understand it correctly? Is there any revision plan afterwards?

    Public void setSchedule(int projectId, int scheduleId) throws RuntimeException{
        Logger.info("set schedule, project id: {}, scheduleId: {}", projectId, scheduleId);


        Schedule schedule = processDao.querySchedule(scheduleId);
        If (schedule == null) {
            Logger.warn("process schedule info not exists");
        }

        Date startDate = schedule.getStartTime();
        Date endDate = schedule.getEndTime();

        String jobName = QuartzExecutors.buildJobName(scheduleId);
        String jobGroupName = QuartzExecutors.buildJobGroupName(projectId);

        Map<String, Object> dataMap = QuartzExecutors.buildDataMap(projectId, scheduleId, schedule);

        QuartzExecutors.getInstance().addJob(ProcessScheduleJob.class, jobName, jobGroupName, startDate, endDate,
                schedule.getCrontab(), dataMap);

    }

---

最近看SchedulerService 实现时候有个疑问， setScheduleState 里面实现设置调度online时候
没有调用master的rpc，而是直接调用：setSchedule。setSchedule 实现是直接修改的QuartzExecutors

感觉在在api与master分布式部署时候有问题 不知道是否理解正确？ 后续有修改计划吗

    public void setSchedule(int projectId, int scheduleId) throws RuntimeException{
        logger.info("set schedule, project id: {}, scheduleId: {}", projectId, scheduleId);


        Schedule schedule = processDao.querySchedule(scheduleId);
        if (schedule == null) {
            logger.warn("process schedule info not exists");
        }

        Date startDate = schedule.getStartTime();
        Date endDate = schedule.getEndTime();

        String jobName = QuartzExecutors.buildJobName(scheduleId);
        String jobGroupName = QuartzExecutors.buildJobGroupName(projectId);

        Map<String, Object> dataMap = QuartzExecutors.buildDataMap(projectId, scheduleId, schedule);

        QuartzExecutors.getInstance().addJob(ProcessScheduleJob.class, jobName, jobGroupName, startDate, endDate,
                schedule.getCrontab(), dataMap);

    }

Front end input box does not have length limit(前端输入框没有做长度限制)

The front-end input box is not limited in length and can be input without restrictions.

---

前端输入框没有做长度限制，可以无限制输入。

Wrong when viewing the log9查看日志时报错)

View the execution log when reporting the following error

---

查看执行日志时报如下错误
logs/escheduler-api-server.log

[INFO] 2019-04-02 21:51:37.395 cn.escheduler.api.service.SessionService:[69] - get session: e6600570-9a5e-4d6f-a954-a9392ba82fa6, ip: 192.168.1.2
[INFO] 2019-04-02 21:51:37.398 cn.escheduler.api.controller.LoggerController:[60] - login user qinlong.yang, view 2 task instance log ,skipLineNum 0 , limit 10000
[INFO] 2019-04-02 21:51:37.399 cn.escheduler.api.service.LoggerService:[57] - log host : 172.20.207.6 , logPath : /opt/escheduler/logs/7/111/2.log , logServer port : 50051
[INFO] 2019-04-02 21:51:37.400 cn.escheduler.api.log.LogClient:[79] - roll view log : path /opt/escheduler/logs/7/111/2.log,skipLineNum 0 ,limit 10000
[WARN] 2019-04-02 21:51:37.407 io.grpc.stub.ClientCalls$ThreadlessExecutor:[628] - Runnable threw exception
java.lang.IllegalAccessError: tried to access field com.google.protobuf.AbstractMessage.memoizedSize from class cn.escheduler.rpc.LogParameter
        at cn.escheduler.rpc.LogParameter.getSerializedSize(LogParameter.java:197)
        at io.grpc.protobuf.lite.ProtoInputStream.available(ProtoInputStream.java:108)
        at io.grpc.internal.MessageFramer.getKnownLength(MessageFramer.java:204)
        at io.grpc.internal.MessageFramer.writePayload(MessageFramer.java:136)
        at io.grpc.internal.AbstractStream.writeMessage(AbstractStream.java:52)
        at io.grpc.internal.DelayedStream$5.run(DelayedStream.java:218)
        at io.grpc.internal.DelayedStream.drainPendingCalls(DelayedStream.java:132)
        at io.grpc.internal.DelayedStream.setStream(DelayedStream.java:101)
        at io.grpc.internal.DelayedClientTransport$PendingStream.createRealStream(DelayedClientTransport.java:414)
        at io.grpc.internal.DelayedClientTransport$PendingStream.access$300(DelayedClientTransport.java:397)
        at io.grpc.internal.DelayedClientTransport$5.run(DelayedClientTransport.java:355)
        at io.grpc.stub.ClientCalls$ThreadlessExecutor.waitAndDrain(ClientCalls.java:626)
        at io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:122)
        at cn.escheduler.rpc.LogViewServiceGrpc$LogViewServiceBlockingStub.rollViewLog(LogViewServiceGrpc.java:310)
        at cn.escheduler.api.log.LogClient.rollViewLog(LogClient.java:88)
        at cn.escheduler.api.service.LoggerService.queryLog(LoggerService.java:63)
        at cn.escheduler.api.controller.LoggerController.queryLog(LoggerController.java:62)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205)
        at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:133)
        at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:116)
        at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:827)
        at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:738)
        at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85)
        at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:963)
        at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:897)
        at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970)
        at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:861)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:687)
        at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
        at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:848)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1772)
        at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:99)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
        at org.springframework.web.filter.HttpPutFormContentFilter.doFilterInternal(HttpPutFormContentFilter.java:105)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
        at org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:81)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
        at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:197)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
        at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
        at org.eclipse.jetty.server.Server.handle(Server.java:534)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:110)
        at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
        at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
        at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
        at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
        at java.lang.Thread.run(Thread.java:748)

Schedule the who am i command, the log does not display（调度who am i命令，日志不显示）

Create a simple shell task in ui and put it in the dag, the content is who am i, can be executed successfully, but the log of the task instance is not printed. No logs are displayed. At the same time, the log of escheduler's log server will print the following error:
16:14:04.366 [grpc-default-executor-8] INFO cn.escheduler.server.rpc.LoggerServer - log parameter path : /app/escheduler-bin/logs/3/9/22.log ,skipLine : 0, Limit : 10000
16:14:04.366 [grpc-default-executor-8] ERROR cn.escheduler.server.rpc.LoggerServer - read file failed : /app/escheduler-bin/logs/3/9/22.log
java.nio.file.NoSuchFileException: /app/escheduler-bin/logs/3/9/22.log
At sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
At sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
At sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
At sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:214)
At java.nio.file.Files.newByteChannel(Files.java:361)
At java.nio.file.Files.newByteChannel(Files.java:407)
At java.nio.file.spi.FileSystemProvider.newInputStream(FileSystemProvider.java:384)
At java.nio.file.Files.newInputStream(Files.java:152)
At java.nio.file.Files.newBufferedReader(Files.java:2784)
At java.nio.file.Files.lines(Files.java:3744)
At java.nio.file.Files.lines(Files.java:3785)
At cn.escheduler.server.rpc.LoggerServer.readFile(LoggerServer.java:166)
At cn.escheduler.server.rpc.LoggerServer.access$200(LoggerServer.java:38)
At cn.escheduler.server.rpc.LoggerServer$LogViewServiceGrpcImpl.rollViewLog(LoggerServer.java:99)
At cn.escheduler.rpc.LogViewServiceGrpc$MethodHandlers.invoke(LogViewServiceGrpc.java:418)
At io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171)
At io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:272)
At io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:650)
At io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
At io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
At java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
At java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
At java.lang.Thread.run(Thread.java:748)
16:14:04.366 [grpc-default-worker-ELG-3-1] DEBUG io.grpc.netty.NettyServerHandler - [id: 0xaa5e2e06, L:/192.168.135.183:50051 - R:/192.168.135.183:52064] OUTBOUND HEADERS: streamId=3 headers=GrpcHttp2OutboundHeaders[:status: 200, content-type: application/grpc, grpc-status: 2] streamDependency=0 weight=16 exclusive=false padding=0 endStream=true
Apr 22, 2019 4:14:04 PM io.grpc.internal.SerializingExecutor run
SEVERE: Exception while executing runnable io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed@2a04a176
java.lang.NullPointerException
At cn.escheduler.server.rpc.LoggerServer$LogViewServiceGrpcImpl.rollViewLog(LoggerServer.java:101)
At cn.escheduler.rpc.LogViewServiceGrpc$MethodHandlers.invoke(LogViewServiceGrpc.java:418)
At io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171)
At io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:272)
At io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:650)
At io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
At io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
At java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
At java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
At java.lang.Thread.run(Thread.java:748)

At present, only the am am i is detected, and other commands such as ls and pwd have no problem.

---

在ui里创建一个最简单的shell 任务并放到dag中，内容为who am i，可以执行成功，但是查看任务实例的日志，并没有打印。显示没有日志。与此同时，escheduler的log server的日志会打印如下错误：
16:14:04.366 [grpc-default-executor-8] INFO cn.escheduler.server.rpc.LoggerServer - log parameter path : /app/escheduler-bin/logs/3/9/22.log ,skipLine : 0, limit : 10000
16:14:04.366 [grpc-default-executor-8] ERROR cn.escheduler.server.rpc.LoggerServer - read file failed : /app/escheduler-bin/logs/3/9/22.log
java.nio.file.NoSuchFileException: /app/escheduler-bin/logs/3/9/22.log
at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
at sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:214)
at java.nio.file.Files.newByteChannel(Files.java:361)
at java.nio.file.Files.newByteChannel(Files.java:407)
at java.nio.file.spi.FileSystemProvider.newInputStream(FileSystemProvider.java:384)
at java.nio.file.Files.newInputStream(Files.java:152)
at java.nio.file.Files.newBufferedReader(Files.java:2784)
at java.nio.file.Files.lines(Files.java:3744)
at java.nio.file.Files.lines(Files.java:3785)
at cn.escheduler.server.rpc.LoggerServer.readFile(LoggerServer.java:166)
at cn.escheduler.server.rpc.LoggerServer.access$200(LoggerServer.java:38)
at cn.escheduler.server.rpc.LoggerServer$LogViewServiceGrpcImpl.rollViewLog(LoggerServer.java:99)
at cn.escheduler.rpc.LogViewServiceGrpc$MethodHandlers.invoke(LogViewServiceGrpc.java:418)
at io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171)
at io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:272)
at io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:650)
at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
16:14:04.366 [grpc-default-worker-ELG-3-1] DEBUG io.grpc.netty.NettyServerHandler - [id: 0xaa5e2e06, L:/192.168.135.183:50051 - R:/192.168.135.183:52064] OUTBOUND HEADERS: streamId=3 headers=GrpcHttp2OutboundHeaders[:status: 200, content-type: application/grpc, grpc-status: 2] streamDependency=0 weight=16 exclusive=false padding=0 endStream=true
Apr 22, 2019 4:14:04 PM io.grpc.internal.SerializingExecutor run
SEVERE: Exception while executing runnable io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed@2a04a176
java.lang.NullPointerException
at cn.escheduler.server.rpc.LoggerServer$LogViewServiceGrpcImpl.rollViewLog(LoggerServer.java:101)
at cn.escheduler.rpc.LogViewServiceGrpc$MethodHandlers.invoke(LogViewServiceGrpc.java:418)
at io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171)
at io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:272)
at io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:650)
at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)

目前测出只有who am i为如此，其他命令如ls和pwd之类的都没有问题

Cron management, process definition need delete function （定时管理、工作流定义添加删除功能）

Timing management, workflow definition add delete function

Batch delete process instance

---

定时管理、工作流定义添加删除功能

批量删除流程实例

If you want to customize the worker, EsScheduler only does the task flow scheduling service, currently supported?（如果想要自定义worker,EsScheduler只做任务流调度服务，当前支持？）

Implement a small distributed file system (supporting stand-alone) for resource uploading instead of hdfs.

Implement a small distributed file system (supporting stand-alone) for resource uploading instead of hdfs.

---

实现一个小巧的分布式文件系统(支持单机)，用于资源上传，替代hdfs