hxsecurity/dongtai-engine

This repository has been merged into https://github.com/HXSecurity/DongTai. DongTai-engine used to analyze the method data collected by the probe, analyze whether there are vulnerabilities in API requests through the algorithm of taint tracking, and is also responsible for timing tasks, including: expired log cleaning, probe state maintenance, data packet replay processing, etc.

DongTai-engine

中文版本(Chinese version)

Whit is DongTai-Engine?

DongTai-Engine is used to process the data collected by the DongTai probe, and its functions are as follows：

1. Analyze whether there are vulnerabilities in HTTP/HTTPS/RPC requests based on method pool data and taint tracking algorithms
2. Handle vulnerability verification requests regularly
3. Regularly update the vulnerabilities in the components
4. Regularly clean up expired log data
5. Regularly maintain the status of the probe detection engine

Deploy

Basic services：MySql、Redis

The basic service configuration is as follows：

service name	ip	port	additional
MySql	127.0.0.1	3306	account：dongtai password：dongtai-iast database name：dongtai_webapi
Redis	127.0.0.1	6379	password：123456 Redis db：0

Official image

1. Pull image

$ docker pull registry.cn-beijing.aliyuncs.com/huoxian_pub/dongtai-engine:latest

2. Create a configuration file：/etc/dongtai/config.ini，The content is as follows：

[mysql]
host = 127.0.0.1
port = 3306
name = dongtai_webapi
user = dongtai
password = dongtai-iast

[redis]
host = 127.0.0.1
port = 6379
password = 123456
db = 0

; The following content unused, keep the default
[engine]
url = http://engine_url


[smtp]
server = server
user = user
password = password
from_addr = from_addr
ssl = False
cc_addr = cc_addr

[aliyun_oss]
access_key = access_key
access_key_secret = access_key

3. Start the dongtai-engine container and map the configuration file

$ docker run -d --name dongtai-engine -v /etc/dongtai/config.ini:/opt/dongtai/engine/conf/config.ini --restart=always secnium/iast-saas-engine:latest

4. Start the dongtai-engine-task container and map the configuration file

$ docker run -d --name dongtai-engine-task -v /etc/dongtai/config.ini:/opt/dongtai/engine/conf/config.ini --restart=always secnium/iast-saas-engine:latest bash /opt/dongtai/engine/docker/entrypoint.sh

Build custom image

1. Build image

$ docker build -t secnium/iast-saas-engine:latest .

2. Create a configuration file：/etc/dongtai/config.ini，The content is as follows：

[mysql]
host = 127.0.0.1
port = 3306
name = dongtai_webapi
user = dongtai
password = dongtai-iast

[redis]
host = 127.0.0.1
port = 6379
password = 123456
db = 0

; The following content unused, keep the default
[engine]
url = http://engine_url

[smtp]
server = server
user = user
password = password
from_addr = from_addr
ssl = False
cc_addr = cc_addr

[aliyun_oss]
access_key = access_key
access_key_secret = access_key

3. Start the dongtai-engine container and map the configuration file

$ docker run -d --name dongtai-engine -v /etc/dongtai/config.ini:/opt/dongtai/engine/conf/config.ini --restart=always secnium/iast-saas-engine:latest

4. Start the dongtai-engine-task container and map the configuration file

$ docker run -d --name dongtai-engine-task -v /etc/dongtai/config.ini:/opt/dongtai/engine/conf/config.ini --restart=always secnium/iast-saas-engine:latest bash /opt/dongtai/engine/docker/entrypoint.sh

Contributing

Contributions are welcomed and greatly appreciated. See CONTRIBUTING.md for details on submitting patches and the contribution workflow.

Any questions? Let's discuss in #DongTai discussions

More resources

• Documentation
• DongTai WebSite

dongtai-engine's Issues

[Bug]: 存在漏洞的位置标示出错

Preflight Checklist

• I agree to follow the Code of Conduct that this project adheres to.
• I have searched the issue tracker for an issue that matches the one I want to file, without success.
• I am not looking for support or already pursued the available support channels without success.

Version

1.1.2

Installation Type

Official Docker Compose

Service Name

Deploy(DongTai-deploy)

Describe the details of the bug and the steps to reproduce it

私有docker化部署，
存在漏洞的位置标示出错，我这显示在HTTP头部的Origin：http://1.1.1.1 其中HTTP标红。

Additional Information

No response

Logs

No response

敏感数据检测的正则存在误报建议使用以下正则（python需要调整，我这边写的是标准语法）

(?<!\d)(86|086|(?<!\d))[1][345789][0-9]{9}(?!\d)

add ISSUE_TEMPLATE/CONTRIBUTING/CODE_OF_CONDUCT、modify deploy file

[Feature]: The exported report does not contain vulnerability information for the component library

Preflight Checklist

• I agree to follow the Code of Conduct that this project adheres to.
• I have searched the issue tracker for an issue that matches the one I want to file, without success.

Problem Description

The exported report does not contain vulnerability information for the component library

Proposed Solution

The exported report does not contain vulnerability information for the component library

Alternatives Considered

The exported report does not contain vulnerability information for the component library

Additional Information

No response

enhancement code style

[Bug]: 在到处报告，报告乱码问题

Preflight Checklist

• I agree to follow the Code of Conduct that this project adheres to.
• I have searched the issue tracker for an issue that matches the one I want to file, without success.
• I am not looking for support or already pursued the available support channels without success.

Version

最新版本

Installation Type

Official SaaS Service

Service Name

Doc(DongTai-Doc)

Describe the details of the bug and the steps to reproduce it

1.导出报告，发现乱码

小BUG

Additional Information

我是tian钧，署名下

Logs

无

Before active verification of vulnerabilities, determine whether to enable active verification

[Bug]: 同一个接口，同一个请求方式，统一个薄弱点，同一种漏洞类型反复多次报漏洞。

Preflight Checklist

• I agree to follow the Code of Conduct that this project adheres to.
• I have searched the issue tracker for an issue that matches the one I want to file, without success.
• I am not looking for support or already pursued the available support channels without success.

Version

1.1.3

Installation Type

Official Docker Compose

Service Name

DongTai-WebAPI

Describe the details of the bug and the steps to reproduce it

建议：同一个接口，同一个请求方式，统一个薄弱点，同一种漏洞类型作为唯一性校验

Additional Information

No response

Logs

No response

Increase the detection of sensitive information leakage

Problem Description

A clear and concise description of the problem you are seeking to solve with this feature request.

With the improvement of web application security, unauthorized access vulnerabilities and unauthorized vulnerabilities account for more and more vulnerabilities. The core of automatic detection of permission vulnerabilities is to identify the sensitive information data returned by the API interface. But how to detect the leakage of sensitive information?

Proposed Solution

A clear and concise description of what would you like to happen.

1. Read sensitive information configuration rules from mysql
2. According to the rules, parse and check whether there is sensitive information from the http response body

Feature: Automatic Versioning

Increase the judgment of whether prompt vulnerabilities are enabled

Increase the judgment of whether prompt vulnerabilities are enabled

github被墙导致代码拉不下来

问题：
由于国内的网络问题，git被墙拉不下来代码，本地部署难度增高；

解决方案：
上传构建好的镜像到公共镜像服务，供社区下载试用

TypeError: the JSON object must be str, bytes or bytearray, not NoneType

DongTai-Engine unsolved exception:

Task core.tasks.vul_recheck[63c005c7-d49a-4636-9168-6fbe59c1aa89] raised unexpected: TypeError('the JSON object must be str, bytes or bytearray, not NoneType')
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/celery/app/trace.py", line 385, in trace_task
    R = retval = fun(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/celery/app/trace.py", line 650, in __protected_call__
    return self.run(*args, **kwargs)
  File "/opt/dongtai/engine/core/tasks.py", line 652, in vul_recheck
    params = json.loads(param_name_value)
  File "/usr/local/lib/python3.7/json/__init__.py", line 341, in loads
    raise TypeError(f'the JSON object must be str, bytes or bytearray, '
TypeError: the JSON object must be str, bytes or bytearray, not NoneType
Received task: core.tasks.search_sink_from_method_pool[d7910aca-24af-403f-84ca-1c3a3f752c9a]  

修改项目构建方式，优化部署流程

同一url三个sink点却只会检出一个漏洞

Preflight Checklist

• I agree to follow the Code of Conduct that this project adheres to.
• I have searched the issue tracker for an issue that matches the one I want to file, without success.
• I am not looking for support or already pursued the available support channels without success.

Version

1.1.1

Installation Type

Official Docker Compose

Service Name

DongTai-agent-java

Describe the details of the bug and the steps to reproduce it

三个xss sink点:

检出漏洞只有一个：

Additional Information

No response

Logs

No response

[Bug]: 相同接口的漏洞没有收敛聚合

Preflight Checklist

• I agree to follow the Code of Conduct that this project adheres to.
• I have searched the issue tracker for an issue that matches the one I want to file, without success.
• I am not looking for support or already pursued the available support channels without success.

Version

1.0.4

Installation Type

Other (specify below)

Service Name

DongTai-Web

Describe the details of the bug and the steps to reproduce it

同一个接口和漏洞，展示多条记录。希望可以根据项目的名称，接口地址，接口方法，漏洞标题，漏洞参数，对这几个列明进行去重和聚合统计

Additional Information

No response

Logs

No response

When the vulnerability status changes, modify the update time of the project