neeky/dbm-agent

dbma -- 数据库管理中心客户端程序

目录

---

• 目录
• dbm-愿景
• 安装-dbm-agent
• MySQL-功能列表
• Redis-功能列表
• 关闭-dbm-agent
• 规范
• 源码安装
• 官方微信公众平台
• 支持的MySQL版本

---

dbm-愿景

dbm 是一个软件套件，包含有 dbm-center 和 dbm-agent 两大组成部分；其中 dbm-center 可以看成一个 web 站点，DBA 通过它可以查看监控、告警、部署各种 MySQL|Redis 环境。 dbm-agent 是 dbm-center 的助手，负责环境的部署、监控的采集与上报、数据库的备份与监控。总的来讲接活的 dbm-center 真正干活的是 dbm-agent。另外 dbm-agent 也集成了若干命令行工具，这使得它也可以单独使用。

愿景

我们希望 DBA 可以从日常的工作中解放出来，做到只要在 web 界面上点一下“同意”系统就能自动、高效、优质的完成所有的操作；同样对于那些我们认为不合理的需求只要点一个“驳回”系统除了向需求方回一个被驳回的响应之外什么都不会干，安静的像一个美男子；更有意思的事，对于所有能用自动化解决的问题，dbm 绝对不会大半夜打 dba 的电话，出现故障自动解决，完成之后记录一下日志就行。

工作原理

dbm-agent 是安装在你机器上的一个 python 程序，它可以工作在两个模式

一：守护进程模式 这个模下我们可以先 dbm-agent 发送 http 请求来要求它完成相应的动作，另外后台线程也会定时上报一些监控信息到管理中心。

二：命令行模式 dbm-agent 除了以 http 方式来暴露功能外，还提供了相应的命令工具，真正做来最小化依赖拿来就用。

---

安装-dbm-agent

1. 安装 dbm-agent

pip3 install dbm-agent

如果你是在国内，推荐使用腾讯云的源对 pip 安装进行加速，配置也非常简单一行命令搞定

pip3 config set global.index-url  https://mirrors.cloud.tencent.com/pypi/simple

---

2. 初始化

sudo su

dbma-cli-init --net-card=eth0 --dbm-center-url-prefix=https://127.0.0.1

# 检查有没有 init 成功
tree /usr/local/dbm-agent/
/usr/local/dbm-agent/
├── etc
│   ├── dbm-agent.json
│   └── templates
│       ├── mysql-8.0.30.cnf.jinja
│       ├── mysql-8.0.31.cnf.jinja
│       ├── mysql-8.0-init-only.jinja
│       ├── mysqld.service.jinja
├── logs
└── pkgs
    └── mysql-8.0.31-linux-glibc2.12-x86_64.tar.xz

3. 启动 不执行启动的话就用不了 http 接口，只能使用命令行工具。

dbm-agent start

4. 检查 dbm-agent 进程在不在

ps -ef | grep dbm-agent
dbma        3520       1  0 16:02 ?        00:00:00 /usr/local/python/bin/python3.11 /usr/local/python/bin/dbm-agent start
root        3556    3273  0 16:02 pts/4    00:00:00 grep --color=auto dbm-agent

5. 检查 http 服务是否正常

curl http://127.0.0.1:8086/apis/dbm-agent 2>/dev/null | jq
{
  "name": "dbm-agent",
  "version": "8.31.9"
}

---

MySQL-功能列表

MySQL 目前支持的功能列表如下

功能名	详细文档
MySQL单机&主从安装配置	安装配置-MySQL
MySQL备份	备份-MySQL
MySQL卸载	卸载-MySQL
MySQL清理	清理-MySQL
MySQLMGR	开发中
高可用	开发中

---

Redis-功能列表

Redis 目前支持的功能列表如下

功能名	详细文档
Redis-master-安装	Redis-master-安装
Redis-replica-安装	Redis-replica-安装

---

关闭-dbm-agent

关闭 dbm-agent 守护进程

dbm-agent stop                                                              
Successful exit

---

规范

dbm-agent 在实现上遵守的规范

规范ID	规范内容
dbm-agent 工作目录位置	/usr/local/dbm-agent
mysql 配置文件保存的位置	/etc/my-{port}.cnf
systemd 配置文件保存的位置	/usr/lib/systemd/system/mysqld-{port}.service
数据目录	/database/mysql/data/{port}
备份目录	/backup/mysql/{port}
binlog目录	/binlog/mysql/binlog/{port}
默认密码	dbma@0352
MySQL 安装目录	/usr/local/
MySQL-shell 安装目录	/usr/local/

---

单元测试

为了尽可能的减少 BUG ，我们尽全力把单元测试的覆盖率推向 100%，目前来看只达到了 87%

# 单元测试用到的依赖
# pip3 install pytest pytest-mock pytest-cov

# 运行单元测试
pytest --cov

... ... ... ... ...
---------- coverage: platform linux, python 3.11.3-final-0 -----------
Name                                             Stmts   Miss  Cover
--------------------------------------------------------------------
dbma/__init__.py                                     0      0   100%
dbma/bil/__init__.py                                 1      0   100%
dbma/bil/cmdexecutor.py                              6      0   100%
dbma/bil/daemon.py                                  66     36    45%
dbma/bil/fs.py                                      56     25    55%
dbma/bil/fun.py                                      4      0   100%
dbma/bil/net.py                                     12      1    92%
dbma/bil/osuser.py                                 139      0   100%
dbma/bil/sudos.py                                   12      0   100%
dbma/components/__init__.py                          0      0   100%
dbma/components/mysql/__init__.py                    0      0   100%
dbma/components/mysql/asserts.py                    24      1    96%
dbma/components/mysql/backends/__init__.py           0      0   100%
dbma/components/mysql/backends/clears.py           119     31    74%
dbma/components/mysql/commons.py                   137     73    47%
dbma/components/mysql/config.py                    387    116    70%
dbma/components/mysql/exceptions.py                 16      0   100%
dbma/components/mysql/install.py                   194     19    90%
dbma/components/orchestrator/__init__.py             0      0   100%
dbma/components/orchestrator/exceptions.py           6      0   100%
dbma/components/orchestrator/install.py             40     22    45%
dbma/core/__init__.py                                0      0   100%
dbma/core/configs.py                                78     14    82%
dbma/core/exception.py                              13      0   100%
dbma/core/messages.py                               13      0   100%
dbma/core/threads/__init__.py                        0      0   100%
dbma/core/threads/backends.py                       31     16    48%
dbma/version.py                                      7      0   100%
--------------------------------------------------------------------
TOTAL                                             2180    361    83%

---

源码安装

# 生成 tar.gz 格式的安装包
cd dbm-agent
python3 setup.py sdist

# 执行安装
cd dist
pip3 install ./dbm-agent-8.33.10.tar.gz

---

官方微信公众平台

---

支持的MySQL版本

目前 MySQL 支持如下这些版本，子版本的话推荐使用最新版本；因为每一个新版本出来的时候都会用它来测试下，老版本就不会再测试了。

1. 8.0.x
2. 8.3.x

---

dbm-agent's Issues

依赖环境不足导致安装失败

背景：
dbm-agent 0.4.6
CentOS Linux release 7.4.1708 (Core)
报错：
2019-11-22 10:10:37,518 - dbm-agent.dbma.mysqldeploy.SingleInstanceInstaller.run - im - ERROR - 1438 - exception occur during install mysql single instance Unknown archive format '/usr/local/dbm-agent/pkg/mysql-8.0.18-linux-glibc2.12-x86_64.tar.xz'
原因：
编译安装python3时，依赖环境没有安装足够，导致python3编译不足，致使最后在单实例安装报错
解决方案：
安装好依赖环境后，重新编译python3，最后再安装工具dbm-agent

MGR 如何扩容咦？

微信交流群加不上，想了解一下后期如何扩容MGR

启动mysql8时报错

2021-07-16T11:14:24.214712+08:00 0 [Warning] [MY-011069] [Server] The syntax '--master-info-repository' is deprecated and will be removed in a future release.
2021-07-16T11:14:24.214731+08:00 0 [Warning] [MY-011069] [Server] The syntax '--relay-log-info-repository' is deprecated and will be removed in a future release.
2021-07-16T11:14:24.214908+08:00 0 [System] [MY-010116] [Server] /usr/local/mysql-8.0.25-linux-glibc2.12-x86_64/bin/mysqld (mysqld 8.0.25) starting as process 157826
2021-07-16T11:14:24.219975+08:00 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started.
2021-07-16T11:14:24.580833+08:00 1 [Warning] [MY-012128] [InnoDB] posix_fallocate(): Failed to preallocate data for file ./#innodb_temp/temp_1.ibt, desired size 81920 Operating system error number 22 - Invalid argument. Check that the disk is not full or a disk quota exceeded. Make sure the file system supports this function. Refer to your operating system documentation for operating system error code information.
2021-07-16T11:14:24.581486+08:00 1 [Warning] [MY-012128] [InnoDB] posix_fallocate(): Failed to preallocate data for file ./#innodb_temp/temp_2.ibt, desired size 81920 Operating system error number 22 - Invalid argument. Check that the disk is not full or a disk quota exceeded. Make sure the file system supports this function. Refer to your operating system documentation for operating system error code information.
2021-07-16T11:14:24.582419+08:00 1 [Warning] [MY-012128] [InnoDB] posix_fallocate(): Failed to preallocate data for file ./#innodb_temp/temp_3.ibt, desired size 81920 Operating system error number 22 - Invalid argument. Check that the disk is not full or a disk quota exceeded. Make sure the file system supports this function. Refer to your operating system documentation for operating system error code information.
2021-07-16T11:14:24.583773+08:00 1 [Warning] [MY-012128] [InnoDB] posix_fallocate(): Failed to preallocate data for file ./#innodb_temp/temp_4.ibt, desired size 81920 Operating system error number 22 - Invalid argument. Check that the disk is not full or a disk quota exceeded. Make sure the file system supports this function. Refer to your operating system documentation for operating system error code information.
2021-07-16T11:14:24.584608+08:00 1 [Warning] [MY-012128] [InnoDB] posix_fallocate(): Failed to preallocate data for file ./#innodb_temp/temp_5.ibt, desired size 81920 Operating system error number 22 - Invalid argument. Check that the disk is not full or a disk quota exceeded. Make sure the file system supports this function. Refer to your operating system documentation for operating system error code information.
2021-07-16T11:14:24.585436+08:00 1 [Warning] [MY-012128] [InnoDB] posix_fallocate(): Failed to preallocate data for file ./#innodb_temp/temp_6.ibt, desired size 81920 Operating system error number 22 - Invalid argument. Check that the disk is not full or a disk quota exceeded. Make sure the file system supports this function. Refer to your operating system documentation for operating system error code information.
2021-07-16T11:14:24.586356+08:00 1 [Warning] [MY-012128] [InnoDB] posix_fallocate(): Failed to preallocate data for file ./#innodb_temp/temp_7.ibt, desired size 81920 Operating system error number 22 - Invalid argument. Check that the disk is not full or a disk quota exceeded. Make sure the file system supports this function. Refer to your operating system documentation for operating system error code information.
2021-07-16T11:14:24.587718+08:00 1 [Warning] [MY-012128] [InnoDB] posix_fallocate(): Failed to preallocate data for file ./#innodb_temp/temp_8.ibt, desired size 81920 Operating system error number 22 - Invalid argument. Check that the disk is not full or a disk quota exceeded. Make sure the file system supports this function. Refer to your operating system documentation for operating system error code information.
2021-07-16T11:14:24.588662+08:00 1 [Warning] [MY-012128] [InnoDB] posix_fallocate(): Failed to preallocate data for file ./#innodb_temp/temp_9.ibt, desired size 81920 Operating system error number 22 - Invalid argument. Check that the disk is not full or a disk quota exceeded. Make sure the file system supports this function. Refer to your operating system documentation for operating system error code information.
2021-07-16T11:14:24.589650+08:00 1 [Warning] [MY-012128] [InnoDB] posix_fallocate(): Failed to preallocate data for file ./#innodb_temp/temp_10.ibt, desired size 81920 Operating system error number 22 - Invalid argument. Check that the disk is not full or a disk quota exceeded. Make sure the file system supports this function. Refer to your operating system documentation for operating system error code information.
2021-07-16T11:14:24.596067+08:00 1 [System] [MY-013577] [InnoDB] InnoDB initialization has ended.
2021-07-16T11:14:24.787415+08:00 0 [System] [MY-011323] [Server] X Plugin ready for connections. Bind-address: '::' port: 33180, socket: /tmp/mysqlx-33180.sock
2021-07-16T11:14:24.933445+08:00 0 [Warning] [MY-010068] [Server] CA certificate ca.pem is self signed.
2021-07-16T11:14:24.933655+08:00 0 [System] [MY-013602] [Server] Channel mysql_main configured to support TLS. Encrypted connections are now supported for this channel.
2021-07-16T11:14:24.934017+08:00 0 [Warning] [MY-013595] [Server] Failed to initialize TLS for channel: mysql_admin. See below for the description of exact issue.
2021-07-16T11:14:24.934154+08:00 0 [Warning] [MY-010069] [Server] Failed to set up SSL because of the following SSL library error: SSL context is not usable without certificate and private key
2021-07-16T11:14:24.934287+08:00 0 [System] [MY-013603] [Server] No TLS configuration was given for channel mysql_admin; re-using TLS configuration of channel mysql_main.
2021-07-16T11:14:24.936635+08:00 0 [Warning] [MY-011810] [Server] Insecure configuration for --pid-file: Location '/tmp' in the path is accessible to all OS users. Consider choosing a different directory.
2021-07-16T11:14:24.975311+08:00 0 [System] [MY-010931] [Server] /usr/local/mysql-8.0.25-linux-glibc2.12-x86_64/bin/mysqld: ready for connections. Version: '8.0.25' socket: '/tmp/mysql-3318.sock' port: 3318 MySQL Community Server - GPL.
2021-07-16T11:14:24.975331+08:00 0 [System] [MY-013292] [Server] Admin interface ready for connections, address: '127.0.0.1' port: 33182

Could not find a version that satisfies the requirement redis>=4.5.4 (from dbm-agent)

执行pip3 install dbm-agent报：
Could not find a version that satisfies the requirement redis>=4.5.4 (from dbm-agent)

testunit syntax error

`[root@slave1 dbm-agent-master]# python3 --version
Python 3.8.0
[root@slave1 dbm-agent-master]# bash autotest.sh
useradd: warning: the home directory already exists.
Not copying any file from skel directory into it.
Creating mailbox file: File exists
.............E.........useradd: user 'mysql' already exists
..............s

ERROR: test_dbmaconfig (unittest.loader._FailedTest)

ImportError: Failed to import test module: test_dbmaconfig
Traceback (most recent call last):
File "/usr/lib64/python3.8/unittest/loader.py", line 436, in _find_test_path
module = self._get_module_from_name(name)
File "/usr/lib64/python3.8/unittest/loader.py", line 377, in _get_module_from_name
import(name)
File "/root/dbm-agent-master/tests/test_dbmaconfig.py", line 10
class TestConfigMixinTestCase(unittest.TestCase)
^
SyntaxError: invalid syntax

---

Ran 38 tests in 7.626s

FAILED (errors=1, skipped=1)
[root@slave1 dbm-agent-master]#
`

配置 innodb-cluster 报错

mysqlsh --uri [email protected]:3306 --password=dbma@0352 < /usr/local/dbm-agent/etc/templates/create-innodb-cluster.js
WARNING: Using a password on the command line interface can be insecure.
A new InnoDB cluster will be created based on the existing replication group on instance '127.0.0.1:3306'.

Creating InnoDB cluster 'production' on 'mysql-one:3306'...

Adding Seed Instance...

Adding Instance 'mysql-three:3306'...
Adding Instance 'mysql-one:3306'...
Adding Instance 'mysql-two:3306'...
Resetting distributed recovery credentials across the cluster...
Dba.createCluster: mysql-one:3306: Access denied; you need SYSTEM_VARIABLES_ADMIN and PERSIST_RO_VARIABLES_ADMIN privileges for this operation (RuntimeError)
at STDIN:2:18
in var cluster = dba.createCluster('production', { adoptFromGR: true });

dbma-cli-single-instance --port=3306 install命令报错/lib/x86_64-linux-gnu/librt.so.1' for IFUNC symbol `clock_gettime'

服务器信息

root@VM-0-7-ubuntu:~# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 18.04.3 LTS
Release: 18.04
Codename: bionic

---

配置信息

root@VM-0-7-ubuntu:~# tree /usr/local/dbm-agent/
/usr/local/dbm-agent/
├── etc
│   ├── dbma.cnf
│   ├── init-users.sql
│   └── templates
│   ├── create-innodb-cluster.js
│   ├── init-users.sql.jinja
│   ├── mysql-8.0-init-only.jinja
│   ├── mysql-8.0.17.cnf.jinja
│   ├── mysql-8.0.18.cnf.jinja
│   └── mysqld.service.jinja
├── logs
└── pkg
├── download_mysql-8.0.sh　//这是我自己写的下载shell
├── mysql-8.0.18-linux-glibc2.12-x86_64.tar.xz
└── mysql-shell-8.0.18-linux-glibc2.12-x86-64bit.tar.gz

4 directories, 11 files

---

报错信息

root@VM-0-7-ubuntu:~# dbma-cli-single-instance --port=3306 install
2019-11-20 13:38:51,723 - dbm-agent.dbma.mysqldeploy.SingleInstanceInstaller.install - im - INFO - 1115 - execute checkings for install mysql
/usr/bin/python3: Relink /lib/x86_64-linux-gnu/libnss_systemd.so.2' with /lib/x86_64-linux-gnu/librt.so.1' for IFUNC symbol `clock_gettime'
Segmentation fault (core dumped)

`

请问 dbm-monitor-gateway 程序在哪里？

请问大佬 dbm-monitor-gateway 程序在哪里可以找到？谢谢。