Golang是一门简单高效的编程语言,我在编写学习的过程中也被其特性所吸引,日志监控系统是生产环境中必备的功能系统,它的作用可以说仅次于核心系统 而Golang的协程实现可以很好的帮我们完成这一核心功能,通过模拟读取nginx输出的日志文件,使用log_proccess.go进行实时读取解析写入到influxdb存储, 再由grafana进行实时展现。mock_data.go是我用于模拟日志输出的一个应用程序。
Golang is a simple and efficient programming language, I am in the process of writing learning is also attracted by its characteristics, log monitoring system is a necessary function of the production environment system, its role can be said after the core system And the Golang implementation can help us to complete this core function, through the simulation of reading Nginx output log files, using Log_proccess.go for real-time read parsing write to influxdb storage, The Grafana is then displayed in real time. Mock_data.go is an application I use to simulate log output.
Golang的并发实现可以通过goroutine执行,而多个goroutine间的数据同步与通信则是channel,且多个channel可以选择数据的读取与写入。 这里需要认真理解下并发与并行。并发:指同一时刻,系统通过调度,来回切换交替的运行多个任务,“看起来”是同时进行;并行:指同一时刻, 两个任务“真正的”同时进行;
Concurrent implementations of Golang can be performed through Goroutine, while data synchronization and communication between multiple goroutine are channel, and multiple channel can choose to read and write data. This requires a careful understanding of concurrency and parallelism. Concurrency: Refers to the same moment, the system through the scheduling, switching back and forth to run multiple tasks, "looks" is the same time; Two tasks "real" at the same time;
1、读取模块的实现
————打开文件
————从文件末尾开始逐行读取
————写入Read Channel
2、解析模块的实现
————从Read Channel中读取每行日志数据
————正则提取所需的监控数据(path、status、method等)
————写入Write Channel
3、写入模块的实现
————初始化influxdb client
————从Write Channel中读取监控数据
————构造数据并写入influxdb
4、绘制监控图
————用grafana
5、监控模块的实现
————总处理日志行数
————系统吞出量
————read channel长度
————write channel长度
————运行总时间
————错误数