fluent-plugins-nursery/fluent-plugin-windows-exporter

Fluentd plugin to collect Windows metrics

fluent-plugin-windows-exporter

Fluentd plugin to collect Windows metrics.

Platform	Support Version
Fluentd	Fluentd v1.x
Ruby	Ruby 2.7.x / 3.1.x / 3.2.x
OS	Windows Server 2008R2 or later

This is a Fluentd port of Prometheus' Windows exporter. This plugin emits metrics as event stream, so can be used in combination with any output plugins.

Installation

% gem install fluent-plugin-windows-exporter

Configuration

List of Options

Option	Description	Default
tag	Tag of the output events	windows.metrics
scrape_interval	The interval time between data collection	60
cpu	Enable cpu collector	true
logical_disk	Enable disk collector	true
memory	Enable memory collector	true
net	Enable network collector	true
os	Enable OS collector	true

Example Configuration

<source>
  @type windows_exporter
  tag windows.metrics  # optional
  scrape_interval 60   # optional
  cpu true             # optional
  memory true          # optional
  os true              # optional
  net true             # optional
  logical_disk true    # optional
</source>

Output format

This plugin is desinged to export the equivalent amount of information with Prometheus. Here is what a typical event looks like:

{
    "type": "gauge",
    "name": "windows_memory_system_code_resident_bytes",
    "desc": "(SystemCodeResidentBytes)",
    "labels": {},
    "value": 8192
}

This is equivalent to the following exposition of Prometheus's Windows exporter:

# HELP windows_memory_system_code_resident_bytes (SystemCodeResidentBytes)
# TYPE windows_memory_system_code_resident_bytes gauge
windows_memory_system_code_resident_bytes 8192

For further details, please refer to Prometheus' exposition format specification.

Copyright

• Copyright(c) 2021- Fujimoto Seiji, Fukuda Daijiro
• License
  • Apache License, Version 2.0

fluent-plugin-windows-exporter's Issues

Perform testing on Windows Server 2008R2

• Windows Server 2008R2 is the minimum version that we support
• We need to verify that our plugin works fine on that platform.

Add basic automated testing using GitHub Actions

This should ensure us that out plugin works on plain WIndows servers.

• Write some simple tests to check if WinFFI/HKPD returns (at least) some values.
• Activate CI on GitHub actions, and run tests on Windows Servers.

Implement "cpu" collector

Prometheus' Windows Exporter

• https://github.com/prometheus-community/windows_exporter/blob/master/collector/cpu.go
• https://github.com/prometheus-community/windows_exporter/blob/master/docs/collector.cpu.md

Implement "net" collector

Prometheus' Windows Exporter

• https://github.com/prometheus-community/windows_exporter/blob/master/collector/net.go
• https://github.com/prometheus-community/windows_exporter/blob/master/docs/collector.net.md

Improve the data collecting performance on Windows

This ticket tracks the status of data collecting performance on Windows.

TODO

1. Get the statistics of the performance of this plugin
   • How many CPU resource this plugin use?
   • How much time it takes to collect metrics?
2. Compare the performance with prometheus-community/windows-exporter
   • Compare CPU usage and time (Prometheus vs Fluentd)
3. If there is a major difference between Prometheus and Fluentd, investigate the reason.
4. Compare the performance of Global vs OLD_Global
   • Check if using OLD_Global improves the CPU usage/collection time or not
   • OLD_Global is an older API that only supports v1 metrics, so it may well be more lightweight than Global
5. Make it sure that fluent-plugin-windows-exporter does not block Fluentd startups
   • Defer HKEY_PERFORMANCE_* loading, so that other input plugins won't get delayed by slow access to WIndows Registry.

Raise Fluent::ConfigError along this repository's readme

Steps to reproduce

1. Install fluent-plugin-windows-exporter gem:

   % gem install fluent-plugin-windows-exporter

2. Edit Fluentd's configuration file along Example Configuration:

   % type test.conf
   <source>
     @type windows_exporter
     tag windows.metrics  # optional
     scrape_interval 60   # optional
     cpu true             # optional
     memory true          # optional
     os trues             # optional
     net true             # optional
     logical_disk true    # optional
   </source>
   <match windows.metrics>
     @type stdout
   </match>

3. Launch Fluentd:

   % fluentd -c test.conf

Expected result

Successfully launched Fluentd

Actual result

Error launching Fluentd with the below command.
(I omitted the leading date and time)

% fluentd -c test.conf
[info]: init supervisor logger path=nil rotate_age=nil rotate_size=nil
[info]: parsing config file is succeeded path="test.conf"
[info]: gem 'fluentd' version '1.15.3'
[info]: gem 'fluent-mixin-rewrite-tag-name' version '0.1.0'
[info]: gem 'fluent-mixin-type-converter' version '0.1.0'
[info]: gem 'fluent-plugin-parser-winevt_xml' version '0.2.5'
[info]: gem 'fluent-plugin-watch-process' version '0.2.0'
[info]: gem 'fluent-plugin-windows-eventlog' version '0.8.3'
[info]: gem 'fluent-plugin-windows-exporter' version '1.0.0'
[info]: Oj isn't installed, fallback to Yajl as json parser
[error]: config error in:
<source>
  @type windows_exporter
  tag "windows.metrics"
  scrape_interval 60
  cpu true
  memory true
  os trues
  net true
  logical_disk true
</source>

[error]: config error file="test.conf" error_class=Fluent::ConfigError error="'os' parameter is required but nil is specified"

Implement HKEY_PERFORMANCE_DATA

• Fetch data from HKEY_PERFORMANCE_DATA and return a Ruby object.
• Official Spec (from msdn)
  • https://docs.microsoft.com/en-us/windows/win32/perfctrs/using-the-registry-functions-to-consume-counter-data
  • https://docs.microsoft.com/en-us/windows/win32/perfctrs/performance-data-format
  • https://docs.microsoft.com/en-us/windows/win32/api/winreg/nf-winreg-regqueryvalueexw
• How to read registry from Ruby
  • https://github.com/ruby/ruby/blob/master/ext/win32/lib/win32/registry.rb
• Related implementation
  • https://github.com/leoluk/perflib_exporter/tree/master/perflib (Golang)

Add support for Ruby 3.1

Introduction

We are plannning to bundle fluent-plugin-windows-exporter in TD Agent,
starting from the next version (the release date will be March 2022).

Since TD Agent is now moving to Ruby 3.x, we also need to confirm that
this plugin works on Ruby 3.1 as well.

Task

• Add Ruby 3.1 to GitHub Actions CI.
  • Use matrix notation to test both on 2.7 and 3.1.
  • Here is an example
• Fix anything that are not compatible with Ruby 3.x.

Strategic priority

See fluent/fluentd#3585 for migration status.

Prepare submittion to RubyGems.org

We'll distribute fluent-plugin-windows-exporter as a Ruby gem.

Note: The installation process should be very streaght foraward, because
this plugin is written in pure Ruby and does not depend on C libraries. So
gem install fluent-plugin-windows-exporter must just work.

Implement "os" collector

Prometheus' Windows Exporter

• https://github.com/prometheus-community/windows_exporter/blob/master/collector/os.go
• https://github.com/prometheus-community/windows_exporter/blob/master/docs/collector.os.md

Implement ffi interface for GlobalMemoryStatusEx() (and others)

It turned out that Prometheus os collector does not use WMI (despite that the documentatoin states otherwise).

• It uses the following API to retrieve the system infroamtion
  • GlobalMemoryStatusEx()
  • NetWkstaGetInfo()
  • HKLM\SYSTEM\CurrentControlSet\Control\Session Manager\Memory Management
  • HKLM\SOFTWARE\Microsoft\Windows NT\CurrentVersion
• See the following implementation for details
  • https://github.com/prometheus-community/windows_exporter/blob/master/collector/os.go#L160

We need to implement the same logic in Ruby to replicate the output.

Implement "logical_disk" collector

Prometheus' Windows Exporter

• https://github.com/prometheus-community/windows_exporter/blob/master/collector/logical_disk.go
• https://github.com/prometheus-community/windows_exporter/blob/master/docs/collector.logical_disk.md

Check the correctness of output metrics

TODO

• Run the following two on the same machine:
  • Prometheus's windows-exporter
  • fluent-plugin-windows-exporter.
• Compare the metrics returned by two, and see if there are any inconsistencies.

Implement "memory" collector

Prometheus' Windows Exporter

• https://github.com/prometheus-community/windows_exporter/blob/master/collector/memory.go
• https://github.com/prometheus-community/windows_exporter/blob/master/docs/collector.memory.md

Investigate missing counter names on RubyInstaller

Branched from #1 (comment).

• When we query HKEY_PERFORMANCE_DATA for counter names, some entries are missing from the data.
• It only occurs when we we run it on Ruby.
• Other languages (C and Python) do not have the same problem.

Add 32bit Windows support

Branched from: #1 (comment)

Document the plugin usage in README.md

We'll need this in order to prepare the initial release.

• How to install this plugin
• Available configuration options, and how to use them
• ...