tinkoff / libvirt-exporter Goto Github PK
View Code? Open in Web Editor NEWPrometheus metrics exporter for libvirt.
License: Apache License 2.0
Prometheus metrics exporter for libvirt.
License: Apache License 2.0
Describe the bug
This appears to be a similar issue to the one resolved in #4. When a libvirt instance has multiple SRIOV attached interfaces, the /metrics resource returns a 500 error with output similar to the following:
An error has occurred while serving metrics:
8 error(s) occurred:
* collected metric "libvirtd_domain_net_rx_bytes" { label:<name:"interface" value:"" > label:<name:"uuid" value:"40e776cf-2187-4b73-9ab5-9168a1c567da" > counter:<value:0 > } was collected before with the same name and label values
* collected metric "libvirtd_domain_net_rx_packets" { label:<name:"interface" value:"" > label:<name:"uuid" value:"40e776cf-2187-4b73-9ab5-9168a1c567da" > counter:<value:0 > } was collected before with the same name and label values
* collected metric "libvirtd_domain_net_rx_errors" { label:<name:"interface" value:"" > label:<name:"uuid" value:"40e776cf-2187-4b73-9ab5-9168a1c567da" > counter:<value:0 > } was collected before with the same name and label values
* collected metric "libvirtd_domain_net_rx_drop" { label:<name:"interface" value:"" > label:<name:"uuid" value:"40e776cf-2187-4b73-9ab5-9168a1c567da" > counter:<value:0 > } was collected before with the same name and label values
* collected metric "libvirtd_domain_net_tx_bytes" { label:<name:"interface" value:"" > label:<name:"uuid" value:"40e776cf-2187-4b73-9ab5-9168a1c567da" > counter:<value:0 > } was collected before with the same name and label values
* collected metric "libvirtd_domain_net_tx_packets" { label:<name:"interface" value:"" > label:<name:"uuid" value:"40e776cf-2187-4b73-9ab5-9168a1c567da" > counter:<value:0 > } was collected before with the same name and label values
* collected metric "libvirtd_domain_net_tx_errors" { label:<name:"interface" value:"" > label:<name:"uuid" value:"40e776cf-2187-4b73-9ab5-9168a1c567da" > counter:<value:0 > } was collected before with the same name and label values
* collected metric "libvirtd_domain_net_tx_drop" { label:<name:"interface" value:"" > label:<name:"uuid" value:"40e776cf-2187-4b73-9ab5-9168a1c567da" > gauge:<value:0 > } was collected before with the same name and label values
This is seen in OpenStack deployments, and it could be that they are failing to insert an interface name where they should be, but a more graceful failure mode in the exporter would be useful if so.
Version of libvirt-exporter
Is libvirt-exporter used in docker?
No
Libvirt version
6.0.0
To Reproduce
Expected behavior
Metrics resource continues to produce output without error
Additional context
I've included a snapshot of the libvirt XML and virsh output below:
<interface type='bridge'>
<mac address='fa:16:3e:2a:68:25'/>
<source bridge='brq862b83ec-b7'/>
<target dev='tapdd8b8ce2-c2'/>
<model type='virtio'/>
<mtu size='1500'/>
<alias name='net0'/>
<address type='pci' domain='0x0000' bus='0x02' slot='0x00' function='0x0'/>
</interface>
<interface type='bridge'>
<mac address='fa:16:3e:f1:a1:ac'/>
<source bridge='brq681aa992-37'/>
<target dev='tapa737598a-b4'/>
<model type='virtio'/>
<mtu size='1500'/>
<alias name='net1'/>
<address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
</interface>
<interface type='hostdev' managed='yes'>
<mac address='fa:16:3e:5d:cd:ce'/>
<driver name='vfio'/>
<source>
<address type='pci' domain='0x0000' bus='0x65' slot='0x02' function='0x2'/>
</source>
<vlan>
<tag id='2021'/>
</vlan>
<alias name='hostdev1'/>
<address type='pci' domain='0x0000' bus='0x08' slot='0x00' function='0x0'/>
</interface>
<interface type='hostdev' managed='yes'>
<mac address='fa:16:3e:ee:35:a1'/>
<driver name='vfio'/>
<source>
<address type='pci' domain='0x0000' bus='0x65' slot='0x04' function='0x6'/>
</source>
<vlan>
<tag id='2022'/>
</vlan>
<alias name='hostdev2'/>
<address type='pci' domain='0x0000' bus='0x09' slot='0x00' function='0x0'/>
</interface>
:~# virsh domiflist 4
Interface Type Source Model MAC
-------------------------------------------------------------------------
tapdd8b8ce2-c2 bridge brq862b83ec-b7 virtio fa:16:3e:2a:68:25
tapa737598a-b4 bridge brq681aa992-37 virtio fa:16:3e:f1:a1:ac
- hostdev - - fa:16:3e:5d:cd:ce
- hostdev - - fa:16:3e:ee:35:a1
Is your feature request related to a problem? Please describe.
A very powerfull metrics would be the stealttime of a vm, for balancing the while VMs between Servers.
Describe the solution you'd like
High Stealtime leads to a low performance of the vm (20% ST means only 80% of CPU is available, and to maximize the hardwareusage this is a powerfull metric)
Describe alternatives you've considered
Load of the libvirt Compute Node is not powerfull enough.
https://grafana.com/grafana/dashboards/13633
仪表盘有些指标和新版本的exporter对不上
At least for:
libvirt_domain_memory_stats_unused_bytes
libvirt_domain_memory_stats_used_percent
I got wrong values. For 1st it is 3,66GB and 2nd is 7,29%, but should be total-used=1,6Gi and 2,2/3,8*100=58%.
System output:
total used free shared buff/cache available
Mem: 3.8Gi 2.2Gi 197Mi 3.0Mi 1.4Gi 1.3Gi
Swap: 0B 0B 0B
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=20.04
DISTRIB_CODENAME=focal
DISTRIB_DESCRIPTION="Ubuntu 20.04.4 LTS"
At the moment it is not possible to determine the version of the binary. It would be helpful to have a -version argument to be able to read the version.
Is it possible to support disk usage on virtual instance? Like it is visible size of disk:
libvirt_domain_block_stats_capacity_bytes{domain="instance-00000222",target_device="sda"}
and it will be useful have also used_bytes.
Thanks a lot!
build_static.sh fails with:
...
make[2]: Leaving directory '/tmp/libvirt-3.8.0/examples'
make[1]: Leaving directory '/tmp/libvirt-3.8.0/examples'
+ sed -i 's/^Libs:.*/& -lnl -ltirpc -lxml2/' /usr/local/lib/pkgconfig/libvirt.pc
+ cd /libvirt-exporter
+ export 'GOPATH=/gopath'
+ go get -d ./...
+ go build --ldflags '-extldflags "-static"' -o libvirt-exporter
# github.com/prometheus/client_golang/prometheus
/gopath/src/github.com/prometheus/client_golang/prometheus/go_collector.go:388:15: undefined: debug.ReadBuildInfo```
Compiled via: build_static.sh
Branch: Master (fresh pull)
OS: Alpine 3.8 (within docker)
HOST: MacOS BigSur
Testtrun (without strip) on ArchLinux & Ubuntu 20.04 (segfault)
make[1]: Entering directory '/tmp/libvirt-3.8.0/po'
if test "libvirt" = "gettext-tools"; then \
../build-aux/install-sh -c -d /usr/local/share/gettext/po; \
for file in Makefile.in.in remove-potcdate.sin quot.sed boldquot.sed [email protected] [email protected] insert-header.sin Rules-quot Makevars.template; do \
/usr/bin/install -c -m 644 ./$file \
/usr/local/share/gettext/po/$file; \
done; \
for file in Makevars; do \
rm -f /usr/local/share/gettext/po/$file; \
done; \
else \
: ; \
fi
make[1]: Leaving directory '/tmp/libvirt-3.8.0/po'
Making install in examples
make[1]: Entering directory '/tmp/libvirt-3.8.0/examples'
make[2]: Entering directory '/tmp/libvirt-3.8.0/examples'
make[2]: Nothing to be done for 'install-exec-am'.
../build-aux/install-sh -c -d ""/usr/local/etc/libvirt/nwfilter""
for f in ./xml/nwfilter/no-other-rarp-traffic.xml ./xml/nwfilter/qemu-announce-self-rarp.xml ./xml/nwfilter/allow-dhcp-server.xml ./xml/nwfilter/no-mac-broadcast.xml ./xml/nwfilter/allow-incoming-ipv4.xml ./xml/nwfilter/no-other-l2-traffic.xml ./xml/nwfilter/allow-ipv4.xml ./xml/nwfilter/no-arp-mac-spoofing.xml ./xml/nwfilter/allow-arp.xml ./xml/nwfilter/no-arp-ip-spoofing.xml ./xml/nwfilter/clean-traffic.xml ./xml/nwfilter/no-arp-spoofing.xml ./xml/nwfilter/allow-dhcp.xml ./xml/nwfilter/no-mac-spoofing.xml ./xml/nwfilter/no-ip-spoofing.xml ./xml/nwfilter/no-ip-multicast.xml ./xml/nwfilter/qemu-announce-self.xml; do \
/usr/bin/install -c -m 644 $f ""/usr/local/etc/libvirt/nwfilter""; \
done
make[2]: Leaving directory '/tmp/libvirt-3.8.0/examples'
make[1]: Leaving directory '/tmp/libvirt-3.8.0/examples'
+ sed -i 's/^Libs:.*/& -lnl -ltirpc -lxml2/' /usr/local/lib/pkgconfig/libvirt.pc
+ cd /libvirt-exporter
+ export 'GOPATH=/gopath'
+ mkdir -p /gopath/
+ ln -s /libvirt-exporter/vendor /gopath/src
+ mkdir -p /gopath/src/github.com/AlexZzz
+ ln -s /libvirt-exporter /gopath/src/github.com/AlexZzz/libvirt-exporter
+ go build --ldflags '-extldflags "-static"' -o libvirt-exporter
go build _/libvirt-exporter: build output "libvirt-exporter" already exists and is not an object file
Hi, I'm new in prometheus and exporter and looking for a openstack hypervisor monitoring.
As I look through source, I can find the exporter offers vm metrics. Then, is it possible to get metrics of hypervisor?
I think the node stats of libvirt is hypervisor stats. (I'm not sure for this because I'm a poor junior dev 😢 )
If i'm right, then the exporter can offer a hypervisor metrics with NodeCPUStats
in libvirt-exporter/vendor/libvirt.org/go/libvirt/connect.go for example?
Feb 25 11:07:07 somehost.example.com libvirt-exporter[780603]: 2021/02/25 11:07:07 Failed to scrape metrics: virError(Code=55, Domain=10, Message='Requested operation is not valid: cannot retrieve vcpu information for inactive domain')
Need to expose disk-related limits.
Something like this:
libvirt_domain_block_limit_read_bytes
libvirt_domain_block_limit_write_bytes
libvirt_domain_block_limit_read_requests
libvirt_domain_block_limit_write_requests
libvirt_domain_block_limit_total_bytes
libvirt_domain_block_limit_total_requests
and burst stats:
libvirt_domain_block_limit_burst_read_bytes
libvirt_domain_block_limit_burst_read_bytes_seconds
libvirt_domain_block_limit_burst_write_bytes
libvirt_domain_block_limit_burst_write_bytes_seconds
libvirt_domain_block_limit_burst_read_requests
libvirt_domain_block_limit_burst_read_requests_seconds
libvirt_domain_block_limit_burst_write_requests
libvirt_domain_block_limit_burst_write_requests_seconds
Dear Aleksei,
I hope you are doing well.
If possible, please add ip address of instance as a label metric.
Best regards,
Alireza
There're no tests right now, except spelling and linter.
Exporter must have a test CI workflow to test most of functionality with real libvirt instances. Not sure about implementation details, but as a result tests must proof that all available metrics are collected and they are collected properly.
It could be useful to check libvirt-exporter data with virsh data when dealing with some metrics.
Might be some issues running libvirt instance during CI.
Dear guys
In the Dockerfile, It doesn't have a specific version; I think it's not good practice for Dockerfile.
Also, in the Dockerfile, it uses the go image, and again it adds (upgrades) the go; It's better to change the docker image instead of it.
I created this PR based on my points.
This exporter is the standard libvirtd exporter for Kolla/Kolla-Ansible.
https://github.com/openstack/kolla/blob/master/docker/prometheus/prometheus-libvirt-exporter/Dockerfile.j2
I deploy kolla-ansible with cloudkitty as rating module and with prometheus as metric source. So cloudkitty has to be configured to use the promQL to retrieve the data from Prometheus. Unfortunately there is no label 'project_id' in the libvirt_domain_info_meta metric. project_id is the standard label name for cloudkitty to filter the domains by. This is configurable, but then other exporters like openstack exporter can't retrieve data.
So for better compatibility i hope you can insert a label with name project_id in the libvirt_domain_info_meta metric.
I'm not a programmer, but think it is not much work to do.
Hi all,
We have recently experienced the following error:
Jul 20 06:13:31 server1 libvirtd[47045]: 2022-07-20 06:13:31.181+0000: 47045: error : qemuProcessReportLogError:1914 : internal error: qemu unexpectedly closed the monitor: 2022-07-20T06:13:31.090440Z qemu-system-x86_64: -device ide-hd,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0,bootindex=1: Failed to get "write" lock
Jul 20 06:13:31 server1 libvirtd[47045]: 2022-07-20 06:13:31.181+0000: 47049: error : qemuProcessReportLogError:1914 : internal error: process exited while connecting to monitor: 2022-07-20T06:13:31.090440Z qemu-system-x86_64: -device ide-hd,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0,bootindex=1: Failed to get "write" lock
Jul 20 07:01:27 server1 libvirt_exporter[94868]: 2022/07/20 07:01:27 Failed to scrape metrics: virError(Code=38, Domain=10, Message='Unable to read from monitor: Connection reset by peer')
Please let us know if any more information is needed.
Describe the bug
2021/04/08 05:53:38 Failed to scrape metrics: virError(Code=1, Domain=10, Message='internal error: block_io_throttle inserted entry was not in expected format')
To Reproduce
Steps to reproduce the behavior:
2021/04/08 05:53:38 Failed to scrape metrics: virError(Code=1, Domain=10, Message='internal error: block_io_throttle inserted entry was not in expected format')
Expected behavior
`# HELP libvirt_domain_block_stats_size_iops_bytes The size of IO operations per second permitted through a block device
libvirt_domain_block_stats_size_iops_bytes{domain="dc533c58-8902-4db6-933b-e626721a5221",target_device="hda"} 0`
Additional context
libvirt prozess startet with:
/usr/bin/kvm -name guest=UUID,process=qemu:UUIS,debug-threads=on -S -object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-49-dc533c58-8902-4db6-9/master-key.aes -machine pc-i440fx-2.4,accel=kvm,usb=off,dump-guest-core=off,memory-backend=pc.ram -cpu Haswell-noTSX,ss=on,pdpe1gb=on,abm=on -m 8192 -object memory-backend-ram,id=pc.ram,size=8589934592 -overcommit mem-lock=off -smp 8,maxcpus=64,sockets=64,cores=1,threads=1 -uuid UUID -smbios type=0,vendor=unspecified,version=1.0,date=10/09/2015,release=166.0 -smbios type=1,manufacturer=unspecified,product=Legacy Virtual Machine,version=1.0,serial=0123456789,uuid=UUID -smbios type=2,manufacturer=unspecified,product=Legacy Virtual Machine,version=1.0,serial=0123456789 -no-user-config -nodefaults -chardev socket,id=charmonitor,fd=98,server=on,wait=off -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -boot strict=on -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -blockdev {"driver":"rbd","pool":"rbd","image":"CEPH-UUID","server":[{"host":"mon1","port":"6789"},{"host":"mon2","port":"6789"},{"host":"mon3","port":"6789"}],"node-name":"libvirt-2-storage","cache":{"direct":false,"no-flush":false},"auto-read-only":true,"discard":"unmap"} -blockdev {"node-name":"libvirt-2-format","read-only":false,"discard":"unmap","cache":{"direct":false,"no-flush":false},"driver":"raw","file":"libvirt-2-storage"} -device ide-hd,bus=ide.0,unit=0,drive=libvirt-2-format,id=ide0-0-0,bootindex=7,write-cache=on -blockdev {"driver":"rbd","pool":"isos","image":"ISO-UUID","server":[{"host":"mon1","port":"6789"},{"host":"mon2","port":"6789"},{"host":"mon3","port":"6789"}],"node-name":"libvirt-1-storage","auto-read-only":true,"discard":"unmap"} -blockdev {"node-name":"libvirt-1-format","read-only":true,"driver":"raw","file":"libvirt-1-storage"} -device ide-cd,bus=ide.1,unit=1,drive=libvirt-1-format,id=ide0-1-1,bootindex=5 -netdev tap,fd=104,id=hostnet0 -device e1000,netdev=hostnet0,id=net0,mac=02:39:3e:56:fa:01,bus=pci.0,addr=0x10,romfile= -device usb-tablet,id=input0,bus=usb.0,port=1 -vnc [::0]:37 -device VGA,id=video0,vgamem_mb=16,bus=pci.0,addr=0x2 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3 -sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny -msg timestamp=on
Want to make libvirt-exporter docker image, command: "docker build -t prometheus-libvirt-exporter -f Dockerbuildimage ."
I use Ubuntu18.04 on HP server where prometheus, prometheus alert manager and other exporters (everything in docker containers) and multiple VMs are running.
Is it mandatory static linking of libvirt library? Or Im doing something in wrong dirrection?
The error is:
rm util/virkeyname-linux.pod util/virkeycode-rfb.pod util/virkeyname-osx.pod util/virkeycode-linux.pod util/virkeycode-osx.pod util/virkeycode-usb.pod util/virkeycode-xt.pod util/virkeycode-win32.pod util/virkeycode-atset3.pod util/virkeycode-atset1.pod util/virkeycode-atset2.pod util/virkeycode-xtkbd.pod util/virkeyname-win32.pod
make[3]: Leaving directory '/tmp/libvirt-3.8.0/src'
make[2]: *** [Makefile:5808: all] Error 2
make[2]: Leaving directory '/tmp/libvirt-3.8.0/src'
make[1]: *** [Makefile:2116: all-recursive] Error 1
make[1]: Leaving directory '/tmp/libvirt-3.8.0'
make: *** [Makefile:2009: all] Error 2
The command '/bin/sh -c ./configure --disable-shared --enable-static --localstatedir=/var --without-storage-mpath && make -j$(nproc) && make install && sed -i 's/^Libs:.*/& -lnl -ltirpc -lxml2/' /usr/local/lib/pkgconfig/libvirt.pc' returned a non-zero code: 2
Could attach whole log if needed.
Is your feature request related to a problem? Please describe.
Would it be possible to build binary release for Linux x86_64/aarch64?
Describe the solution you'd like
As for this exporter: https://github.com/openstack-exporter/openstack-exporter/releases
I'd like to be able to download an archive of the statically build exporter for linux.
Describe alternatives you've considered
I'll have to get a two step build phase as our platform is an airgapped platform that can't reach internet and so can't build correctly the exporter without cloning the whole github platform.
Additional context
Any offline platform won't be able to build this exporter as dependencies are refering to github or internet hosted repos.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.