rochaporto/collectd-openstack

collectd plugins for openstack

collectd-openstack's Issues

Interval is not taken into account

I am currently running collectd plugins on the basis of your base.py

I am writing to the log what the interval is in the subclass and also when the config callback is executed.
The read_callback will be initialized with the default interval, as the config is not yet read.

May 28 21:06:14 host collectd[23258]: plugin_load: plugin "python" successfully loaded.
May 28 21:06:14 host collectd[23258]: subclass_interval: 10.0
May 28 21:06:14 host collectd[23258]: reading config
May 28 21:06:14 host collectd[23258]: base_interval: 60.0

running collectd 5.7.1-1.1 on Debian 9

Seems like #11 is not solved, what could be a solution to this initialize the read callback after reading the config?

interval doesn't seem to be taken into account

Plugins seem to run more often than what is specified in the Interval config param.

add missing __init__.py files

So that we can reference the plugins location as a package.

glance plugin should get all tenant data in a single query

Right now it's doing a separate query per tenant, even if reusing the client.

floating ip listing is not ordered

Nuff said.

Need to check why.

floating ip data not showing correctly

We're relying on the floating ip data from nova/limits, which unfortunately is not published correctly.

Need to ask neutron directly for this sort of info instead.

volumes data missing

Quota queries in nova does not provide volumes used/quota data.

Need to query cinder, but the way we currently do it (by asking for all volumes and processing them) does not work - asking for a volume list only gives by the ones for the current tenant.

Should do it the same way we do for nova instead: list volumes for each tenant, passing the tenant_id explicitly.

Abandoned?

Has this project been abandoned? It not I will fork and make patches to get it working with kilo and liberty but I wanted to be sure before I fork and keep all the patches in my own repo.

allow specifying reserved values for different metrics

Things like reserved cpu or ram, per node or total.

neutron plugin crashes on missing quotas

If a tenant does not have any quotas being retrieved, the plugin crashes.

openstack-neutron: failed to get stats :: list index out of range :: Traceback (most recent call last):#0
12 File "/usr/lib/collectd/plugins/openstack/base.py", line 107, in read_callback#012 stats = self.get_stats()#12 File "/usr/lib/collectd/plugins/openstack
/neutron_plugin.py", line 77, in get_stats#012 quota = client.list_quotas(tenant_id=tenant.id)['quotas'][0]#012IndexError: list index out of range
Jul 15 04:44:49 icinga collectd[8858]: openstack-neutron: failed to get stats :: list index out of range :: Traceback (most recent call last):#12 File "/usr/li
b/collectd/plugins/openstack/base.py", line 107, in read_callback#012 stats = self.get_stats()#12 File "/usr/lib/collectd/plugins/openstack/neutron_plugin.p
y", line 77, in get_stats#012 quota = client.list_quotas(tenant_id=tenant.id)['quotas'][0]#012IndexError: list index out of range

quota values incorrect

The quota values showing for instances (as an example) are bogus.

Seems the tenant is not being passed correctly in the queries.

plugin failures when tenant_id is missing

If a network or volume is attached to a tenant_id which in the meantime disappeared, the plugins crash:

openstack-neutron: failed to get stats :: u'231bf8653e7646d198a8afb2353823a5' :: Traceback (most
recent call last):#12 File "/usr/lib/collectd/plugins/openstack/base.py", line 107, in read_callback#012 stats = self.get_stats()#12 File "/usr/lib/collec
td/plugins/openstack/neutron_plugin.py", line 70, in get_stats#012 tenant = tenants[network['tenant_id']]#012KeyError: u'231bf8653e7646d198a8afb2353823a5'
Jul 17 10:25:43 cat-wgtn-mon0 collectd[21553]: openstack-neutron: failed to get stats :: u'231bf8653e7646d198a8afb2353823a5' :: Traceback (most recent call last)
:#12 File "/usr/lib/collectd/plugins/openstack/base.py", line 107, in read_callback#012 stats = self.get_stats()#12 File "/usr/lib/collectd/plugins/openst
ack/neutron_plugin.py", line 70, in get_stats#012 tenant = tenants[network['tenant_id']]#012KeyError: u'231bf8653e7646d198a8afb2353823a5'
2014-07-17T10:25:43.854469+12:00 cat-wgtn-mon0 collectd[21553]: Unhandled python exception in read callback: UnboundLocalError: local variable 'stats' referenced before assignment
Jul 17 10:25:43 cat-wgtn-mon0 collectd[21553]: Unhandled python exception in read callback: UnboundLocalError: local variable 'stats' referenced before assignment
2014-07-17T10:25:43.862936+12:00 cat-wgtn-mon0 collectd[21553]: read-function of plugin `python.openstack.neutron_plugin' failed. Will suspend it for 640.000 seconds.

This situation can happen when there are some inconsistencies in the dbs, and we should protect against it.

Allow specifying the overcommit ratios

Need a couple additional configuration parameters to pass the overcommit ratios to the plugins.

Probably needed at least for vcpus and memory.

enable decent verbose logging

For every plugin, enable logging of start/stop of a run when verbose is on.

Currently we dump a bunch of detailed graphite, we might want an intermediate logging level somehow.

Unhandled python exception in read callback: UnboundLocalError: local variable 'stats' referenced before assignment

./collectd -T
openstack-keystone: failed to get stats :: Authorization Failed: The resource could not be found. (HTTP 404) (Request-ID: req-56dd66a4-b85b-446c-8f29-05e6a622f2eb) :: Traceback (most recent call last):
  File "/home/fedora/collectd-openstack/plugins/base.py", line 138, in read_callback
    stats = self.get_stats()
  File "/home/fedora/collectd-openstack/plugins/keystone_plugin.py", line 45, in get_stats
    keystone = self.get_keystone()
  File "/home/fedora/collectd-openstack/plugins/base.py", line 50, in get_keystone
    tenant_name=self.tenant, auth_url=self.auth_url)
  File "/usr/lib/python2.7/site-packages/keystoneclient/v2_0/client.py", line 152, in __init__
    self.authenticate()
  File "/usr/lib/python2.7/site-packages/keystoneclient/utils.py", line 318, in inner
    return func(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/keystoneclient/httpclient.py", line 503, in authenticate
    resp = self.get_raw_token_from_identity_service(**kwargs)
  File "/usr/lib/python2.7/site-packages/keysto
Unhandled python exception in read callback: UnboundLocalError: local variable 'stats' referenced before assignment
Traceback (most recent call last):
  File "/home/fedora/collectd-openstack/plugins/keystone_plugin.py", line 78, in read_callback
    plugin.read_callback()
  File "/home/fedora/collectd-openstack/plugins/base.py", line 144, in read_callback
    self.dispatch(stats)
UnboundLocalError: local variable 'stats' referenced before assignment
read-function of plugin `python.keystone_plugin' failed.
openstack-nova: failed to get stats :: Authorization Failed: The resource could not be found. (HTTP 404) (Request-ID: req-8a5e2dbc-98cf-41b0-8442-f1c0c7f41a47) :: Traceback (most recent call last):
  File "/home/fedora/collectd-openstack/plugins/base.py", line 138, in read_callback
    stats = self.get_stats()
  File "/home/fedora/collectd-openstack/plugins/nova_plugin.py", line 48, in get_stats
    keystone = self.get_keystone()
  File "/home/fedora/collectd-openstack/plugins/base.py", line 50, in get_keystone
    tenant_name=self.tenant, auth_url=self.auth_url)
  File "/usr/lib/python2.7/site-packages/keystoneclient/v2_0/client.py", line 152, in __init__
    self.authenticate()
  File "/usr/lib/python2.7/site-packages/keystoneclient/utils.py", line 318, in inner
    return func(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/keystoneclient/httpclient.py", line 503, in authenticate
    resp = self.get_raw_token_from_identity_service(**kwargs)
  File "/usr/lib/python2.7/site-packages/keystoneclient
Unhandled python exception in read callback: UnboundLocalError: local variable 'stats' referenced before assignment
Traceback (most recent call last):
  File "/home/fedora/collectd-openstack/plugins/nova_plugin.py", line 113, in read_callback
    plugin.read_callback()
  File "/home/fedora/collectd-openstack/plugins/base.py", line 144, in read_callback
    self.dispatch(stats)
UnboundLocalError: local variable 'stats' referenced before assignment

cinder queries require multiple client instances

This is problematic as we end up with many keystone calls.

Issue is with the limits.get() call in cinderclient not taking the tenant.id.

A workaround that worked for the novaclient was to set the client.tenant_id variable after instantiating the client:
client = NovaClient('2', self.username, self.password, self.tenant, self.auth_url)
for tenant in tenant_list:
# FIX: nasty but works for now (tenant.id not being taken below :()
client.tenant_id = tenant.id

Might work here too.

Unable to monitor a single project with nova

It seems that this plugin fails to cover the use case where you only have access to a project, throwing this error:

failed to get stats :: You are not authorized to perform the requested action: admin_required

I think that this rules out most of the users because they would not be able to monitor resources usages inside their own project. Very few of them will have admin access over an entire cloud.

nova_plugin fails on openstack-kilo

My setup:
OpenStack Compute (Nova): 2015.1
OS: CentOS 7.1
collectd: 5.5.0

Log message:

Aug 27 11:01:06 controller collectd[6660]: openstack-nova: failed to get stats :: 'NovaPlugin' object has no attribute 'AllocationRatioCores' :: Traceback (most recent call last):
  File "/usr/lib64/collectd/plugins/openstack/base.py", line 138, in read_callback
    stats = self.get_stats()
  File "/usr/lib64/collectd/plugins/openstack/nova_plugin.py", line 79, in get_stats
    data[self.prefix]['cluster']['config'][item] = getattr(self, item)
AttributeError: 'NovaPlugin' object has no attribute 'AllocationRatioCores'
Aug 27 11:01:06 controller collectd[6660]: Unhandled python exception in read callback: UnboundLocalError: local variable 'stats' referenced before assignment
Aug 27 11:01:06 controller collectd[6660]: read-function of plugin `python.nova_plugin' failed. Will suspend it for 120.000 seconds.

Thanks in advance for your help!

nova maxTotalRAMSize value is bogus

The current plugin gives a bogus value for this field.

This seems to be related to an inconsistency in nova-api. In the python novaclient limits.py the tenant_id is passed as a querystring param (which seems to match the api):
http://developer.openstack.org/api-ref-compute-v2-ext.html#ext-compute_limits_admins

but in reality the service is ignoring it (at least in havana).

Can be reproduced in the command line too:

nova --os-tenant-name absolute-limits | grep maxTotalRAMSize
| maxTotalRAMSize | 136000 |

nova absolute-limits --tenant | grep maxTotalRAMSize
| maxTotalRAMSize | 50000 |

dpkg-query -l python-novaclient
python-novaclient 1:2.15.0-0ubuntu1~cloud0

dpkg-query -l nova-common
nova-common 1:2013.2.2-0ubuntu1~cloud0

snapshot project_id attr not always returned

Seems the project_id is not always returned as a os-extended-attr in the snapshots query:

{
  "status": "available",
  "display_name": "snapshot for base-jenkins-agent",
  "created_at": "2014-03-27T05:55:15.000000",
  "display_description": "",
  "volume_id": "e3b40183-8752-4b9d-948f-e9e7a0187a38",
  "metadata": {

  },
  "id": "6d123b0a-c226-4509-b30e-429c914fae37",
  "size": 40
},
{
  "status": "available",
  "display_name": "snapshot for snap2",
  "created_at": "2014-04-14T05:28:21.000000",
  "display_description": "",
  "os-extended-snapshot-attributes:progress": "100%",
  "volume_id": "d7f3856d-327c-4c01-82ac-889ddf565780",
  "os-extended-snapshot-attributes:project_id": "94b566de52f9423fab80ceee8c0a4a23",
  "metadata": {

  },
  "id": "8db14c13-ed5b-4e53-9a03-fb7f51d91764",
  "size": 40
},

Need to check why... the query is being done with all_tenants.

provide one line installer (curl pipe bash)

Before building packages it would be very helpful to have a one line installer for these plugins.

Or another idea: how about a Makefile with an install target?

ImportError: dynamic module does not define init function (inituuid)

I copied the py files into the /usr/lib64/collectd/ directory (where the other collectd plugins are located) and then used the following conf file (just showing the keystone one):

LoadPlugin python

<Plugin "python">
ModulePath "/usr/lib64/collectd"

Import "keystone_plugin"

<Module "keystone_plugin">
    Username "admin"
    Password "Emnz"
    TenantName "admin"
    AuthURL "http://96.239.248.100:5000/v2.0"
    Verbose "False"
</Module>

When I then attempt to restart collectd I get an error and it will not actually start. So to isolate the error I ran the collectd module test using the following command:

/usr/sbin/collectd -T -C /etc/collectd.conf

When I do this the messages I get back are:

plugin_load: plugin "write_http" successfully loaded.
plugin_load: plugin "cpu" successfully loaded.
plugin_load: plugin "disk" successfully loaded.
plugin_load: plugin "memory" successfully loaded.
plugin_load: plugin "netlink" successfully loaded.
plugin_load: plugin "virt" successfully loaded.
plugin_load: plugin "ethstat" successfully loaded.
ethstat plugin: Registered interface eth0
plugin_load: plugin "load" successfully loaded.
plugin_load: plugin "swap" successfully loaded.
plugin_load: plugin "python" successfully loaded.
python plugin: Error importing module "keystone_plugin".
Unhandled python exception in importing module: ImportError: dynamic module does not define init function (inituuid)
python plugin: Found a configuration for the "keystone_plugin" plugin, but the plugin isn't loaded or didn't register a configuration callback.
Error: Reading the config file failed!
Read the logs for details.

All of my conf files are similar and if I attempt to use the other openstack modules for glance, cinder, nova or neutron, I get the same exact error for each one.