cloudfoundry-community/collectd-boshrelease

Run collectd with BOSH

A super basic release for collectd.

• Concourse CI
• Pull requests will be automatically tested against a bosh-lite (see testflight-pr job)
• Latest upstream project versions are automatically added as blobs via CI
• Discussions and CI notifications at #collectd-boshrelease channel on https://slack.cloudfoundry.org

With basic config

properties:
  collectd:
    hostname_prefix: cf.collectd.warden.
    config: |
      LoadPlugin "df"
      LoadPlugin "disk"
      LoadPlugin "cpu"
      LoadPlugin "load"
      LoadPlugin "write_graphite"
      <Plugin "write_graphite">
        <Node "">
          EscapeCharacter "."
          Host "localhost"
          Port "2003"
        </Node>
      </Plugin>

You will end up with metrics that looks like

cf.collectd.warden.collectd_z1.0.cpu-1.cpu-steal 0.000000 1446825506
cf.collectd.warden.collectd_z1.0.cpu-2.cpu-user 0.799932 1446825506
cf.collectd.warden.collectd_z1.0.cpu-2.cpu-nice 0.000000 1446825506
cf.collectd.warden.collectd_z1.0.cpu-2.cpu-system 0.199982 1446825506
cf.collectd.warden.collectd_z1.0.cpu-2.cpu-idle 98.791028 1446825506
cf.collectd.warden.collectd_z1.0.cpu-2.cpu-wait 0.000000 1446825506
cf.collectd.warden.collectd_z1.0.cpu-2.cpu-interrupt 0.000000 1446825506
cf.collectd.warden.collectd_z1.0.cpu-2.cpu-softirq 0.000000 1446825506
cf.collectd.warden.collectd_z1.0.cpu-2.cpu-steal 0.000000 1446825506
cf.collectd.warden.collectd_z1.0.disk-sda1.disk_ops.read 0.000000 1446825506
cf.collectd.warden.collectd_z1.0.disk-sda1.disk_ops.write 0.000000 1446825506
cf.collectd.warden.collectd_z1.0.disk-sda1.disk_time.read 0.000000 1446825506

Usage

To use this bosh release:

git clone https://github.com/cloudfoundry-community/collectd-boshrelease.git

export BOSH_ENVIRONMENT=<alias>
export BOSH_DEPLOYMENT=haproxy
bosh deploy collectd-boshrelease/manifests/collectd.yml

To speed up your deployment, you can use the pre-compiled BOSH release:

bosh deploy kafkatree-boshrelease/manifests/kafka.yml \
  -o kafka-boshrelease/manifests/operators/use-compiled-releases.yml

To specify your own collectd configuration, then apply an operator file to modify the default.

Create a file, say tmp/config.yml:

---
- type: replace
  path: /instance_groups/name=collectd/jobs/name=collectd/properties/collectd/config
  value: |
      LoadPlugin "df"
      LoadPlugin "disk"
      LoadPlugin "cpu"
      LoadPlugin "load"
      LoadPlugin "write_graphite"
      <Plugin "write_graphite">
        <Node "">
          EscapeCharacter "."
          Host "localhost"
          Port "2003"
        </Node>
      </Plugin>

Merge your change into the deployment:

bosh deploy collectd-boshrelease/manifests/collectd.yml -o tmp/config.yml

The default manifest emits cpu, disk, df and load data every 5 sec to graphite on localhost. If you want to see this data simply:

$ bosh ssh -d collectd
# nc -l -p 2003

collectd-boshrelease's Issues

Update to v5.9.1 blocked

Waiting on .tar.bz2 to be included in upstream v5.9.1 release

collectd/collectd#3255

issue building release with new blobs

Trying to build the new release with bosh, it fails:

# bosh create release
Syncing blobs...
collectd/collectd-5.7.1.tar.bz2 downloading 1.7M (0%)
RSA 1024 bit CA certificates aredownloading 1.7M (1%)
/usr/local/rvm/gems/ruby-2.1.7/gems/blobstore_client-1.3262.24.0/lib/blobstore_client/sha1_verifiable_blobstore_client.rb:38:in `check_sha1': sha1 mismatch expected=bc77d2493b26e5c38e167a8a44fedfe287742c09 actual=173003d00e2fe95ac99e217668c44392c9525e2b (Bosh::Blobstore::BlobstoreError)

I'm not really sure how to dissect this release... The expected sha1 seems to come from config/blobs.yml, but I don't see the source location for that guy specified anywhere... In releases/collectd/collectd-2.1.0.yml there is:

packages:
- name: collectd
  version: f1eb5a60a7eebc2e182b8a42761bc282108d4f64
  fingerprint: f1eb5a60a7eebc2e182b8a42761bc282108d4f64
  sha1: e0999c521e7c45a61b79c678cccfa59e7d533308

which in .final_builds/packages/collectd/index.yml maps to:

  f1eb5a60a7eebc2e182b8a42761bc282108d4f64:
	    version: f1eb5a60a7eebc2e182b8a42761bc282108d4f64
	    blobstore_id: e69cf0e4-cfc3-4d0f-7d55-88afe3f195ef
	    sha1: e0999c521e7c45a61b79c678cccfa59e7d533308

when I try to curl that blob out of the s3 bucket, i get:

# curl http://collectd-boshrelease.s3.amazonaws.com/e69cf0e4-cfc3-4d0f-7d55-88afe3f195ef
<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>AccessDenied</Code><Message>Access Denied</Message><RequestId>8D71B6458648E57F</RequestId><HostId>sxcrDOewzBmqcCt8ZpAsMdZQAoTdmq1ZAoHnsrxG0U64qXTb0oaXI/cruuOntFN5TJEg10rP/zU=</HostId></Error>

Open to new maintainers?

This repo hasn't been updated for a while, so we've been using a fork at https://github.com/18F/cg-collectd-boshrelease. Would you all be open to adding us as maintainers so we can review open issues and merge in our changes?

@simonjohansson

Unauthorised to download blobs

What

We're hitting an issue, where attempt to build new boshrelease results in 403 errors whilst attempting to download the blobs.

→ bosh create-release
Blob download 'collectd/collectd-5.7.1.tar.bz2' (1.8 MB) (id: 32ecd5f6-7b34-49e2-5f46-0be95fd0b7be sha1: bc77d2493b26e5c38e167a8a44fedfe287742c09) started

Blob download 'collectd/collectd-5.7.1.tar.bz2' (id: 32ecd5f6-7b34-49e2-5f46-0be95fd0b7be) failed

Blob download 'yajl/yajl-2.1.0.tar.gz' (84 kB) (id: 8aaed8b3-9ded-4568-6fb3-7bee36e3da2a sha1: fe6b3c7439b26175aee59cabf8c4923b9eb3650d) started

Blob download 'yajl/yajl-2.1.0.tar.gz' (id: 8aaed8b3-9ded-4568-6fb3-7bee36e3da2a) failed

- Getting blob '32ecd5f6-7b34-49e2-5f46-0be95fd0b7be' for path 'collectd/collectd-5.7.1.tar.bz2':
    Getting blob from inner blobstore:
      Getting blob from inner blobstore:
        AccessDenied: Access Denied
	status code: 403, request id: 91FAB841CED959AC
- Getting blob '8aaed8b3-9ded-4568-6fb3-7bee36e3da2a' for path 'yajl/yajl-2.1.0.tar.gz':
    Getting blob from inner blobstore:
      Getting blob from inner blobstore:
        AccessDenied: Access Denied
	status code: 403, request id: 915D831670D4BCA6

Exit code 1

Would it be possible for that to be resolved?

Thanks.

bump collectd version

G'day,

Collectd 5.5 introduced some nice features for CPU aggregation. I've tested it locally with local blob store and it works fine. How can i assist with updating the boshrelease to include the newer version of collectd?

can i simply be given put access to the collectd-boshrelease bucket and then raise a PR for upgrade to this repo?

Cheers,
Michael

Add collectd release to the official bosh releases list

Hello here,

would it be possible to add the collectd bosh release to the official bosh release list.
It would make its usage more convenient.

In alternative, the project github releases could contain a proper ready-to-use bosh release gzipped tar, instead of the standard github project archive

Thanks,
Claudio

Usage setup

Hi,
I was trying to understand and use the collectd setup for an existing Cloud Foundry installation. Looking at the repository I have a few questions. I my setup collectd should report the data via the network plugin to influxdb. So the questions are:

1. Do I really need the collectd instances (2 of them) and the collectd_leader?
2. If not is it enough to just use the basic setup upload the release, compile it and add the property configuration to my existing CF deployment?

Thanks a lot for your help. I will try to help and improve the documentation. :-)

Regards,
Johannes