choria-legacy/mcollective-choria

Distribution of plugins for MCollective as found in Puppet 6

mcollective-choria's Issues

move the orchestrator bits out of the application

Support both the NATS clients

Today and after #48 we'll support the basic NATS client but we should also some way support the Streaming client, less urgent now till streaming gets working clustering but probably a requirements for #33

Version the playbook schema in use

playbook data should include a indicator of what version schema they are

figure out how to install eventmachine with ssl on windows

All the windows hassles should now be sorted, except knowing how to install eventmachine with ssl support on windows

Add reports to the playbook run

There was a basic report in the #32 POC but it was more a whatif than something useful, design and flesh out and build

• Record metadata
• Record all inputs
• Record all discovered node sets
• For every task record a generic status and let task types store arbitrary stuff like raw rpc results
• Store all the logs
• Versioned

Ruby error when using mcollective-package-agent: Could not autoload puppet/provider/service/windows: no 'environments'

We just installed a brand new 1 node mcollective-choria system (ubuntu 16.04), and latest puppet-agent/puppetserver.

When we run the mcollective puppet agent, no problems, but if we try the mcollective service or package agent we get the following error:

error 2016/12/19 16:50:49: agent.rb:108:in `rescue in handlemsg' service#status failed: Puppet::Error: Could not autoload puppet/type/service: Could not autoload puppet/provider/service/windows: no 'environments' in {:current_environment=><Puppet::Node::Environment:70261004253860 @name="*root*" @manifest="no_manifest" @modulepath="" >, :root_environment=><Puppet::Node::Environment:70261004253860 @name="*root*" @manifest="no_manifest" @modulepath="" >} at top of [[0, nil, nil]]
error 2016/12/19 16:50:49: agent.rb:109:in `rescue in handlemsg' /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/context.rb:55:in `lookup'
	/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet.rb:235:in `lookup'
	/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/autoload.rb:131:in `module_directories'
	/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/autoload.rb:166:in `search_directories'
	/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/autoload.rb:95:in `get_file'
	/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/autoload.rb:64:in `load_file'
	/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/autoload.rb:194:in `load'
	/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/type.rb:1790:in `provider'
	/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/type.rb:1841:in `provide'
	/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/provider/service/windows.rb:3:in `<top (required)>'
	/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/autoload.rb:68:in `load'
	/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/autoload.rb:68:in `load_file'
	/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/autoload.rb:83:in `block in loadall'
	/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/autoload.rb:81:in `each'
	/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/autoload.rb:81:in `loadall'
	/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/autoload.rb:208:in `loadall'
	/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/metatype/manager.rb:126:in `newtype'
	/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/type/service.rb:10:in `<module:Puppet>'
	/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/type/service.rb:8:in `<top (required)>'
	/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/autoload.rb:68:in `load'
	/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/autoload.rb:68:in `load_file'
	/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/autoload.rb:194:in `load'
	/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/metatype/manager.rb:171:in `type'
	/opt/puppetlabs/mcollective/plugins/mcollective/util/service/puppetservice.rb:36:in `service_provider'
	/opt/puppetlabs/mcollective/plugins/mcollective/util/service/puppetservice.rb:30:in `status'
	/opt/puppetlabs/mcollective/plugins/mcollective/agent/service.rb:59:in `do_service_action'
	/opt/puppetlabs/mcollective/plugins/mcollective/agent/service.rb:32:in `block in <class:Service>'
	/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/mcollective/rpc/agent.rb:86:in `handlemsg'
	/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/mcollective/agents.rb:126:in `block (2 levels) in dispatch'
	/opt/puppetlabs/puppet/lib/ruby/2.1.0/timeout.rb:90:in `block in timeout'
	/opt/puppetlabs/puppet/lib/ruby/2.1.0/timeout.rb:33:in `block in catch'
	/opt/puppetlabs/puppet/lib/ruby/2.1.0/timeout.rb:33:in `catch'
	/opt/puppetlabs/puppet/lib/ruby/2.1.0/timeout.rb:33:in `catch'
	/opt/puppetlabs/puppet/lib/ruby/2.1.0/timeout.rb:105:in `timeout'
	/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/mcollective/agents.rb:125:in `block in dispatch'

It all comes down to how choria initializes puppet in https://github.com/ripienaar/mcollective-choria/blob/master/lib/mcollective/util/choria.rb#L367.

When this is used as is and followed by what the service agent does (in https://github.com/puppetlabs/mcollective-service-agent/blob/master/util/service/puppetservice.rb#L36), it produces the above stacktrace.

Puppet doesn't seem to be correctly initialized.

If I add the following to the choria.rb puppet initialization function, based on what is done in the mcollective-puppet-agent (found here: https://github.com/puppetlabs/mcollective-puppet-agent/blob/master/util/puppet_agent_mgr/mgr_v3.rb#L21):

            # This check is to keep backwards compatibility
            # with Puppet versions < 3.5.0
            if Puppet.respond_to?(:base_context) \
                && Puppet.respond_to?(:push_context)
              Puppet.push_context(Puppet.base_context(Puppet.settings))
            end

Then it works correctly :)

Obviously, I'm not using puppet < 3.5.0, so either the comment is plainly inaccurate or there's something else at play here which I haven't found.

Any idea?
Thanks!

Install EM in AIO on Ubuntu 16.04 fails

I tried to set up mcollective with your module, but the compilation of the eventmachine during the installation of the nats gem seems to produce some wrong code or there is a runtime lib missing.

Starting the mcollectived after the module run:
`# /opt/puppetlabs/puppet/bin/mcollectived --config=/etc/puppetlabs/mcollective/server.cfg --pidfile=/var/run/puppetlabs/mcollective.pid --no-daemonize

/opt/puppetlabs/puppet/bin/ruby: symbol lookup error: /opt/puppetlabs/puppet/lib/ruby/gems/2.1.0/extensions/x86_64-linux/2.1.0/eventmachine-1.2.0.1/rubyeventmachine.so: undefined symbol: ZNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEC1EPKcRKS3`

From my old days of c++ development in the last millenium, it looks like there is a C++ lib missing...

I tried to remove the gem, install it by hand using/opt/puppetlabs/puppet/bin/gem install nats to use the right ruby version, but that does not help.

Any clues?

Some Facts:
`# facter os ruby

{
architecture => "amd64",
distro => {
codename => "xenial",
description => "Ubuntu 16.04.1 LTS",
id => "Ubuntu",
release => {
full => "16.04",
major => "16.04"
}
},
family => "Debian",
hardware => "x86_64",
name => "Ubuntu",
release => {
full => "16.04",
major => "16.04"
},
selinux => {
enabled => false
}
}
ruby => {
platform => "x86_64-linux",
sitedir => "/opt/puppetlabs/puppet/lib/ruby/site_ruby/2.1.0",
version => "2.1.9"
}
`

filter out deactivated nodes

the discovery plugin does not filter out deactivated nodes

Add a shell script node set data source

Shell script should just output a bunch of node names to stdout

add PQL discovery language to PuppetDB discovery

with PQL becoming visible in PE hopefully more people will actually grok it, so be good to add PQL not just internally to PuppetDB discovery but also expose it on the CLI.

Discovery plugins cannot add CLI options for discovery but we can overload the -W one - because -W already has a query language: -W processing happens at a higher layer, so we'll have to use -I - which is actually closer to what this will do, it finds identities.

-I pql:<pql query here>

when the puppetdb discovery is active this will run the query and extract certnames from the results and target those nodes, if PuppetDB isn't active this will error safely

These queries will be mashed in along with the rest of the PQL query the discovery plugin generates so there is some restrictions, you need pql: nodes[certname] { facts_environment = "production" } and not just some random query - it has to return a list of certname which is obvious if you think this is a identity query it makes sense but docs needed

Add a shell script task type

Just run a shell script, STDOUT = info logs, STDERR = error logs, exit code is success/fail for the task

error loading nats connector

Following deployment wiki:

error 2016/08/09 14:11:07: pluginmanager.rb:171:in `rescue in loadclass' Failed to load Mcollective::Connector::Nats: cannot load such file -- mcollective/connector/nats.rb
Error: Facter: error while resolving custom fact "mcollective": cannot load such file -- mcollective/connector/nats.rb

Make last task failure/success message available on metadata

In post tasks you might want to log why failures happened, this would be the message from the last run task generally, this should be avilable on the metadata or similar

use SRV for puppet, ca and puppetdb

The choria application failed to run: uninitialized constant MCollective::Util::Choria::Resolv

I'm testing this module with ActiveMQ for the middleware which is mostly working fine so far.

However, mco choria request_cert is failing.

$ mco choria request_cert -v
Certificate /home/<USER>/.puppetlabs/etc/puppet/ssl/certs/<USER>.mcollective.pem has already been requested, attempting to retrieve it
Waiting up to 240 seconds for it to be signed

Attempting to download certificate /home/<USER>/.puppetlabs/etc/puppet/ssl/certs/<USER>.mcollective.pem: 0 / 24
The choria application failed to run: uninitialized constant MCollective::Util::Choria::Resolv

uninitialized constant MCollective::Util::Choria::Resolv (NameError)
    from /opt/puppetlabs/mcollective/plugins/mcollective/util/choria.rb:28:in `resolver'  <----
    from /opt/puppetlabs/mcollective/plugins/mcollective/util/choria.rb:83:in `block in query_srv_records'
    from /opt/puppetlabs/mcollective/plugins/mcollective/util/choria.rb:82:in `map'
    from /opt/puppetlabs/mcollective/plugins/mcollective/util/choria.rb:82:in `query_srv_records'
    from /opt/puppetlabs/mcollective/plugins/mcollective/util/choria.rb:281:in `try_srv'
    from /opt/puppetlabs/mcollective/plugins/mcollective/util/choria.rb:315:in `puppetca_server'
    from /opt/puppetlabs/mcollective/plugins/mcollective/util/choria.rb:580:in `attempt_fetch_cert'
    from /opt/puppetlabs/mcollective/plugins/mcollective/application/choria.rb:117:in `block in request_cert_command'
    from /opt/puppetlabs/mcollective/plugins/mcollective/application/choria.rb:114:in `times'
    from /opt/puppetlabs/mcollective/plugins/mcollective/application/choria.rb:114:in `request_cert_command'
    from /opt/puppetlabs/mcollective/plugins/mcollective/application/choria.rb:78:in `main'
    from /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/mcollective/application.rb:293:in `run'
    from /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/mcollective/applications.rb:23:in `run'
    from /opt/puppetlabs/bin/mco:33:in `<main>'

To get around it temporarily, I hacked /opt/puppetlabs/mcollective/plugins/mcollective/util/choria.rb with a require "resolv" at the top and that seemed to work fine.

import mcollective-connector-nats

DRY up and move SRV logic to Util::Choria for later reuse to find puppetserver/ca/puppetdb etc

add docs for puppetdb discovery

add some wiki page, especially since #61

discovery find client only installs of the moduels

at present the puppetdb discovery just checks for the mcollective_agent_foo classes but actually it should further limit it to only ones with $server true else client only machines are discovered as having the agent

incorporate mcollective-discovery-puppetdb

DRYup via Util::Choria

Add arbitrary named task lists and make them reusable

Imagine you want to do blue/green deploys, the logic for the deployment will be identical but you want to run the 2nd group only when the first completed

To do this you'd make 2 node sets, one per blue nodes and one per green nodes. You'd invoke the task list with a different variable set

This is pretty much conceptually exactly the same as the current hook lists but I dont think I have any way to pass in a variable into a task list - so perhaps the older idea of calling another (or the same) playbook with a different set of inputs will have more legs?

incorporate mcollective-security-puppet

reuse the various ssl functions and DRY up

check result sets better

When for example runonce fails either due to action policy or other reasons this is just silently not logged and ignored.

This is bad, it should log and fail. Additionally when this happens the nodes should be left disabled since basically the whole stack is inconsistent and user intervention is needed

change DDL licences to Apache-2.0

some have older ASL 2.0 - nats connector for eg. reported by @igalic

handle cyclic catalogs

work around this idiocy https://tickets.puppetlabs.com/browse/PUP-6537

fix botched discovery fix

discovery fix in #27 messed up and now it wont find rpcutil at all and it's apparently never supported finding classes in the form foo::bar

Simplify fact queries in choria discovery plugin

PuppetDB 4.2.0 can do PQL queries for facts using dot notation rather than this stuff, use that

Problem with package plugin

I have just install mcollective with choria.
I could find and ping nodes, but when I try to use the package plugin I get an error (other plugins like service does not work too).
In the client this is what I get:

amateo@joshua:~/puppetcode/mcollective$ mco rpc package status package=puppet-agent -I noctua10.um.es

 * [ ============================================================> ] 1 / 1


noctua10.um.es                           Unknown Request Status
   undefined method `properties' for nil:NilClass

and in the server log I have:

Dec 15 13:32:09 noctua10 mcollectived[12988]: choria.rb:147:in `validrequest?' Received valid request 6b342da51acc5efa901bb4fb61dde39a from choria=amateo.mcollective
Dec 15 13:32:09 noctua10 mcollectived[12988]: agent.rb:108:in `rescue in handlemsg' package#status failed: NoMethodError: undefined method `properties' for nil:NilClass
Dec 15 13:32:09 noctua10 mcollectived[12988]: agent.rb:109:in `rescue in handlemsg' /opt/puppetlabs/mcollective/plugins/mcollective/util/package/puppetpackage.rb:39:in `status'#012#011/opt/puppetlabs/mcollective/plugins/mcollective/agent/package.rb:105:in `do_pkg_action'#012#011/opt/puppetlabs/mcollective/plugins/mcollective/agent/package.rb:22:in `block in <class:Package>'#012#011/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/mcollective/rpc/agent.rb:86:in `handlemsg'#012#011/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/mcollective/agents.rb:126:in `block (2 levels) in dispatch'#012#011/opt/puppetlabs/puppet/lib/ruby/2.1.0/timeout.rb:90:in `block in timeout'#012#011/opt/puppetlabs/puppet/lib/ruby/2.1.0/timeout.rb:33:in `block in catch'#012#011/opt/puppetlabs/puppet/lib/ruby/2.1.0/timeout.rb:33:in `catch'#012#011/opt/puppetlabs/puppet/lib/ruby/2.1.0/timeout.rb:33:in `catch'#012#011/opt/puppetlabs/puppet/lib/ruby/2.1.0/timeout.rb:105:in `timeout'#012#011/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/mcollective/agents.rb:125:in `block in dispatch'

I have also tried running directly the agent, with the same result:

mateo@joshua:~/puppetcode/main$ mco package status puppet-agent -I noctua10.um.es

 * [ ==========================================================> ] 1 / 1

   noctua10.um.es: undefined method `properties' for nil:NilClass

Summary of Arch:

     No aggregate summary could be computed

Summary of Ensure:

     No aggregate summary could be computed


Finished processing 1 / 1 hosts in 21.42 ms

and the same log in the server.

These same commands works without problems in my previous platform configured with puppet/mcollective module instead of choria.

release 0.0.8

Error requesting cert for choria

Following the deploymen wiki, the default puppetserver auth.conf doesn't seem to have any permissions for /puppet/v3/environment, only /puppet/v3/environments.

/etc/puppetlabs/mcollective]# mco choria request_cert

The choria application failed to run, use -v for full error backtrace details: Failed to make request to Puppet: 403: Forbidden: Forbidden request: /puppet/v3/environment/production (method :get). Please see the server logs for details.

Is this something I should have configured somehow otherwise in puppetserver, or should choria take care of this for me?

add a YAML node set data source

foo_nodes:
  - node1
  - node2
  - node3

user should be able to point at the YAML and say a node list is foo_nodes

windows wants paths quoted, quote path to facter

import the packager from puppet-mcollective

remove packager

think this belongs better in ripienaar-mcollective

add a Next Steps section to docs

Docs kind of assume you know whats up with mcollective and what its for, but as choria installs various plugins it might be handy to add a quick intro to using it focussing on these plugins and then link to the official 'using the cli' doc

Problem with a client only installation

I have tried to install only the client on a node, so I have run:

class {'::mcollective':
  client => true,
  server => false,
}

Although puppet executions works without any problem, I don't have the /etc/puppetlabs/mcollective/plugin.d/choria.cfg file, so mco choria request_cert doesn't work.

I have had to manually create this file in order to work.

Wiki should mention mcollective default config assumes gnatsd running on 'puppet' node

Following the instructions in the wiki, it seems mcollective's default configuration tries to connect to a node named puppet. That seems worth pointing out in the wiki.

I'm not completely sure why that is. The server.cfg names host as stomp. My puppet.conf is

[agent]
server=servername

add a basic module dependant on ripienaar-mcollective

Add a direct PQL node set source

Allow for direct PQL queries as a node source

move docs to choria.io

Use nats gem 0.8.0

NATS daemon.latest has a feature where on connect to any cluster member it will announce the whole cluster topology to clients and keep it up to date, this is a huge improvement in config complexity and surviving in a dynamic environment

The 0.8.0 gem supports this feature, should look at moving to that.

Add user pluggable task types

Add a REST task type

Make a generic REST client task type, get/put/post etc, just send JSON

Add user pluggable node discovery types

Use eventmachine pure

This can be done by setting $eventmachine_library = :pure_ruby before loading event machine, also update eventmachine in the Gemfile

Create a non EM based nats client

The only viable NATS client now is using EM which is a pain, getting EM+SSL on windows is basically a no go on the Puppet packaging and it would be very hard to make non interactive.

NATS protocol though is quite trivial and we need a small amount of what it supports, in theory a thread based client can be written with no dependencies.

Set the NATS name property to the identity of the connection

NATS.latest and the gem has a name field that you can set to something user supplied, set this to the identity of the mcollective using the library so the connections can be mapped on the NATS side

check the crl for client certs

puppet agent caches the CRL locally it seems, we should check that and detect revoked certs

fix data files

various data files have incorrect keys - not correctly namespaced for the module

create a playbook like system

Full spec for the feature @ http://choria.io/docs/roadmap/playbooks/

Mcollective server.cfg fix for puppet agent

Hi, so the mcollective-puppet-agent won't actually run puppet (the runonce or runall actions) as it can't find the puppet agent in the new location (/opt/puppetlabs/bin). The agent simply runs 'puppet agent' but it resets the path so the custom puppetlabs path set in /etc/profile.d is ignored. There is a simple fix, which is to add this line to mcollective server.cfg:

plugin.puppet.command = /opt/puppetlabs/bin/puppet agent

This is then read in by mcollective_agent_puppet/files/mcollective/agent/puppet.rb:

@puppet_command = @config.pluginconf.fetch("puppet.command", "puppet agent")

Ref:
https://tickets.puppetlabs.com/browse/MCOP-496?jql=text%20~%20%22mcollective-puppet-agent%22