choria-legacy / mcollective-choria Goto Github PK
View Code? Open in Web Editor NEWDistribution of plugins for MCollective as found in Puppet 6
License: Apache License 2.0
Distribution of plugins for MCollective as found in Puppet 6
License: Apache License 2.0
In post tasks you might want to log why failures happened, this would be the message from the last run task generally, this should be avilable on the metadata or similar
Today and after #48 we'll support the basic NATS client but we should also some way support the Streaming client, less urgent now till streaming gets working clustering but probably a requirements for #33
the discovery plugin does not filter out deactivated nodes
Allow for direct PQL queries as a node source
This can be done by setting $eventmachine_library = :pure_ruby
before loading event machine, also update eventmachine in the Gemfile
Docs kind of assume you know whats up with mcollective and what its for, but as choria installs various plugins it might be handy to add a quick intro to using it focussing on these plugins and then link to the official 'using the cli' doc
I have tried to install only the client on a node, so I have run:
class {'::mcollective':
client => true,
server => false,
}
Although puppet executions works without any problem, I don't have the /etc/puppetlabs/mcollective/plugin.d/choria.cfg file, so mco choria request_cert
doesn't work.
I have had to manually create this file in order to work.
I have just install mcollective with choria.
I could find and ping nodes, but when I try to use the package plugin I get an error (other plugins like service does not work too).
In the client this is what I get:
amateo@joshua:~/puppetcode/mcollective$ mco rpc package status package=puppet-agent -I noctua10.um.es
* [ ============================================================> ] 1 / 1
noctua10.um.es Unknown Request Status
undefined method `properties' for nil:NilClass
and in the server log I have:
Dec 15 13:32:09 noctua10 mcollectived[12988]: choria.rb:147:in `validrequest?' Received valid request 6b342da51acc5efa901bb4fb61dde39a from choria=amateo.mcollective
Dec 15 13:32:09 noctua10 mcollectived[12988]: agent.rb:108:in `rescue in handlemsg' package#status failed: NoMethodError: undefined method `properties' for nil:NilClass
Dec 15 13:32:09 noctua10 mcollectived[12988]: agent.rb:109:in `rescue in handlemsg' /opt/puppetlabs/mcollective/plugins/mcollective/util/package/puppetpackage.rb:39:in `status'#012#011/opt/puppetlabs/mcollective/plugins/mcollective/agent/package.rb:105:in `do_pkg_action'#012#011/opt/puppetlabs/mcollective/plugins/mcollective/agent/package.rb:22:in `block in <class:Package>'#012#011/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/mcollective/rpc/agent.rb:86:in `handlemsg'#012#011/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/mcollective/agents.rb:126:in `block (2 levels) in dispatch'#012#011/opt/puppetlabs/puppet/lib/ruby/2.1.0/timeout.rb:90:in `block in timeout'#012#011/opt/puppetlabs/puppet/lib/ruby/2.1.0/timeout.rb:33:in `block in catch'#012#011/opt/puppetlabs/puppet/lib/ruby/2.1.0/timeout.rb:33:in `catch'#012#011/opt/puppetlabs/puppet/lib/ruby/2.1.0/timeout.rb:33:in `catch'#012#011/opt/puppetlabs/puppet/lib/ruby/2.1.0/timeout.rb:105:in `timeout'#012#011/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/mcollective/agents.rb:125:in `block in dispatch'
I have also tried running directly the agent, with the same result:
mateo@joshua:~/puppetcode/main$ mco package status puppet-agent -I noctua10.um.es
* [ ==========================================================> ] 1 / 1
noctua10.um.es: undefined method `properties' for nil:NilClass
Summary of Arch:
No aggregate summary could be computed
Summary of Ensure:
No aggregate summary could be computed
Finished processing 1 / 1 hosts in 21.42 ms
and the same log in the server.
These same commands works without problems in my previous platform configured with puppet/mcollective module instead of choria.
Full spec for the feature @ http://choria.io/docs/roadmap/playbooks/
discovery fix in #27 messed up and now it wont find rpcutil
at all and it's apparently never supported finding classes in the form foo::bar
work around this idiocy https://tickets.puppetlabs.com/browse/PUP-6537
Following the instructions in the wiki, it seems mcollective's default configuration tries to connect to a node named puppet
. That seems worth pointing out in the wiki.
I'm not completely sure why that is. The server.cfg names host as stomp
. My puppet.conf is
[agent]
server=servername
NATS.latest and the gem has a name field that you can set to something user supplied, set this to the identity of the mcollective using the library so the connections can be mapped on the NATS side
Following deployment wiki:
error 2016/08/09 14:11:07: pluginmanager.rb:171:in `rescue in loadclass' Failed to load Mcollective::Connector::Nats: cannot load such file -- mcollective/connector/nats.rb
Error: Facter: error while resolving custom fact "mcollective": cannot load such file -- mcollective/connector/nats.rb
foo_nodes:
- node1
- node2
- node3
user should be able to point at the YAML and say a node list is foo_nodes
at present the puppetdb discovery just checks for the mcollective_agent_foo
classes but actually it should further limit it to only ones with $server
true else client only machines are discovered as having the agent
We just installed a brand new 1 node mcollective-choria system (ubuntu 16.04), and latest puppet-agent/puppetserver.
When we run the mcollective puppet agent, no problems, but if we try the mcollective service or package agent we get the following error:
error 2016/12/19 16:50:49: agent.rb:108:in `rescue in handlemsg' service#status failed: Puppet::Error: Could not autoload puppet/type/service: Could not autoload puppet/provider/service/windows: no 'environments' in {:current_environment=><Puppet::Node::Environment:70261004253860 @name="*root*" @manifest="no_manifest" @modulepath="" >, :root_environment=><Puppet::Node::Environment:70261004253860 @name="*root*" @manifest="no_manifest" @modulepath="" >} at top of [[0, nil, nil]]
error 2016/12/19 16:50:49: agent.rb:109:in `rescue in handlemsg' /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/context.rb:55:in `lookup'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet.rb:235:in `lookup'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/autoload.rb:131:in `module_directories'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/autoload.rb:166:in `search_directories'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/autoload.rb:95:in `get_file'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/autoload.rb:64:in `load_file'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/autoload.rb:194:in `load'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/type.rb:1790:in `provider'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/type.rb:1841:in `provide'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/provider/service/windows.rb:3:in `<top (required)>'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/autoload.rb:68:in `load'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/autoload.rb:68:in `load_file'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/autoload.rb:83:in `block in loadall'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/autoload.rb:81:in `each'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/autoload.rb:81:in `loadall'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/autoload.rb:208:in `loadall'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/metatype/manager.rb:126:in `newtype'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/type/service.rb:10:in `<module:Puppet>'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/type/service.rb:8:in `<top (required)>'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/autoload.rb:68:in `load'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/autoload.rb:68:in `load_file'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/autoload.rb:194:in `load'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/metatype/manager.rb:171:in `type'
/opt/puppetlabs/mcollective/plugins/mcollective/util/service/puppetservice.rb:36:in `service_provider'
/opt/puppetlabs/mcollective/plugins/mcollective/util/service/puppetservice.rb:30:in `status'
/opt/puppetlabs/mcollective/plugins/mcollective/agent/service.rb:59:in `do_service_action'
/opt/puppetlabs/mcollective/plugins/mcollective/agent/service.rb:32:in `block in <class:Service>'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/mcollective/rpc/agent.rb:86:in `handlemsg'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/mcollective/agents.rb:126:in `block (2 levels) in dispatch'
/opt/puppetlabs/puppet/lib/ruby/2.1.0/timeout.rb:90:in `block in timeout'
/opt/puppetlabs/puppet/lib/ruby/2.1.0/timeout.rb:33:in `block in catch'
/opt/puppetlabs/puppet/lib/ruby/2.1.0/timeout.rb:33:in `catch'
/opt/puppetlabs/puppet/lib/ruby/2.1.0/timeout.rb:33:in `catch'
/opt/puppetlabs/puppet/lib/ruby/2.1.0/timeout.rb:105:in `timeout'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/mcollective/agents.rb:125:in `block in dispatch'
It all comes down to how choria
initializes puppet in https://github.com/ripienaar/mcollective-choria/blob/master/lib/mcollective/util/choria.rb#L367.
When this is used as is and followed by what the service agent does (in https://github.com/puppetlabs/mcollective-service-agent/blob/master/util/service/puppetservice.rb#L36), it produces the above stacktrace.
Puppet doesn't seem to be correctly initialized.
If I add the following to the choria.rb puppet initialization function, based on what is done in the mcollective-puppet-agent (found here: https://github.com/puppetlabs/mcollective-puppet-agent/blob/master/util/puppet_agent_mgr/mgr_v3.rb#L21):
# This check is to keep backwards compatibility
# with Puppet versions < 3.5.0
if Puppet.respond_to?(:base_context) \
&& Puppet.respond_to?(:push_context)
Puppet.push_context(Puppet.base_context(Puppet.settings))
end
Then it works correctly :)
Obviously, I'm not using puppet < 3.5.0, so either the comment is plainly inaccurate or there's something else at play here which I haven't found.
Any idea?
Thanks!
I'm testing this module with ActiveMQ for the middleware which is mostly working fine so far.
However, mco choria request_cert is failing.
$ mco choria request_cert -v
Certificate /home/<USER>/.puppetlabs/etc/puppet/ssl/certs/<USER>.mcollective.pem has already been requested, attempting to retrieve it
Waiting up to 240 seconds for it to be signed
Attempting to download certificate /home/<USER>/.puppetlabs/etc/puppet/ssl/certs/<USER>.mcollective.pem: 0 / 24
The choria application failed to run: uninitialized constant MCollective::Util::Choria::Resolv
uninitialized constant MCollective::Util::Choria::Resolv (NameError)
from /opt/puppetlabs/mcollective/plugins/mcollective/util/choria.rb:28:in `resolver' <----
from /opt/puppetlabs/mcollective/plugins/mcollective/util/choria.rb:83:in `block in query_srv_records'
from /opt/puppetlabs/mcollective/plugins/mcollective/util/choria.rb:82:in `map'
from /opt/puppetlabs/mcollective/plugins/mcollective/util/choria.rb:82:in `query_srv_records'
from /opt/puppetlabs/mcollective/plugins/mcollective/util/choria.rb:281:in `try_srv'
from /opt/puppetlabs/mcollective/plugins/mcollective/util/choria.rb:315:in `puppetca_server'
from /opt/puppetlabs/mcollective/plugins/mcollective/util/choria.rb:580:in `attempt_fetch_cert'
from /opt/puppetlabs/mcollective/plugins/mcollective/application/choria.rb:117:in `block in request_cert_command'
from /opt/puppetlabs/mcollective/plugins/mcollective/application/choria.rb:114:in `times'
from /opt/puppetlabs/mcollective/plugins/mcollective/application/choria.rb:114:in `request_cert_command'
from /opt/puppetlabs/mcollective/plugins/mcollective/application/choria.rb:78:in `main'
from /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/mcollective/application.rb:293:in `run'
from /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/mcollective/applications.rb:23:in `run'
from /opt/puppetlabs/bin/mco:33:in `<main>'
To get around it temporarily, I hacked /opt/puppetlabs/mcollective/plugins/mcollective/util/choria.rb
with a require "resolv"
at the top and that seemed to work fine.
Shell script should just output a bunch of node names to stdout
with PQL becoming visible in PE hopefully more people will actually grok it, so be good to add PQL not just internally to PuppetDB discovery but also expose it on the CLI.
Discovery plugins cannot add CLI options for discovery but we can overload the -W one - because -W already has a query language: -W processing happens at a higher layer, so we'll have to use -I - which is actually closer to what this will do, it finds identities.
-I pql:<pql query here>
when the puppetdb discovery is active this will run the query and extract certnames from the results and target those nodes, if PuppetDB isn't active this will error safely
These queries will be mashed in along with the rest of the PQL query the discovery plugin generates so there is some restrictions, you need pql: nodes[certname] { facts_environment = "production" }
and not just some random query - it has to return a list of certname
which is obvious if you think this is a identity query it makes sense but docs needed
DRY up and move SRV logic to Util::Choria for later reuse to find puppetserver/ca/puppetdb etc
Just run a shell script, STDOUT = info logs, STDERR = error logs, exit code is success/fail for the task
various data files have incorrect keys - not correctly namespaced for the module
NATS daemon.latest has a feature where on connect to any cluster member it will announce the whole cluster topology to clients and keep it up to date, this is a huge improvement in config complexity and surviving in a dynamic environment
The 0.8.0 gem supports this feature, should look at moving to that.
The only viable NATS client now is using EM which is a pain, getting EM+SSL on windows is basically a no go on the Puppet packaging and it would be very hard to make non interactive.
NATS protocol though is quite trivial and we need a small amount of what it supports, in theory a thread based client can be written with no dependencies.
When for example runonce
fails either due to action policy or other reasons this is just silently not logged and ignored.
This is bad, it should log and fail. Additionally when this happens the nodes should be left disabled
since basically the whole stack is inconsistent and user intervention is needed
Make a generic REST client task type, get/put/post etc, just send JSON
All the windows hassles should now be sorted, except knowing how to install eventmachine with ssl support on windows
reuse the various ssl functions and DRY up
DRYup via Util::Choria
Hi, so the mcollective-puppet-agent won't actually run puppet (the runonce or runall actions) as it can't find the puppet agent in the new location (/opt/puppetlabs/bin). The agent simply runs 'puppet agent' but it resets the path so the custom puppetlabs path set in /etc/profile.d is ignored. There is a simple fix, which is to add this line to mcollective server.cfg:
plugin.puppet.command = /opt/puppetlabs/bin/puppet agent
This is then read in by mcollective_agent_puppet/files/mcollective/agent/puppet.rb:
@puppet_command = @config.pluginconf.fetch("puppet.command", "puppet agent")
Ref:
https://tickets.puppetlabs.com/browse/MCOP-496?jql=text%20~%20%22mcollective-puppet-agent%22
some have older ASL 2.0 - nats connector for eg. reported by @igalic
puppet agent caches the CRL locally it seems, we should check that and detect revoked certs
Following the deploymen wiki, the default puppetserver auth.conf doesn't seem to have any permissions for /puppet/v3/environment, only /puppet/v3/environments.
/etc/puppetlabs/mcollective]# mco choria request_cert
The choria application failed to run, use -v for full error backtrace details: Failed to make request to Puppet: 403: Forbidden: Forbidden request: /puppet/v3/environment/production (method :get). Please see the server logs for details.
Is this something I should have configured somehow otherwise in puppetserver, or should choria take care of this for me?
add some wiki page, especially since #61
playbook data should include a indicator of what version schema they are
There was a basic report in the #32 POC but it was more a whatif than something useful, design and flesh out and build
think this belongs better in ripienaar-mcollective
I tried to set up mcollective with your module, but the compilation of the eventmachine during the installation of the nats gem seems to produce some wrong code or there is a runtime lib missing.
Starting the mcollectived after the module run:
`# /opt/puppetlabs/puppet/bin/mcollectived --config=/etc/puppetlabs/mcollective/server.cfg --pidfile=/var/run/puppetlabs/mcollective.pid --no-daemonize
/opt/puppetlabs/puppet/bin/ruby: symbol lookup error: /opt/puppetlabs/puppet/lib/ruby/gems/2.1.0/extensions/x86_64-linux/2.1.0/eventmachine-1.2.0.1/rubyeventmachine.so: undefined symbol: ZNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEC1EPKcRKS3`
From my old days of c++ development in the last millenium, it looks like there is a C++ lib missing...
I tried to remove the gem, install it by hand using/opt/puppetlabs/puppet/bin/gem install nats
to use the right ruby version, but that does not help.
Any clues?
Some Facts:
`# facter os ruby
{
architecture => "amd64",
distro => {
codename => "xenial",
description => "Ubuntu 16.04.1 LTS",
id => "Ubuntu",
release => {
full => "16.04",
major => "16.04"
}
},
family => "Debian",
hardware => "x86_64",
name => "Ubuntu",
release => {
full => "16.04",
major => "16.04"
},
selinux => {
enabled => false
}
}
ruby => {
platform => "x86_64-linux",
sitedir => "/opt/puppetlabs/puppet/lib/ruby/site_ruby/2.1.0",
version => "2.1.9"
}
`
PuppetDB 4.2.0
can do PQL queries for facts using dot notation rather than this stuff, use that
Imagine you want to do blue/green deploys, the logic for the deployment will be identical but you want to run the 2nd group only when the first completed
To do this you'd make 2 node sets, one per blue nodes and one per green nodes. You'd invoke the task list with a different variable set
This is pretty much conceptually exactly the same as the current hook lists but I dont think I have any way to pass in a variable into a task list - so perhaps the older idea of calling another (or the same) playbook with a different set of inputs will have more legs?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.