Giter Site home page Giter Site logo

mcorpc-agent-provider's Introduction

Choria mcorpc Agent Provider

This repository is now archived, code is now part of github.com/choria-io/go-choria

Go Report Card GoDoc

This is an Agent Provider for the Choria Server that provides compatability with The Marionette Collective (mcollective) Simple RPC system.

Agent Features:

  • Agent DDLs represented in memory and loaded from JSON files
  • Agents, Actions and Agent Metadata that maps to the same terminology and behavior as MCollective
  • Auditing that is compatible with the Ruby based Choria auditing plugin
  • A framework for writing new Agents in Go that can be compiled into the Choria Server (see the docs for more information on this.)
  • Agents written in Go and compiled into the Choria Server:
    • choria_util - Utilities used to interrogate version information about the server
    • rpcutil - General RPC utilities like extracting facts and statistics, compatible with MCollective as far as possible
    • discovery - Agent used to assist broadcast based discovery
  • A wrapper around the historical Ruby MCollective agent system that can run a MCollective agent in a sandbox without requiring the deprecated mcollectived daemon
  • A fully compatible Go implementation of the action policy framework. In a future Choria release, this will become the default action policy implementation.

Client Features:

  • A fully featured Go client to the MCollective RPC system that is compatible with Ruby and Go nodes
    • Broadcast discovery
    • RPC Requests with the usual features like batches, direct, broadcast and more

TODO

  • Authorization system that matches the action policy feature
  • Support limiting compiled in agents using the new Authorization feature above
  • Ability to execute ruby data plugins during discovery to enable compound filters

mcorpc-agent-provider's People

Contributors

ripienaar avatar vjanelle avatar

Stargazers

 avatar

Watchers

 avatar  avatar

mcorpc-agent-provider's Issues

Add a generic external agent provider

We need a way to implement agents as external processes. The ruby provider is kind of the general approach but it does it via a shim which starts up old mco, we dont need all that for the generic case.

We should have a thing that:

  • delivers a request to STDIN or temp file of any external process
  • receives reply on STDOUT or temp file of any external process

We kind of do this for ruby, so we can use something similar to:

https://github.com/choria-io/mcorpc-agent-provider/blob/58eedaa63fe933b2830463fc0e6b83f455cb881f/mcorpc/ruby/agent.go#L28-L46

For the request and reply is just https://github.com/choria-io/mcorpc-agent-provider/blob/58eedaa63fe933b2830463fc0e6b83f455cb881f/mcorpc/mcorpc.go#L65-L70

With an accompanying DDL and some kind of logic for where to go look for external process delivered agents we should be able to make it work.

We'll need a few other things:

  • Agent program should respond to --should-activate by exiting 0 for yes 1 for no
  • Auditing is done in the wrapper prior to calling the agent/action
  • Authorization will be supported once choria has a authorizer which it doesnt have now

Today this doesnt exist, we'd build a new agent provider plugin that delivers this.

/cc @optiz0r and @vjanelle

support callbacks before and after discovery

Clients need to know how many nodes were discovered (and indeed they can supply the targets) but we also support limiting so we should add 2 callbacks:

  • 1 called before discovery is done - not called when targets is supplied, allowing clients to print Discovering...
  • 1 called after discovery and limits, it gets passed the discovered count and limited count and can return error. When it returns error the request ends, allowing clients to print totals and adjust progress reports etc

parse ddl input sections and support validation

In order to support the new choria req CLI we need to be able to convert map[string]string as received from the CLI into the correct data types that the DDL would expect on the receiving end else we cant call anything with int inputs etc

This means a bunch of things:

  • We need to parse the input sections, today we leave as json.RawMessage
  • We need to understand the type property for all typical JSON types and support validation
  • We need to understand things like type list and string with maxlength and validate those
  • We need to understand validators like ipv4address and validate using go-validator
  • We need to be able to convert "10" into 10.0 float etc
  • We need to be able to convert map[string]string into map[string]interface{} after first turning each value into its right type

add reply formatters

the choria req command has a bunch of nasty code for formatting replies either as json or text, we should make some helpers here instead to assist with doing that and for reuse in other apps

include the agent config path in the environment

Agents can have config in the plugin.d dir but the directory isnt a fixed point - different OS, different deployment methods etc.

We should set CHORIA_EXTERNAL_CONFIG in the environment that points to a path of the agent config - which might not exist if there is no config.

Validation failures for external agent with no inputs

Summary

Using https://github.com/choria-io/go-external I created a simple choria agent that should list users with a list action (user list) that doesn't take any inputs (none are declared in the ddl or json), but when I try to invoke it via mco rpc I get an error about missing inputs:

$ mco rpc --verbose -I csqmgmt-puppetmaster02.grass.corp user list
 * [ ============================================================> ] 1 / 1
csqmgmt-puppetmaster02.grass.corp       : Validation failed: request contains inputs while none are declared in the DDL
    {}

I suspect that this is because somewhere along the line the lack of input is converted into an empty hash which is not the same as no input and thus fails validation in the mcorpc-agent-provider library.

Note that I verified that my agent generally works - the echo action which I copied from example in the README works fine within the same agent:

[alexander.hermes@csqmgmt-puppetmaster02 ~]$ mco rpc user echo -I csqmgmt-puppetmaster02.grass.corp "message=foo"

 * [ ============================================================> ] 1 / 1

csqmgmt-puppetmaster02.grass.corp        
   
   Echo: foo

Logs

choria-audit.log extract showing the two actions being submitted.

{"timestamp":"2019-11-25T02:09:45.182154+0000","request_id":"39619513748a511fa60dd547c94af119","request_time":1574647785,"caller":"choria=alexander.hermes.mcollective","sender":"csqmgmt-puppetmaster02.grass.corp","agent":"user","action":"echo","data":{"message":"foo","process_results":true}}
{"timestamp":"2019-11-25T02:09:50.854554+0000","request_id":"f7c4e19db756502191196f9903ba1f39","request_time":1574647790,"caller":"choria=alexander.hermes.mcollective","sender":"csqmgmt-puppetmaster02.grass.corp","agent":"user","action":"list","data":{"process_results":true}}

choria.log extract showing the successful echo action followed by a failed list action:

{"agent":"user","component":"server","identity":"csqmgmt-puppetmaster02.grass.corp","level":"debug","msg":"Attempting to call external agent user#echo (/opt/puppetlabs/mcollective/plugins/mcollective/agent/user) with a timeout 15","subsystem":"agents","time":"2019-11-25T10:09:45+08:00"}
{"level":"debug","msg":"Sending a broadcast message to NATS target 'mcollective.reply.csqmgmt-puppetmaster02.grass.corp.3381.1' for message 39619513748a511fa60dd547c94af119 type reply","time":"2019-11-25T10:09:45+08:00"}
{"level":"debug","msg":"Publishing 785 bytes to mcollective.reply.csqmgmt-puppetmaster02.grass.corp.3381.1","time":"2019-11-25T10:09:45+08:00"}

<snip validation stuff>


{"component":"server","identity":"csqmgmt-puppetmaster02.grass.corp","level":"debug","msg":"Matching request f7c4e19db756502191196f9903ba1f39 with agent filters '[]string{\"user\"}'","subsystem":"discovery","time":"2019-11-25T10:09:50+08:00"}
{"component":"server","identity":"csqmgmt-puppetmaster02.grass.corp","level":"debug","msg":"Handling message f7c4e19db756502191196f9903ba1f39 with timeout 15000000000","subsystem":"agents","time":"2019-11-25T10:09:50+08:00"}
{"agent":"user","component":"server","identity":"csqmgmt-puppetmaster02.grass.corp","level":"info","msg":"Handling message f7c4e19db756502191196f9903ba1f39 for user#list from choria=alexander.hermes.mcollective","subsystem":"agents","time":"2019-11-25T10:09:50+08:00"}
{"agent":"user","component":"server","identity":"csqmgmt-puppetmaster02.grass.corp","level":"debug","msg":"Attempting to call external agent user#list (/opt/puppetlabs/mcollective/plugins/mcollective/agent/user) with a timeout 15","subsystem":"agents","time":"2019-11-25T10:09:50+08:00"}
{"agent":"user","component":"server","identity":"csqmgmt-puppetmaster02.grass.corp","level":"error","msg":"Validation failed: request contains inputs while none are declared in the DDL","subsystem":"agents","time":"2019-11-25T10:09:50+08:00"}
{"level":"debug","msg":"Sending a broadcast message to NATS target 'mcollective.reply.csqmgmt-puppetmaster02.grass.corp.3412.1' for message f7c4e19db756502191196f9903ba1f39 type reply","time":"2019-11-25T10:09:50+08:00"}
{"level":"debug","msg":"Publishing 853 bytes to mcollective.reply.csqmgmt-puppetmaster02.grass.corp.3412.1","time":"2019-11-25T10:09:50+08:00"}

Details of my test setup

System

$ uname -a
Linux csqmgmt-puppetmaster02.grass.corp 3.10.0-957.21.3.el7.x86_64 #1 SMP Tue Jun 18 16:35:19 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

$ lsb_release -a
LSB Version:	:core-4.1-amd64:core-4.1-noarch
Distributor ID:	CentOS
Description:	CentOS Linux release 7.6.1810 (Core) 
Release:	7.6.1810
Codename:	Core

Agent source

See the repo at https://github.com/ExalDraen/choria-plugin-user-agent

DDL

metadata :name        => "user",
         :description => "Interact with users on a system",
         :author      => "Alexander Hermes (@ExalDraen)",
         :license     => "GNU General Public License v3.0",
         :version     => "0.1.0",
         :url         => "https://github.com/exaldraen/choria-plugin-user-agent",
         :provider    => "external",
         :timeout     => 15


action "echo", :description => "Echo text back" do
  display :ok

  input :message,
        :prompt      => "The message to send for echoing",
        :description => "This message will be echoed back",
        :type        => :string,
        :validation  => :shellsafe,
        :maxlength   => 512,
        :optional    => false




  output :message,
         :description => "Echo of the original message",
         :display_as  => "Echo",
         :type        => "string"

end

action "list", :description => "List logged-in users on a system" do
  display :ok



  output :user_list,
         :description => "List of logged-in users",
         :display_as  => "User LIst",
         :type        => "string"

end

JSON

{
  "$schema": "https://choria.io/schemas/mcorpc/ddl/v1/agent.json",
  "metadata": {
    "license": "GNU General Public License v3.0",
    "author": "Alexander Hermes (@ExalDraen)",
    "timeout": 15,
    "name": "user",
    "version": "0.1.0",
    "url": "https://github.com/exaldraen/choria-plugin-user-agent",
    "description": "Interact with users on a system",
    "provider": "external"
  },
  "actions": [
    {
      "action": "echo",
      "input": {
        "message": {
          "prompt": "The message to send for echoing",
          "description": "This message will be echoed back",
          "type": "string",
          "optional": false,
          "validation": "shellsafe",
          "maxlength": 512
        }
      },
      "output": {
        "message": {
          "description": "Echo of the original message",
          "display_as": "Echo",
          "type": "string"
        }
      },
      "display": "ok",
      "description": "Echo text back"
    },
    {
      "action": "list",
      "input": {},
      "output": {
        "user_list": {
          "description": "List of logged-in users",
          "display_as": "User LIst",
          "type": "string"
        }
      },
      "display": "ok",
      "description": "List logged-in users on a system"
    }
  ]
}

support hashes and array input types

Ruby mco would let someone type :type => Hash in the DDL and then check if the thing is a type Hash, thats quite annoying and open ended.

We should formalize and support types "Hash", "hash", "Array" and "array" and validate and convert those.

If the map[string]string being passed in has a input of either of those types try to JSON parse it and validate that the things in there is right, this would allow CLI users to type JSON in the CLI automatically

add limit related options to client

We need to support analogues of limit, limit_targets and limit_seed

Limit method can be one of random or first
When random and if limit_seed is set the random number generator is seeded with that
limit can either be a integer number of a percentage like 10%

See MCollective::RPC::Client#pick_nodes_from_discovered

This is to support choria-io/go-choria#651

support generating CSRs and storing signed certs

A new action should be added gencsr which should write a private key and csr to either dir(configfile)/ssl/{private,csr}.pem or in the configured SSL dir and returns the CSR along with the effective dir the SSL should live in.

Later in the configure action it should take an optional certificate, ca and ssldir that will then trigger a save.

The returning of the ssldir bit is so that an machine appropriate config can be generated for the manual provider

support basic aggregators

Data aggregators are little ruby plugins that live in the data stream of RPC replies and aggregate what they see, these are used in CLI to summarize output especially when only failures are shown but people still want a overview of what happened.

While super userful it turns out only a number of key ones seem to be widely used:

% grep -h aggregate /opt/puppetlabs/mcollective/plugins/mcollective/agent/*ddl| tr -s '[:blank:]'|awk -F "(" '{print $1}'|sort |uniq -c
      5  aggregate average
      5  aggregate boolean_summary
      1  aggregate nagios_states
      1  aggregate process_summary
     45  aggregate summary

Thus if we supported summary (which can also be boolean_summary) we'll hit most, possibly add average time permitting.

create a plugin factory

Today with the new plugin system each plugin needs someting like this:

package agent

import "github.com/choria-io/go-choria/plugin"

// Plugin is a choria plugin
type Plugin struct{}

// ChoriaPlugin produces the plugin for choria
func ChoriaPlugin() plugin.Pluggable {
	return &Plugin{}
}

// PluginInstance implements plugin.Pluggable
func (p *Plugin) PluginInstance() interface{} {
	return New
}

// PluginVersion implements plugin.Pluggable
func (p *Plugin) PluginVersion() string {
	return Metadata.Version
}

// PluginName implements plugin.Pluggable
func (p *Plugin) PluginName() string {
	return Metadata.Name
}

// PluginType implements plugin.Pluggable
func (p *Plugin) PluginType() plugin.Type {
	return plugin.AgentPlugin
}

This should be easy to create using a factory that takes the metadata and the New and this package should provide that factory

various json ddl conversion issues

When converting JSON ddls to ruby we have a few issues:

  • all numbers from json files are floats when handled as interface{} so we need to improve some things there to avoid panics
  • all validations are shown as :foo when its a regex, so that becomes :.+ which is bad
  • output type is always shown, sometimes as ""
  • the word aggregate should be before the names of aggregate funcs

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.