purpleidea/mgmt

Next generation distributed, event-driven, parallel config management!

mgmt: next generation config management!

About:

Mgmt is a real-time automation tool. It is familiar to existing configuration management software, but is drastically more powerful as it can allow you to build real-time, closed-loop feedback systems, in a very safe way, and with a surprisingly small amout of our mcl code. For example, the following code will ensure that your file server is set to read-only when it's friday.

import "datetime"
$is_friday = datetime.weekday(datetime.now()) == "friday"
file "/srv/files/" {
	state => $const.res.file.state.exists,
	mode => if $is_friday { # this updates the mode, the instant it changes!
		"0550"
	} else {
		"0770"
	},
}

It can run continuously, intermittently, or on-demand, and in the first case, it will guarantee that your system is always in the desired state for that instant! In this mode it can run as a decentralized cluster of agents across your network, each exchanging information with the others in real-time, to respond to your changing needs. For example, if you want to ensure that some resource runs on a maximum of two hosts in your cluster, you can specify that as well:

import "sys"
import "world"

# we'll set a few scheduling options:
$opts = struct{strategy => "rr", max => 2, ttl => 10,}

# schedule in a particular namespace with options:
$set = world.schedule("xsched", $opts)

if sys.hostname() in $set {
	# use your imagination to put something more complex right here...
	print "i got scheduled" {} # this will run on the chosen machines
}

As you add and remove hosts from the cluster, the real-time schedule function will dynamically pick up to two hosts from the available pool. These specific functions aren't intrinsic to the core design, and new ones can be easily added.

Please read on if you'd like to learn more...

Community:

Come join us in the mgmt community!

Medium	Link
Matrix	#mgmtconfig on Matrix.org
IRC	#mgmtconfig on Libera.Chat
Twitter	@mgmtconfig & #mgmtconfig
Mailing list	looking for a new home, suggestions welcome
Patreon	purpleidea on Patreon

Status:

Mgmt is a next generation automation tool. It has similarities to other tools in the configuration management space, but has a fast, modern, distributed systems approach. The project contains an engine and a language. Please have a look at an introductory video or blog post.

Mgmt is a fairly new project. It is usable today, but not yet feature complete. With your help you'll be able to influence our design and get us to 1.0 sooner! Interested users should read the quick start guide.

Documentation:

Please read, enjoy and help improve our documentation!

Documentation	Additional Notes
quick start guide	for everyone
frequently asked questions	for everyone
general documentation	for everyone
language guide	for everyone
function guide	for mgmt developers
resource guide	for mgmt developers
style guide	for mgmt developers
godoc API reference	for mgmt developers
prometheus guide	for everyone
puppet guide	for puppet sysadmins
development	for mgmt developers
videos	for everyone
blogs	for everyone

Questions:

Please ask in the community! If you have a well phrased question that might benefit others, consider asking it by sending a patch to the FAQ section. I'll merge your question, and a patch with the answer!

Get involved:

Feel free to grab one of the straightforward #mgmtlove issues if you're a first time contributor to the project or if you're unsure about what to hack on! Please get involved by working on one of these items or by suggesting something else! There are some lower priority issues and harder issues available in our TODO file. Please have a look.

Bugs:

Please set the DEBUG constant in main.go to true, and post the logs when you report the issue. Feel free to read my article on debugging golang programs.

Patches:

We'd love to have your patches! Please send them by email, or as a pull request.

On the web:

Blog posts and recorded talks about mgmt are listed here!

Happy hacking!

mgmt's Issues

is etcd a dependency?

If I am not mistaken, etcd should be a Require: of the spec file

Ctrl+C doesn't quit mgmt

igalic@levix ~/s/p/mgmt (master|✔)> ./mgmt run -f examples/graph1b.yaml 
14:50:16 main.go:65: This is: mgmt, version: 0.0.1-70-g3cf8c4a
14:50:16 main.go:66: Main: Start: 1454421016718582027
14:50:16 main.go:196: Main: Running...
14:50:16 main.go:106: Etcd: Starting...
14:50:16 etcd.go:132: Etcd: Watching...
14:50:16 configwatch.go:54: Watching: examples/graph1b.yaml
14:50:16 main.go:149: Graph: Vertices(2), Edges(1)
14:50:16 main.go:152: Graphviz: No filename given!
14:50:16 main.go:163: State: graphNil -> graphStarting
14:50:16 etcd.go:159: Etcd: Waiting 1000 ms for connection...
14:50:16 main.go:165: State: graphStarting -> graphStarted
14:50:16 file.go:340: File[file2]: Apply
14:50:16 file.go:340: File[file3]: Apply
14:50:17 etcd.go:132: Etcd: Watching...
14:50:17 etcd.go:159: Etcd: Waiting 2000 ms for connection...
14:50:19 etcd.go:132: Etcd: Watching...
14:50:19 etcd.go:159: Etcd: Waiting 4000 ms for connection...
14:50:23 etcd.go:132: Etcd: Watching...
14:50:23 etcd.go:159: Etcd: Waiting 8000 ms for connection...
14:50:31 etcd.go:132: Etcd: Watching...
14:50:31 etcd.go:159: Etcd: Waiting 16000 ms for connection...
^C14:50:46 main.go:51: Interrupted by ^C
14:50:47 etcd.go:132: Etcd: Watching...
14:50:47 etcd.go:159: Etcd: Waiting 16000 ms for connection...
^C14:51:03 etcd.go:132: Etcd: Watching...
14:51:03 etcd.go:159: Etcd: Waiting 16000 ms for connection...

virt emulator path is distro specific and unnecessary

In the virt resource the emulator path is specified implicitly which is incorrect and unnecessary for at least Fedora, Ubuntu, CentOS, and Arch (all that I had handy to check).

For example, if the domain type is kvm, libvirt will use its configured emulator for that type:

From capabilities (on arch):

      <domain type='qemu'/>
      <domain type='kvm'>
        <emulator>/usr/sbin/qemu-system-x86_64</emulator>
      </domain>

so as long as the domain type is set to kvm, it will already know to use /usr/sbin/qemu-system-x86_64.

implement noop functionality

We should implement a no-operation (noop) flag which causes a resource to not run apply.

This should be available as a metaparam (and default to false) for per resource setting, and as a --noop flag, which would set all noop values to true.

Since this should be fairly straight forward, I will tag this is an mgmtlove bug for new contributors.

Allow mgmt to run from Puppet manifests

Still on that, but no usable results yet. I had used my first PoC Ruby script to reproduce issue #4.

My initial approach of using the output of puppet master --compile cannot work. Not only does it not include actual dependency edges, it's also much too raw. I will blog about this with details.

What I will try to do instead is to rely on the agent side graph generating code, to let Puppet help with validation, munging and the computation of actual edges.

Tests (can) fail with Ruby 2.1.5

The YAML format test works by making Ruby parse the YAML, then re-serialize the result. However, on my machine (Debian 8) the output of the YAML module is a little arbitrary. Observe:

 irb(main):009:0> puts YAML.dump( { :all => { :content => "This\nis\nstupid.\n", :path => "/this/is/not" } } )

---
:all:
  :content: |
    This
    is
    stupid.
  :path: "/this/is/not"
=> nil
irb(main):010:0> puts YAML.dump( { :all => { :content => "This\n", :path => "/this/is/not" } } )

---
:all:
  :content: 'This

'
  :path: "/this/is/not"
=> nil
irb(main):011:0> puts YAML.dump( { :all => { :content => "This is demo\n", :path => "/this/is/not" } } )

---
:all:
  :content: 'This is demo

'
  :path: "/this/is/not"
=> nil
irb(main):012:0> puts YAML.dump( { :all => { :content => "This is demo\n\n", :path => "/this/is/not" } } )

---
:all:
  :content: "This is demo\n\n"
  :path: "/this/is/not"
=> nil
irb(main):013:0> puts YAML.dump( { :all => { :content => "This is demo\nWhat\n", :path => "/this/is/not" } } )

---
:all:
  :content: |
    This is demo
    What
  :path: "/this/is/not"
=> nil
irb(main):014:0> RUBY_VERSION
=> "2.1.5"

Specifically, the strings in the example graphs are almost always represented in quoted form:

$ ruby -e "require 'yaml'; puts YAML.load_file('$i').to_yaml.each_line.map(&:rstrip).join(10.chr)+10.chr" 2>/dev/null

---
graph: mygraph
resources:
  file:
  - name: file1
    meta:
      autoedge: true
    path: "/tmp/foo/bar/f1"
    content: 'i am f1

'
    state: exists
  - name: file2
    meta:
      autoedge: true
    path: "/tmp/foo/"
    content: 'i am f2

'
    state: exists
edges: []

Seems to me that Ruby is not a safe way to run this test in this particular form. But then, I'm not even sure this is a very useful test to have, honestly. I'd rather see unit or integration tests that make sure that mgmt behaves correctly given the respective graph.

implement a systemd-timers resource ("cron")

We should probably call this cron, because we already have a "timer" resource, but we could rename that one if someone has a suggestion.

file resource needs improvements

These are the remaining non-obvious issues with the File resource after closing #13 and after 598c746

1) Should this API still be used?

$content - for files, this is a string with the contents of the file
$content - for directories, this is the path to an existing directory, the structure/contents of which we'll want to copy in
$content - for directories, if this is an empty string, it means an empty directory

My original notes in #13 mentioned this, but I forgot about it when I wrote the code. If it's still logical, we should change it, and remove the use of Source which I used for dirs.

2) Are nested file resources an issue?

As pointed out by @ffrank What if we have two File resources (including dirs) where one is nested inside the other? In particular when the nested member is either a file or another directory. Similarly, the same question could apply if a nested directory itself contains another nested file or directory...

Well, after some consideration I think the answer should be as follows:

We limit the CheckApply() and also the Watch() functions to only work on a subset of their recursive contents if indeed they have recursive == true, and also if there is another file resource found to be contained within them. To get this information, the BaseRes will have to grow a method to either return all the file resources, OR more generally to return all other resources. I think I prefer the later. It can then loop through this list and find which paths are subsets.

Is this still the right approach? It needs to be investigated. The autoedges API now exists, so it might be worth considering that it could be helpful.

3) The recurse variable isn't respected for some deletions. This is a simple enhancement that should be added.

I used os.RemoveAll to keep things simple. We'll want to do this recursively with os.Remove, so that we can add a recurseMax (or similarly named) parameter.

/cc @ffrank @witlessbird

make fails to find the version or sversion if not built from a git clone

When build from the tarball:
make build VERSION=0.0.3 SVERSION=0.0.3 PROGRAM=mgmt
make fails to set the version and sversion in the Makefile because the := operation fails to assign the variable since the $(shell ...) command errors.

file resource needs improvements

The file resource can (currently) only set file contents for a single file. It can't make directories or do anything fancy yet. That's okay, because the remaining parts are left as an introductory patch for a new golang hacker. Here's the design, people working on this code are welcome to suggest alternatives:

All the work should probably happen in Apply() and StateOK():
https://github.com/purpleidea/mgmt/blob/master/file.go#L339
NOTE: i'm considering changing the API to merge the two functions-- comments welcome. It might be preferable to delay this patch if you're not willing to rebase your work. Such is a pre 1.x release.

The resource offers up a list of file properties, and does "the right thing":
$name - if ends in / then it's a directory, otherwise it's a file
$content - for files, this is a string with the contents of the file
$content - for directories, this is the path to an existing directory, the structure/contents of which we'll want to copy in
$content - for directories, if this is an empty string, it means an empty directory
$recursive - if resource would require changing more than a single thing, this must be true, otherwise error.
we'll have to reconsider if this is necessary or not. only applies to directories.
$force - if resource would require a switch from file to directory or vice versa, then force must be true. otherwise it's an error. it can probably be thought of as the -f flag on rm.

Lastly (and can be a separate patch) the Watch() function will need support for watching recursively into the directories. Adding fanotify support ( https://github.com/go-fsnotify/fsnotify/issues/114 ) would also be welcome in addition.

Comments welcome by anyone who is going to hack on this!

mgmt name

I think we should look for a better name -- and I have an idea.

I think that instead of focusing on one name, we should focus on multiple names -- flavors.

What I mean is that if we only ship one flavor, then everyone will need e.g libvirt installed. That is a huge requirement.

My proposal is to find a name that we can decline in multiple flavour, so we can ship multiple binaries with different names.

e.g.:

MusicMgmt

Pick your flavour:

Name	Package	File	Service	Livbvirt	Augeas
Bagpipe	X	X	X
Paixiao	X	X	X		X
Bandura	X	X
Kadlong	X	X	X	X	X

ofc music is just an example.

Regression: deadlock in file handling

mgmt can get stuck managing a file resource.

How to reproduce:

1. Run the file1 example

   mkdir /tmp/mgmt
   mgmt run --tmp-prefix --file examples/file1.yaml
   

2. Remove /tmp/mgmt/f3

3. Try and Ctrl-C out of mgmt.

It does work if all files existed anyway. In this case, removing f3 twice in a row will cause the lock-up. (Or something along those lines.)

VERSION:
   0.0.4-30-g9368c7e

$ git submodule status
 e2d51961dd4abe6540c7d8e4aac0da7addf2e16e vendor/github.com/coreos/etcd (v3.1.0-alpha.1-13-ge2d5196)
 acebe0f9ff5993e130b141ee60e83e592839ca22 vendor/github.com/grpc-ecosystem/grpc-gateway (v1.1.0-13-gacebe0f)
 231b4cfea0e79843053a33f5fe90bd4d84b23cd3 vendor/google.golang.org/grpc (v1.0.0-183-g231b4cf)
 7be54206639f256967dd82fa767397ba5f8f48f5 vendor/gopkg.in/fsnotify.v1 (v1.2.1)

can't build mgmt.static on debian (golang docker image with libvirt-dev)

I was trying to generate .deb packages for mgmt, and I could get this. Unfortunately the binary in this package is not mgmt.static but is the dynamically linked version. This is because on debian, I don't think there is a static version of libvirt.

I'm using Docker to build mgmt before packaging it. I'm using the following Dockerfile as a build image:

FROM golang
RUN export DEBIAN_FRONTEND=noninteractive; \
  apt-get update && apt-get install -y --no-install-recommends \
  libvirt-dev \
  && rm -rf /var/lib/apt/lists/*

But the make mgmt.static fails with:

Jan 24 23:02:43 perrin fpmbot2[31881]: + GOPATH=/src/gopath2 make PROGRAM=mgmt mgmt.static
Jan 24 23:02:44 perrin fpmbot2[31881]: Building: mgmt.static, version: 0.0.8-19-g74435aa...
Jan 24 23:02:44 perrin fpmbot2[31881]: go generate
Jan 24 23:02:45 perrin fpmbot2[31881]: go build -a -installsuffix cgo -tags netgo -ldflags '-extldflags "-static" -X main.program=mgmt -X main.version=0.0.8-19-g74435aa' -o mgmt.static;

Jan 24 23:04:41 perrin fpmbot2[31881]: # _/src
Jan 24 23:04:41 perrin fpmbot2[31881]: /usr/local/go/pkg/tool/linux_amd64/link: running gcc failed: exit status 1
Jan 24 23:04:41 perrin fpmbot2[31881]: /usr/bin/ld: cannot find -lvirt-qemu
Jan 24 23:04:41 perrin fpmbot2[31881]: /usr/bin/ld: cannot find -lvirt
Jan 24 23:04:41 perrin fpmbot2[31881]: /tmp/go-link-207154733/000004.o: In function `_cgo_a5acef59ed3f_Cfunc_dlopen':
Jan 24 23:04:41 perrin fpmbot2[31881]: /src/gopath2/src/github.com/coreos/pkg/dlopen/dlopen.go:100: warning: Using 'dlopen' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
Jan 24 23:04:41 perrin fpmbot2[31881]: /tmp/go-link-207154733/000000.o: In function `mygetgrouplist':
Jan 24 23:04:41 perrin fpmbot2[31881]: /usr/local/go/src/os/user/getgrouplist_unix.go:15: warning: Using 'getgrouplist' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
Jan 24 23:04:41 perrin fpmbot2[31881]: /tmp/go-link-207154733/000000.o: In function `mygetgrgid_r':
Jan 24 23:04:41 perrin fpmbot2[31881]: /usr/local/go/src/os/user/lookup_unix.go:38: warning: Using 'getgrgid_r' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
Jan 24 23:04:41 perrin fpmbot2[31881]: /tmp/go-link-207154733/000000.o: In function `mygetgrnam_r':
Jan 24 23:04:41 perrin fpmbot2[31881]: /usr/local/go/src/os/user/lookup_unix.go:43: warning: Using 'getgrnam_r' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
Jan 24 23:04:41 perrin fpmbot2[31881]: /tmp/go-link-207154733/000000.o: In function `mygetpwnam_r':
Jan 24 23:04:41 perrin fpmbot2[31881]: /usr/local/go/src/os/user/lookup_unix.go:33: warning: Using 'getpwnam_r' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
Jan 24 23:04:41 perrin fpmbot2[31881]: /tmp/go-link-207154733/000000.o: In function `mygetpwuid_r':
Jan 24 23:04:41 perrin fpmbot2[31881]: /usr/local/go/src/os/user/lookup_unix.go:28: warning: Using 'getpwuid_r' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
Jan 24 23:04:41 perrin fpmbot2[31881]: collect2: error: ld returned 1 exit status
Jan 24 23:04:41 perrin fpmbot2[31881]: Makefile:114: recipe for target 'mgmt.static' failed
Jan 24 23:04:41 perrin fpmbot2[31881]: make: *** [mgmt.static] Error 2

(make mgmt works).

To build this, I'm using fpm combined with a tool I wrote, fpmbot that I execute inside the mgmt checkout with fpmbuild -config ../mgmt.yaml -f -o /tmp -t deb using the following mgmt.yaml:

git: https://github.com/purpleidea/mgmt.git
clean: -fdx
ref: master
fpm:
- -s
- dir
- mgmt=/usr/bin/mgmt
- misc/mgmt.service=/usr/lib/systemd/system/mgmt.service
build:
  prepare: |
    mkdir -p gopath2/src/github.com/purpleidea;
    ln -s ../../../.. gopath2/src/github.com/purpleidea/mgmt;
    GOPATH=$PWD/gopath2 go get github.com/purpleidea/mgmt
  build: |
    GOPATH=$PWD/gopath2 make PROGRAM=mgmt mgmt
  install: |
    true
env:
  docker:
    Dockerfile: |
      FROM golang
      RUN export DEBIAN_FRONTEND=noninteractive; \
        apt-get update && apt-get install -y --no-install-recommends \
        libvirt-dev \
        && rm -rf /var/lib/apt/lists/*

[feat] Clean up on resource removal

Hi,

I was wondering if this solves some of the problems that chef doesn't.

For example, let's say I configure this to install package A and make sure that is run in one node. Then, let's say I remove that part of the "recipe" and just install package B (or just use a whole new "recipe" and stop using the other) and then run it again. Will A still be there?

Or, is it possible (like with chef) to modify a "recipe" in a way that an already running node will run ok but it will fail on a new starting from scratch node? Or is there some way to make this impossible?

Installing mgmt without having libvirt installed yields spurious error messages

I did a go get... but forgot to install libvirt first. The result was this:

[I] 11:32:18 ~ $ go get -u github.com/purpleidea/mgmt
# pkg-config --cflags libvirt libvirt libvirt libvirt libvirt libvirt libvirt libvirt libvirt libvirt libvirt libvirt libvirt libvirt libvirt libvirt libvirt libvirt libvirt libvirt libvirt libvirt libvirt libvirt libvirt libvirt libvirt libvirt libvirt libvirt libvirt libvirt libvirt libvirt libvirt
Package libvirt was not found in the pkg-config search path.
Perhaps you should add the directory containing `libvirt.pc'
to the PKG_CONFIG_PATH environment variable
No package 'libvirt' found
Package libvirt was not found in the pkg-config search path.
Perhaps you should add the directory containing `libvirt.pc'
to the PKG_CONFIG_PATH environment variable
No package 'libvirt' found
Package libvirt was not found in the pkg-config search path.
Perhaps you should add the directory containing `libvirt.pc'
to the PKG_CONFIG_PATH environment variable
No package 'libvirt' found
Package libvirt was not found in the pkg-config search path.
Perhaps you should add the directory containing `libvirt.pc'
to the PKG_CONFIG_PATH environment variable
No package 'libvirt' found
Package libvirt was not found in the pkg-config search path.
Perhaps you should add the directory containing `libvirt.pc'
to the PKG_CONFIG_PATH environment variable
No package 'libvirt' found
Package libvirt was not found in the pkg-config search path.
Perhaps you should add the directory containing `libvirt.pc'
to the PKG_CONFIG_PATH environment variable
No package 'libvirt' found
Package libvirt was not found in the pkg-config search path.
Perhaps you should add the directory containing `libvirt.pc'
to the PKG_CONFIG_PATH environment variable
No package 'libvirt' found
Package libvirt was not found in the pkg-config search path.
Perhaps you should add the directory containing `libvirt.pc'
to the PKG_CONFIG_PATH environment variable
No package 'libvirt' found
Package libvirt was not found in the pkg-config search path.
Perhaps you should add the directory containing `libvirt.pc'
to the PKG_CONFIG_PATH environment variable
No package 'libvirt' found
Package libvirt was not found in the pkg-config search path.
Perhaps you should add the directory containing `libvirt.pc'
to the PKG_CONFIG_PATH environment variable
No package 'libvirt' found
Package libvirt was not found in the pkg-config search path.
Perhaps you should add the directory containing `libvirt.pc'
to the PKG_CONFIG_PATH environment variable
No package 'libvirt' found
Package libvirt was not found in the pkg-config search path.
Perhaps you should add the directory containing `libvirt.pc'
to the PKG_CONFIG_PATH environment variable
No package 'libvirt' found
Package libvirt was not found in the pkg-config search path.
Perhaps you should add the directory containing `libvirt.pc'
to the PKG_CONFIG_PATH environment variable
No package 'libvirt' found
Package libvirt was not found in the pkg-config search path.
Perhaps you should add the directory containing `libvirt.pc'
to the PKG_CONFIG_PATH environment variable
No package 'libvirt' found
Package libvirt was not found in the pkg-config search path.
Perhaps you should add the directory containing `libvirt.pc'
to the PKG_CONFIG_PATH environment variable
No package 'libvirt' found
Package libvirt was not found in the pkg-config search path.
Perhaps you should add the directory containing `libvirt.pc'
to the PKG_CONFIG_PATH environment variable
No package 'libvirt' found
Package libvirt was not found in the pkg-config search path.
Perhaps you should add the directory containing `libvirt.pc'
to the PKG_CONFIG_PATH environment variable
No package 'libvirt' found
Package libvirt was not found in the pkg-config search path.
Perhaps you should add the directory containing `libvirt.pc'
to the PKG_CONFIG_PATH environment variable
No package 'libvirt' found
Package libvirt was not found in the pkg-config search path.
Perhaps you should add the directory containing `libvirt.pc'
to the PKG_CONFIG_PATH environment variable
No package 'libvirt' found
Package libvirt was not found in the pkg-config search path.
Perhaps you should add the directory containing `libvirt.pc'
to the PKG_CONFIG_PATH environment variable
No package 'libvirt' found
Package libvirt was not found in the pkg-config search path.
Perhaps you should add the directory containing `libvirt.pc'
to the PKG_CONFIG_PATH environment variable
No package 'libvirt' found
Package libvirt was not found in the pkg-config search path.
Perhaps you should add the directory containing `libvirt.pc'
to the PKG_CONFIG_PATH environment variable
No package 'libvirt' found
Package libvirt was not found in the pkg-config search path.
Perhaps you should add the directory containing `libvirt.pc'
to the PKG_CONFIG_PATH environment variable
No package 'libvirt' found
Package libvirt was not found in the pkg-config search path.
Perhaps you should add the directory containing `libvirt.pc'
to the PKG_CONFIG_PATH environment variable
No package 'libvirt' found
Package libvirt was not found in the pkg-config search path.
Perhaps you should add the directory containing `libvirt.pc'
to the PKG_CONFIG_PATH environment variable
No package 'libvirt' found
Package libvirt was not found in the pkg-config search path.
Perhaps you should add the directory containing `libvirt.pc'
to the PKG_CONFIG_PATH environment variable
No package 'libvirt' found
Package libvirt was not found in the pkg-config search path.
Perhaps you should add the directory containing `libvirt.pc'
to the PKG_CONFIG_PATH environment variable
No package 'libvirt' found
Package libvirt was not found in the pkg-config search path.
Perhaps you should add the directory containing `libvirt.pc'
to the PKG_CONFIG_PATH environment variable
No package 'libvirt' found
Package libvirt was not found in the pkg-config search path.
Perhaps you should add the directory containing `libvirt.pc'
to the PKG_CONFIG_PATH environment variable
No package 'libvirt' found
Package libvirt was not found in the pkg-config search path.
Perhaps you should add the directory containing `libvirt.pc'
to the PKG_CONFIG_PATH environment variable
No package 'libvirt' found
Package libvirt was not found in the pkg-config search path.
Perhaps you should add the directory containing `libvirt.pc'
to the PKG_CONFIG_PATH environment variable
No package 'libvirt' found
Package libvirt was not found in the pkg-config search path.
Perhaps you should add the directory containing `libvirt.pc'
to the PKG_CONFIG_PATH environment variable
No package 'libvirt' found
Package libvirt was not found in the pkg-config search path.
Perhaps you should add the directory containing `libvirt.pc'
to the PKG_CONFIG_PATH environment variable
No package 'libvirt' found
Package libvirt was not found in the pkg-config search path.
Perhaps you should add the directory containing `libvirt.pc'
to the PKG_CONFIG_PATH environment variable
No package 'libvirt' found
Package libvirt was not found in the pkg-config search path.
Perhaps you should add the directory containing `libvirt.pc'
to the PKG_CONFIG_PATH environment variable
No package 'libvirt' found
pkg-config: exit status 1

The amount of error messages seems to stem from the fact that we end up with # pkg-config --cflags libvirt libvirt libvirt libvirt libvirt libvirt libvirt libvirt libvirt libvirt libvirt libvirt libvirt libvirt libvirt libvirt libvirt libvirt libvirt libvirt libvirt libvirt libvirt libvirt libvirt libvirt libvirt libvirt libvirt libvirt libvirt libvirt libvirt libvirt libvirt. That's a bit much of libvirt flags...

I'm not sure if this is under our control and/or if we can tone that down a bit as then you'd just get a single error message and be done.

It would be nice to be able to go get mgmt without that operation blowing up on not having libvirt installed though. I realise that's probably impossible though since the libvirt package is just a set of bindings to the C library and this problem essentially stems from upstream.

Build doesn't check for old golang

I tried to build mgmt using the default golang in debian jessie, which I later realised is 1.3.3. It would be helpful for the build scripts to check the golang version and exit with a useful message if it's too old.

mgmt needs a systemd service file

Make a systemd service file for mgmt. Create it in misc/ folder. Add it to RPM spec file. Submit as a patch :)

This should be a fairly straightforward patch for a beginner. Hence #mgmtlove tag.

System package dependencies in installation documentation

I tried installing mgmt by following the quickstart guide on an Ubuntu 16.04 LTS machine. I get the following error:

➜  ~ go get -u github.com/purpleidea/mgmt
# github.com/rgbkrk/libvirt-go
dev/go/src/github.com/rgbkrk/libvirt-go/cfuncs.go:6:29: fatal error: libvirt/libvirt.h: No such file or directory
compilation terminated.

After running sudo apt-get install libvirt-dev this message disappears, but when attempting go get ./... I get arrors about pcap.h and systemd/sd-journal.h. After sudo apt-get install libpcap-dev libsystemd-dev these errors went away.

Should these dependencies be listed in the docs, or are they so system dependent that the user will have to figure this out for their system?

nspawn terminate may hang without failing

systemd-machined can sometimes receive the terminate command but not stop nspawn. It will not allow another attempt to terminate the machine for 90 seconds and gives no signal regarding this situation.

hostname resource should allow system changes

We now have a hostname resource! d039006 w00t!

However, because the hostname is a special value used as a UUID by mgmt, it might be useful if we could transactionally update our etcd entries to reflect the changes and avoid having to re-populate etcd.

Think about this and figure out how to plumb it in, and what would break without it. Perhaps it makes sense to trigger a restart or a network reload when it happens?

TBD...

ENOSPC in file.go

We talked about this, but because the question is in the source:

        } else if err == syscall.ENOSPC {
            // XXX: i sometimes see: no space left on device
            // XXX: why causes this to happen ?
            log.Printf("Strange file[%v] error: %+v\n", obj.Name, err.Error) // 0x408da0
            log.Fatal(err)
        } else {

The ENOSPC error happens when the system runs out of available inotify watches, a limit that is controlled by the fs.inotify.max_user_watches sysctl (which defaults to 8192). I run backup software that uses inotify for filesystem monitoring, and in these environments I often crank that value way up:

sysctl -w fs.inotify.max_user_watches=819200

Automatic build

Would it be possible to publish automated builds generated by travis on GitHub Releases ?

https://docs.travis-ci.com/user/deployment/releases/

This would allow us to download the binary for easy deploy on servers.

Declarative DSL

Consider a DSL that is similar to make with explicitly set defaults to map to fqdn's (via import/whatever).
This way targets could be handled by an external program of some sort that 'schedules' hosts (e.g host with least latency to $someplace start http server).

Rename "mgmtmain" to libmgmt or lib?

I suck at naming, so I'd like your advice. I'm considering renaming "mgmtmain" (which I think is a bad name) to either "libmgmt", or most likely "lib".

https://github.com/purpleidea/mgmt/tree/master/mgmtmain

This is the package which gets imported and used by the main stub,

https://github.com/purpleidea/mgmt/blob/master/main.go#L24

and also is what is imported if we use mgmt as a library.

https://github.com/purpleidea/mgmt/tree/master/examples/lib

So I'd love to hear your comments, suggestions, etc... If you have any alternate ideas, please LMK ASAP!

/cc @ffrank @vinzenz @joejulian @jumanjiman @tuxmea @BinaryTiger

Thanks!

Make some resources optional

Some systems do not have all the libraries required to run mgmt. For example, CoreOS doesn't have libvirt. Have the ability to build mgmt without the virt resource would allow mgmt to run on CoreOS.

create a timer resource

We could have a timer resource in mgmt!

The idea would be to basically simulate some of the timer stuff I wrote in puppet, but more elegantly of course:
https://ttboj.wordpress.com/2012/11/14/setting-timed-events-in-puppet/

The timer resource could trigger after N seconds, and in addition could support different algorithms for incrementing, eg: linear, exponential, etc... so that you can trigger timers in different ways.

You might want to have different options about when to reset the timer, eg: on recompile, or not, and so on...

Experiment, and have a little fun! I hope someone gets to this before I do, because it should be a pleasure to write!

How is deltaDepth supposed to be computed?

There seems to be a bug in the logic that computes the deltaDepth value in FileRes.Watch().

Consider these four lines. To put it into words: if the path that raised an event has the currently focused path as a prefix, then subtract the length of the event path from that prefix. Otherwise, if the current focus has the event path as a prefix, subtract the length of the focus path from its prefix.

In both cases, the length of the longer path is subtracted. Each will yield a value below zero. So what's going on? I suppose this is a bug, but my mind boggles whenever I try and trace its implications. Halp!

moving the directory where a file belongs breaks mgmt

if we move the parent directory of a file mgmt watches, mgmt will still watch the old file.

implement a libvirt resource

Here is some background: https://www.redhat.com/archives/libvir-list/2016-April/msg00169.html

This looks like one of the most promising libraries to use, but alternative suggestions are encouraged: https://github.com/rgbkrk/libvirt-go ("encouraged", because the amount of analysis I've done is minimal).

Let me know if you're interested. This is most definitely an intermediate level project, but suitable for someone not very familiar with mgmt internals.

mgmt on Mesos?

The DAG picture shown on your blog looks very similar to a Hadoop/Spark DAG. Will it be reasonable if mgmt becomes a framework running on Mesos?

nspawn terminate may throw a temporary error causing CheckApply to fail

The machine1.Manage.Terminate method may throw a "Resource deadlock avoided" error. This error is transitory and should cause the termination to be retried. Since it's, instead, returned as an error from CheckApply, the resource fails permanently.

implement an http resource

We should have an http resource. Alternate names for it welcome. I considered calling it "rest", but I think http is best.

It should probably support long-polling to watch. It could use the Poll metaparam.

It should store the output as an Output var to be used by send/recv https://ttboj.wordpress.com/2016/12/07/sendrecv-in-mgmt/ so that it can be put into a file.

It could probably take a map[string]string as input (eg: a query string). It should probably support at least GET and POST.

Correct state determination could be based on some sort of matching of the results. Eg: status: 200, or value == success. More ideas needed here.

In some ways, it could be thought of as the "exec" of network resources, and in a positive manner. This is meant to replace traditional exec { "wget foo > /var/bar": } patterns that puppet users are familiar with.

It will probably get used in composite resources to implement higher level resources that act over an API.

Comments welcome!

(Disclaimer: Updated since original posting!)

filter managed attributes

There are use cases where one does not want to manage all attributes.
e.g. create a file in case that it is missing, manage user and mode but don't change the content when the file is there.
This could be achieved by filtering attributes on resource level making "mask" a new meta parameter.
In discussion with @purpleidea

subscription_grouping feature

Hello.
I would like to propose grouping mechanism for a exported-collected 'resources'.
In my opinion, it must be looks like 'subscriptions' for the mgmt-instance.
Each instance might be run with optional parameter '--subscription'. If parameter is empty the subscription == 'default'
Each 'exported resource' from client comes to clients subscription(if other subscription(s) is not specified in resource declaration).

After it, 'collect' works in reverse way. It is not breaking change for your examples/graph3c.yaml but add more flexibility to mgmt.

If you have interest about it, I can write code and send PR to you.

Feel free to discus about it. ;)

DEPRECATED Action signature

mgmt finishes with the following deprecation warning:
DEPRECATED Action signature. Must be cli.ActionFunc. This is an error in the application. Please contact the distributor of this application if this is not you. See https://github.com/codegangsta/cli/blob/master/CHANGELOG.md#deprecated-cli-app-action-signature

[LOVE] package not found on centos 7.x

cli used:
sudo ./mgmt run --file examples/pkg1.yaml --tmp-prefix

15:55:14 pkg.go:268: Pkg[powertop]: CheckApply(true)
15:55:16 pgraph.go:738: Pkg[powertop]: CheckApply errored: The FilterState method failed with: Can't find package named 'powertop'..
15:55:16 pgraph.go:881: Pkg[powertop]: Exited with failure: The FilterState method failed with: Can't find package named 'powertop'..

pkcon works:
[vagrant@testem01z1 mgmt] sudo pkcon install powertop
Resolving [=========================]
Querying [=========================]
Testing changes [=========================]
Finished [=========================]
Installing [=========================]
Testing changes [=========================]
Installing packages [=========================]
Finished [=========================]

run pkg resource file get the error 'Can't run PackagesToPackageIDs: Resolve error'

on my CentOS 6.8 , my pkg resource file

---
graph: mygraph
resources:
  pkg:
  - name: git
    state: installed
edges: []

and run ./mgmt run --file examples/pkg1.yaml
get the error

[root@master mgmt]#  ./mgmt run --file examples/pkg1.yaml
02:02:39 main.go:73: This is: mgmt, version: 0.0.4-28-gf09db49-dirty
02:02:39 main.go:74: Main: Start: 1474999359789282662
02:02:39 main.go:173: Main: Working prefix is: /var/lib/mgmt/
02:02:39 main.go:196: Main: Seeds: No seeds specified!
02:02:39 etcd.go:387: Etcd: Bootstrapping...
02:02:39 etcd.go:1390: Etcd: Nominated: master.example.net=http://127.0.0.1:2380
02:02:39 etcd.go:1405: Etcd: StartServer(newCluster: true): master.example.net=http://127.0.0.1:2380
02:02:39 etcd.go:1655: Etcd: StartServer: Starting server...
02:02:39 etcd.go:302: Etcd: Connect: Endpoints: []
02:02:39 etcd.go:316: Etcd: Connect: CtxError...
02:02:39 etcd.go:585: Etcd: CtxError: Reason: CtxDelayErr(1s): No endpoints available yet!
02:02:40 etcd.go:1662: Etcd: StartServer: Done starting server!
02:02:40 etcd.go:1672: Etcd: StartServer: Server running...
02:02:40 etcd.go:1424: Etcd: Addresses are: http://127.0.0.1:2379
02:02:40 etcd.go:300: Etcd: Connect: Endpoints: master.example.net=http://127.0.0.1:2379
02:02:40 etcd.go:2248: Etcd: ApplyDeltaEvents: Event(PUT): master.example.net
02:02:40 etcd.go:1390: Etcd: Nominated: master.example.net=http://127.0.0.1:2380
02:02:40 etcd.go:399: Etcd: Startup: Volunteering...
02:02:40 etcd.go:1585: Etcd: Ideal cluster size is now: 5
02:02:40 etcd.go:946: Etcd: Set(/_mgmt/idealClusterSize): &{cluster_id:2037210783374497686 member_id:13195394291058371180 revision:80 raft_term:15  <nil>}
02:02:40 main.go:378: Main: Running...
02:02:40 main.go:250: Etcd: Starting...
02:02:40 etcd.go:1190: Etcd: Members: List: [master.example.net]
02:02:40 main.go:254: Main: Waiting...
02:02:40 etcd.go:1213: Etcd: Leader: master.example.net
02:02:40 etcd.go:1232: Etcd: Volunteers: [master.example.net]
02:02:40 etcd.go:1236: Etcd: Quitters: []
02:02:40 etcd.go:1248: Etcd: Candidates: []
02:02:40 etcd.go:1190: Etcd: Members: List: [master.example.net]
02:02:40 etcd.go:1213: Etcd: Leader: master.example.net
02:02:40 etcd.go:1232: Etcd: Volunteers: [master.example.net]
02:02:40 etcd.go:1236: Etcd: Quitters: []
02:02:40 etcd.go:1248: Etcd: Candidates: []
02:02:40 autoedge.go:71: Compile: Adding AutoEdges...
02:02:40 autogroup.go:346: Compile: Grouping: Algorithm: nonReachabilityGrouper...
02:02:40 main.go:321: Graph: Vertices(1), Edges(0)
02:02:40 main.go:324: Graphviz: No filename given!
02:02:40 pgraph.go:864: State: 0 -> 1
02:02:40 pgraph.go:913: State: 1 -> 2
02:02:40 main.go:254: Main: Waiting...
02:02:40 pkg.go:268: Pkg[git]: CheckApply(true)
02:02:40 pgraph.go:738: Pkg[git]: CheckApply errored: The pkgMappingHelper failed with: Can't run PackagesToPackageIDs: Resolve error: The name org.freedesktop.PackageKit was not provided by any .service files.
02:02:40 pgraph.go:881: Pkg[git]: Exited with failure: The pkgMappingHelper failed with: Can't run PackagesToPackageIDs: Resolve error: The name org.freedesktop.PackageKit was not provided by any .service files.
^C02:04:08 main.go:61: Interrupted by ^C

Cannot build latest master

Using go1.5.4, this is what I'm getting:

Building: mgmt.static, version: 0.0.3-61-ga1ed034...
go generate
stringer: checking package: etcd.go:119:15: OpOption not declared by package etcd
event.go:20: running "stringer": exit status 1
Makefile:78: recipe for target 'mgmt.static' failed
make: *** [mgmt.static] Error 1

Add DHT for file storage and config deploys

Design

Mgmt usually runs as a distributed system. As a result there is no central file store or single location to pull the configuration (DSL) or data files from. Here's how we plan to do this in mgmt.
(NOTE: this is all subject to discussion, and constructive comments are definitely welcome!)

Each mgmt agent will embed a bittorrent client and server. We'll use DHT to keep things distributed.

1. When we want to push a new configuration, the sysadmin uses the mgmt tool containing the auth mechanism (currently un-described/unimplemented) to sign a new config/file "blob" and push it to any host in the cluster. The cluster then uses torrent to replicate this throughout the mesh of mgmt peers.

1a) Whether the initial push to the first host is done via torrent itself is undecided.
1b) Whether torrent is always running, or starts up on demand based on a message set in etcd is also undecided.
1c) Whether DHT functions relatively independently from the etcd cluster, or uses it to bootstrap some configuration is always undecided, although I suspect using etcd less is a smart decision as it keeps things more independent.

2. For accessing files (potentially even very large ones) which would typically come from a fileserver (eg: the puppet fileserver). Instead, these get stored in torrent. This allows us to:

• avoid requiring a central server
• distribute large amounts of data quickly
• keep a certain replica count easily (ensure N hosts have a copy)

If our torrent client can selectively download only certain files, then this means we don't have to have every file replicated everywhere. Whether this means we create a single torrent for each file, or if we have a single torrent with groups of things will be up for discussion.

Prior art

Some of the inspiration for this idea came from Twitter's use of a similar mechanism called "murder": https://blog.twitter.com/2010/murder-fast-datacenter-code-deploys-using-bittorrent Unfortunately, since we want to have everything in golang this can't be re-used.

Existing libs

A quick search for golang torrent/dht libs was done a while back. I found:

https://github.com/nictuku/dht
https://github.com/jackpal/Taipei-Torrent

to be the most promising. It might make sense to use something completely different. This is definitely up for discussion. There's also (although my earlier notes skipped over it):

https://github.com/anacrolix/torrent

Bootstrapping

It might be sane to use etcd for initial discovery of peers, and for the alert that a new torrent file is available. We might push a magnet link up to etcd which peers are watching for to declare that a new config is available.

Implementation notes

All the torrent plumbing should probably exist inside a new mgmt package named torrent or dht or similar. This package should get used inside mgmtmain/main.go to plumb everything in to our Main function. Lastly, the flags needed should be passed into cli.go

Other thoughts

This will eventually make way for deploy scenarios where we might want to make sure that all of the new code and configuration hits everyone in the cluster before we switch over to the new config, or instead roll out as soon as possible, or have a staged roll out where we do 10% at a time, and so on... None of this needs to be done in the initial implementation, but it's something worth mentioning here anyways.

As always, please discuss specifics in #mgmtconfig on Freenode IRC, and put the notes about consensus here.

File resource on OS X fails

When managing a file which is not yet available:

• on Linux watcher.Add(current) returns no Error.
• on OS X it returns (*os.PathError)

https://github.com/purpleidea/mgmt/blob/master/file.go#L142

Switch to upstream virt bindings?

Daniel Berrange published this:

https://www.berrange.com/posts/2016/12/15/announce-new-libvirt-project-go-language-bindings/

Should we switch? Who wants to write the patch?

Cheers

implement a network resource

Managing a network interface is a fundamental part of managing any system. We should provide a native interface to do so. Some ideas:

• It might be worth basing this on systemd-networkd. (It looks like a safe, stable base!)
• Since mgmt works in a decentralized manner, so transient network "blips" shouldn't disrupt it.
• During CheckApply() if we don't transition to an acceptable state, we could revert and error.
• With the powerful Resource (Res) API, other resources will be able to add automatic edges to this network resource in case they require connectivity. Yet another reason this is an important primitive to write.
• I'd focus on getting simple networking working initially, with support for bridges or other systemd-networkd features as an exciting second step.

Patches welcome! Please feel free to make yourself known if you'd like to work on this

help does not show a list of commands

According to the help text:

$ mgmt --help
NAME:
   mgmt - next generation config management

USAGE:
   mgmt [global options] command [command options] [arguments...]

VERSION:
   0.0.3

COMMANDS:
     run, r   run
     help, h  Shows a list of commands or help for one command

GLOBAL OPTIONS:
   --help, -h     show help
   --version, -v  print the version

So my expectation is that mgmt help will show me a list of commands. Instead it shows the exact same help text.

mgmt version 0.0.3

implement a systemd-nspawn resource

title says it all. help wanted :)

package resource bugs

mgmt does not error when there is an unsupported resource it its yaml file

moving a file does not release the inotify watch

File watches can become excessive?

Currently tinkering with the inotify code, I have not yet grasped all that is going on. I am marveling at the inclusion of base directories, however. There is a conspicuous comment that suggests that this can lead to unexpected behavior.

For example, running the current file example, and removing /tmp/mgmt/f1 leads to the following debug output.

02:11:29 file.go:160: File[file1]: Watch(/tmp/mgmt/f1), Event(/tmp/mgmt/f1): 4
02:11:29 resources.go:361: File[file1]: Process()
02:11:29 resources.go:212: File[file1]: State: resStateWatching -> resStateEvent
02:11:29 resources.go:371: File[file1]: OKTimestamp(1459123861358615846)
02:11:29 resources.go:212: File[file1]: State: resStateEvent -> resStateCheckApply
02:11:29 file.go:325: File[file1]: CheckApply(true)
02:11:29 file.go:363: File[file1]: Apply
02:11:29 resources.go:381: File[file1]: CheckApply(): false, <nil>
02:11:29 resources.go:212: File[file1]: State: resStateCheckApply -> resStatePoking
02:11:29 resources.go:261: File[file1]: Poke: File[file2]
02:11:29 file.go:133: File[file1]: Watching: /tmp/mgmt
02:11:29 file.go:160: File[file4]: Watch(/tmp/mgmt), Event(/tmp/mgmt/f1): 4
02:11:29 resources.go:361: File[file2]: Process()
02:11:29 resources.go:212: File[file2]: State: resStateWatching -> resStateEvent
02:11:29 resources.go:240: File[file2]: OKTimestamp: (1459123861359371820) >= File[file1](1459123889979722631): !false
02:11:29 resources.go:371: File[file2]: OKTimestamp(1459123861359371820)
02:11:29 resources.go:212: File[file2]: State: resStateEvent -> resStateCheckApply
02:11:29 file.go:325: File[file2]: CheckApply(true)
02:11:29 resources.go:381: File[file2]: CheckApply(): true, <nil>
02:11:29 resources.go:212: File[file2]: State: resStateCheckApply -> resStatePoking
02:11:29 resources.go:261: File[file2]: Poke: File[file3]
02:11:29 file.go:133: File[file2]: Watching: /tmp/mgmt/f2
02:11:29 resources.go:212: File[file2]: State: resStatePoking -> resStateWatching
02:11:29 resources.go:361: File[file3]: Process()
02:11:29 resources.go:212: File[file3]: State: resStateWatching -> resStateEvent
02:11:29 resources.go:240: File[file3]: OKTimestamp: (1459123861359855622) >= File[file2](1459123889980055223): !false
02:11:29 resources.go:371: File[file3]: OKTimestamp(1459123861359855622)
02:11:29 resources.go:212: File[file3]: State: resStateEvent -> resStateCheckApply
02:11:29 file.go:325: File[file3]: CheckApply(true)
02:11:29 resources.go:381: File[file3]: CheckApply(): true, <nil>
02:11:29 resources.go:212: File[file3]: State: resStateCheckApply -> resStatePoking
02:11:29 file.go:133: File[file3]: Watching: /tmp/mgmt/f3
02:11:29 resources.go:212: File[file3]: State: resStatePoking -> resStateWatching
02:11:29 file.go:133: File[file4]: Watching: /tmp/mgmt
02:11:29 resources.go:212: File[file4]: State: resStateWatching -> resStateWatching
02:11:29 file.go:160: File[file4]: Watch(/tmp/mgmt), Event(/tmp/mgmt/f1): 1
02:11:29 file.go:133: File[file4]: Watching: /tmp/mgmt
02:11:29 resources.go:212: File[file4]: State: resStateWatching -> resStateWatching
02:11:29 resources.go:212: File[file1]: State: resStatePoking -> resStateWatching
02:11:29 file.go:160: File[file4]: Watch(/tmp/mgmt), Event(/tmp/mgmt/f1): 2
02:11:29 file.go:133: File[file4]: Watching: /tmp/mgmt
02:11:29 resources.go:212: File[file4]: State: resStateWatching -> resStateWatchin

Then, a touch /tmp/mgmt/x3 (not managed) yields

02:12:22 file.go:160: File[file4]: Watch(/tmp/mgmt), Event(/tmp/mgmt/x3): 1
02:12:22 file.go:133: File[file4]: Watching: /tmp/mgmt
02:12:22 resources.go:212: File[file4]: State: resStateWatching -> resStateWatching
02:12:22 file.go:160: File[file4]: Watch(/tmp/mgmt), Event(/tmp/mgmt/x3): 16
02:12:22 file.go:133: File[file4]: Watching: /tmp/mgmt
02:12:22 resources.go:212: File[file4]: State: resStateWatching -> resStateWatching
02:12:22 file.go:160: File[file1]: Watch(/tmp/mgmt), Event(/tmp/mgmt/x3): 1
02:12:22 file.go:133: File[file1]: Watching: /tmp/mgmt
02:12:22 resources.go:212: File[file1]: State: resStateWatching -> resStateWatching
02:12:22 file.go:160: File[file1]: Watch(/tmp/mgmt), Event(/tmp/mgmt/x3): 16
02:12:22 file.go:133: File[file1]: Watching: /tmp/mgmt
02:12:22 resources.go:212: File[file1]: State: resStateWatching -> resStateWatching

So I guess we're even firing events for unrelated files in the same directory.

What's the idea behind adding directories to the watch in the first place?

pkg resource should support disabling autostart on included services

For some reason, debian/ubuntu style systems automatically start daemons after their package is installed. redhat style systems do not. We believe waiting is the sane approach, because it give you the chance to configure your service before it gets exposed to the wild, and also helps avoid other issues such as the mysql problem. [1]

Starting and stopping services is our job as configuration management, it shouldn't be up to the package manager. Luckily, we could disable this autostart per service in our pkg resource! Disabling autostart should be the default. We should add such an option and call it autostart.

mgmt/resources/pkg.go

Line 36 in 81daf10

// PkgRes is a package resource for packagekit.

This guide explains how to disabling each service:

https://major.io/2016/05/05/preventing-ubuntu-16-04-starting-daemons-package-installed/

In particular one commenter @joejulian mentions:

If you *do* for some reason want deb-systemd-invoke to start or restart a specific service, test $1 against the systemd unit name. "exit 104" for services you do want deb-systemd-invoke to manage.

#!/bin/bash
[[ "$1" = "salt-minion.service ]] && exit 104
exit 101

This should be a mostly straight forward patch, except it might need slightly more work to add a basic OS type detection lib as a pre-requisite. Perhaps an existing lib could be found and used.

One small trick will be to make sure the policy-rc.d exception file lists all exempt services. This can be done by storing each item individually in a third folder /var/mgmt/autostart/*.autostart, and calling out from that one policy-rc.d to see which items are exempt.

Comments, alternative designs, and in particular: patches, are welcome! Let us know here if you want to work on this or need some help starting!

[1] https://github.com/purpleidea/mgmt/blob/master/docs/documentation.md#controlling-autoedges

add lxc support to virt resource

libvirt can manage lxc containers with the url lxc:/// which does not work with the virt resource.