kriansa/ansible-bundler

Turn Ansible playbooks into executable files

📦 Ansible Bundler

Ansible Bundler embeds together a full playbook and its dependencies so you can run it it from a single binary on any* computer, having just Python as a host dependency - you don't even need Ansible! Think of it as makeself for Ansible playbooks.

The closest that Ansible provides natively for this is ansible-pull, but it requires the host to have Ansible properly installed, and you need to manage the playbook location yourself.

While playbooks were never meant to be used as standalone packages, Ansible offers tools to help on more complex deployments, such as Tower and AWX (Tower's open-source upstream project).

_{* Well, we currently only support Unix based OSes.}

Use case

Ansible is awesome. It's so powerful and flexible that we can use from server provisioning to automating mundane tasks such as bootstraping your own computer.

One thing that it lacks though is the ability to be used for simple auto scaling deployments where you just want to pull a playbook and run it easily. Currently, you need to setup Ansible, ensure you have a repository to download the files, manage the right permissions to it and then run ansible-pull. This can get harder when you have more complex playbooks with several dependencies.

Ansible Bundler makes these steps easier by having a single binary that takes care of setting up Ansible on the host and executing the playbook without having to do anything globally (such as installing ansible). You can simply pull the playbook binary and execute it right away.

Usage

Generate a new self-contained playbook:

$ bundle-playbook -f playbook.yml

Run it on the host:

$ ./playbook.run

You will need Python on the host in order to run the final executable. 👍

Advanced build

$ bundle-playbook --playbook-file=playbook.yml \
  --requirements-file=requirements.yml \
  --vars-file=vars.yml \
  --ansible-version=2.8.0 \
  --python-package=boto3 \
  --extra-deps=files

By default, all files on roles folder in the same path as the playbook.yml are automatically included. If you need more dependent files, you can specify them using --extra-deps (short -d).

Run bundle-playbook --help to get a list of all possible parameters.

Binary interface

The built playbook binary has a few options that you can use at runtime. Here are the options you can currently use:

--help            Show this help message and exit
--debug           Run the packaged bundler with verbose logging
--keep-temp       Keep extracted files into the tempfolder after finishing. This is 
                  useful for debugging purposes
-e <EXTRA_VARS>, --extra-vars=<EXTRA_VARS>
                  Set additional variables as key=value or YAML/JSON, or a filename if
                  prepended with @. You can pass this parameter multiple times. This will
                  take precedence on the variables that have been previously defined on
                  the packaged playbook.

Installation

Currently you can download and install it using the pre-built packages that are available in RPM and DEB formats on Github releases. They should work on most RHEL-based distros (CentOS, Fedora, Amazon Linux, etc) as well as on Debian-based distros (Ubuntu, Mint, etc). There's also a AUR available if you're using Arch.

If your distro is not compatible with the prebuilt packages, please refer to Building below.

Building

You will need Docker installed on your machine. When you have it installed, you can proceed installing the dependencies with:

$ make deps

This is only required once. After that you're good to go. You can currently build the package in a directory structure that you can later copy to your root filesystem. This is very useful as a base for building OS packages for most package managers such as RPM or DEB.

$ make

The output will be at build/pkg

In fact, we offer support for building deb and rpm artifacts out of the box:

$ make deb rpm

The output will be at build/dist

Contributing

Pull requests are welcome. For major changes, please open an issue first to discuss what you would like to change.

Please make sure to update tests as appropriate. For more information, please refer to Contributing.

License

This project is licensed under the BSD 3-Clause License - see the LICENSE.md file for details.

ansible-bundler's Issues

No pre-commit and subsequently minimal code enforcement

Would you like me to add a pre-commit config for this repo?

This will allow you to get a lot more particular about people's contributions with respect to linting, code standards, etc.

Examples:

https://github.com/Azure/counterfit/blob/main/.pre-commit-config.yaml
https://github.com/l50/goutils/blob/main/.pre-commit-config.yaml
https://github.com/CowDogMoo/Walls-of-Excellence/blob/main/.pre-commit-config.yaml

Cannot build/install on ubuntu 20.04

$ make deps
Using default tag: latest
latest: Pulling from skandyla/fpm
9811207f4eba: Pull complete 
aaee8a25dd44: Pull complete 
Digest: sha256:23b12f00e06774e4e5b0775ef2ac0e8d8b50565b6369b93caa62e9684089180e
Status: Downloaded newer image for skandyla/fpm:latest
docker.io/skandyla/fpm:latest

$ make deb
make: *** [Makefile:45: deb] Error 1

$ make
cp: cannot create directory 'build/pkg/etc/ansible-bundler': No such file or directory
make: *** [Makefile:33: package] Error 1

How to use inventory file

I want to use the ansible to run commands also on multiple servers.

There is a way to use an inventory file?

Like ansible-playbook --inventory <host.ini file>

Parameters in the binary file '-e <EXTRA_VARS>, --extra-vars=<EXTRA_VARS>' do not work

Played with this a bit and noticed that you list a way to override vars in the created binary file, however it does not work because you do not account for it in your case statement and in other places.

Need to be able to specify the pip index to use.

I am operating behind a proxy and have a local pip mirror which I use to install packages normally. Executing the playbook.run file causes it to try and reach out to pypi.org instead of my index. I can work around this by copying a pip.conf to the system prior to executing the binary, but that is not optimal. I would like to be able to build the pip.conf into the package. Is this possible?

RFE: Make privileged run optional

Currently when i run the binary its running as a privileged user.
however, most of the times its not necessary and it should be optional to run the binary as a privileged user.
if user would like to run under sudo permissions they can always do it by appending sudo to binary or with --become an option

"Unexpected Exception" while using digits for name tasks in role

Hi.
How to repeat:

1. Create role with ansible-galaxy
2. Create any task
3. Include task in tasks/main.yml like this:

• name: 1
  include_tasks: tasks/any_task.yml

Create bundle and run it on target host:

PLAY [localhost] *****************************************************************************************************************************************
ERROR! Unexpected Exception, this is probably a bug: argument of type 'int' is not iterable

"[DEPRECATION WARNING]: ALLOW_WORLD_READABLE_TMPFILES"

$ ./testplaybook.run 
Installing playbook Python dependencies...
Collecting ansible
  Downloading ansible-3.0.0.tar.gz (30.8 MB)
     |████████████████████████████████| 30.8 MB 13.7 MB/s 
Collecting ansible-base<2.11,>=2.10.5
  Downloading ansible-base-2.10.6.tar.gz (5.7 MB)
     |████████████████████████████████| 5.7 MB 40.6 MB/s 
Requirement already satisfied: PyYAML in /usr/lib/python3/dist-packages (from ansible-base<2.11,>=2.10.5->ansible->-r /tmp/ansible-bundle.Ofbv1/requirements.txt (line 1)) (5.3.1)
Requirement already satisfied: cryptography in /usr/lib/python3/dist-packages (from ansible-base<2.11,>=2.10.5->ansible->-r /tmp/ansible-bundle.Ofbv1/requirements.txt (line 1)) (2.8)
Requirement already satisfied: jinja2 in /usr/lib/python3/dist-packages (from ansible-base<2.11,>=2.10.5->ansible->-r /tmp/ansible-bundle.Ofbv1/requirements.txt (line 1)) (2.10.1)
Collecting packaging
  Downloading packaging-20.9-py2.py3-none-any.whl (40 kB)
     |████████████████████████████████| 40 kB 37.4 MB/s 
Collecting pyparsing>=2.0.2
  Downloading pyparsing-2.4.7-py2.py3-none-any.whl (67 kB)
     |████████████████████████████████| 67 kB 43.8 MB/s 
Building wheels for collected packages: ansible, ansible-base
  Building wheel for ansible (setup.py) ... done
  Created wheel for ansible: filename=ansible-3.0.0-py3-none-any.whl size=51827969 sha256=4c2ef3032289551e6284176972e140f2871ac624ea80ffeaaca76182b2001da0
  Stored in directory: /tmp/pip-ephem-wheel-cache-z69e3yqs/wheels/dd/a3/3b/aa47a96ff8b08c4479fd574b7de49477bdc0665339ad0a1f7d
  Building wheel for ansible-base (setup.py) ... done
  Created wheel for ansible-base: filename=ansible_base-2.10.6-py3-none-any.whl size=1871040 sha256=8adecb5759cef3c85f9c73e2bb99685073627956853e6f5562105d0389ea8bab
  Stored in directory: /tmp/pip-ephem-wheel-cache-z69e3yqs/wheels/70/90/be/131d90414ecdfbeaffaed50785401f6f393cb6ac225d6f9488
Successfully built ansible ansible-base
Installing collected packages: pyparsing, packaging, ansible-base, ansible
Successfully installed ansible-3.0.0 ansible-base-2.10.6 packaging-20.9 pyparsing-2.4.7
[DEPRECATION WARNING]: ALLOW_WORLD_READABLE_TMPFILES option, moved to a per plugin approach that is more flexible, use mostly the same config will work, but now controlled from the 
plugin itself and not using the general constant. instead. This feature will be removed from ansible-base in version 2.14. Deprecation warnings can be disabled by setting 
deprecation_warnings=False in ansible.cfg.

PLAY [localhost] **************************************************************************************************************************************************************************

TASK [Gathering Facts] ********************************************************************************************************************************************************************
ok: [localhost]

TASK [foobar] *****************************************************************************************************************************************************************************
ok: [localhost] => {
    "msg": "foobar123"
}

PLAY RECAP ********************************************************************************************************************************************************************************
localhost                  : ok=2    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

RFE: Passing extra-vars at runtime to bundled binaries

Though we have an option to pass variables as var-files argument
It kind of means that we are hardcoding variables which is as good as declaring variables inside the playbook.

I would like to see a way where the extra-vars are passed to binary-file generated.
for example :
if the following is the playbook

---
- hosts: localhost
  tasks:
    - name: test
      debug:
        msg: "{{ test_var | default('hello') }}"

and we create a binary by

bundle-playbook  test_playbook.yaml

creates:
test_playbook.run

I would like to pass extravars are runtime parameter as follows:
./test_playbook.run -e "testkey=testvalue"

env: ‘ansible-playbook’: No such file or directory

OS: Ubuntu 18.04

Steps:

0. Clean Ubuntu 18.04 and bundle my playbook to "playbook.run"
1. install python and python-pip: sudo apt-get install python python-pip
2. Run playbook.run: ./playbook.run

Output:

Installing ansible...
Collecting ansible
.... # ignore installing messages...
env: ‘ansible-playbook’: No such file or directory

Try to find out why:

0. Extract tarball attached in playbook.run to /tmp/foo
1. Change directory to /tmp/foo, then run "./run-playbook.sh"
2. The modules are installed successfully in "python-deps", but "python-deps/usr/bin" is not existed. "ansible-playbook" is installed in "python-deps/home/$USER/.local/bin", not "python-deps/usr/bin", that's why "No such file or directory" appeared.
3. If I use sudo to run: "sudo -H ./run-playbook.sh", "ansible-playbook" is installed in "python-deps/usr/local/bin"

Failed Dependencies

Server:

NAME="Red Hat Enterprise Linux Server"
VERSION="7.9 (Maipo)"
ID="rhel"
ID_LIKE="fedora"
VARIANT="Server"
VARIANT_ID="server"
VERSION_ID="7.9"

Python:

python -V
Python 3.8.6

ansible-bundler:

bundle-playbook
Ansible Bundler - v1.10.2

Error:

./batch-migrations.run --debug
[2021-08-19 10:10:37 MST] Extracting bundle contents to /tmp/ansible-bundle.4x060...
[2021-08-19 10:10:39 MST] Running entrypoint...
Installing playbook Python dependencies...
Collecting ansible (from -r /tmp/ansible-bundle.4x060/requirements.txt (line 1))
  Downloading https://files.pythonhosted.org/packages/b5/01/dd6bf3cb7d834c7493d10fa1f0720c34c7703fc9bf12c93f294def465bb0/ansible-4.4.0.tar.gz (35.4MB)
    100% |████████████████████████████████| 35.4MB 77.6MB/s 
Collecting ansible-core<2.12,>=2.11.3 (from ansible->-r /tmp/ansible-bundle.4x060/requirements.txt (line 1))
  Downloading https://files.pythonhosted.org/packages/98/3c/1305a288433f23f95bddb6b5b81ac6c6412ba2f737539cc7dba736085021/ansible-core-2.11.4.tar.gz (6.8MB)
    100% |████████████████████████████████| 6.9MB 60.1MB/s 
Collecting jinja2 (from ansible-core<2.12,>=2.11.3->ansible->-r /tmp/ansible-bundle.4x060/requirements.txt (line 1))
  Downloading https://files.pythonhosted.org/packages/80/21/ae597efc7ed8caaa43fb35062288baaf99a7d43ff0cf66452ddf47604ee6/Jinja2-3.0.1-py3-none-any.whl (133kB)
    100% |████████████████████████████████| 143kB 46.6MB/s 
Collecting PyYAML (from ansible-core<2.12,>=2.11.3->ansible->-r /tmp/ansible-bundle.4x060/requirements.txt (line 1))
  Downloading https://files.pythonhosted.org/packages/7a/5b/bc0b5ab38247bba158504a410112b6c03f153c652734ece1849749e5f518/PyYAML-5.4.1-cp36-cp36m-manylinux1_x86_64.whl (640kB)
    100% |████████████████████████████████| 645kB 73.8MB/s 
Collecting cryptography (from ansible-core<2.12,>=2.11.3->ansible->-r /tmp/ansible-bundle.4x060/requirements.txt (line 1))
  Downloading https://files.pythonhosted.org/packages/9b/77/461087a514d2e8ece1c975d8216bc03f7048e6090c5166bc34115afdaa53/cryptography-3.4.7.tar.gz (546kB)
    100% |████████████████████████████████| 552kB 74.7MB/s 
    Complete output from command python setup.py egg_info:
    
            =============================DEBUG ASSISTANCE==========================
            If you are seeing an error here please try the following to
            successfully install cryptography:
    
            Upgrade to the latest pip and try again. This will fix errors for most
            users. See: https://pip.pypa.io/en/stable/installing/#upgrading-pip
            =============================DEBUG ASSISTANCE==========================
    
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-build-5ze_t_48/cryptography/setup.py", line 14, in <module>
        from setuptools_rust import RustExtension
    ModuleNotFoundError: No module named 'setuptools_rust'
    
    ----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-build-5ze_t_48/cryptography/
Error: Python dependencies could not be installed.
[2021-08-19 11:11:19 MST] Finished, removing temp content...
[2021-08-19 11:11:19 MST] Done.

pip3 install:

pip3 install setuptools-rust                                 
Collecting setuptools-rust                                                     
  Using cached https://files.pythonhosted.org/packages/82/2b/349ad916a2f032506a2c7c0810950a299f96e05d88b21797c2170bd6b2c6/setuptools_rust-0.12.1-py3-none-any.
whl                   
Collecting semantic-version>=2.6.0 (from setuptools-rust)                                                                                                     
  Using cached https://files.pythonhosted.org/packages/a5/15/00ef3b7888a10363b7c402350eda3acf395ff05bebae312d1296e528516a/semantic_version-2.8.5-py2.py3-none-
any.whl                                                                                                                                                       
Collecting toml>=0.9.0 (from setuptools-rust)                                                                                                                 
  Using cached https://files.pythonhosted.org/packages/44/6f/7120676b6d73228c96e17f1f794d8ab046fc910d781c8d151120c3f1569e/toml-0.10.2-py2.py3-none-any.whl
Collecting setuptools>=46.1 (from setuptools-rust)                                                                                                            
  Using cached https://files.pythonhosted.org/packages/bd/25/5bdf7f1adeebd4e3fa76b2e2f045ae53ee208e40a4231ad0f0c3007e4353/setuptools-57.4.0-py3-none-any.whl
Installing collected packages: semantic-version, toml, setuptools, setuptools-rust                 
Exception:                                                                                                                                                    
Traceback (most recent call last):                                                                                                                            
  File "/usr/lib/python3.6/site-packages/pip/basecommand.py", line 215, in main 
    status = self.run(options, args)                                                                                                                          
  File "/usr/lib/python3.6/site-packages/pip/commands/install.py", line 365, in run               
    strip_file_prefix=options.strip_file_prefix,                                                                                                              
  File "/usr/lib/python3.6/site-packages/pip/req/req_set.py", line 789, in install                
    **kwargs                                                                                                                                                  
  File "/usr/lib/python3.6/site-packages/pip/req/req_install.py", line 854, in install                
    strip_file_prefix=strip_file_prefix                                                                                                                       
  File "/usr/lib/python3.6/site-packages/pip/req/req_install.py", line 1069, in move_wheel_files 
    strip_file_prefix=strip_file_prefix,                                                                                                                      
  File "/usr/lib/python3.6/site-packages/pip/wheel.py", line 345, in move_wheel_files             
    clobber(source, lib_dir, True)                                                                                                                            
  File "/usr/lib/python3.6/site-packages/pip/wheel.py", line 287, in clobber
    ensure_dir(dest)  # common for the 'include' path                     
  File "/usr/lib/python3.6/site-packages/pip/utils/__init__.py", line 83, in ensure_dir
    os.makedirs(path)                                                          
  File "/usr/lib64/python3.6/os.py", line 210, in makedirs                    
    makedirs(head, mode, exist_ok)                                                                                                                            
  File "/usr/lib64/python3.6/os.py", line 220, in makedirs               
    mkdir(name, mode)                                                          
PermissionError: [Errno 13] Permission denied: '/usr/local/lib/python3.6'

pip3 install --user

pip3 install --user setuptools-rust                      
Collecting setuptools-rust
  Using cached https://files.pythonhosted.org/packages/82/2b/349ad916a2f032506a2c7c0810950a299f96e05d88b21797c2170bd6b2c6/setuptools_rust-0.12.1-py3-none-any.
whl                
Collecting semantic-version>=2.6.0 (from setuptools-rust)
  Using cached https://files.pythonhosted.org/packages/a5/15/00ef3b7888a10363b7c402350eda3acf395ff05bebae312d1296e528516a/semantic_version-2.8.5-py2.py3-none-
any.whl
Collecting toml>=0.9.0 (from setuptools-rust)
  Using cached https://files.pythonhosted.org/packages/44/6f/7120676b6d73228c96e17f1f794d8ab046fc910d781c8d151120c3f1569e/toml-0.10.2-py2.py3-none-any.whl
Collecting setuptools>=46.1 (from setuptools-rust)
  Using cached https://files.pythonhosted.org/packages/bd/25/5bdf7f1adeebd4e3fa76b2e2f045ae53ee208e40a4231ad0f0c3007e4353/setuptools-57.4.0-py3-none-any.whl
Installing collected packages: semantic-version, toml, setuptools, setuptools-rust
Successfully installed semantic-version-2.8.5 setuptools-57.4.0 setuptools-rust-0.12.1 toml-0.10.2

Issue:

It appears that ansible-bundler is trying to use pip install without the --user switch which fails to run successfully as a normal user.

How do we pass extra-vars into the binaries created ?

I created a playbook

---
- hosts: localhost
  tasks:
    - name: test
      debug:
        msg: "{{ test_var | default('hello') }}"

I created the binary using bundle-playbook
I was wondering how to pass extra-vars into the binary ??

Bundle ansible-playbooks with all folders included ( like filter_plugins, callback_plugins, etc)

ansible playbooks often include multiple filter plugins
for example:
there can be filter_plugins, callback_plugins etc.,

here is an example playbook that includes a filter_plugin the following is the folder structure.

.
├── filter_plugins
│   ├── test_filter.py
└── test_ansible.yml

1 directory, 3 files

code for test_filter.py
The functionality of this filter is to append "something" sting to any value that is passed to filter

#!/usr/bin/env python

def test_filter(testvar):
    return "something"+str(testvar)


class FilterModule(object):
    ''' A filter to fetch ann attr from dict '''
    def filters(self):
        return {
            'test_filter': test_filter
        }

the playbook is as follows:

---
- hosts: localhost
  tasks:
    - name: test
      debug:
        msg: "{{ somevar | test_filter }}"

The following is the usage of the playbook:

[srallaba@samvaranpc ansible_dir]$ ansible-playbook -vvvv test_ansible.yml -e"somevar=testvalue"
ansible-playbook 2.4.3.0
  config file = None
  configured module search path = [u'/home/srallaba/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python2.7/site-packages/ansible
  executable location = /usr/bin/ansible-playbook
  python version = 2.7.16 (default, Apr 30 2019, 15:54:43) [GCC 9.0.1 20190312 (Red Hat 9.0.1-0.10)]
No config file found; using defaults
setting up inventory plugins
 [WARNING]: Unable to parse /etc/ansible/hosts as an inventory source

 [WARNING]: No inventory was parsed, only implicit localhost is available

 [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all'

Loading callback plugin default of type stdout, v2.0 from /usr/lib/python2.7/site-packages/ansible/plugins/callback/default.pyc

PLAYBOOK: test_ansible.yml *******************************************************************************************************************************************************************************************************************
1 plays in test_ansible.yml

PLAY [localhost] *****************************************************************************************************************************************************************************************************************************

TASK [Gathering Facts] ***********************************************************************************************************************************************************************************************************************
Using module file /usr/lib/python2.7/site-packages/ansible/modules/system/setup.py
<127.0.0.1> ESTABLISH LOCAL CONNECTION FOR USER: srallaba
<127.0.0.1> EXEC /bin/sh -c 'echo ~ && sleep 0'
<127.0.0.1> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /home/srallaba/.ansible/tmp/ansible-tmp-1566571609.56-75746724107344 `" && echo ansible-tmp-1566571609.56-75746724107344="` echo /home/srallaba/.ansible/tmp/ansible-tmp-1566571609.56-75746724107344 `" ) && sleep 0'
<127.0.0.1> PUT /tmp/tmpcu3aCc TO /home/srallaba/.ansible/tmp/ansible-tmp-1566571609.56-75746724107344/setup.py
<127.0.0.1> EXEC /bin/sh -c 'chmod u+x /home/srallaba/.ansible/tmp/ansible-tmp-1566571609.56-75746724107344/ /home/srallaba/.ansible/tmp/ansible-tmp-1566571609.56-75746724107344/setup.py && sleep 0'
<127.0.0.1> EXEC /bin/sh -c '/usr/bin/python2 /home/srallaba/.ansible/tmp/ansible-tmp-1566571609.56-75746724107344/setup.py; rm -rf "/home/srallaba/.ansible/tmp/ansible-tmp-1566571609.56-75746724107344/" > /dev/null 2>&1 && sleep 0'
ok: [localhost]
META: ran handlers

TASK [test] **********************************************************************************************************************************************************************************************************************************
task path: /home/srallaba/ansible_dir/test_ansible.yml:4
ok: [localhost] => {
    "msg": "somethingtestvalue"
}
META: ran handlers
META: ran handlers

PLAY RECAP ***********************************************************************************************************************************************************************************************************************************
localhost                  : ok=2    changed=0    unreachable=0    failed=0 

Currently when i bundle this playbook through bundle-playbook.
There is option to include folders into the bundle.
for example if we could add multiple folders(filter_plugins, callback_plugins via) to the bundle-playbook it would unlock potential of bundling customized playbooks.

example usage would be :

$ bundle-playbook --playbook-file=playbook.yml \
  --requirements-file=requirements.yml \
  --vars-file=vars.yml \
  --extra-deps=files
  --extra-folder=folder_name1 folder_name2 

This will help us including many kinds of folders/files that can be helpful in the playbook run.

Dependencies

According to the description in the README, it says that ansible-bundler bundles everything needed into a single executable and all that is needed on the target machine that will be run on is python and the bundle.

Long story short, I bundled my playbook and attempted to run it on a machine that only has python installed and it attempts to download everything required via pip, so ansible and all the dependencies listed in requirements.txt. This seems contradictory to the documentation.

Am I doing something wrong or is that how it is designed to work? Reason I'm asking, if it has to download all the modules, then it's not as portable as I was hoping. Thank you. :)