cdalvaro/docker-salt-master

🐳🧂 Dockerized Salt Master

Hi, I'm Carlos 👋

• 👨‍🎓 I'm a theoretical physicist who enjoys programming and contributing to open-source projects
• 👨‍💻 I work as software engineer and team lead at @Meteologica, developing and improving Meteologica's infrastructure and forecast algorithms
• 🚀 I actively maintain my docker project docker-salt-master up-to-date with every new release of saltstack/salt
• ⌨️ Some open-source projects I've contributed to are: @saltstack, @Homebrew or sameersbn/docker-gitlab
• 🌱 I'm currently learning:
  • Machine Learning: master thesis and notebooks
  • Swift programming language: hacking with Swift
  • Ruby programming language: notebooks, advent of code 2023
• 🎽 I love all kind of sports, but especially calisthenics and running
• 🎮 Beginner Fortnite player

Languages and Tools

docker-salt-master's Issues

Additional dependencies

Discussed in #233

^{Originally posted by abhi1693 March 7, 2024}
What is the best way to install additional dependencies into the container so that it persists across reboots?

As requested by @abhi1693, it's a good enhancement for this project to add support for additional dependencies.

The main idea is to be able to install python packages when the container starts before running salt-master so modules that require extra packages can fulfill their requirements.

Two approaches can be taken:

• Install dependencies providing a requirements.txt file
• Specify python packages using an environment variable

Both approaches can be implemented. Since one method can be more suitable than the other depending each configuration.

I think we can use the environment variables PYTHON_PACKAGES_FILE and PYTHON_PACKAGES to specify the requirements.txt file and a list of packages, respectively.

Allowing both systems at the same time can be confusing, so the first one will take priority over the second one. So, when the env variable PYTHON_PACKAGES_FILE is defined, PYTHON_PACKAGES will be ignored.

PYTHON_PACKAGES will be directly forwarded to the pip install command. So, as long it respects pip install format, all pip functionality will be supported.

Add support for installing saltstack-formulas

Install specified saltstack-formulas repos and make them available directly from the docker image.

Another interesting source of formulas could be the unofficial salt-formulas account.

Add label for multi-arch images

Follow the following link: https://docs.github.com/en/packages/working-with-a-github-packages-registry/working-with-the-container-registry#adding-a-description-to-multi-arch-images to properly set the description when using multi-arch images.

The 3006.3 release has /home/salt owned by root, causing crash at startup

Describe the bug
In the 3006.2 release, the /home/salt directory was owned by the salt user:

root@f6964e308ee1:/home/salt# ls -la
total 32
drwxrwxr-x 1 salt salt 4096 Aug 10 20:10 .
drwxr-xr-x 1 root root 4096 Aug 10 20:09 ..
-rw-r--r-- 1 salt salt  220 Jan  6  2022 .bash_logout
-rw-r--r-- 1 salt salt 3771 Jan  6  2022 .bashrc
drwxrwxr-x 3 salt salt 4096 Aug 10 20:10 .cache
-rw-r--r-- 1 salt salt  849 Aug 10 20:09 .profile
drwxr-xr-x 7 root root 4096 Aug 10 20:10 data
root@f6964e308ee1:/home/salt# exit

However, in the 3006.3 release, this directory is now owned by root:

root@48671de9e0fb:/home/salt# ls -la
total 28
drwxr-x--- 1 root root 4096 Sep  8 11:46 .
drwxr-xr-x 1 root root 4096 Sep  8 11:46 ..
-rw-r--r-- 1 root root  220 Jan  6  2022 .bash_logout
-rw-r--r-- 1 root root 3771 Jan  6  2022 .bashrc
-rw-r--r-- 1 root root  849 Sep  8 11:46 .profile
drwxr-xr-x 7 root root 4096 Sep  8 11:46 data

This causes the system to fail to startup, since the salt user cannot cd to /home/salt:

2023-09-14 18:03:06,255 INFO supervisord started with pid 1
2023-09-14 18:03:07,257 INFO spawned: 'salt-master' with pid 99
2023-09-14 18:03:07,259 INFO spawned: 'cron' with pid 100
supervisor: couldn't chdir to /home/salt: EACCES
supervisor: child process was not spawned
2023-09-14 18:03:07,262 INFO exited: salt-master (exit status 127; not expected)
2023-09-14 18:03:08,264 INFO spawned: 'salt-master' with pid 101
2023-09-14 18:03:08,265 INFO success: cron entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
supervisor: couldn't chdir to /home/salt: EACCES
supervisor: child process was not spawned
2023-09-14 18:03:08,269 INFO exited: salt-master (exit status 127; not expected)
2023-09-14 18:03:10,273 INFO spawned: 'salt-master' with pid 102
supervisor: couldn't chdir to /home/salt: EACCES
supervisor: child process was not spawned
2023-09-14 18:03:10,279 INFO exited: salt-master (exit status 127; not expected)
^C2023-09-14 18:03:11,280 WARN received SIGINT indicating exit request
2023-09-14 18:03:11,281 INFO waiting for salt-master, cron to die
2023-09-14 18:03:11,281 INFO stopped: cron (terminated by SIGTERM)

To Reproduce
Steps to reproduce the behavior:

1. Just run docker: docker run --rm -ti cdalvaro/docker-salt-master:3006.3

Expected behavior
salt-master starts up

Additional info
The full log with the outputted error.

[INFO] - Configuring directories ...
'/srv' -> '/home/salt/data/srv'
'/var/log/salt' -> '/home/salt/data/logs/salt'
[INFO] - Configuring logrotate ...
[INFO] - Configuring container timezone ...
[INFO] - Setting TimeZone -> UTC ...
[INFO] - Configuring salt-master service ...
[INFO] - Configuring 3rd-party salt-formulas ...
[INFO] - Setting up salt keys ...
[INFO] -  ==> Setting up master keys ...
[INFO] - Generating master keys ...
Error: Write access denied to "/home/salt/data/keys" for user "salt".
[INFO] -  ==> Setting up GPG keys ...
[INFO] - Could not find GPG keys. GPG setup skipped.
[INFO] - Setting up salt keys permissions ...
Salt Version:
          Salt: 3006.3

Python Version:
        Python: 3.10.13 (main, Sep  6 2023, 02:11:27) [GCC 11.2.0]

Dependency Versions:
          cffi: 1.14.6
      cherrypy: 18.6.1
      dateutil: 2.8.1
     docker-py: Not Installed
         gitdb: Not Installed
     gitpython: Not Installed
        Jinja2: 3.1.2
       libgit2: 1.6.4
  looseversion: 1.0.2
      M2Crypto: Not Installed
          Mako: Not Installed
       msgpack: 1.0.2
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     packaging: 22.0
     pycparser: 2.21
      pycrypto: Not Installed
  pycryptodome: 3.9.8
        pygit2: 1.12.2
  python-gnupg: 0.4.8
        PyYAML: 6.0.1
         PyZMQ: 23.2.0
        relenv: 0.13.10
         smmap: Not Installed
       timelib: 0.2.4
       Tornado: 4.5.3
           ZMQ: 4.3.4

System Versions:
          dist: ubuntu 22.04.3 jammy
        locale: utf-8
       machine: x86_64
       release: 6.5.2-arch1-1
        system: Linux
       version: Ubuntu 22.04.3 jammy

[INFO] - Starting supervisord ...
2023-09-14 18:04:53,145 INFO Included extra file "/etc/supervisor/conf.d/cron.conf" during parsing
2023-09-14 18:04:53,145 INFO Included extra file "/etc/supervisor/conf.d/salt-master.conf" during parsing
2023-09-14 18:04:53,145 INFO Set uid to user 0 succeeded
2023-09-14 18:04:53,148 INFO RPC interface 'supervisor' initialized
2023-09-14 18:04:53,148 CRIT Server 'unix_http_server' running without any HTTP authentication checking
2023-09-14 18:04:53,148 INFO supervisord started with pid 1
2023-09-14 18:04:54,150 INFO spawned: 'salt-master' with pid 99
2023-09-14 18:04:54,151 INFO spawned: 'cron' with pid 100
supervisor: couldn't chdir to /home/salt: EACCES
supervisor: child process was not spawned
2023-09-14 18:04:54,154 INFO exited: salt-master (exit status 127; not expected)
2023-09-14 18:04:55,157 INFO spawned: 'salt-master' with pid 101
2023-09-14 18:04:55,158 INFO success: cron entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
supervisor: couldn't chdir to /home/salt: EACCES
supervisor: child process was not spawned
2023-09-14 18:04:55,163 INFO exited: salt-master (exit status 127; not expected)
2023-09-14 18:04:57,167 INFO spawned: 'salt-master' with pid 102
supervisor: couldn't chdir to /home/salt: EACCES
supervisor: child process was not spawned
2023-09-14 18:04:57,173 INFO exited: salt-master (exit status 127; not expected)
2023-09-14 18:05:00,178 INFO spawned: 'salt-master' with pid 103
supervisor: couldn't chdir to /home/salt: EACCES
supervisor: child process was not spawned
2023-09-14 18:05:00,183 INFO exited: salt-master (exit status 127; not expected)
2023-09-14 18:05:01,184 INFO gave up: salt-master entered FATAL state, too many start retries too quickly

Version report (please complete the following information):

• Host OS:
  • Linux aragorn 6.5.2-arch1-1 # 1 SMP PREEMPT_DYNAMIC Wed, 06 Sep 2023 21:01:01 +0000 x86_64 GNU/Linux
  • Linux cluster-pi-3 6.1.21-v8+ #1642 SMP PREEMPT Mon Apr 3 17:24:16 BST 2023 aarch64 GNU/Linux
• Docker:
  • Docker version 24.0.5, build ced0996600
  • k3s version v1.27.3+k3s1 (fe9604ca)
• Image tag:
  • 3006.3

Additional context
Add any other context about the problem here.

Fix gitfs tests

Gitfs tests fail at their first attempt.

Permission denied on start

Describe the bug
When i try to start the container using the docker-compose file, I get permission denied errors
I also tried starting the container with this command, but same output.

docker run --name salt_stack -it --rm \
    --publish 4505:4505 --publish 4506:4506 \
    --env "PUID=$(id -u)" --env "PGID=$(id -g)" \
    --volume $(pwd)/roots/:/home/salt/data/srv/ \
    --volume $(pwd)/keys/:/home/salt/data/keys/ \
    ghcr.io/cdalvaro/docker-salt-master:latest

To Reproduce
Clone repo and do docker-compose up

Expected behavior
The container should generate keys etc and start.

Additional info

[+] Building 0.0s (0/0)                                                                                      
[+] Running 1/0
 ✔ Container salt_master  Created                                                                       0.0s 
Attaching to salt_master
salt_master  | [INFO] - Configuring directories ...
salt_master  | '/srv' -> '/home/salt/data/srv'
salt_master  | '/var/log/salt' -> '/home/salt/data/logs/salt'
salt_master  | [INFO] - Configuring logrotate ...
salt_master  | [INFO] - Configuring container timezone ...
salt_master  | [INFO] - Setting TimeZone -> Europe/Madrid ...
salt_master  | [INFO] - Configuring salt-master service ...
salt_master  | [INFO] - Configuring 3rd-party salt-formulas ...
salt_master  | [INFO] - Setting up salt keys ...
salt_master  | [INFO] -  ==> Setting up master keys ...
salt_master  | [INFO] - Generating master keys ...
salt_master  | Error: [Errno 13] Permission denied: '/home/salt/data/keys/master.pem'
salt_master  | [INFO] -  ==> Setting up GPG keys ...
salt_master  | [INFO] - Could not find GPG keys. GPG setup skipped.
salt_master  | [INFO] - Setting up salt keys permissions ...
salt_master  | chmod: changing permissions of '/home/salt/data/keys/master.pem': Permission denied
salt_master exited with code 1
salt_master  | chown: changing ownership of '/home/salt/data/keys/master.pem': Permission denied
salt_master exited with code 1
salt_master  | chown: changing ownership of '/home/salt/data/keys/master.pem': Permission denied
salt_master exited with code 1
salt_master  | chown: changing ownership of '/home/salt/data/keys/master.pem': Permission denied
salt_master exited with code 1
salt_master  | chown: changing ownership of '/home/salt/data/keys/master.pem': Permission denied

Version report (please complete the following information):

• Host OS: [e.g. uname -a]
  Darwin MBP 23.2.0 Darwin Kernel Version 23.2.0: Wed Nov 15 21:55:06 PST 2023; root:xnu-10002.61.3~2/RELEASE_ARM64_T6020 arm64
• Docker: [e.g. docker --version]
  Docker version 25.0.2, build 29cf629
• Image tag: [e.g. 3006.6]
  ghcr.io/cdalvaro/docker-salt-master:latest

Additional context
Add any other context about the problem here.

Optionally enable salt-minion

Discussed in #235

^{Originally posted by abhi1693 March 8, 2024}
I need to be able to fetch basic information about the master via the API such as version it's running. This is needed by 3rd party integrations to ensure we can begin the minion upgrade process when this version changes. One idea was to simply install a minion inside the master. I tried adding a minion.conf in the config directory with just master: localhost and it started throwing errors about missing directories.

I was wondering if this is a hidden feature of this repo so that I can simply enable this via environment variable and start managing our master too?

As requested by @abhi1693, having a salt-minion running inside the container can be useful in some circumstances. So, I'm going to add support for enabling a salt-minion instance inside the container with a basic configuration to be paired with the master instance.

The minion will be disabled by default and could be activated using the environment variable: SALT_MINION_ENABLED with value True.

Additionally, the minion will be configurable using a config directory.

I'll add more details when the development begins.

Add support for salt-api

As suggested by @jothibasu-kamaraj, salt-api support is going to be added.

SSL version error in pepper with 3001.1 image

Describe the bug
I use 3001.1 image for up a master with api. When I done I try to connect via pepper(0.7.6) installed from pip. But faced with an error:

Error with request: <urlopen error [SSL: WRONG_VERSION_NUMBER] wrong version number (_ssl.c:1108)>
<urlopen error [SSL: WRONG_VERSION_NUMBER] wrong version number (_ssl.c:1108)>
Uncaught Pepper error (increase verbosity for the full traceback).

Also api.log has errors like this:

2020-10-02 15:23:45,895 [cherrypy.error   :219 ][INFO    ][181] [02/Oct/2020:15:23:45] ENGINE Listening for SIGTERM.
2020-10-02 15:23:45,896 [cherrypy.error   :219 ][INFO    ][181] [02/Oct/2020:15:23:45] ENGINE Listening for SIGHUP.
2020-10-02 15:23:45,896 [cherrypy.error   :219 ][INFO    ][181] [02/Oct/2020:15:23:45] ENGINE Listening for SIGUSR1.
2020-10-02 15:23:45,896 [cherrypy.error   :219 ][INFO    ][181] [02/Oct/2020:15:23:45] ENGINE Bus STARTING
2020-10-02 15:23:45,896 [cherrypy.error   :219 ][INFO    ][181] [02/Oct/2020:15:23:45] ENGINE Started monitor thread '_TimeoutMonitor'.
2020-10-02 15:23:46,003 [cherrypy.error   :219 ][INFO    ][181] [02/Oct/2020:15:23:46] ENGINE Serving on https://0.0.0.0:8000
2020-10-02 15:23:46,003 [cherrypy.error   :219 ][INFO    ][181] [02/Oct/2020:15:23:46] ENGINE Bus STARTED
2020-10-02 15:23:46,005 [cherrypy.error   :219 ][ERROR   ][181] [02/Oct/2020:15:23:46] ENGINE Error in HTTPServer.tick
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/cherrypy/wsgiserver/__init__.py", line 2021, in start
    self.tick()
  File "/usr/lib/python3/dist-packages/cherrypy/wsgiserver/__init__.py", line 2090, in tick
    s, ssl_env = self.ssl_adapter.wrap(s)
  File "/usr/lib/python3/dist-packages/cherrypy/wsgiserver/ssl_builtin.py", line 66, in wrap
    s = self.context.wrap_socket(sock,do_handshake_on_connect=True,
  File "/usr/lib/python3.8/ssl.py", line 500, in wrap_socket
    return self.sslsocket_class._create(
  File "/usr/lib/python3.8/ssl.py", line 1040, in _create
    self.do_handshake()
  File "/usr/lib/python3.8/ssl.py", line 1309, in do_handshake
    self._sslobj.do_handshake()
OSError: [Errno 0] Error

So I think we need to do something about it. Doesn't work out of the box or documentation needs clarification

That's is not critical for me at all.
I will try to figure out what's going on.

Version report (please complete the following information):

• OS: [e.g. 5.4.0-47-generic]
• Docker: [Docker version 19.03.13, build 4484c46d9d]
• Image tag: [e.g. 3001.1]

Add support for configurating the reactor in master config

As per the documentation, there is a way for telling the master to sync custom types on minions' start. Please refer to: https://docs.saltproject.io/en/latest/topics/reactor/index.html#minion-start-reactor

It would be good to have a config option and directory mapping for configuring the reactor, much like other options are (mounting the keys, roots, etc).

An alternative could be enhancing the config system so a master config template (or zero or more small config files like Apache's conf.d/* files, for example) can be used. This way, future or yet unsupported options could be covered easily.

Improve support for managing keys

Instead of using /etc/salt-docker/keys as a source for master and minions keys, this directory will be used as the default pki/master directory.

/etc/salt-docker/keys directory will be able to be mounted as a Docker volume and will contain the master and minion keys.

In addition support for signed keys will be added.

Api port should be exposed by default.

As a new user I just edit compose file and bring up a container. But unfortunately forget about exposing api port in docker compose. That's why I faced with ssl error when trying to connect to api.

Should be added at:

docker-salt-master/docker-compose.yml

Line 12 in 581808b

ports:

Of course it can be added as commented row, because it is not enabled by default. But I think it should be. That's just an example.

@cdalvaro what do you think about this ?

Add support for log rotation

In order to maintain logs tidy and easy to read, it is a good idea to add support for log rotation.

_pygit2.GitError: Failed to authenticate SSH session: Unable to send userauth-publickey request

Hi.

Have you experienced this issue ?

Describe the bug
When using Gitfs2 for remote git repo, i keep getting errors.
When doing it via cli, it works fine, and i can see, that it's using the same ssh key files.

To Reproduce
Steps to reproduce the behavior:
Create new github repo
Create ssh key, without password
put the keys in the keys folder, and name them accordingly
restart container, and watcht the logs.
test using cli to clone the repo, using ssh, and verify it works.

Expected behavior
No errors in logs, and able to access the gitfs repo

Additional info
The full log with the outputted error.

Version report (please complete the following information):

• Host OS: Linux salt-statefulset-0 5.4.0-88-generic #99-Ubuntu SMP Thu Sep 23 17:29:00 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
• Docker: K8S 1.20
• Image tag: latest

Additional context

gitfs.conf

gitfs_provider: pygit2
gitfs_privkey: /home/salt/data/keys/gitfs_ssh
gitfs_pubkey: /home/salt/data/keys/gitfs_ssh.pub

gitfs_remotes:
  - [email protected]:rhjensen79/salt.git:
    - root: salt
    - base: master

question

I had to downgrade docker-compose file to 3.3. Like the file below. I'm getting connection refused for the ports and /var/log/salt/master is logging the messages below, tell me if this looks like a bug, I can report it as so (using the template). I've also noticed that, there's no configuration in /home/salt/data/config/. I've added a master.conf with default ports but no luck. Thanks for any help.

version: '3.3'
volumes:
  roots:
  keys:
  logs:
services:
  master:
    container_name: salt_master_engage1
    image: ghcr.io/cdalvaro/docker-salt-master:latest
    restart: always
    volumes:
      - "roots/:/home/salt/data/srv"
      - "keys/:/home/salt/data/keys"
      - "logs/:/home/salt/data/logs"
    ports:
      - "4505:4505"
      - "4506:4506"
      ### salt-api port
      # - "8000:8000"
    healthcheck:
      test: ["CMD", "/usr/local/sbin/healthcheck"]
      #start_period: 30s
    environment:
      DEBUG: 'false'
      TZ: America/Chicago
      PUID: 1000
      PGID: 1000
      SALT_LOG_LEVEL: info
      ### salt-api settings
      # SALT_API_SERVICE_ENABLED: 'True'
      # SALT_API_USER: salt_api
      # SALT_API_USER_PASS: 4wesome-Pass0rd

2022-10-31 10:12:57,968 [salt.modules.network:2143][ERROR   ][12952] Exception while creating a ThreadPoolExecutor for resolving FQDNs: can't start new thread
2022-10-31 10:12:57,986 [salt.modules.network:2143][ERROR   ][12951] Exception while creating a ThreadPoolExecutor for resolving FQDNs: can't start new thread
2022-10-31 10:12:57,995 [salt.modules.network:2143][ERROR   ][12950] Exception while creating a ThreadPoolExecutor for resolving FQDNs: can't start new thread
2022-10-31 10:12:57,996 [salt.modules.network:2143][ERROR   ][12949] Exception while creating a ThreadPoolExecutor for resolving FQDNs: can't start new thread
2022-10-31 10:12:58,007 [salt.modules.network:2143][ERROR   ][12948] Exception while creating a ThreadPoolExecutor for resolving FQDNs: can't start new thread
2022-10-31 10:12:58,066 [salt.utils.process:998 ][ERROR   ][13192] An un-handled exception from the multiprocessing process 'FileserverUpdate' was caught:
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/salt/utils/process.py", line 993, in wrapped_run_func
    return run_func()
  File "/usr/local/lib/python3.10/dist-packages/salt/master.py", line 508, in run
    self.update_threads[interval].start()
  File "/usr/lib/python3.10/threading.py", line 935, in start
    _start_new_thread(self._bootstrap, ())
RuntimeError: can't start new thread

Versions of `python` and `python-dev`, etc. don't necessarily match

python${PYTHON_VERSION} python3-dev libpython3-dev

You're only specifying the minor version for python, and not any of the related packages. They will instead use the default for the distribution, ignoring PYTHON_VERSION.

You either need to specify the same version for all of them (and use e.g. the deadsnakes repo), or not bother with PYTHON_VERSION and use the distro default for everything.