saltstack-formulas/redis-formula

Redis state

redis-formula

Formula to install and configure redis.

Table of Contents

• General notes
• Contributing to this repo
• Special notes
• Todo
• Available states
• Testing

General notes

See the full SaltStack Formulas installation and usage instructions.

If you are interested in writing or contributing to formulas, please pay attention to the Writing Formula Section.

If you want to use this formula, please pay attention to the FORMULA file and/or git tag, which contains the currently released version. This formula is versioned according to Semantic Versioning.

See Formula Versioning Section for more details.

Contributing to this repo

Commit message formatting is significant!!

Please see How to contribute for more details.

pre-commit

pre-commit is configured for this formula, which you may optionally use to ease the steps involved in submitting your changes. First install the pre-commit package manager using the appropriate method, then run bin/install-hooks and now pre-commit will run automatically on each git commit.

$ bin/install-hooks
pre-commit installed at .git/hooks/pre-commit
pre-commit installed at .git/hooks/commit-msg

Special notes

None.

Todo

• merge redis 2.4 and 2.6 templates to generic redis.conf.jinja

Available states

• redis
• redis.clean
• redis.common
• redis.server
• redis.server.clean
• redis.sentinel

redis

Meta-state (This is a state that includes other states).

This state installs the Redis server solution (see https://redis.io)

redis.clean

Meta-state (This is a state that includes other states).

Stop Redis daemon and remove redis server solution.

redis.common

Install redis only

redis.server

Install redis and start up the service ( Ubuntu + RedHat based systems should work ).

For a list of all available options, look at: redis/files/redis-{version}.conf.jinja - also have a look at the pillar.example and map.jinja.

To use the new redis 3.2 template (more structured) add the following line to the pillar:

source_path: salt://redis/files/redis-3.2-ng.conf.jinja

For options that aren't in the template (timeout, for example in redis-2.8 template), you can use extra_opts:

extra_opts:
  timeout: 10

redis.server.clean

Meta-state (This is a state that includes other states).

Stop Redis daemon and remove redis package/archive.

redis.sentinel

Install redis sentinel. Compatible with Ubuntu 14/16 and CentOS 6/7.

Testing

Linux testing is done with kitchen-salt.

Requirements

• Ruby
• Docker

$ gem install bundler
$ bundle install
$ bin/kitchen test [platform]

Where [platform] is the platform name defined in kitchen.yml, e.g. debian-9-2019-2-py3.

bin/kitchen converge

Creates the docker instance and runs the redis main states, ready for testing.

bin/kitchen verify

Runs the inspec tests on the actual instance.

bin/kitchen destroy

Removes the docker instance.

bin/kitchen test

Runs all of the stages above in one go: i.e. destroy + converge + verify + destroy.

bin/kitchen login

Gives you SSH access to the instance for manual testing.

redis-formula's Issues

Having trouble running tests

When I attempt to run the tests based on the instructions in the readme:

bundle install
kitchen converge
kitchen verify

I get this error.

Class: Kitchen::ClientError
Message: Could not load the 'salt_solo' provisioner from the load path. Please ensure that your provisioner is installed as a gem or included in your Gemfile if using Bundler.

My version of ruby is:

ruby 2.1.5p273 (2014-11-13 revision 48405) [x86_64-darwin14.0]

Obviously it can't find salt_solo, but I was unable to find information about salt_solo and how to install it. Is that documented somewhere?

[BUG] redis_disable_transparent_huge_pages runs on every highstate and reports a change

Your setup

Formula commit hash / release tag

v0.26.2

Versions reports (master & minion)

3002.2

Pillar / config used

In salt code:
include:
- redis.server

In pillar:
redis:
disable_transparent_huge_pages: true

---

Bug details

Describe the bug

When redis:disable_transparent_huge_pages is set, the redis_disable_transparent_huge_pages cmd.run state runs and reports a change on every highstate.

Steps to reproduce the bug

Set aforementioned configuration. Run highstate,

Expected behaviour

Only report a change on the first run, when the configuration actually changes.

Attempts to fix the bug

Additional context

Support later versions of Redis

There are native configurations supported by later versions of Redis (i.e. 6) that are not reflected in any of the files/ jinja templates that I can see. In particular, the configurations that I need to have supported are related to TLS. At this time, here is the following schedule of versions and their respective release / eol dates

Version - Release date	End of Life (EOL)
7.4 – February 2024	-
7.2 – August 2023	August 31, 2025
6.4 – February 2023	February 28, 2025
6.2 – August 2021	August 31, 2024
6.0 – May 2020	May 31, 2022
5.6 – April 2020	October 31, 2021
5.4 – December 2018	December 31, 2020
5.2 – June 2018	December 31, 2019

Describe the solution you'd like

I would like to see updates for support that reflect newer versions of Redis that are out there and being used.

Describe alternatives you've considered

Writing my own formula, which I have done for numerous things in the past.

Additional context

For Reference, here is the Redis page documenting its product lifecycle:
Redis Product-Lifecycle

[BUG] Jinja2 TemplateNotFound error with using salt-ssh

Your setup

Formula commit hash / release tag

Commit hash: 90500f9 (master branch, at 2020/06/29)

Versions reports (master & minion)

salt-ssh 3001 (From the ubuntu focal repos)
salt-minion 3000.3 (From pip)

salt-ssh is run from ubuntu 20.04 desktop
salt-minion is run on a minimal 20.04 focal image for LXC containers.

Pillar / config used

Nothing relevant here. Bug relates to formula itself. When server.sls tries to render redis-3.2-ng.conf.jinja a file not found error appears relating to map.jinja

---

Bug details

Describe the bug

If you try use the redis formula with salt-ssh it fails with a file not found error (the file is there). The command

salt-ssh -i <minion> state.apply redis.server

fails with

          ID: redis_config
    Function: file.managed
        Name: /etc/redis/redis.conf
      Result: False
     Comment: Unable to manage file: Jinja error: redis/map.jinja
              Traceback (most recent call last):
                File "/var/tmp/.salt_e05380_salt/pyall/salt/utils/templates.py", line 400, in render_jinja_tmpl
                  output = template.render(**decoded_context)
                File "/var/tmp/.salt_e05380_salt/pyall/jinja2/environment.py", line 1090, in render
                  self.environment.handle_exception()
                File "/var/tmp/.salt_e05380_salt/pyall/jinja2/environment.py", line 832, in handle_exception
                  reraise(*rewrite_traceback_stack(source=source))
                File "/var/tmp/.salt_e05380_salt/pyall/jinja2/_compat.py", line 28, in reraise
                  raise value.with_traceback(tb)
                File "<template>", line 4, in top-level template code
                File "/var/tmp/.salt_e05380_salt/pyall/salt/utils/jinja.py", line 204, in get_source
                  raise TemplateNotFound(template)
              jinja2.exceptions.TemplateNotFound: redis/map.jinja

              ; line 4

              ---
              #
              # This file is managed by salt. Do not edit by hand.
              #
              {% from "redis/map.jinja" import redis_settings with context %}    <======================
              ################################## INCLUDES ###################################
              {% if redis_settings.include is defined %}
              {%- for include in redis_settings.include %}
              include {{ include }}
              {%- endfor %}
              [...]
              ---
     Started: 10:17:55.643913
    Duration: 98.964 ms
     Changes:

The "redis/map.jinja" file is there.

Steps to reproduce the bug

Just try using the redis formula with salt-ssh in any way.

Expected behaviour

Installed redis, with appropriately rendered config

Attempts to fix the bug

Some other formulas have the same issue. I cannot recall which ones, but quite a few mainstream ones do fail in the same way with the map.jinja error. The formula works fine with regular salt command

Additional context

N/A, ask and I will do my best to answer

Ability to specify an interface in 'bind'

And have it translated to interface's IP, which would allow further automation

redis-stable.tar.gz is a moving target

File sum set for file /usr/local/redis-stable.tar.gz of 12755897666792eb9e1a0b7e4589eb1cb8e229d0 does not match real sum of d944c90d87e4cf2f382506c3e155335dd31da82e

If this forumla is going to hardcode in a checksum then it can't also point at stable, but instead at a fixed version.

Typo in redis/map.jinja

--- redis-formula-orig/redis/map.jinja 2015-04-09 13:56:16.586711242 +0000
+++ redis-formula/redis/map.jinja 2015-04-09 13:56:29.994625004 +0000
@@ -17,7 +17,7 @@
'cfg_name': '/etc/redis.conf',
'cfg_version': '2.4',
'logfile': '/var/log/redis/redis.log',

•    'piddfile': '/var/run/redis.pid'
  

•    'pidfile': '/var/run/redis.pid'
  

  },
  }, merge=salt'pillar.get') %}

update-rc.d not support on rhel in server.sls

update-rc.d not support on rhel in server.sls

  cmd.wait:
    - name: update-rc.d -f redis-server remove
    - watch:
      - file: redis-init-script

Problem getting this working on CentOS

Worked like a charm for Unbuntu but I cannot get this formula to work for my CentOS boxes. Error installing "The following packages were not found, and no possible matches were found in the package db: redis-server".

I see from the formula source code that the package name should be "redis" when grains data matches OsFamily:RedHat, but it appears that the wrong package "redis-server" is being called.

I cloned the formula to a directory on my salt master /srv/fromulas/redis-formula and added that path to file_roots.

Any suggestions?

Redis config template assumes a relatively new version of redis

Many of the options listed in the file are dependent on a relatively new version of redis. Ideally options would be included or not by version and also commented out if the value is default otherwise.

Rewrite needed

This needs a rewrite.

The config files don't work for example on debian, and it's generally ugly and hard to work with. Also, it seems to assume ubuntu.

[BUG] Cannot extend ID 'redis-server-clean' in 'base:redis.server.clean'. It is not part of the high state.

salt version report

Salt Version:
          Salt: 3005.1

Dependency Versions:
          cffi: Not Installed
      cherrypy: unknown
      dateutil: 2.7.3
     docker-py: Not Installed
         gitdb: 2.0.6
     gitpython: 3.0.7
        Jinja2: 2.10.1
       libgit2: 0.28.3
      M2Crypto: Not Installed
          Mako: Not Installed
       msgpack: 0.6.2
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     pycparser: Not Installed
      pycrypto: 2.6.1
  pycryptodome: 3.6.1
        pygit2: 1.0.3
        Python: 3.8.10 (default, Jun 22 2022, 20:18:18)
  python-gnupg: 0.4.5
        PyYAML: 5.3.1
         PyZMQ: 20.0.0
         smmap: 2.0.5
       timelib: Not Installed
       Tornado: 4.5.3
           ZMQ: 4.3.2

System Versions:
          dist: ubuntu 20.04 focal
        locale: utf-8
       machine: aarch64
       release: 5.4.0-126-generic
        system: Linux
       version: Ubuntu 20.04 focal

redis state works fine

sudo salt 'app' state.sls redis
app:
----------
          ID: install-redis
    Function: pkg.installed
        Name: redis-server
      Result: True
     Comment: All specified packages are already installed
     Started: 17:41:31.159161
    Duration: 16.001 ms
     Changes:
----------
          ID: redis_config
    Function: file.managed
        Name: /etc/redis/redis.conf
      Result: True
     Comment: File /etc/redis/redis.conf is in the correct state
     Started: 17:41:31.177212
    Duration: 57.645 ms
     Changes:
----------
          ID: redis_service
    Function: service.running
        Name: redis-server
      Result: True
     Comment: The service redis-server is already running
     Started: 17:41:31.235464
    Duration: 18.743 ms
     Changes:
----------
          ID: redis_overcommit_memory
    Function: sysctl.present
        Name: vm.overcommit_memory
      Result: True
     Comment: Sysctl value vm.overcommit_memory = 1 is already set
     Started: 17:41:31.255109
    Duration: 2.953 ms
     Changes:

Summary for app
------------
Succeeded: 4
Failed:    0
------------
Total states run:     4
Total run time:  95.342 ms

state redis clean doesn't work fine

 sudo salt 'app' state.sls redis.clean

app:
    Data failed to compile:
----------
    Cannot extend ID 'redis-server-clean' in 'base:redis.server.clean'. It is not part of the high state.
This is likely due to a missing include statement or an incorrectly typed ID.
Ensure that a state with an ID of 'redis-server-clean' is available
in environment 'base' and to SLS 'redis.server.clean'
ERROR: Minions returned with non-zero exit code

show_sls

sudo salt 'app' state.show_sls redis.clean
app:
    ----------
    redis-server-clean:
        ----------
        __env__:
            base
        __sls__:
            redis.server.clean
        pkg:
            |_
              ----------
              name:
                  redis-server
            - removed
            |_
              ----------
              order:
                  10001
    redis-server-service-clean:
        ----------
        __env__:
            base
        __sls__:
            redis.server.clean
        service:
            |_
              ----------
              name:
                  redis-server
            |_
              ----------
              enable:
                  False
            |_
              ----------
              require_in:
                  |_
                    ----------
                    user:
                        redis-server-clean
                  |_
                    ----------
                    file:
                        redis-server-clean
            - dead
            |_
              ----------
              order:
                  10000

Multiple Instances

Hey,

I just wnet throw the states & pillar and it seems like it doesnt support multiple instances in it's current form. Or did I miss anything?

Thanks in advance :)

Installation of not needed init file

Usually service file comes with package and Installation not needed init file /etc/init.d/redis
is mistake and was add by 15d2516

Running redis.server I have enabled svc_state to 'running' but doesnt work

I have enabled svc_state to running. The status of the redis says that it has been restarted( from salt-master) but when I go to the instance it isnt running. I still need to run redis-server. I am using Ubuntu.

`vm.overcommit_memory` is in incorrect block in pillar

Hi,

Problem: Changing the overcommit_memory value does not overwrite the defaults.yaml value

Cause: The overcommit_memory option was added to the incorrect block (part of lookup)

Resolution: Dropping overcommit_memory out of the lookup block allows the option to work as expected

If anyone else can confirm this, I will create a Pull Request in the next few days.

db.conf configuration file

Hi,

Let me know if this could be usable for a pull request.

I added a generation of the configuration file (/etc/db/db.conf).
The file contains a machine IP address, server role (master/slave) etc..., this informations could be used by some administration script (provided by you, for example, which changing roles of master and slave instances, or doing backup etc... ).

The db.conf.jinja:

<redis>
    <user>{{ salt['pillar.get']('redis:dbconf:user', '') }}</user>
    <password>{{ salt['pillar.get']('redis:dbconf:password', '') }}</password>
    <backup>{{ salt['pillar.get']('redis:dbconf:backup', '') }}</backup>
    <master_ip>{{ salt['pillar.get']('redis:dbconf:master_ip', '') }}</master_ip>
    <master_netmask>{{ salt['pillar.get']('redis:dbconf:master_netmask', '') }}</master_netmask>
    <hardwar_ip>{{ salt['pillar.get']('redis:dbconf:hardwar_ip', '') }}</hardwar_ip>
    <hardwar_mask>{{ salt['pillar.get']('redis:dbconf:hardwar_mask', '') }}</hardwar_mask>
    <role>{{ salt['pillar.get']('redis:dbconf:role', '') }}</role>
</redis>

and you can define the variables via pillar:

  dbconf:
    user: admin666
    master_ip: 123.123.123.123

Technically, there are only a few modifications in the original formula code.

Link to "Salt Formulas installation and usage instructions" in readme is busted

The link to "Salt Formulas installation and usage instructions" in the read me is busted.

(And it would be nice to see a YAML example of how to use this in the read me)

Support for sentinel setup with redis for HA

Hi,

Any plans to add support for redis with sentinel to have a high availability redis cluster ? Or should this be in a separate formula ?

overcommit_memory

Should we add the sysctl vm.overcommit_memory=1 to this package, as described in https://redis.io/topics/faq#background-saving-is-failing-with-a-fork-error-under-linux-even-if-i39ve-a-lot-of-free-ram? It could be optional, but I've been bit by this countless times myself, and can't see a reason any Linux server running Redis wouldn't want this.

[BUG] service fails on archlinux

Versions reports (master & minion)

Pillar / config used

None

---

Bug details

Describe the bug

● redis.service - Advanced key-value store
     Loaded: loaded (/usr/lib/systemd/system/redis.service; disabled; vendor preset: disabled)
     Active: failed (Result: exit-code) since Sat 2020-12-05 20:00:58 UTC; 4min 43s ago
   Main PID: 1150 (code=exited, status=1/FAILURE)

Dec 05 20:00:58 archlinux systemd[1]: Starting Advanced key-value store...
Dec 05 20:00:58 archlinux redis-server[1150]: *** FATAL CONFIG FILE ERROR (Redis 6.0.9) ***
Dec 05 20:00:58 archlinux redis-server[1150]: Reading the configuration file, at line 19
Dec 05 20:00:58 archlinux redis-server[1150]: >>> 'logfile /var/log/redis/redis.log'
Dec 05 20:00:58 archlinux redis-server[1150]: Can't open the log file: No such file or directory
Dec 05 20:00:58 archlinux systemd[1]: redis.service: Main process exited, code=exited, status=1/FAILURE
Dec 05 20:00:58 archlinux systemd[1]: redis.service: Failed with result 'exit-code'.
Dec 05 20:00:58 archlinux systemd[1]: Failed to start Advanced key-value store.

Steps to reproduce the bug

Expected behaviour

It should work on Archlinux

Attempts to fix the bug

#86

Additional context