linux-system-roles/logging

An ansible role which configures logging.

logging's Issues

Generated variables file when setting logging_debug to true for the facts set by the logging role.

Generate a facts file with the facts that are set by the logging role and passed to the sub role, when setting logging_debug to true.

Update Rsyslog role README based on the structure changes

Update Rsyslog README based on the structure changes

[RFE] set Template, Tag, and Severity

We forward logs to graylog which requires us to set parameters not currently available in this role in order to extract and filter specific log messages in searches and use them in automated alerting rules. Sample configuration:

For now we are using this role plus another which creates additional files like these in /etc/rsyslog.d:

ruleset(name="forwardToLogServer") {
  *.* action(
    name="forwardToLogServer"
    type="omfwd"
    Target="psyslog.example.com"
    Port="514"
    Protocol="UDP"
    Template="RSYSLOG_SyslogProtocol23Format"
  )
}

input(type="imfile"
      File="/var/log/php_errors.log"
      Tag="php:"
      Severity="err"
      Ruleset="onlySendToLogServer"
)

[RFE] Rename rsyslog_viaq_prereq_packages, rsyslog_viaq_packages and rsyslog_viaq_rules to be general

I think we should rename rsyslog_viaq_prereq_packages and rsyslog_viaq_packages to be general.
Like : rsyslog_extra_prereq_packages and rsyslog_extra_packages.

rsyslog_viaq_rules - need to consider if we should rename it as well.

In oVirt use case for example I will need to update these list to keep the elasticsearch packages and rsyslog-mmnormalize but I don't need the other packages viaq uses for transformations (like 'rsyslog-mmjsonparse', 'rsyslog-mmkubernetes')

We can keep the default to fit viaq but only rename the variables.

@richm @nhosoi

missing quotes around immark module interval option

quotes are missing arround {{ logging_mark_interval }} variable in roles/rsyslog/tasks/main.yml, which leads to the following error:

[root@centos-8 /]# rsyslogd -N1
rsyslogd: version 8.1911.0-7.el8_4.2, config validation run (level 1), master config /etc/rsyslog.conf
rsyslogd: error during parsing file /etc/rsyslog.d/10-local-modules.conf, on or before line 5: invalid character '6' in object definition - is there an invalid escape sequence somewhere? [v8.1911.0-7.el8_4.2 try https://www.rsyslog.com/e/2207 ]
rsyslogd: error during parsing file /etc/rsyslog.d/10-local-modules.conf, on or before line 5: invalid character '0' in object definition - is there an invalid escape sequence somewhere? [v8.1911.0-7.el8_4.2 try https://www.rsyslog.com/e/2207 ]
rsyslogd: error during parsing file /etc/rsyslog.d/10-local-modules.conf, on or before line 5: invalid character '0' in object definition - is there an invalid escape sequence somewhere? [v8.1911.0-7.el8_4.2 try https://www.rsyslog.com/e/2207 ]
rsyslogd: error during parsing file /etc/rsyslog.d/10-local-modules.conf, on or before line 5: syntax error on token ')' [v8.1911.0-7.el8_4.2 try https://www.rsyslog.com/e/2207 ]
rsyslogd: could not interpret master config file '/etc/rsyslog.conf'. [v8.1911.0-7.el8_4.2 try https://www.rsyslog.com/e/2207 ]

[root@centos-8 rsyslog.d]# systemctl status rsyslog
● rsyslog.service - System Logging Service
   Loaded: loaded (/usr/lib/systemd/system/rsyslog.service; enabled; vendor preset: enabled)
   Active: active (running) since Wed 2021-11-03 16:49:01 UTC; 22s ago
     Docs: man:rsyslogd(8)
           https://www.rsyslog.com/doc/
 Main PID: 2283 (rsyslogd)
   CGroup: /user.slice/user-1000.slice/[email protected]/user.slice/podman-15978.scope/36a5613bf95e1ea833164b186b0b675fef23b51506fe444aa8d432bd98f1d790/system.slice/rsyslog.service
           └─2283 /usr/sbin/rsyslogd -n

Nov 03 16:49:01 centos-8 systemd[1]: Starting System Logging Service...
Nov 03 16:49:01 centos-8 rsyslogd[2283]: error during parsing file /etc/rsyslog.d/10-local-modules.conf, on or before line 5: invalid character '6' in object definition - is there an invalid escape sequence somewhere? [v8.1911.0-7.el8_4.2>
Nov 03 16:49:01 centos-8 systemd[1]: Started System Logging Service.
Nov 03 16:49:01 centos-8 rsyslogd[2283]: error during parsing file /etc/rsyslog.d/10-local-modules.conf, on or before line 5: invalid character '0' in object definition - is there an invalid escape sequence somewhere? [v8.1911.0-7.el8_4.2>
Nov 03 16:49:01 centos-8 rsyslogd[2283]: error during parsing file /etc/rsyslog.d/10-local-modules.conf, on or before line 5: invalid character '0' in object definition - is there an invalid escape sequence somewhere? [v8.1911.0-7.el8_4.2>
Nov 03 16:49:01 centos-8 rsyslogd[2283]: error during parsing file /etc/rsyslog.d/10-local-modules.conf, on or before line 5: syntax error on token ')' [v8.1911.0-7.el8_4.2 try https://www.rsyslog.com/e/2207 ]
Nov 03 16:49:01 centos-8 rsyslogd[2283]: could not interpret master config file '/etc/rsyslog.conf'. [v8.1911.0-7.el8_4.2 try https://www.rsyslog.com/e/2207 ]
Nov 03 16:49:01 centos-8 rsyslogd[2283]: [origin software="rsyslogd" swVersion="8.1911.0-7.el8_4.2" x-pid="2283" x-info="https://www.rsyslog.com"] start

How to reproduce:

- name: Testing immark
  hosts: all
  tasks:
    - name: include linux-system-roles.logging
      include_role:
        name: linux-system-roles.logging
      vars:
        logging_inputs:
          - name: system_input
            type: basics
        logging_mark: true
        logging_mark_interval: 600

Suggested fix:

                  module(load="immark" interval="{{ logging_mark_interval }}")

Setting rsyslog_group and rsyslog_user to root

Instead of setting rsyslog_group and rsyslog_user to root, can we instead set rsyslog_unprivileged to false?

@nhosoi

Indirect role recursion (due to callback) + lazy evaluation in variables prevents using vars.

Problem description:
In the top level task, set internal variables to the corresponding external ones to pass them to the included role as follows, in the top level default file, the default value is set to external_var.

vars:
  __internal_var: "{{ exteranl_var }}"
include_role:
  name: "{{ role_path }}/roles/rsyslog"

Then, in the role, it includes another level of role as follows. Please note that the deploy.yml is not in the top level.

  include_role:
    name: "{{ role_path }}/../../../"
    tasks_from: deploy.yml

In deploy.yml, __internal_var is referred. Due to the ansible's lazy variable evaluation, external_var is accessed at the first time, then it issues an error - the external_var variable is not defined when the external_var is not set in the inventory file. If it is set in the inventory file, the value is correctly evaluated and used.

For more details, see also the conversation starting with #99 (comment)

Replace double underlines(__) with only a single underline (_)

I think we should replace the double underlines(__) to have only a single underline (_).
It is not standard in the linux-system-roles. Is there any special reason this was left as is?

@richm @nhosoi @pcahyna

RELP: the port definition for client side is not honored

No matter what, the resulting port for RELP is 20514

    logging_outputs:
      - name: relp_output
        type: relp
        server_host: 10.0.138.114
        port: 6514
        tls: true
        ca_cert_src: /tmp/tmp.YOFEgWl40t/ca.pem
        cert_src: /tmp/tmp.YOFEgWl40t/client-cert.pem
        private_key_src: /tmp/tmp.YOFEgWl40t/client-key.pem
        permitted_servers:
          - hostX

Should logs formatting be specific to output type?

For now I'm assuming that we have single logs formatting for each input (logs_collection).

Do we want to have different logs formatting for each output type?
Example, have different Viaq formatting when outputting to Elasticsearch and another to a Kafka output and another for remote rsyslog, etc.

Alternatively, we can have a default formatting and if it should be different it can be passed to the role using the rsyslog specific parameter.
For example, in the viaq use case we have the "rsyslog_conf_viaq_formatting" .

I would assume that we don't need this functionality at this point, But I would like your opinion and to think how/if this will affect the logging role API.

@richm @nhosoi @pcahyna @nhosoi @tabowling

Issues during installation of rsyslog example role on fedora 28

I run the rsyslog role on Fedora28.
I run it with the example rsyslog vars.yaml.

rsyslog__enabled: true
# install example packages & config files
rsyslog__example: true
rsyslog__capabilities: [ 'network', 'remote-files', 'tls' ]
rsyslog__forward: [ '' ]
rsyslog__group: root
rsyslog__user: root

I had a few issues:

1. I was missing the python2-libselinux package
   TASK [rsyslog : Generate main rsyslog configuration] *****************************************************************************************************************************************
   fatal: [localhost]: FAILED! => {"changed": false, "checksum": "6d5d22e905ff75f31050d24cc0adb3ca18f57430", "msg": "Aborting, target uses selinux but python bindings (libselinux-python) aren't installed!"}

2. TASK [rsyslog : Install/Update required packages] ********************************************************************************************************************************************
   [DEPRECATION WARNING]: Invoking "yum" only once while using a loop via squash_actions is deprecated. Instead of using a loop to supply multiple items and specifying name: {{ item }},
   please use name: [u'{{ rsyslog__base_packages }}', u'{{ rsyslog__viaq_prereq_packages if rsyslog__viaq|bool else [] }}', u'{{ rsyslog__viaq_packages if rsyslog__viaq|bool else [] }}', u'{{ rsyslog__tls_packages if rsyslog__pki|bool else [] }}', u'{{ rsyslog__packages }}'] and remove the loop. This feature will be removed in version 2.11. Deprecation warnings can be disabled
   by setting deprecation_warnings=False in ansible.cfg.

3. TASK [rsyslog : Moving the contents of /etc/rsyslog.d to the backup dir] *********************************************************************************************************************
   [WARNING]: Consider using the unarchive module rather than running tar. If you need to use command because unarchive is insufficient you can add warn=False to this command task or set
   command_warnings=False in ansible.cfg to get rid of this message.

@richm @nhosoi

[RFE] Set default parameters based on the required logs user want to collect

In the README we address specific use cases:

1. Viaq
2. Viaq-k8s
3. Example

I think we should use the use cases above as the supported logs to collect, along with ovirt-host and ovirt-engine and custom conf and set the rsyslog and fluentd defaults based on them. So the user will not need to configure the following vars :

• rsyslog__enabled
• rsyslog__viaq
• rsyslog__capabilities
• rsyslog__group
• rsyslog__user

But only specify the required logs.

For example:
logging_logs_list: ['viaq']

Will set in the defaults file:
rsyslog__viaq: true
rsyslog__capabilities: [ 'viaq' ]
rsyslog__group: root
rsyslog__user: root

logging_logs_list: [’viaq-k8s’ ] or logging_logs_list: ['viaq', ’viaq-k8s’ ]

Will set in the defaults file:
rsyslog__viaq: true
rsyslog__capabilities: [ 'viaq',’viaq-k8s’ ]
rsyslog__group: root
rsyslog__user: root

This will allow the user to set only 1 variable instead of 4.
@richm @nhosoi @pcahyna @nkinder

[RFE] Add to each logs collection a state, default is present

Similar to what is done in the selinux role
https://github.com/linux-system-roles/selinux/blob/a19c83b94c62dd858d0a3df5081b954e070cdd8e/tasks/main.yml#L96
https://github.com/linux-system-roles/selinux/blob/a19c83b94c62dd858d0a3df5081b954e070cdd8e/selinux-playbook.yml#L9

We want to add a "state" for each logs_collection.
So we can know, when running the playbook multiple times with different logging_output_list , if to keep or remove the existent files base on the state. Default is "present" which means to keep files. If state is set to false then files will be removed.

This affects the current design API.

@pcahyna @richm

Handling custom files if removed from list

In other roles we remove the files relevant for the role in case they where removed from the logging_output_list or in case rsyslog_enabled is false.

What do we want to do if a custom file that was deployed to /etc/rsyslog.d was removed from the list or in case rsyslog_enabled is false?

@richm @nhosoi @pcahyna

[RFE] Add documentation of what logs are collected when choosing each of the optional capabilities

Add documentation on what logs are collected when selection the optional capabilities.
There are currently viaq, viaq-k8s, example rsyslog and default conf.

@nhosoi

When using rsyslog_purge_original_conf true it does not delete the sub directories

When using rsyslog_purge_original_conf true, the files are deleted from the sub-directories, but the directory itself is not deleted.

rsyslog__forward - documentation is not clear enough

When running the rsyslog role with the example vars.yaml

rsyslog__enabled: true
# install example packages & config files
rsyslog__example: true
rsyslog__capabilities: [ 'network', 'remote-files', 'tls' ]
rsyslog__forward: [ '*.info @10.10.10.1:514' ]
rsyslog__group: root
rsyslog__user: root
```

For using the rsyslog forward do we only need to set the rsyslog__forward parameter ? 

Where are the logs saved by default? This is missing from documentation.

Does it also support viaq and viaq-k8s? or are they pre set to send data only to elasticsearch?

@richm @nhosoi 

RELP vs TCP/UDP differences

I noticed that there's a different approach to configure RELP vs TCP/UDP even though it is quite similar.
E.g. different port(s) definition, different keys definition. Theoretically, there should be just a different type needed.

Actually I like the way RELP is done.

@nhosoi, I'm just curious why the difference? Is there a technical reason not to keep the same sematic?

Document supported flows and what logs are collected

Please add to documentation the supported flows.

Is this already documented anywhere?

OpenShift logs (viaq, viaq-k8s) - Default output is Viaq elasticsearch. Optional is sending to a second ES and Kibana cluster dedicated to infrastructure logs, by setting openshift_logging_use_ops to true.
Can these logs be sent to remote rsyslog as well at this point?

Rsyslog example - what are the optional outputs at this point?

Default conf - where is data collected to ?

What other outputs are currently supported?

@nhosoi @richm

[RFE] Add logs handling in case the target Elasticsearch is unavailable

Add logs handling in case the target Elasticsearch is unavailable, queuing.

@richm @nhosoi

support for rsyslog omamqp1

Need ability to set up an output using omamqp1 - https://github.com/rsyslog/rsyslog/tree/master/contrib/omamqp1
Should be added to RHEL 8.2 - https://bugzilla.redhat.com/show_bug.cgi?id=1713427

rsyslog - Don't move all confs to backup by default when rerunning rsyslog role

Currently when running rsyslog role we move all conf files to a backup dir and deploy new confs.

We may want to adopt this kind of approach: https://github.com/linux-system-roles/selinux#purge-local-modifications.

That is only purge the files if requested by the user.
This should allow rerunning of the role and have it immutable.

@richm @nhosoi @nkinder

Update logging_output_list to logging_outputs

Update logging_output_list to logging_outputs, due to ansible conventions.

Add support for multiline logs in oVirt vdsm.log

Some logs in vdsm.log include a long stack trace. Need to add handling for these logs.

Separate viaq-k8s to a new input_role

We want to create for the viaq-k8s a new input_role, so that we will not need to add a "when" condition when viaq-k8s is in the logging_logs_collections.

Set omelastocsearch variables for different use cases

@richm @nhosoi

In oVirt we have 2 use cases:

1. logs collected from files
2. metrics collected from collectd using the imtcp rsyslog input plugin.

For logging we would like to retry on failure , but for metrics no.

How should we set the following parameters?
bulkmode=
writeoperation=
bulkid=
dynbulkid=
retryfailures=
retryruleset=

I don't believe we have a message ID ... How do you add message id for logs collected from file?

I think it should be
bulkmode="on"
writeoperation="index"
bulkid=""
dynbulkid="off"
retryfailures= for logs -> "on" and for metrics -> "off"
retryruleset= for logs -> "try_es" and for metrics -> ""

**The issue with "try_es" is that it tries to set $.es_msg_id even if "bulkid" is "" ....

logging/roles/rsyslog/roles/output_roles/elasticsearch/defaults/main.yaml

Line 105 in 3bc23d0

if strlen($!es_msg_id) > 0 then {

[RFE] rsyslog - need to add a generic elasticsearch output

Comment by @richm :

actually, this elasticsearch output is specific to viaq rsyslog__conf_viaq_elasticsearch - so it should be specific to viaq - we'll have to add another elasticsearch output that is more generic in the future

Should be an "or" condition instead of "and"

logging/roles/rsyslog/tasks/main.yaml

Line 134 in a72b50f

(item.state|d() and item.state == 'absent') and

@nhosoi Hi, Should the condition here be with an "or" instead of "and"?

Default remote_log_path should use FROMHOST instead of HOSTNAME

When rsyslog receives a log from a remote host using one of the remote input modules like imfwd, imrelp, etc. the fields that identify the remote host are FROMHOST and FROMHOST-IP. The current docs/code use HOSTNAME which is the hostname of the server host. The code should use e.g.

roles/rsyslog/templates/output_remote_files.j2:  string="{{ __remote_log_path }}/msg/%FROMHOST%/%PROGRAMNAME:::secpath-replace%.log"

This will have to be changed in the README, in roles/rsyslog/templates/output_remote_files.j2, and in tests/tests_remote.yml

Why is private key required when sending metrics to Viaq?

Hi,

Why is private key required when sending metrics to Viaq?

@richm @nhosoi

[RFE] Enable logging collector service

When running the logging role it should enable the logging collector service.

[RFE] Add logging_collector parameter for setting collector as rsyslog (default) or fluentd

Add logging_collector parameter for setting collector as rsyslog (default) ot fluentd.
Add to fluentd role docs that this role is only used for backwards compatibility of ovirt 4.2.z and below.
When running the logging role in ovirt 4.2.z need to set this parameter to 'fluentd'.
For all other cases role will not require this parameter since its default is rsyslog.

Possible bug in 00-global.conf when running will example rsyslog

I running the rsyslog role on Fedora28.

I run it with the example rsyslog vars.yaml.

rsyslog__enabled: true
# install example packages & config files
rsyslog__example: true
rsyslog__capabilities: [ 'network', 'remote-files', 'tls' ]
rsyslog__forward: [ '' ]
rsyslog__group: root
rsyslog__user: root

When looking at the defaults file I see

  - filename: '00-global.conf'
    comment: 'Global options'
    options: |-
      global(
        defaultNetstreamDriver="{{ rsyslog__default_netstream_driver }}"
        workDirectory="{{ rsyslog__work_dir }}"
      {% if rsyslog__pki|bool and "tls" in rsyslog__capabilities %}
        defaultNetstreamDriverCAFile="{{ rsyslog__pki_path + '/' + rsyslog__pki_realm + '/' + rsyslog__pki_ca }}"
      {%   if rsyslog__default_driver_authmode != "anon" or "network" in rsyslog__capabilities %}
        defaultNetstreamDriverCertFile="{{ rsyslog__pki_path + '/' + rsyslog__pki_realm + '/' + rsyslog__pki_crt }}"
        defaultNetstreamDriverKeyFile="{{ rsyslog__pki_path + '/' + rsyslog__pki_realm + '/' + rsyslog__pki_key }}"
      {%   endif %}
      {% endif %}
      )

But even though "network" is in rsyslog__capabilities I dont see the lines
defaultNetstreamDriverCertFile="..."
defaultNetstreamDriverKeyFile="..."

@richm @nhosoi

Add clean up tasks and run them for all subroles

Currently we only run the roles that are in the inputs or outputs list.
This means the cleanup tasks won't run.
We need to have a separate cleanup task for each sub role and run it the input or output is not in the list.

Split rsyslog__capabilities to inputs and outputs to support multiple outputs

Split rsyslog__capabilities to outputs so we can configure each input to each output/s.

Inputs - viaq, viaq-k8s, rsyslog example, ovirt-host, ovirt-engine, custom
Outputs - P1 - Viaq, P1 - Local (File/Journal), P1 - Remote Rsyslog, P2 - Elasticsearch, P2 - Fluentd Forward, P2 - Message Queue (kafka, amqp)

@richm @nhosoi @pcahyna @nkinder

Integration with journald/auditd

are there any plans to also integrate with journald/auditd to enable remote logging and/or general configuration or would you see this as independent roles?

RELP: wrong reference to the input on server side

the following flow will create bad condition in the resulting config

   logging_flows:
      - name: flows
        inputs: [system_input, remote_input]
        outputs: [files_output]

if ($inputname == "remote_input_1" ) ..

the suffix _1 should not be there.

[RFE] Add tags to inputs based on the output name

We need to add a tag to the inputs, so we can later filter the records based on the output name and direct then to the correct output.

@pcahyna

The code path when rsyslog_in_image=true is not tested.

The issue was brought up in testing the value of rsyslog_version. Instead of checking the value itself, we should test the behaviour derived from the value. But the scenario is only available when rsyslog_in_image=true, which test environment is missing now.

For more details, please see the discussion starting with:
#91 (comment)

[RFE] Support Copying and Splitting the log stream

Copying: The ability to copy logs to multiple destinations. For example, I want to send a copy of my records both to the viaq elasticsearch and some other destination such as
splunk/kafka.

Splitting: The ability to split my log stream into different subsets and send each subset to different destinations. For example, I want to send logs from the audit subsystem to a super
secret elasticsearch, and send other non-security related logs to the regular elasticsearch, and send a copy to some other destination such as splunk/kafka.

That is, I should be able to Copy and Split at the same time.

[RFE] Update example naming to be meaningful

Currently setting collection for "example" data does not reflect what logs it actually collects.
Need to update its name to be meaningful.
Depends on: [RFE] Support Copying and Splitting the log stream #9

RELP: cert and key files should be deployed in the /etc/pki/tls folder

Currently, the cert and key files are stored in /etc/rsyslog.d.
This is not a functional issue but the files does not get the usual selinux labels. It gets syslog_conf_t which grants rsyslogd the read permissions. It is just not a good practice.

[RFE] Support for rsyslog startmsg.regex option

Our application sometimes produces a log with multi-line messages which we need to process with startmsg.regex option
e.x:

input(type="imfile"
      file="/var/log/php_errors.log"
      tag="php:"
      severity="err"
      startmsg.regex="^[[:digit:]]{4}(-[[:digit:]]{2}){2}(T)([[:digit:]]{2}:){2}[[:digit:]]{2}.[[:digit:]]{6}-[[:digit:]]{2}:[[:digit:]]{2}[[:space:]]"

)

Property-Based Filters

Are the property-based filters supported? If yes, what is the syntax for adding them to a playbook?

In an rsyslog.conf file I would use the following:
:msg, startswith, "somefilter" -/var/log/output.log

Rsyslog doc: https://www.rsyslog.com/doc/v8-stable/configuration/filters.html

For a simple rsyslog configuration basic input to 9 file outputs, role repeats same tasks over and over ( 433 tasks )

PLAY RECAP *** hostname : ok=221 changed=3 unreachable=0 failed=0 skipped=209 rescued=0 ignored=0

variables :

logging_inputs:
  - name: system
    type: basics
logging_outputs:
  - name: localfs
    type: files
    state: absent
  - name: fw_arcsight
    type: forwards
    severity: info
    target: 1.1.1.1
    udp_port: 514
  - name: console
    type: files
    facility: kernel
    path: /dev/console
  - name: secure
    type: files
    facility: authpriv
    path: /var/log/secure
  - name: messages_exclude_oracle_audit
    type: files
    severity: info
    exclude:
      - authpriv.none
      - cron.none
      - mail.none
      - local1.none
    path: /var/log/messages
  - name: mail
    type: files
    facility: mail
    path: -/var/log/maillog
  - name: cron
    type: files
    facility: cron
    path: -/var/log/cron
  - name: emergency
    type: files
    severity: emerg
    path: :omusrmsg:*
  - name: boot
    type: files
    facility: local7
    path: /var/log/boot.log
  - name: oracle_audit
    type: files
    facility: local1
    severity: warning
    path: /arcsight/audit.log

In the rsyslog README es-ops is mentioned but no explanation about it.

@nhosoi - Please add to documentation what does es-ops enabled mean.
I think it should not be part of the default installation and should be mentioned in the context of Viaq.

Include meta information

As of now, pulling this role using ansible-galaxy fails with:

 [WARNING]: - linux-system-roles-logging was NOT installed successfully: this
role does not appear to have a meta/main.yml file.

Please include a meta/main.yml file to prevent this.

When to cleanup an output

We tag each new log with its output name.
Then we create the outputs configurations in a way that the data if filtered based on the output name it was tagged with.
This allows controlling the data flow.

Currently we decided that the default of a logs_collection configuration is "present", which means that the configuration files for this logs_collection will be removed only when it is marked as "state: absent"

In case the user reruns the role with different logs_collections. There is an option that a specific output will not be in use anymore in the current run, but it is still used in previous configurations that still exists and where not removed.

How do we know if an output is not used anymore and remove the configurations for it?

One option is to decide that each run should include all needed logs_collections and if a collection is not in the logs_collection, then its configurations should be removed (implicit "state:absent").

@pcahyna @tabowling @richm @nhosoi