kwoodson/ansible-role-yedit

Ansible Role for Yaml editing

yedit repository

• Ansible Role: Yedit
• Install
• Examples
• Development
• Documentation

Ansible Role: Yedit

This repository contains an ansible module for modifying yaml files.

I didn’t see a good method of editing yaml files and config managing them through ansible. This is my attempt.

Install

You can install via Ansible Galaxy:

$ ansible-galaxy install kwoodson.yedit

If you do this, you should also add a requirements.yml so other users of your playbook know what dependencies to install:

---
- src: kwoodson.yedit

You can then reference it in a play by importing it before use:

  roles:
    - kwoodson.yedit
    - role-that-uses-yedit

Examples

Sometimes it is necessary to config manage .yml files.

- hosts: localhost
  gather_facts: no
  roles:
  - kwoodson.yedit
  tasks:
  - name: manage yaml files
    yedit:
      src: /tmp/test.yaml
      key: a.b.c
      value:
        d:
          e:
            f:
              this is a test

  - name: get a specific value
    yedit:
      src: /tmp/test.yaml
      state: list
      key: a.b.c.d.e.f
    register: yeditout
  - debug: var=yeditout

Development

As this is a role, just copy it into any roles directory recognized by Ansible. For details, see Ansible documentation:

• Embedding Modules and Plugins In Roles

• module_utils

Documentation

Full documentation is available inline here.

ansible-role-yedit's Issues

Preserve comments when editing yaml

When editing a yaml all of its comments are removed. It should be possible to preserve it.

key validation failure

Having the following test.yaml file

:port: 1234
:host: 127.0.0.1

The following task will not update the :host, silently.

---
- name: update host
  yedit:
    src: 'test.yml'
    state: present
    key: ':host'
    value: '0.0.0.0'

[test append to top level] fails on macos

The last test is currently broken and I am not sure why:

TASK [test append to top level] **********************************************************************************************************************************************************************************
An exception occurred during task execution. To see the full traceback, use -vvv. The error was: __main__.YeditException: Error adding to object at path: [1]
fatal: [localhost]: FAILED! => {
    "changed": false,
    "rc": 1
}

MSG:

MODULE FAILURE
See stdout/stderr for the exact error


MODULE_STDERR:

Traceback (most recent call last):
  File "<stdin>", line 113, in <module>
  File "<stdin>", line 105, in _ansiballz_main
  File "<stdin>", line 48, in invoke_module
  File "/var/folders/3q/pc1jcyjj3qqbj6_x7n4kbtqw0000gn/T/ansible_yedit_payload_eHOw09/__main__.py", line 1022, in <module>
  File "/var/folders/3q/pc1jcyjj3qqbj6_x7n4kbtqw0000gn/T/ansible_yedit_payload_eHOw09/__main__.py", line 1014, in main
  File "/var/folders/3q/pc1jcyjj3qqbj6_x7n4kbtqw0000gn/T/ansible_yedit_payload_eHOw09/__main__.py", line 931, in run_ansible
  File "/var/folders/3q/pc1jcyjj3qqbj6_x7n4kbtqw0000gn/T/ansible_yedit_payload_eHOw09/__main__.py", line 833, in process_edits
  File "/var/folders/3q/pc1jcyjj3qqbj6_x7n4kbtqw0000gn/T/ansible_yedit_payload_eHOw09/__main__.py", line 719, in put
  File "/var/folders/3q/pc1jcyjj3qqbj6_x7n4kbtqw0000gn/T/ansible_yedit_payload_eHOw09/__main__.py", line 385, in add_entry
__main__.YeditException: Error adding to object at path: [1]

Supplying a dict as value to a key/value yedit-call

On my system with ansible 2.1.1.0 the following playbook generates - in my opinion - not the expected yml.

Playbook

- hosts: localhost
  gather_facts: no
  tasks:
  - name: manage yaml files
    yedit:
      src: /tmp/test1.yaml
      key: a
      value:
        d:
          this is a test

This playbook is a simplified example from the projects README-file.

Generated yml

a: '{''d'': ''this is a test''}'

Note that there is no yml-dictionary within a here. There is just a string containing a JSON/YML-dictionary.

Expected yml

a:
  d: this is a test

Is this the correct behavior?

Better support of comments

Take this file.yml:

variables:
  var1: VAR1

#This is another definition
other:
  var2: VAR2

Use this task:

- name: add variable
  yedit:
    src: file.yml
    key: variables.var3
    value: VAR3

You end up with file.yml:

variables:
  var1: VAR1

#This is another definition
  var3: VAR3
other:
  var2: VAR2

It's correct in a functional point of view but ugly. It would be better to have:

variables:
  var1: VAR1
  var3: VAR3

#This is another definition
other:
  var2: VAR2

Error with string value beginning from "!" sign

I use ansible 2.7.9 (Ansible AWX 4.0.0.0), latest yedit.
I found that yedit cannot parse the string if it begins from "!" sign, for example:

somerole/tasks/main.yml

- name: import kwoodson.yedit YAML editor
  import_role:
    name: ansible-role-yedit

- name: Update environment
  yedit:
    src: '{{ deploypath }}/file.yml'
    separator: '|'
    state: present
    edits:
    - { key: a|b, value: '{{ somevalue }}' }

host_vars/somehost
somevalue: '!sometext'

I receive an error:

File \"/tmp/ansible_yedit_payload_ECyASG/__main__.py\", line 969, in <module>\r\n  File \"/tmp/ansible_yedit_payload_ECyASG/__main__.py\", line 961, in main\r\n  File \"/tmp/ansible_yedit_payload_ECyASG/__main__.py\", line 889, in run_ansible\r\n  File \"/tmp/ansible_yedit_payload_ECyASG/__main__.py\", line 784, in process_edits\r\n  File \"/tmp/ansible_yedit_payload_ECyASG/__main__.py\", line 775, in parse_value\r\n__main__.YeditException: Could not determine type of incoming value. value=[<type 'str'>] vtype=[]\r\n"

Removing an item from a list

- name: remove apples from the grocery list
  yedit:
    state: absent
    index: 1
    key: groceries
    content: { 'groceries': [ 'oranges', 'apples' ] }
  register: res
- debug: var=res

I would expect res.result to be equal to { 'groceries': [ 'oranges' ] }, instead it's just null. It seems like absent ignores the index parameter entirely and just deletes the whole groceries key.

@kwoodson can you confirm that { 'groceries': [ 'oranges' ] } is the expected value here? If so, I'll prepare a PR.

Ensure array contains element (aka kubectl patch -p)

Hi you,

I am using yedit to do a patch like operation on a kube ServiceAccount.
Command :
kubectl patch serviceaccount default -n monitoring -p '{"imagePullSecrets": [{"name": "monitoring-de-icr-io"}]}'

When I run this twice, it tells me no change and I have what I aim for which means

kubectl get sa test -n monitoring -o yaml
apiVersion: v1
imagePullSecrets:
- name: monitoring-de-icr-io
kind: ServiceAccount
metadata:
  creationTimestamp: "2020-05-20T15:25:38Z"
  name: test
  namespace: monitoring
  resourceVersion: "876792"
  selfLink: /api/v1/namespaces/monitoring/serviceaccounts/test
  uid: 48eedc79-3751-49e4-a519-7be8cea59ea2
secrets:
- name: test-token-8v5v8

Now what am I doing with yedit is :

    - name: Patch service account to use the image pull secret - Patch
      yedit:
        content: "{{ monitoring_sa.resources[0] }}"
        key: imagePullSecrets
        value: "name: monitoring-de-icr-io"
        append: true
      register: msg

The whole ope is

    - name: Patch service account to use the image pull secret - Get
      k8s_info:
        api_version: v1
        kind: ServiceAccount
        name: test
        namespace: monitoring
      register: monitoring_sa
    - debug: var=monitoring_sa
    - name: Patch service account to use the image pull secret - Patch
      yedit:
        content: "{{ monitoring_sa.resources[0] }}"
        key: imagePullSecrets
        value: "name: monitoring-de-icr-io"
        append: true
      register: msg
    - debug: var=msg.result[0].edit
    - name: Patch service account to use the image pull secret - Apply
      k8s:
        state: present
        definition: "{{ msg.result[0].edit }}"

But runing it twice gives me

kubectl get sa test -n monitoring -o yaml
apiVersion: v1
imagePullSecrets:
- name: monitoring-de-icr-io
- name: monitoring-de-icr-io
kind: ServiceAccount
metadata:
  creationTimestamp: "2020-05-20T15:25:38Z"
  name: test
  namespace: monitoring
  resourceVersion: "876792"
  selfLink: /api/v1/namespaces/monitoring/serviceaccounts/test
  uid: 48eedc79-3751-49e4-a519-7be8cea59ea2
secrets:
- name: test-token-8v5v8

the use of update gives empty result.

What is the best way to get this done please?

Append to list of dicts in a specific location

I've been reading the doc and trying all the parameters but I wonder if I can go from this with this module:

my_var:
  - name: 'name_1'
    id: 111
  - name: 'name_2'
    id: 112
    programs:
      - program_name: "HTTP"
        port: 80
      - program_name: "HTTPS"
        port: 443
  - name: 'name_3'
    id: 113

To this, in one task:

my_var:
  - name: 'name_1'
    id: 111
  - name: 'name_2'
    id: 112
    programs:
      - program_name: "HTTP"
        port: 80
      - program_name: "HTTPS"
        port: 443
      - program_name: "new_program_1"
        port: 111
      - program_name: "new_program_2"
        port: 222
  - name: 'name_3'
    id: 113

I need it to be exactly ordered like that, so inside item with value name: 'name_2' and inside the sublist programs
@kwoodson

Tag with semantic versions for Ansible Galaxy

Please tag each release with semantic versions in GitHub. This will allow users of this role (via Ansible Galaxy) to pin versions, and I think it will help import updates to the role automatically.

Add item to list, creating list on demand

Hi, I would like to ensure that an item is in a list. For that I can use update: yes:

- name: Test
  hosts: [localhost]
  gather_facts: no

  roles:
    - kwoodson.yedit

  tasks:
  - name: Modify YAML
    yedit:
      content:
        foo: bar
        users:
          - alice
      key: users
      value: "bob"
      update: yes
      # append: yes
      state: present
    register: res

  - debug: var=res

However, this does not work when the list 'users' doesn't exist. It just succeeds without doing anything. If I use append: yes instead, it will create the list if neccessary. But then I get duplicates if the item does exist.

I think the source of this difference is that append has the following code, but update doesn't:

ansible-role-yedit/library/yedit.py

Lines 622 to 624 in 0ae7b9c

	if entry is None:
	self.put(path, [])
	entry = Yedit.get_entry(self.yaml_dict, path, self.separator)

Could this be added to update, or is there another way to achive this?

Please update to the openshift version, including the timestamp fix

Hi,

The version of yedit used in openshift has and additional feature (edits), but it also (re?)introduces a bug (invalid timestamps).

Could you update the version here to include the feature, but exclude the bug ?

Thanks.

PS. I couldn't find a way to make yedit just leave the timestamps alone.

Make writes to the filesytem more durable

Could we add some flush and sync calls to the _write method, to make writes a little bit more reliable?

Manipulate lists of dictionaries

I was trying this very useful role( hopefully a future module ;) ) and I'm not able to manipulate a list of dictionaries.

What I need to manipulate is something like:

nexus_roles:
  - id: users
    name: users_roles
    description: Role to all users
    privileges:
      - nx-search-read
      - all-repos-read
    roles: []
  - id: devs
    name: devs_roles
    description: Role to all devs
    privileges:
      - nx-search-read
      - nx-repository-view-docker-*-*
      - all-repos-read
    roles: []

It would be useful to be able to create, delete or update parts or the full dictionary.
I was trying several variants but did not manage to get this to work.

Was this use case not implemented?

Thanks

Issues with --- disappearing

I have the following in an ansible task:

- block:
...
    - name: Save egress interface to host_vars
      yedit:
        src: "{{ inventory_dir }}/host_vars/{{ inventory_hostname }}.yml"
        key: egress.iface
        value: "{{ egress['iface'] }}"
      delegate_to: 127.0.0.1

The key/value is added properly, but the --- at the beginning of the src file disappears. I have confirmed this by copying the file immediately before and after this runs as well as with the built in backup option.

Fail to parse multiple document in yaml file

I have yaml file with multiple document . I am trying to read a value from one of the document and I am getting this error while trying to access "spec.selector.name" .

Below is the yaml:

apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  name: epgs.aci.aw
spec:
  group: aci.aw
  names:
    kind: Epg
    listKind: EpgList
    plural: epgs
  scope: Namespaced
  version: v1
---
apiVersion: v1
kind: Service
metadata:
  name: ep-registry
  namespace: kube-system
  labels:
    app: gbpserver
spec:
  clusterIP: 10.96.0.2
  ports:
  - port: 14443
    targetPort: 14443
  selector:
    name: aci-containers-controller

Error:

Traceback (most recent call last):
  File "/tmp/ansible_yedit_payload_v_m8fh76/__main__.py", line 499, in load
AttributeError: module 'yaml' has no attribute 'RoundTripLoader'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/tmp/ansible_yedit_payload_v_m8fh76/__main__.py", line 501, in load
  File "/usr/lib/python3/dist-packages/yaml/__init__.py", line 94, in safe_load
    return load(stream, SafeLoader)
  File "/usr/lib/python3/dist-packages/yaml/__init__.py", line 72, in load
    return loader.get_single_data()
  File "/usr/lib/python3/dist-packages/yaml/constructor.py", line 35, in get_single_data
    node = self.get_single_node()
  File "/usr/lib/python3/dist-packages/yaml/composer.py", line 43, in get_single_node
    event.start_mark)
yaml.composer.ComposerError: expected a single document in the stream
  in "<unicode string>", line 1, column 1:
    apiVersion: apiextensions.k8s.io ... 
    ^
but found another document
  in "<unicode string>", line 14, column 1:
    ---
    ^

Is there a way to resolve this ?

Replaces empty values with the string "null"

If I have a file like the following, the password value will actually be changed to null.

This:

db:
  host: example.com
  user: test
  password:

Becomes:

db:
  host: example.com
  password: null
  user: test

This happens even when editing totally unrelated values in other parts of the config which should obviously not be happening.

rename module to make it easily discoverable

Bases on an IRC discussion we had on #ansible-devel it was suggested to rename the module to make it easy for users to use it, especially as we aim to make it an ansible core module.

Proposals so far:

• yaml
• yaml_file - similar to ini_file, only downside is that it will be an issue if we ever want to add support for "content" as alternative to "src" as calling parameter.

Someone noted that JSON support could be wanted in the future and that the module could do both. Still, I think that this is outside the current scope and even if we endup implementing that we could create a module alias that does that for json, so we can reuse the codebase.

Please comment with your suggestions/votes.

Doc is not readable

The doc is not in an easy to read format

Validate functionality

lineinfile, blockinfile, replace, etc support a validate command that tests the resulting yaml before committing it. It'd be great if this could too, especially as netplan is becoming the standard way to manage network interfaces.

Support datetime-stamps for the backup file

It would be nice to use datetime information in the backup file name.

Reports failed, but is successful

When running the following code it reports as failed. The value is correctly updated.

When the 'hash' already has the matching value it reports "OK" or unchanged.

- name: "manage {{username}} in internal_users.yml file"
  yedit:
    src: "{{od4es_internal_users_file}}"
    key: "{{username}}.hash"
    value: "{{ userhash }}"

The full traceback is:
Traceback (most recent call last):
  File "/home/vagrant/.ansible/tmp/ansible-tmp-1554493645.47-213942729249153/AnsiballZ_yedit.py", line 113, in <module>
    _ansiballz_main()
  File "/home/vagrant/.ansible/tmp/ansible-tmp-1554493645.47-213942729249153/AnsiballZ_yedit.py", line 105, in _ansiballz_main
    invoke_module(zipped_mod, temp_path, ANSIBALLZ_PARAMS)
  File "/home/vagrant/.ansible/tmp/ansible-tmp-1554493645.47-213942729249153/AnsiballZ_yedit.py", line 48, in invoke_module
    imp.load_module('__main__', mod, module, MOD_DESC)
  File "/tmp/ansible_yedit_payload_p5Dtgd/__main__.py", line 969, in <module>
  File "/tmp/ansible_yedit_payload_p5Dtgd/__main__.py", line 961, in main
  File "/tmp/ansible_yedit_payload_p5Dtgd/__main__.py", line 893, in run_ansible
  File "/tmp/ansible_yedit_payload_p5Dtgd/__main__.py", line 446, in write
  File "/tmp/ansible_yedit_payload_p5Dtgd/__main__.py", line 422, in _write
OSError: [Errno 22] Invalid argument

fatal: [localhost]: FAILED! => {
    "changed": false, 
    "module_stderr": "Traceback (most recent call last):\n  File \"/home/vagrant/.ansible/tmp/ansible-tmp-1554493645.47-213942729249153/AnsiballZ_yedit.py\", line 113, in <module>\n    _ansiballz_main()\n  File \"/home/vagrant/.ansible/tmp/ansible-tmp-1554493645.47-213942729249153/AnsiballZ_yedit.py\", line 105, in _ansiballz_main\n    invoke_module(zipped_mod, temp_path, ANSIBALLZ_PARAMS)\n  File \"/home/vagrant/.ansible/tmp/ansible-tmp-1554493645.47-213942729249153/AnsiballZ_yedit.py\", line 48, in invoke_module\n    imp.load_module('__main__', mod, module, MOD_DESC)\n  File \"/tmp/ansible_yedit_payload_p5Dtgd/__main__.py\", line 969, in <module>\n  File \"/tmp/ansible_yedit_payload_p5Dtgd/__main__.py\", line 961, in main\n  File \"/tmp/ansible_yedit_payload_p5Dtgd/__main__.py\", line 893, in run_ansible\n  File \"/tmp/ansible_yedit_payload_p5Dtgd/__main__.py\", line 446, in write\n  File \"/tmp/ansible_yedit_payload_p5Dtgd/__main__.py\", line 422, in _write\nOSError: [Errno 22] Invalid argument\n", 
    "module_stdout": "", 
    "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error", 
    "rc": 1
}

The file being updated is https://github.com/opendistro-for-elasticsearch/security/blob/master/securityconfig/internal_users.yml

Integer coming through as String

With yedit ansible task like this :

    yedit:
      src: "zones/my_dns_zone.yaml"
      key: "pilot1"
      value:
        ttl: "60"
        type: "A"
        value: "10.1.1.1"

i get Yaml output like this where the integer 60 is a string :

pilot1:
  ttl: '60'
  type: A
  value: 10.1.1.1

how do I get it so the ttl comes out as an integer like so :

pilot1:
  ttl: 60
  type: A
  value: 10.1.1.1

Upstream this module?

Any particular reason as to why this isn't submitted upstream to be a part of the core modules?

Fails reading or writing certain values

I love what you've made here but I'm running into some trouble. I'm using ansible-vault and yedit is failing with the encrypted strings generated by it. It seems to be with the "!vault |" part. Perhaps it's just not able to handle YAML paragraphs? Here's an example of a value I can't read or write with yedit:

token: !vault |
          $ANSIBLE_VAULT;1.1;AES256
          63383134376666393265353163363631626437623532656262643138333830613764393834643265
          3630303866623662306563393936333130623366613864360a643963353661363465386239613131
          38373632303332633439633239386363663963626363613434393166653565363138353032383931
          6337353636623035370a323663643833636232343965393730626331623831303237636237373030
          6431

Add tests for ansible-code

Currently this module only has tests for the underlying Yedit-class. There is a rather complete set of ansible-tasks within the repository, but their output is never checked, nor are they executed by the test framework.

Just like the underlying class the ansible-code, which handles most of the parameter parsing, should also be tested.

• add assert-statements to the existing playbooks, to quickly get some basic coverage
• split the big playbooks into smaller ones, grouped by functionality
• write a simplistic test-runner to run all of them, either via the ansible API or via a dump shellscript

Error importing module

Hello,

First of all, good job, this role it's awesome.

I'm trying to use it but I have the next error.
FAILED! => {"failed": true, "msg": "ERROR! error importing module in /mnt/c/Users/jlp/repos/openshift/ansible/roles/kwoodson.yedit/library/yedit.py, expecting format like 'from ansible.module_utils.basic import *'"}

I've imported the role an copy to my roles folder, after that I made a playbook where I use a module (master-config) to edit a yaml file.

Do you know what could it be happening? I've also tried to copy your role files to /usr/lib/python2.7/dist-packages/ansible/ but didn't work either.

Playbook:

• hosts:
  • capmachine
    become: yes

  vars_files:

  - vars/{{env}}.yml

  roles:
  • roles/kwoodson.yedit
  • role: master-config

Role/master-config/tasks/main.yml:

• yedit:
  src: /etc/origin/master/master-config.yaml
  state: present
  key: controllerArguments#node-monitor-grace-period
  value:
  "40s"

Thanks a lot for your help.

Multiple simultaneous yml accesses

Hi,

There could be 'lock' option in yedit which would lock the file if there are multiple simultaneous accesses to the file.

I tested code below and seems to be working. Though I didn't make any exhaustive testing for it.

    module = AnsibleModule(
        argument_spec=dict(
            state=dict(default='present', type='str',
                       choices=['present', 'absent', 'list']),
            debug=dict(default=False, type='bool'),
            src=dict(default=None, type='str'),
            content=dict(default=None),
            content_type=dict(default='dict', choices=['dict', 'str']),
            key=dict(default='', type='str'),
            value=dict(),
            value_type=dict(default='', type='str'),
            update=dict(default=False, type='bool'),
            append=dict(default=False, type='bool'),
            index=dict(default=None, type='int'),
            curr_value=dict(default=None, type='str'),
            curr_value_format=dict(default='yaml', choices=['yaml', 'json', 'str'], type='str'),
            backup=dict(default=True, type='bool'),
            lock=dict(default=False, type='bool'),
        ),
        mutually_exclusive=[["curr_value", "index"], ['update', "append"]],
        required_one_of=[["content", "src"]],

        supports_check_mode=True,
    )
    state = module.params['state']

    yamlfile = Yedit(filename=module.params['src'], backup=module.params['backup'])

    if module.params['lock'] == True:
        lock_file = open(yamlfile.filename + ".yedit_lock", "w")
        fcntl.flock(lock_file.fileno(), fcntl.LOCK_EX)

use molecule to test role and module

We should use ansible-molecule for testing the code.

I will take care of this.

The readme is full of errors

Wrong install instruction (wrong role name)
Wrong example (second task)

missing information

I'm testing the module atm so I'm not going to open a PR for now (still trying to figure out how all of this work)

how to use seprator

---

• hosts: localhost
  gather_facts: false
  roles:
  • yedit
    tasks:
  • name: editing files
    yedit:
    src: '/tmp/versions.yaml'
    edits:
    - key: "war-1.0-dev"
    value: "1.15.0-2019061723.23313320"

Desire file should be like this:-
war-1.0-dev : 1.15.0-2019061723.23313320

but it showing
war-1
0-dev: 1.15.0-2019061723.23313320

Uploading to Ansible Galaxy

Would you consider setting your repo to be a proper Ansible Galaxy role so that we can use it and stay up to date more easily ?

allow prepending to a list

Simmilar to append: yes there should be a prepend: yes to add something as the first entry on a list.

- name: add oranges to the end, because they are kinda meh
  yedit:
    append: yes
    content: [ 'kiwi', 'peppers' ]
    value: oranges
# generates: [ 'kiwi', 'peppers', 'oranges' ]

- name: add apples before everything else, because they are AWESOME
  yedit:
    prepend: yes
    content: [ 'kiwi', 'peppers' ]
    value: apples
# generates: [ 'apples', 'kiwi', 'peppers' ]

• implement yedit.prepend and its tests, simmilar to the existing yedit.append
• add the mutually exclusive prepend parameter to the ansible parameters and write the necessary glue-code
• add parameter documentation
• add an example to the readme

Remote host module not found 'yaml'

Having an issue where a remote host is failing when running the yedit module. I'm getting the below error message:

fatal: [local-web-server]: FAILED! => {"changed": false, "failed": true, "module_stderr": "Shared connection to 172.17.3.160 closed.\r\n", "module_stdout": "Traceback (most recent call last):\r\n File \"/tmp/ansible_3Vz6cf/ansible_module_yedit.py\", line 191, in <module>\r\n import yaml # noqa: F401\r\nImportError: No module named yaml\r\n", "msg": "MODULE FAILURE", "rc": 0}

The remote host in question is on a restricted network and as well as limited on what it can get via apt-get. It is a Debian Jessie OS. Ideally, I want to avoid using 'apt-get install python-yaml' if at all possible.

Defaulting to an empty list with `append=true`

Assuming some.yml is empty or at least does not yet contain the key domains, the following yml does nothing:

- name: Add default domain
  yedit:
    src: some.yml
    key: domains
    append: yes
    value: "google.com"

To my knowledge, there is no way to easily default domains to an empty array, when the state of the file in unknown to begin with. The current code for yedit.append() explicitly aborts, when the key is not found. It also aborts, when the key is of some other type, which makes sense.

I would like the change the behavior within yedit.append(), so that if the key does not yet exist, so it defaults to an empty array, to which the new value is added.

@kwoodson what are your thoughts on this?

yedit module will modify the file permissions after execution

Hi, I recently encountered a problem when using the yedit module. First, I executed the file module to modify the permissions of a file and changed it to 0600. Then I executed the yedit module to inject a key into this file, and the result was this file. The permissions changed to 0644 again.

Will the yedit module modify the file permissions after execution?

Example:
At first, the permission of /etc/origin/master/master-config.yaml are 0644

[root@ocp-node01 etc]# stat /etc/origin/master/master-config.yaml
  File: ‘/etc/origin/master/master-config.yaml’
  Size: 6940      	Blocks: 16         IO Block: 4096   regular file
Device: fd00h/64768d	Inode: 101353485   Links: 1
Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)

Then ,I execute the task below to modify file permission.

- name: "Verify Permissions on the OpenShift Master Configuration File"
  file:
    path: /etc/origin/master/master-config.yaml
    mode: 0600
    state: file

and sucess.

[root@ocp-node01 etc]# stat /etc/origin/master/master-config.yaml
  File: ‘/etc/origin/master/master-config.yaml’
  Size: 6940      	Blocks: 16         IO Block: 4096   regular file
Device: fd00h/64768d	Inode: 101353485   Links: 1
Access: (0600/-rw-------)  Uid: (    0/    root)   Gid: (    0/    root)

After that, I execute the task below to modify file content.

- name: "Enable service-account-lookup on the API Server"
  yedit:
    src: /etc/origin/master/master-config.yaml
    key: kubernetesMasterConfig.apiServerArguments.service-account-lookup
    value:
      - 'true'

With the file content changed, the permissions of the file also changed.

TASK [common : Enable service-account-lookup on the API Server] ****************************************************************************************************************
changed: [192.168.62.121]
[root@ocp-node01 etc]# stat /etc/origin/master/master-config.yaml
  File: ‘/etc/origin/master/master-config.yaml’
  Size: 6940      	Blocks: 16         IO Block: 4096   regular file
Device: fd00h/64768d	Inode: 101355612   Links: 1
Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)

I know I can reverse the order of tasks, but my scenario here is not recommended.
Is there any other way to avoid this?

python exception while running yedit

Hello everyone,

I have a simple yml:

- hosts: localhost
  roles:
  - roles/kwoodson.yedit
  tasks:
  - name: manage yaml files
    yedit:
      src: ~/workspace/docker-machines/test/docker-compose.yml
      key: services.atb.image
      value:
        wakka

However it fails with

module 'yaml' has no attribute 'RoundTripDumper'

Here's the full stack trace:

fatal: [localhost]: FAILED! => {
    "changed": false,
    "module_stderr": "/Users/dmitry/.ansible/tmp/ansible-tmp-1566064416.420654-155937174493441/AnsiballZ_yedit.py:18: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses\n  import imp\nTraceback (most recent call last):\n  File \"/var/folders/l7/y1c6vkdx02l2mrw0_z90ggcr0000gn/T/ansible_yedit_payload_5wbjyf6s/__main__.py\", line 444, in write\nAttributeError: module 'yaml' has no attribute 'RoundTripDumper'\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n  File \"/Users/dmitry/.ansible/tmp/ansible-tmp-1566064416.420654-155937174493441/AnsiballZ_yedit.py\", line 114, in <module>\n    _ansiballz_main()\n  File \"/Users/dmitry/.ansible/tmp/ansible-tmp-1566064416.420654-155937174493441/AnsiballZ_yedit.py\", line 106, in _ansiballz_main\n    invoke_module(zipped_mod, temp_path, ANSIBALLZ_PARAMS)\n  File \"/Users/dmitry/.ansible/tmp/ansible-tmp-1566064416.420654-155937174493441/AnsiballZ_yedit.py\", line 49, in invoke_module\n    imp.load_module('__main__', mod, module, MOD_DESC)\n  File \"/usr/local/Cellar/ansible/2.8.4/libexec/lib/python3.7/imp.py\", line 234, in load_module\n    return load_source(name, filename, file)\n  File \"/usr/local/Cellar/ansible/2.8.4/libexec/lib/python3.7/imp.py\", line 169, in load_source\n    module = _exec(spec, sys.modules[name])\n  File \"<frozen importlib._bootstrap>\", line 630, in _exec\n  File \"<frozen importlib._bootstrap_external>\", line 728, in exec_module\n  File \"<frozen importlib._bootstrap>\", line 219, in _call_with_frames_removed\n  File \"/var/folders/l7/y1c6vkdx02l2mrw0_z90ggcr0000gn/T/ansible_yedit_payload_5wbjyf6s/__main__.py\", line 969, in <module>\n  File \"/var/folders/l7/y1c6vkdx02l2mrw0_z90ggcr0000gn/T/ansible_yedit_payload_5wbjyf6s/__main__.py\", line 961, in main\n  File \"/var/folders/l7/y1c6vkdx02l2mrw0_z90ggcr0000gn/T/ansible_yedit_payload_5wbjyf6s/__main__.py\", line 893, in run_ansible\n  File \"/var/folders/l7/y1c6vkdx02l2mrw0_z90ggcr0000gn/T/ansible_yedit_payload_5wbjyf6s/__main__.py\", line 446, in write\n  File \"/var/folders/l7/y1c6vkdx02l2mrw0_z90ggcr0000gn/T/ansible_yedit_payload_5wbjyf6s/__main__.py\", line 408, in _write\nFileNotFoundError: [Errno 2] No such file or directory: '~/workspace/docker-machines/test/docker-compose.yml.yedit'\n",
    "module_stdout": "",
    "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error",
    "rc": 1
}

However I do believe I have the latest Yaml module:

~/w/a/docker ❯❯❯ python3.7                                                                                                                                                                                                    master ◼
Python 3.7.4 (default, Jul  9 2019, 18:13:23)
[Clang 10.0.1 (clang-1001.0.46.4)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import yaml
>>> yaml.__version__
'5.1.2'

This is on:

• MacOS 10.14
• Ansible 2.8.4

Licensing the project

It would be much clearer if your project could advertise its license. Software projects with unknown licences can be hard to use in some cases.

Thank you !

Overwriting an item within a list crashes ansible

Given the list [1, 2, 3], this playbook should change it to [5, 2, 3].

- hosts: localhost
  gather_facts: no
  tasks:

  - copy:
      dest: /tmp/test1.yml
      content: |
        - 1
        - 2
        - 3

  - name: overwrite first entry in list
    yedit:
      src: /tmp/test1.yml
      update: yes
      key: 0  # using `index` instead of `key` yields the same result
      value: 5

Instead it crashes ansible:

TASK [overwrite first entry in list] *******************************************
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/ansible/executor/process/result.py", line 143, in run
    elif result.is_failed():
  File "/usr/lib/python2.7/site-packages/ansible/executor/task_result.py", line 56, in is_failed
    'results' in self._result and True in [True for x in self._result['results'] if 'failed_when_result' in x]:
TypeError: argument of type 'int' is not iterable

This only seems to affect lists. Overwriting a key within a dictionary works fine.

Error: ImportError: No module named yaml

Using Ubuntu 18.04 I get the following error:

An exception occurred during task execution. To see the full traceback, use -vvv. The error was: ImportError: No module named yaml
fatal: [my-host]: FAILED! => changed=false 
  module_stderr: |-
    Traceback (most recent call last):
      File "<stdin>", line 114, in <module>
      File "<stdin>", line 106, in _ansiballz_main
      File "<stdin>", line 49, in invoke_module
      File "/tmp/ansible_yedit_payload_fkAMVf/__main__.py", line 191, in <module>
    ImportError: No module named yaml
  module_stdout: ''
  msg: |-
    MODULE FAILURE
    See stdout/stderr for the exact error
  rc: 1

I already tried various variants of pip install pyaml, (python2, python3, earlier version) but to no success. Any ideas?

Backslashes get escaped

Given this YAML passed to yedit:

exclude_lines:
- ^\s*(\(|\))\s*$
- ^\s+$

I end up with the following result:

exclude_lines:
- ^\\s*(\\(|\\))\\s*$
- ^\\s+$

This, naturally, breaks the regular expression that I am trying to provide in this particular value.

I get the same result if I wrap the values in single-quotes.
If I wrap the values in double quotes, it's a YAML syntax error (unless I escape the slashes myself, in which case, the end result is still the same).

Is there a way I can provide a blackslash to yedit without it being escaped for me?

Add a flag to not re-order the yaml file it is adding items to?

Please? I think ruamel.yaml can keep the order of the yaml if you tell it to?

explicit_start support

Hi,

We need explicit_start inside our yaml , is it possible to set somehow? :)

swap out PyYaml in favor of ruamel.yaml

Based on a bunch of the comments on stackoverflow our current library, PyYaml, is not being currently worked on.

PyYaml only supports yaml 1.1 which came out in 2004. The relevant issue was last touched two years ago.
Ruamel.yaml also supports yaml 1.2 which is more recent, 2009. In addition, it enables us to preserve comments and formatting.

Periods in keys

I've got a YAML file (from Elasticsearch) that comes with periods in its keys:

filebeat.inputs:
- enabled: false
  paths:
  - /var/log/*.log
  type: log

In trying to write to this key, I'm using:

- name: Add log file inputs.
  yedit:
    src: /etc/filebeat/filebeat.yml
    key: filebeat.inputs
    value:
      - type: log
        enabled: true
        paths:
          - /var/log/*.log
          - /var/log/cron
          - /var/log/maillog
          - /var/log/messages
          - /opt/log/20*/*/*/*.log
  notify: restart beats

Unfortunately this results in duplicating the filebeat.inputs key in the destination file, which means the latter (the original one) overrides my new key:

filebeat:
  inputs:
  - enabled: true
    paths:
    - /var/log/*.log
    - /var/log/cron
    - /var/log/maillog
    - /var/log/messages
    - /opt/log/20*/*/*/*.log
    type: log
filebeat.inputs:
- enabled: false
  paths:
  - /var/log/*.log
  type: log

To try to get yedit to recognise/replace the original filebeats.input key, I've tried specifying the key in the following ways:

• filebeat.inputs (as above)
• "filebeat.inputs" (same result as above)
• "[filebeat.inputs]" (idea from here; no match - no changes made to file)
• filebeat\.inputs (no match - no changes made to file)
• "filebeat\.inputs" (YAML syntax error)

Do you have any ideas for how I could achieve this?

If it's not easy to match a key with a period in it, alternatively I could look at ways to completely remove that duplicate key, although I think that would be a last resort.

issues with dots in keys

Hi kwoodson,

I have the following problem when my keys contain dots such as in domain names

My target yaml structure I want to write:

lab00001:
  hosts:
    mydomainname.example.com
       horst

Now using quotes I try to escape the dots in the domain name

- hosts: localhost
  gather_facts: no
  roles:
  - kwoodson.yedit
  tasks:
  - name: insert simple key, value
    yedit:
      src: /tmp/xxx.yaml
      key: 
        lab00001
          hosts
            "mydomainname.example.com"
      value: horst
      state: present

this does not write anything into the file.

Other combinations of escaping dots in yaml do not work either for me, e.g

            "mydomainname\.example\.com"

If I leave the dots and quotes, it writes content to the file

- hosts: localhost
  gather_facts: no
  roles:
  - kwoodson.yedit
  tasks:
  - name: insert simple key, value
    yedit:
      src: /tmp/xxx.yaml
      key: 
        lab01
          hosts
            mydomainnameXXX
      value: horst
      state: present

Is there a way to use dots in keys (escape) with your module?

Thanks in advance,
Oliver

libraries missing?

Hi,

We saw we need pyyaml on the server, wouldn't it be nice to fix it so the requirements get auto installed ?

Incompatible with CentOS 6

CentOS 6 uses Python 2.6, which requires the position of format field to be explicitly specified. Otherwise the module will fail with the error message:
File \"/tmp/ansible_jeh8mY/ansible_module_yedit.py\", line 215, in Yedit\r\n backup_ext=\".{}\".format(time.strftime(\"%Y%m%dT%H%M%S\")),\r\nValueError: zero length field name in format\r\n", "msg": "MODULE FAILURE", "rc": 1

enable ci - travis

@kwoodson Please enable Travis CI on this repository so I can test changes. The "tox" branch already has a travis file on it. Once you do this, reassign the task to me to complete the task.

Release the Yedit-class on pypi

As there doesn't seem to be any other module to edit yml files safely: Could you please publish the Yedit backend class as an independent module on pypi?