opensearch-project/ansible-playbook

🤖 A community repository for Ansible Playbook of OpenSearch Project.

• OpenSearch Project Ansible-Playbook
• Version and Branching
• OpenSearch Installation with Dashboards
  • Prerequisite
  • Configure
    • Multi-node Installation
    • Single Node Installation
  • Install
  • OpenID authentification
  • Custom configuration files
  • IaC (Infrastructure-as-Code)
• Contributing
• Getting Help
• Code of Conduct
• Security
• License
• Copyright

OpenSearch Project Ansible-Playbook

A community repository for Ansible Playbook of OpenSearch Project.

Version and Branching

As of now, this ansible-playbook repository maintains 2 branches:

• main (Version is 2.x.x for both os_version and os_dashboards_version in inventories/opensearch/group_vars/all/all.yml)
• 1.x (Version is 1.x.x for both os_version and os_dashboards_version in inventories/opensearch/group_vars/all/all.yml)

Contributors should choose the corresponding branch(es) when commiting their change(s):

• If you have a change for a specific version, only open PR to specific branch
• If you have a change for all available versions, first open a PR on main, then open a backport PR with [backport 1.x] in the title, with label backport 1.x, etc.

OpenSearch Installation with Dashboards

This ansible playbook supports the following,

• Can be deployed on baremetal and VMs(AWS EC2)
• Supports most popular Linux distributions(Centos7, RHEL7, Amazon Linux2, Ubuntu 20.04)
• Install and configure the Apache2.0 opensource OpenSearch
• Configure TLS/SSL for OpenSearch transport layer(Nodes to Nodes communication) and REST API layer
• Generate self-signed certificates to configure TLS/SSL for opensearch
• Configure the Internal Users Database with limited users and user-defined passwords
• Configuration of authentication and authorization via OpenID
• Overriding default settings with your own
• Install and configure the Apache2.0 opensource OpenSearch Dashboards

Prerequisite

• Ansible 2.9+
• Java 8

Configure

Refer the file inventories/opensearch/group_vars/all/all.yml to change the default values.

For example if we need to increase the java memory heap size for opensearch,

xms_value: 8
xmx_value: 8

In inventories/opensearch/hosts file, you can configure the node details. ansible_host is used for ansible to connect the nodes to run this playbook. ip is used in OpenSearch and Dashboards configuration.

In AWS EC2,

os1 ansible_host=<Elastic/Public IP> address ansible_user=root ip=<Private IP address>

Multi-node Installation

By default, this playbook will install five nodes opensearch cluster with respective roles (3 master, 5 data and 2 ingest nodes).

os1 ansible_host=10.0.1.1 ip=10.0.1.1 roles=data,master
os2 ansible_host=10.0.1.2 ip=10.0.1.2 roles=data,master
os3 ansible_host=10.0.1.3 ip=10.0.1.3 roles=data,master
os4 ansible_host=10.0.1.4 ip=10.0.1.4 roles=data,ingest
os5 ansible_host=10.0.1.5 ip=10.0.1.5 roles=data,ingest

Note: You need to add additional nodes details in inventories/opensearch/hosts file for creating opensearch cluster with different node sizes.

For example, if you want to create seven nodes cluster with two additional data nodes (os6 and os7) then you need to include the below entries.

os6 ansible_host=10.0.1.6 ip=10.0.1.6 roles=data
os7 ansible_host=10.0.1.7 ip=10.0.1.5 roles=data

You have to mention the opensearch node roles details in roles variable.

Single Node Installation

For single node installation, you need to change the cluster_type variable in inventory file inventories/opensearch/group_vars/all/all.yml

cluster_type: single-node

Install

# Deploy with ansible playbook - run the playbook as root
ansible-playbook -i inventories/opensearch/hosts opensearch.yml --extra-vars "admin_password=myStrongPassword@123! kibanaserver_password=Test@6789 logstash_password=Test@456"

You should set the reserved users(admin, kibanaserver, and logstash) password using admin_password, kibanaserver_password, and logstash_password variables.

Note: Starting OpenSearch 2.12, a strong password is required for admin user, i.e. myStrongPassword123!. The cluster will fail to start with a weak password (i.e. admin) or no password.

If you define your own internal users (in addition to the reserved admin, kibanaserver, and logstash) in custom configuration files, then passwords to them should be set via variables on the principle of <username>_password

It will install and configure the opensearch. Once the deployment completed, you can access the opensearch Dashboards with user admin and password which you provided for variable admin_password.

# Deploy with ansible playbook - run the playbook as non-root user which have sudo privileges,
ansible-playbook -i inventories/opensearch/hosts opensearch.yml --extra-vars "admin_password=myStrongPassword@123! kibanaserver_password=Test@6789 logstash_password=Test@456" --become

Note: Change the user details in ansible_user parameter in inventories/opensearch/hosts inventory file.

OpenID authentification

To enable authentication via OpenID, you need to change the auth_type variable in the inventory file inventories/opensearch/group_vars/all/all.yml by setting the value oidc and prescribe the necessary settings in the oidc: block.

Custom configuration files

To override the default settings files, you need to put your settings in the files directory. The files should be named exactly the same as the original ones (internal_users.yml, roles.yml, tenants.yml, etc.)

Especially note the file files/internal_users.yml. If it exists and the copy_custom_security_configs: true setting is enabled, then only in this case the task of setting passwords for internal users from variables is started. If the file internal_users.yml is not located in the files directory, but, for example, in one of its subdirectories, then playbook will not work correctly

IaC (Infrastructure-as-Code)

If you want to use the role not only for the initial deployment of the cluster, but also for further management of it, then set the iac_enable parameter to true.

By default, if the /tmp/opensearch-nodecerts directory with certificates exists on the server from which the playbook is launched, it is assumed that the configuration has not changed and some settings are not copied to the target servers.

Conversely, if the /tmp/opensearch-nodecerts directory does not exist on the server from which the playbook is launched, then new certificates and settings are generated and they are copied to the target servers.

If you use this repository not only for the initial deployment of the cluster, but also for its automatic configuration via CI/CD, then new certificates will be generated every time the pipeline is launched, overwriting existing ones, which is not always necessary if the cluster is already in production.

When iac_enable enabling, and all the cluster servers have all the necessary certificates, they will not be copied again. If at least on one server (for example, when adding a new server to the cluster) if there is not at least one certificate from the list, then all certificates on all cluster servers will be updated

Also, if the option is enabled, the settings files will be updated with each execution (previously, the settings were updated only if the /tmp/opensearch-nodecerts directory was missing on the server from which the playbook was launched and new certificates were generated)

Contributing

See developer guide and how to contribute to this project.

Getting Help

If you find a bug, or have a feature request, please don't hesitate to open an issue in this repository.

For more information, see project website and documentation. If you need help and are unsure where to open an issue, try forums.

Code of Conduct

This project has adopted the Amazon Open Source Code of Conduct. For more information see the Code of Conduct FAQ, or contact [email protected] with any additional questions or comments.

Security

If you discover a potential security issue in this project we ask that you notify AWS/Amazon Security via our vulnerability reporting page. Please do not create a public GitHub issue.

License

This project is licensed under the Apache v2.0 License.

Copyright

Copyright OpenSearch Contributors. See NOTICE for details.

ansible-playbook's Issues

[CI] Need automate version bump PR to be opened once OpenSearch/Dashboards release new versions

As of now, we are relying on human efforts to bump OS/OSD versions once there is a new release.
This could delay the release on these platforms and could add to human errors.
We need a github action to automatically open a PR after detecting the new releases.

[Enhancement] Switch from Tar to Rpm for Ansible Playbook

With RPM being released on 2.0.0-rc1 (preview) and 1.3.2 today (2022/05/05), we can finally switch from Tar to Rpm for RH-based linux distro.

@saravanan30erd has talked to me about before to have this switch happen.

We can discuss about the approach as we currently dont have deb yet to support Ubuntu.

Thanks.

[BUG][opensearch]

Describe the bug
running the playbook fails because user logstash password is not defined.

To Reproduce
Steps to reproduce the behavior:

1. Run the playbook with the given example in the readme
2. Delete the logstash user and run the playbook,
3. It works.

Expected behavior
Playbook work out of the box, providing only the admin and kibanaserver password in the --extra-vars
Maybe it is just the documentation which was not very clear to me, but that case, the logstash user should be provided as well, as part of the example.

[BUG][Security Plugin configuration | Initialize the opensearch security index in opensearch with custom configs]

Describe the bug
REST-layer traffic not use tls,but run securityadmin.sh failed.
According to the official documentation (https://opensearch.org/docs/latest/security/configuration/tls/), the REST layer does not need to enable TLS, but an error will be reported when executing securityadmin.sh.

To Reproduce
Steps to reproduce the behavior:

1. turn off https for rest ,The configuration is as follows:
   plugins.security.ssl.http.enabled: false
2. restart opensearch
3. run securityadmin.sh ,command is as follows:
   bash /usr/share/opensearch/plugins/opensearch-security/tools/securityadmin.sh -diagnose -cacert /usr/share/opensearch/config/root-ca.pem -cert /usr/share/opensearch/config/admin.pem -key /usr/share/opensearch/config/admin.key -f /usr/share/opensearch/config/opensearch-security/internal_users.yml -nhnv -icl -h 10.200.200.21
4. the errors message :
   Security Admin v7
   Will connect to 10.200.200.21:9200 ... done
   ERR: An unexpected IOException occured: Unrecognized SSL message, plaintext connection?
   Trace:
   java.io.IOException: Unrecognized SSL message, plaintext connection?
   at org.opensearch.client.RestClient.extractAndWrapCause(RestClient.java:959)
   at org.opensearch.client.RestClient.performRequest(RestClient.java:333)
   at org.opensearch.client.RestClient.performRequest(RestClient.java:321)
   at org.opensearch.security.tools.SecurityAdmin.execute(SecurityAdmin.java:573)
   at org.opensearch.security.tools.SecurityAdmin.main(SecurityAdmin.java:163)
   Caused by: javax.net.ssl.SSLException: Unrecognized SSL message, plaintext connection?
   at java.base/sun.security.ssl.SSLEngineInputRecord.bytesInCompletePacket(SSLEngineInputRecord.java:145)
   at java.base/sun.security.ssl.SSLEngineInputRecord.bytesInCompletePacket(SSLEngineInputRecord.java:64)
   at java.base/sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:612)
   at java.base/sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:506)
   at java.base/sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:482)
   at java.base/javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:679)
   at org.apache.http.nio.reactor.ssl.SSLIOSession.doUnwrap(SSLIOSession.java:279)
   at org.apache.http.nio.reactor.ssl.SSLIOSession.doHandshake(SSLIOSession.java:333)
   at org.apache.http.nio.reactor.ssl.SSLIOSession.isAppInputReady(SSLIOSession.java:545)
   at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:120)
   at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:162)
   at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:337)
   at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:315)
   at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:276)
   at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
   at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:591)
   at java.base/java.lang.Thread.run(Thread.java:833)

Screenshots

Host/Environment (please complete the following information):

• Ansible Version: [ 2.15.6]
• Playbook Version: [2.10.0]

solution
if not use tls for rest layer, How to do?

[BUG][Security Plugin Configuration] local action requesting unnecessary privilege escalation

Describe the bug
local action requesting unnecessary privilege escalation.

To Reproduce
Steps to reproduce the behavior:

1. checkout the latest version from this repository
2. Apply the fix for #80
3. run the playbook as instructed on the README:

$ ansible-playbook -i inventories/opensearch/hosts opensearch.yml --extra-vars "admin_password=Test@123 kibanaserver_password=Test@6789 logstash_password=Test@123" --become

5. See the error:

TASK [linux/opensearch : Security Plugin configuration | Check that the files/internal_users.yml exists] ************************************************************************************
fatal: [aos -> localhost]: FAILED! => {"changed": false, "module_stderr": "sudo: a password is required\n", "module_stdout": "", "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error", "rc": 1}

Expected behavior
Finish the deployment without errors.l

Playbook Name
Specify the Playbook which is affected?
roles/linux/opensearch/tasks/security.ym

Screenshots
N/A

Host/Environment (please complete the following information):

• Ansible Version: 2.12.6
• Playbook Version: 2.1.0

Additional context
As the task does not need elevate privileges to be performed it's better to set it to become: false other then force privilege escalation for a simple local file existence check.

[CVE] CVE-2021-45046 in preparation for new OpenSearch patch

Describe the bug
New CVE: https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2021-45046

To Reproduce
Steps to reproduce the behavior:

1. Go to '...'
2. Click on '....'
3. Scroll down to '....'
4. See error

Expected behavior
A clear and concise description of what you expected to happen.

Playbook Name
Specify the Playbook which is affected?

Screenshots
If applicable, add screenshots to help explain your problem.

Host/Environment (please complete the following information):

• Ansible Version: [e.g. 2.x.x]
• Playbook Version: [e.g. 1.x.x]

Additional context
Add any other context about the problem here.

Add a linter to the ansible playbook as workflow

I am currently adding a new linter to the ansible:
https://github.com/peterzhuamazon/ansible-playbook/blob/opensearch-ansible-linter/.github/workflows/lint.yml

I see a lot of warnings and errors being exposed:
https://github.com/peterzhuamazon/ansible-playbook/runs/6335919476?check_suite_focus=true

I am wondering if this is worth adding as it probably will better serve our code.

@saravanan30erd @prudhvigodithi thoughts?

Thanks.

[Question] Interesting fork of Ansible Playbook by remil1000

Hi all,

@saravanan30erd I have noticed @remil1000 has a interesting fork which change a lot of the structure based on our initial setup of ansible playbook, even the support of Debian 11 is added.
https://github.com/remil1000/opensearch-ansible-playbook

@remil1000 we would love to have your changes port back in the upstream we have here.
Is it possible for you to contribute directly to this repo?

Thanks.

cc: @stockholmux

[BUG][Security Plugin] Can't upload configuration

Describe the bug

• ERR: Seems /usr/share/opensearch/plugins/opensearch-security/securityconfig/config.yml is not in OpenSearch Security 7 format: java.io.FileNotFoundException: /usr/share/opensearch/plugins/opensearch-security/securityconfig/config.yml (No such file or directory)
• ERR: Seems /usr/share/opensearch/plugins/opensearch-security/securityconfig/whitelist.yml is not in OpenSearch Security 7 format: java.io.FileNotFoundException: /usr/share/opensearch/plugins/opensearch-security/securityconfig/whitelist.yml (No such file or directory)
• ERR: cannot upload configuration, see errors above

This tool will be deprecated in the next major release of OpenSearch **
** https://github.com/opensearch-project/security/issues/1755           **
**************************************************************************
Security Admin v7
Will connect to 10.10.1.77:9200 ... done
Connected as *****************"
OpenSearch Version: 2.3.0
Contacting opensearch cluster 'opensearch' and wait for YELLOW clusterstate ...
Clustername: *****
Clusterstate: GREEN
Number of nodes: 5
Number of data nodes: 5
.opendistro_security index already exists, so we do not need to create one.
Populate config from /usr/share/opensearch/plugins/opensearch-security/securityconfig/
ERR: Seems /usr/share/opensearch/plugins/opensearch-security/securityconfig/config.yml is not in OpenSearch Security 7 format: java.io.FileNotFoundException: /usr/share/opensearch/plugins/opensearch-security/securityconfig/config.yml (No such file or directory)
Will update '/roles' with /usr/share/opensearch/plugins/opensearch-security/securityconfig/roles.yml 
   SUCC: Configuration for 'roles' created or updated
Will update '/rolesmapping' with /usr/share/opensearch/plugins/opensearch-security/securityconfig/roles_mapping.yml 
   SUCC: Configuration for 'rolesmapping' created or updated
Will update '/internalusers' with /usr/share/opensearch/plugins/opensearch-security/securityconfig/internal_users.yml 
   SUCC: Configuration for 'internalusers' created or updated
ERR: Seems /usr/share/opensearch/plugins/opensearch-security/securityconfig/action_groups.yml is not in OpenSearch Security 7 format: java.io.FileNotFoundException: /usr/share/opensearch/plugins/opensearch-security/securityconfig/action_groups.yml (No such file or directory)
Will update '/tenants' with /usr/share/opensearch/plugins/opensearch-security/securityconfig/tenants.yml 
   SUCC: Configuration for 'tenants' created or updated
ERR: Seems /usr/share/opensearch/plugins/opensearch-security/securityconfig/nodes_dn.yml is not in OpenSearch Security 7 format: java.io.FileNotFoundException: /usr/share/opensearch/plugins/opensearch-security/securityconfig/nodes_dn.yml (No such file or directory)
ERR: Seems /usr/share/opensearch/plugins/opensearch-security/securityconfig/whitelist.yml is not in OpenSearch Security 7 format: java.io.FileNotFoundException: /usr/share/opensearch/plugins/opensearch-security/securityconfig/whitelist.yml (No such file or directory)
ERR: cannot upload configuration, see errors above

To Reproduce
Steps to reproduce the behavior:

1. Clone https://github.com/opensearch-project/ansible-playbook.git
2. Customize Variables in inventories/opensearch/hosts and inventories/opensearch/group_vars/all/all.yml
   2.1 Fix various other problems with the Playbook, already mentioned in the Issues, which hinder the Playbook getting this far
3. Run ansible-playbook -i inventories/opensearch/hosts opensearch.yml --extra-vars "admin_password=Test@123 kibanaserver_password=Test@6789" --become
4. Get stuck on Security Plugin configuration | Initialize the opensearch security index in opensearch

Expected behavior
I expect the Playbook to run successfully and not into errors.

Playbook Name
roles/linux/opensearch/tasks/security.yml

Host/Environment (please complete the following information):

• Freshly installed linux systems
• OpenSearch & Dashboards v2.3.0
• Playbook a2a7f28

[Enhancement] Bump Ansible Playbook to run OpenSearch/Dashboards 1.1.0.

Is your feature request related to a problem? Please describe.
[Enhancement] Bump Ansible Playbook to run OpenSearch/Dashboards 1.1.0.

1.2.0 is coming and 1.1.0 already released.
We should start setting the default version to 1.1.0 for playbooks.
Also suggest having long live branches such as 1.x, 2.x so main will be the unstable branch.

Describe the solution you'd like
See above.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

[Enhancement][IaC] Ability to manage ISM policies and index patterns via role configuration files

Is your feature request related to a problem? Please describe.
PR #63 allows you to configure through the roles of internal users, tenants, roles and roles mappings.

The next step is to expand the functionality available through the configuration of the "opensearch" and "dashboards" roles.

For example, creating index management policies in OpenSearch and creating index patterns in Dashboards

[Enhancement] debian/ubuntu installation via. package manager/apt

Is your feature request related to a problem? Please describe.
Managing the OpenSearch installation via the package manager apt under Debian/Ubuntu is a better way for the version control and especially any future updates.

Describe the solution you'd like
Installing OpenSearch from an APT repository for Ubuntu/Debian: https://opensearch.org/docs/latest/install-and-configure/install-dashboards/debian/#installing-opensearch-dashboards-from-an-apt-repository

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

Rename parameter kibanaserver to dashboardserver - Ansible

Refer: opensearch-project/security#1443

Rename parameter kibanaserver to dashboardserver in Ansible scripts.
We have to wait for this change merged in security plugin, then we should do the same change in ansible.

[BUG][roles/centos7/opensearch/tasks/security.yml]

Describe the bug
In security.yml playbook, the extract command is based on tar but the file is 'zip'.

To Reproduce
Steps to reproduce the behavior:

1. Go to '...'
2. Click on '....'
3. Scroll down to '....'
4. See error

Expected behavior
In task: "Security Plugin Configuration | Extract the certificates generation tool" the command uses tar to unzip file, but tar can't do that. Use 'unzip' command or 'unarchive' ansible module.

Playbook Name
security.yml inside: roles/centos7/opensearch/tasks.

Screenshots

TASK [centos7/opensearch : Security Plugin configuration | Extract the certificates generation tool] *****************************************************[WARNING]: Consider using the unarchive module rather than running 'tar'.  If you need to use command because unarchive is insufficient you can add
'warn: false' to this command task or set 'command_warnings=False' in ansible.cfg to get rid of this message.
fatal: [os1]: FAILED! => {"changed": true, "cmd": ["tar", "-xvf", "search-guard-tlstool.zip"], "delta": "0:00:00.009987", "end": "2021-12-13 08:54:04.439241", "msg": "non-zero return code", "rc": 2, "start": "2021-12-13 08:54:04.429254", "stderr": "tar: Esto no parece un archivo tar\ntar: Saltando a la siguiente cabecera\ntar: Se sale con estado de fallo debido a errores anteriores", "stderr_lines": ["tar: Esto no parece un archivo tar", "tar: Saltando a la siguiente cabecera", "tar: Se sale con estado de fallo debido a errores anteriores"], "stdout": "", "stdout_lines": []}

Host/Environment (please complete the following information):

• Ansible Version: ansible 2.10.8
• Playbook Version: latest from repo
• Debian 11 amd64 OS

Additional context

[BUG][Security Plugin Configuration]

Describe the bug

The Playbook execution fails at the task Security Plugin configuration | Copy the opensearch security internal users template execution on a single-node deployment. Tested on an EC2 Ubuntu instance.

To Reproduce
Steps to reproduce the behavior:

1. checkout the latest version from this repository
2. Change the cluster_type variable to single-node unnecessary step ref: #81 (comment)
3. configure the inventory hosts as below

aos ansible_host=<public_ip> ansible_user=ubuntu ip=<private_ip> roles=data,master

dashboards1 ansible_host=<2nd_instance_public_ip> ansible_user=ubuntu ip=<second_instance_private_ip>

# List all the nodes in the os cluster
[os-cluster]
aos

# List all the Master eligible nodes under this group
[master]
aos

[dashboards]
dashboards1

4. See error

TASK [linux/opensearch : Security Plugin configuration | Copy the opensearch security internal users template] *******************************************************************************
fatal: [aos]: FAILED! => {"changed": false, "checksum": "17c615cd8e9089aedb31891e7aedd1ff8fa32318", "msg": "Destination directory /usr/share/opensearch/plugins/opensearch-security/securityconfig does not exist"}

Expected behavior
Finish the deployment without errors.

Playbook Name
Specify the Playbook which is affected?
roles/linux/opensearch/tasks/security.yml

Screenshots
N/A

Host/Environment (please complete the following information):

• Ansible Version: 2.12.6
• Playbook Version: 2.1.0

Additional context
The task Security Plugin configuration | Copy the opensearch security internal users template execution assumes that the {{ os_sec_plugin_conf_path }} already exists when copying the internal_users template. However, the template module does not create parent directories.

[BUG][No module named 'selinux' during executing of task "Disable the selinux"]

Describe the bug
When task "Disable the selinux" runs on OS without installed SELinux, for example Debian, it is failed with error: "An exception occurred during task execution. The error was: ModuleNotFoundError: No module named 'selinux'".
Fixed by PR 113

To Reproduce
Run playbook opensearch.yaml to install system on debian OS without installed selinux.
The playbook is failed with error "ModuleNotFoundError: No module named 'selinux'"

Expected behavior
Playbook ignore missing selinux

Playbook Name
opensearch.yml playbook - roles linux - opensearch - tasks main - task "Disable the selinux"
opensearch.yml playbook - roles linux - dashboards - tasks main - task "Disable the selinux"

Host/Environment (please complete the following information):
ansible [core 2.14.2]
python version = 3.9.2
jinja version = 3.1.2
Playbook Version: 2.3.0

[Enhancement] Bump Ansible Playbook to run OpenSearch/Dashboards 1.2.0.

Is your feature request related to a problem? Please describe.
[Enhancement] Bump Ansible Playbook to run OpenSearch/Dashboards 1.2.0.

1.3.0 is coming and 1.2.0 already released.
We should start setting the default version to 1.2.0 for playbooks.

Describe the solution you'd like
See above.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

[BUG][linux/opensearch : Security Plugin configuration | Initialize the opensearch security index in opensearch]task failed

Describe the bug
Failed with this playbook when run cluster install.All the opensearch nodes running.

TASK [linux/opensearch : Security Plugin configuration | Initialize the opensearch security index in opensearch]

bash /usr/share/opensearch/plugins/opensearch-security/tools/securityadmin.sh -cacert /usr/share/opensearch/config/root-ca.pem -cert /usr/share/opensearch/config/admin.pem -key /usr/share/opensearch/config/admin.key -f /usr/share/opensearch/plugins/opensearch-security/securityconfig/internal_users.yml -nhnv -icl -h 192.168.20.231

Security Admin v7
Will connect to 192.168.20.231:9300 ... done
Connected as CN=admin.gzhc.local,OU=Ops,O=gzhc.local\, Inc.,DC=gzhc.local
OpenSearch Version: 1.2.3
OpenSearch Security Version: 1.2.3.0
Diagnostic trace written to: /usr/share/opensearch/plugins/opensearch-security/tools/securityadmin_diag_trace_2022-Jan-14_19-07-52.txt
Contacting opensearch cluster 'opensearch' and wait for YELLOW clusterstate ...
ERR: Timed out while waiting for a green or yellow cluster state.
   * Try running securityadmin.sh with -icl (but no -cl) and -nhnv (If that works you need to check your clustername as well as hostnames in your TLS certificates)
   * Make also sure that your keystore or PEM certificate is a client certificate (not a node certificate) and configured properly in opensearch.yml
   * If this is not working, try running securityadmin.sh with --diagnose and see diagnose trace log file)
   * Add --accept-red-cluster to allow securityadmin to operate on a red cluster.

To Reproduce
Steps to reproduce the behavior:

1. inventories/opensearch/hosts file:

gzites231 ansible_host=192.168.20.231  ansible_user=inception ip=192.168.20.231 roles=master,ingest
gzites232 ansible_host=192.168.20.232  ansible_user=inception ip=192.168.20.232 roles=master,ingest
gzites233 ansible_host=192.168.20.233  ansible_user=inception ip=192.168.20.233 roles=master,ingest
gzites225 ansible_host=192.168.20.225  ansible_user=inception ip=192.168.20.225 roles=data
gzites226 ansible_host=192.168.20.226  ansible_user=inception ip=192.168.20.226 roles=data
gzites227 ansible_host=192.168.20.227  ansible_user=inception ip=192.168.20.227 roles=data
gzites228 ansible_host=192.168.20.228  ansible_user=inception ip=192.168.20.228 roles=data
gzites229 ansible_host=192.168.20.229  ansible_user=inception ip=192.168.20.229 roles=data
gzites230 ansible_host=192.168.20.230  ansible_user=inception ip=192.168.20.230 roles=data

dashboards1 ansible_host=192.168.20.231  ansible_user=inception ip=192.168.20.231

# List all the nodes in the os cluster
[os-cluster]
gzites231
gzites232
gzites233
gzites225
gzites226
gzites227
gzites228
gzites229
gzites230
# List all the Master eligible nodes under this group
[master]
gzites231
gzites232
gzites233

[dashboards]
dashboards1

2. The inventories/opensearch/group_vars/all/all.yml file:

## Common opensearch configuration parameters ##

os_cluster_name: opensearch

# opensearch download
os_download_url: http://192.168.20.23:8000/opensearch

# opensearch version
os_version: "1.2.3"

# opensearch dashboards version
os_dashboards_version: "1.2.0"

# Configure hostnames for opensearch nodes
# It is required to configure SSL
# Example es1.example.com, es2.example.com
domain_name: gzhc.local

os_user: opensearch

# Java memory heap values(GB) for opensearch
# You can change it based on server specs
xms_value: 4
xmx_value: 4

# Cluster type whether its single node or multi-node
cluster_type: multi-node

3. run the playbook ansible-playbook -i inventories/opensearch/hosts opensearch.yml --extra-vars "admin_password=Pass@000 kibanaserver_password=Pass@000" -b --become-method=sudo -K -k
4. See error

Expected behavior
A clear and concise description of what you expected to happen.
The cluster should be initialized successfully,but not.
Playbook Name
Specify the Playbook which is affected?
TASK [linux/opensearch : Security Plugin configuration | Initialize the opensearch security index in opensearch] is affected.

Host/Environment (please complete the following information):

• OS: Ubuntu 20.04
• JDK: 11.0.12
• Ansible Version: 2.11.7
• Playbook Version: 1.1.0

Additional context

curl https://192.168.20.231:9200 -k

shown:

OpenSearch Security not initialized.

[Enhancement] Important Settings - Opensearch

Refer: https://opensearch.org/docs/latest/opensearch/install/important-settings/

There are few recommendations in opensearch for production workloads, we need to configure these settings in playbook.

[CVE] CVE-2021-45105 log4j with infinite loop needs to update to 1.2.3.

Describe the bug
https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2021-45105

To Reproduce
Steps to reproduce the behavior:

1. Go to '...'
2. Click on '....'
3. Scroll down to '....'
4. See error

Expected behavior
A clear and concise description of what you expected to happen.

Playbook Name
Specify the Playbook which is affected?

Screenshots
If applicable, add screenshots to help explain your problem.

Host/Environment (please complete the following information):

• Ansible Version: [e.g. 2.x.x]
• Playbook Version: [e.g. 1.x.x]

Additional context
Add any other context about the problem here.

[Enhancement][opensearch] how to use pre created corporate certificates for https

Is your feature request related to a problem? Please describe.
Ability to use pre created corporate certificates, because of security requirements.

Describe the solution you'd like
Put your certificates into some folder, playbook will be using it during opensearch installation

[BUG][ansible-playbook] conflicting variable name used in different roles

Describe the bug
Variable os_plugin_bin_path is used for different purposes in the dashboards role and the opensearch role.
For the dashboards plays, it's intended to point to the opensearch-dashboards-plugin executable, whilst for the opensearch plays, it should point to the opensearch-plugin executable.

This means that you're basically forced to install everything under /usr/share/ (unless you want to clone the repo twice, and maintain separate trees for opensearch and for opensearch-dashboards),
or to edit the ./roles/linux/dashboards/defaults/main.yml and ./roles/linux/opensearch/defaults/main.yml separately for your custom settings.

I would expect ./inventories/opensearch/group_vars/all/all.yml to be the place for custom overrides of the defaults ?

To Reproduce
Steps to reproduce the behavior:

1. Clone the ansible-playbook repository
2. Tweak inventories/opensearch/group_vars/all/all.yml to your liking, adding something like this to the bottom:

iac_enable: true
os_home: /opt/opensearch
os_conf_dir: /opt/opensearch/config

4. Add some hosts to the inventory, with at least 1 dashboards host and at least 1 os-cluster host
5. Run the playbook according to README.md
6. See the task TASK [linux/opensearch : Security Plugin configuration | Initialize the opensearch security index in opensearch with default configs] ***** fail with:

bash: /usr/share/opensearch/plugins/opensearch-security/tools/securityadmin.sh: No such file or directory

8. Add to inventories/opensearch/group_vars/all/all.yml:

os_plugin_bin_path: /opt/opensearch/bin/opensearch-plugin
os_sec_plugin_conf_path: /opt/opensearch/config/opensearch-security
os_sec_plugin_tools_path: /opt/opensearch/plugins/opensearch-security/tools

9. Run the playbook again, like in step 5.
10. See the task TASK [linux/dashboards : Get all the installed dashboards plugins] fail
    with sudo: /opt/opensearch/bin/opensearch-plugin: command not found.

Expected behavior
I expected it to be possible to install opensearch-dashboards with this playbook, even with opensearch data/master data installed on it's own file system mounted somewhere else than underneath /usr/share/.

Playbook Name
opensearch.yml

Role name
linux/dashboards

Screenshots

TASK [linux/dashboards : Get all the installed dashboards plugins] ************************************************************************
fatal: [dashboards-hostname]: FAILED! => {"changed": true, "cmd": ["sudo", "-u", "opensearch-dashboards", "/opt/opensearch/bin/opensearch-plugin", "list"], "delta": "0:00:00.018502", "end": "2023-09-08 12:24:12.839548", "msg": "non-zero return code", "rc": 1, "start": "2023-09-08 12:24:12.821046", "stderr": "sudo: /opt/opensearch/bin/opensearch-plugin: command not found", "stderr_lines": ["sudo: /opt/opensearch/bin/opensearch-plugin: command not found"], "stdout": "", "stdout_lines": []}

Host/Environment (please complete the following information):

• Ansible Version: 2.14.5
• Playbook Version: commit 1b9c5f9

[Enhancement]Need to separate OpenSearch vs Dashboards version in Ansible Playbook

Is your feature request related to a problem? Please describe.
It is possible that a OpenSearch and Dashboards binaries having different version.
Ex: 1.1.1 Dashboards can still use for OpenSearch 1.1.0.

As of now it seems their versions are controlled by a single version number.

Describe the solution you'd like
A clear and concise description of what you want to happen.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

[BUG] OpenSearch and Dashboards - User and Groups settings in systemd service file

Describe the bug

Hard coded user and group in service templates dashboards.service and opensearch.service.

To Reproduce

Expected behavior

Use settings from inventories/opensearch/group_vars/all/all.yml

Playbook Name

Opensearch and Dashboards

Screenshots

Host/Environment (please complete the following information):

Any

Additional context

Solution

Add {{ os_group }} to inventories/opensearch/group_vars/all/all.yml and replace constants in service files with {{ os_user }} and {{ os_group }}. Further recommendation: Introduce {{ dashboards_user }} and {{ dashboards_groups }}

Ensure DCO Workflow Check

Coming from opensearch-project/project-meta#17

A Developer Certificate of Origin is required on commits in the OpenSearch-Project.

See doc.yml for an example workflow. Ensure CONTRIBUTING.md to has a section on the DCO per the project template.

• DCO Check Workflow
• CONTRIBUTING.md DCO Section

[Enhancement][Add the ability to authenticate to OIDC when configuring the cluster]

========== EN ===========

Current situation:

Currently, when the OpenSearch cluster is turned around, authentication is configured based on the internal user database. 2 users are added: admin and kibanaserver. There is no possibility to add new users (in security.yml it is strictly prescribed to set passwords only for these two users). It is also not possible to change other settings (e.g. tenants, roles, role_mapping) or configure authentication via OIDC using these roles

What would you like:

Be able to flexibly configure the above parameters using these roles

========== RU ===========

Текущая ситуация:

В настоящее время при развороте кластера OpenSearch настраивается аутентификация на основе внутренней БД пользователей. Добавляются 2 пользователя: admin и kibanaserver. Возможность добавить новых пользователей отсутствует (в security.yml жёстко прописана установка паролей только для этих двух пользователей). Также отсутствует возможность менять другие настройки (e.g. tenants, roles, roles_mapping) или настроить с помощью этих ролей аутентификацию через OIDC

Чего бы хотелось:

Иметь возможность при помощи данных ролей гибко настраивать указанные выше параметры

[BUG][opensearch]The error was: 'ansible.vars.hostvars.HostVarsVars object' has no attribute 'roles'

Describe the bug
The error was: 'ansible.vars.hostvars.HostVarsVars object' has no attribute 'roles'

Run playbook

git clone https://github.com/opensearch-project/ansible-playbook.git opensearch-project
ansible-playbook -vvv -i host.ini opensearch-project/opensearch.yml -b

Inventory

master0 ansible_host=51.250.110.211 ip=192.168.10.5 roles=master,ingest
data0 ansible_host=51.250.103.164 ip=192.168.10.18 roles=data
data1 ansible_host=51.250.103.114 ip=192.168.10.19 roles=data
dashboard0 ansible_host=51.250.100.220 ip=192.168.10.14

[os-cluster]
master0
data0
data1
dashboard0

[master]
master0

[dashboard]
dashboard0

[all:vars]
ansible_user=ubuntu
ansible_ssh_private_key_file=~/.ssh/id_rsa
domain_name=opensearch.local
os_download_url=https://artifacts.opensearch.org/releases/bundle/opensearch
os_version=1.3.0
os_user=opensearch
cluster_type=multi-node
os_cluster_name=opensearch
xms_value=8
xmx_value=8

error:

TASK [linux/opensearch : OpenSearch Install | Copy Configuration File]
************************************************************************
fatal: [dashboard0]: FAILED! =>
  msg: |-
    The task includes an option with an undefined variable. The error was: 'ansible.vars.hostvars.HostVarsVars object' has no attribute 'roles'

    The error appears to be in '/home/user/github/infrastructure-as-a-code-example/opensearch-cluster/opensearch-project/roles/linux/opensearch/tasks/opensearch.yml': line 28, column 3, but may
    be elsewhere in the file depending on the exact syntax problem.

    The offending line appears to be:


    - name: OpenSearch Install | Copy Configuration File
      ^ here

Version

ansible [core 2.12.2]
  python version = 3.8.10 (default, Nov 26 2021, 20:14:08) [GCC 9.3.0]

Server where ansible

cat /etc/*release*
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=20.04
DISTRIB_CODENAME=focal
DISTRIB_DESCRIPTION="Ubuntu 20.04.3 LTS"
NAME="Ubuntu"
VERSION="20.04.3 LTS (Focal Fossa)"

Remote server

cat /etc/*release*
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=20.04
DISTRIB_CODENAME=focal
DISTRIB_DESCRIPTION="Ubuntu 20.04.3 LTS"
NAME="Ubuntu"
VERSION="20.04.3 LTS (Focal Fossa)"

[Enhancement] Publish role for install opensearch to galaxy

Hello! Thanks for opensearch role

Could developers publish role for install opensearch to galaxy ?

Please create additional "good first issues" to reach the suggested minimum of 15 by 9/15.

Open Source Contributor Initiative (OSCI) launches on September 20th. Java client repo lacks the minimum of 15 "good first issues" that can be picked up by participants of OSCI.

Please create additional "good first issues" to reach the suggested minimum of 15 by 9/15.

[BUG] upgrading opensearch cluster (eg. from 2.2.1 to 2.3.0)

Describe the bug

When upgrading opensearch, eg. from 2.2.1 to 2.3.0, opensearch fails to restart with a "jar hell" error

To Reproduce

1. a full (multi-node) opensearch installation was previously executed, eg os_version: "2.2.1"

2. Change all.yml: os_version: "2.3.0"

3. Run the playbook

4. See the error on the server:

journalctl -u opensearch

Sep 15 11:06:29 [redacted] systemd[1]: Stopping opensearch...
Sep 15 11:06:29 [redacted] systemd[1]: opensearch.service: Deactivated successfully.
Sep 15 11:06:29 [redacted] systemd[1]: Stopped opensearch.
Sep 15 11:06:29 [redacted] systemd[1]: opensearch.service: Consumed 28.873s CPU time.
Sep 15 11:06:29 [redacted] systemd[1]: Started opensearch.
Sep 15 11:06:32 [redacted] opensearch[28098]: WARNING: A terminally deprecated method in java.lang.System has been called
Sep 15 11:06:32 [redacted] opensearch[28098]: WARNING: System::setSecurityManager has been called by org.opensearch.bootstrap.OpenSearch (file:/usr/share/opensearch>
Sep 15 11:06:32 [redacted] opensearch[28098]: WARNING: Please consider reporting this to the maintainers of org.opensearch.bootstrap.OpenSearch
Sep 15 11:06:32 [redacted] opensearch[28098]: WARNING: System::setSecurityManager will be removed in a future release
Sep 15 11:06:35 [redacted] opensearch[28098]: uncaught exception in thread [main]
Sep 15 11:06:35 [redacted] opensearch[28098]: java.lang.IllegalStateException: jar hell!
Sep 15 11:06:35 [redacted] opensearch[28098]: class: org.opensearch.tools.launchers.JvmErgonomics
Sep 15 11:06:35 [redacted] opensearch[28098]: jar1: /usr/share/opensearch/lib/opensearch-launchers-2.3.0.jar
Sep 15 11:06:35 [redacted] opensearch[28098]: jar2: /usr/share/opensearch/lib/opensearch-launchers-2.2.1.jar
Sep 15 11:06:35 [redacted] opensearch[28098]:         at org.opensearch.bootstrap.JarHell.checkClass(JarHell.java:314)
Sep 15 11:06:35 [redacted] opensearch[28098]:         at org.opensearch.bootstrap.JarHell.checkJarHell(JarHell.java:213)
Sep 15 11:06:35 [redacted] opensearch[28098]:         at org.opensearch.bootstrap.JarHell.checkJarHell(JarHell.java:100)
Sep 15 11:06:35 [redacted] opensearch[28098]:         at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:227)
Sep 15 11:06:35 [redacted] opensearch[28098]:         at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:404)
Sep 15 11:06:35 [redacted] opensearch[28098]:         at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:180)
Sep 15 11:06:35 [redacted] opensearch[28098]:         at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:171)
Sep 15 11:06:35 [redacted] opensearch[28098]:         at org.opensearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:104)
Sep 15 11:06:35 [redacted] opensearch[28098]:         at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138)
Sep 15 11:06:35 [redacted] opensearch[28098]:         at org.opensearch.cli.Command.main(Command.java:101)
Sep 15 11:06:35 [redacted] opensearch[28098]:         at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:137)
Sep 15 11:06:35 [redacted] opensearch[28098]:         at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:103)
Sep 15 11:06:35 [redacted] opensearch[28098]: For complete error details, refer to the log at /usr/share/opensearch/logs/aiv-cluster.log

Playbook Name Specify the Playbook which is affected?

• roles/linux/opensearch/tasks/opensearch.yml

Screenshots N/A

Host/Environment (please complete the following information):

• Ansible Version: 2.9.6
• Playbook Version: 2.1.0

Additional context

probably because the files from the new tar are extracted to existing os_home dir, and land next to older existing files, there's this "jar hell" problem ?

[BUG] opensearch.yml: Failure if curl not available

Describe the bug

If curl is not available, the task Check the opensearch status fails with the error

TASK [linux/opensearch : Check the opensearch status] *************************************************************************
[WARNING]: Consider using the get_url or uri module rather than running 'curl'.  If you need to use command because get_url or
uri is insufficient you can add 'warn: false' to this command task or set 'command_warnings=False' in ansible.cfg to get rid
of this message.
fatal: [os1]: FAILED! => {"changed": false, "cmd": "curl 'https://os1:9200/_cluster/health?pretty' -u 'admin:********@123' -k", "msg": "[Errno 2] No such file or directory: b'curl'", "rc": 2}
...

To Reproduce
Steps to reproduce the behavior:

1. Provide machines without curl installed
2. Execute the playbook opensearch.yml using ansible-playbook -i inventories/opensearch/hosts opensearch.yml --extra-vars "admin_password=Test@123 kibanaserver_password=Test@6789" --become
3. See the above error

Expected behavior
No error.

Playbook Name
opensearch.yml

Host/Environment (please complete the following information):

• Ansible Version:2.10.8
• Playbook Version: commit heiderich@c4b2156 (I expect the same to happen with with current main (69b1bd4).

Additional context
As already indicated in the error message, using the uri module would be preferred over calling the external command curl. I guess heiderich@b41ae14 will resolve this issue. However currently I cannot completely test it due to another issue.

Independently of this issue it would be a good idea to validate the certs. But I consider this a separate issue.

[BUG]Opensearch playbook issue with SSL configuration

I followed the steps in the README.

The playbook fails on TASK [linux/opensearch : Wait for opensearch to startup]

When inspecting one of the servers, I find that Opensearch is not running. Its logs end in

Caused by: org.opensearch.OpenSearchException: plugins.security.ssl.transport.keystore_filepath or plugins.security.ssl.transport.server.pemcert_filepath and plugins.security.ssl.transport.client.pemcert_filepath must be set if transport ssl is requested.
	at org.opensearch.security.ssl.DefaultSecurityKeyStore.initTransportSSLConfig(DefaultSecurityKeyStore.java:422) ~[?:?]
	at org.opensearch.security.ssl.DefaultSecurityKeyStore.initSSLConfig(DefaultSecurityKeyStore.java:258) ~[?:?]
	at org.opensearch.security.ssl.DefaultSecurityKeyStore.<init>(DefaultSecurityKeyStore.java:179) ~[?:?]
	at org.opensearch.security.ssl.OpenSearchSecuritySSLPlugin.<init>(OpenSearchSecuritySSLPlugin.java:218) ~[?:?]
	at org.opensearch.security.OpenSearchSecurityPlugin.<init>(OpenSearchSecurityPlugin.java:252) ~[?:?]
	at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
	at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:64) ~[?:?]
	at jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
	at java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:500) ~[?:?]
	at java.lang.reflect.Constructor.newInstance(Constructor.java:481) ~[?:?]
	at org.opensearch.plugins.PluginsService.loadPlugin(PluginsService.java:781) ~[opensearch-1.2.4.jar:1.2.4]
	at org.opensearch.plugins.PluginsService.loadBundle(PluginsService.java:726) ~[opensearch-1.2.4.jar:1.2.4]
	at org.opensearch.plugins.PluginsService.loadBundles(PluginsService.java:528) ~[opensearch-1.2.4.jar:1.2.4]
	at org.opensearch.plugins.PluginsService.<init>(PluginsService.java:194) ~[opensearch-1.2.4.jar:1.2.4]
	at org.opensearch.node.Node.<init>(Node.java:396) ~[opensearch-1.2.4.jar:1.2.4]
	at org.opensearch.node.Node.<init>(Node.java:319) ~[opensearch-1.2.4.jar:1.2.4]
	at org.opensearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:242) ~[opensearch-1.2.4.jar:1.2.4]
	at org.opensearch.bootstrap.Bootstrap.setup(Bootstrap.java:242) ~[opensearch-1.2.4.jar:1.2.4]
	at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:412) ~[opensearch-1.2.4.jar:1.2.4]
	at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:178) ~[opensearch-1.2.4.jar:1.2.4]

Looking at the documentation for TLS, I can't see any reference to plugins.security.ssl.transport.client.pemcert_filepath nor plugins.security.ssl.transport.server.pemcert_filepath.

The config dir contains a number of *.key and *.pem files.

-rw------- 1 opensearch opensearch  1704 Mar  3 10:11 admin.key
-rw------- 1 opensearch opensearch  1537 Mar  3 10:12 admin.pem
-rw------- 1 opensearch opensearch  2204 Mar  3 13:42 jvm.options
drwxr-x--- 2 opensearch opensearch  4096 Jan 14 03:38 jvm.options.d
-rw-rw---- 1 opensearch opensearch 11646 Jan 14 03:42 log4j2.properties
-rw-rw---- 1 opensearch opensearch   196 Mar  3 10:28 opensearch.keystore
drwxr-x--- 2 opensearch opensearch  4096 Jan 14 03:59 opensearch-observability
drwxr-x--- 2 opensearch opensearch  4096 Jan 14 03:59 opensearch-reports-scheduler
-rw------- 1 opensearch opensearch   198 Mar  3 13:43 opensearch.yml
-rw------- 1 opensearch opensearch  1704 Mar  3 10:11 os2_http.key
-rw------- 1 opensearch opensearch  1602 Mar  3 10:11 os2_http.pem
-rw------- 1 opensearch opensearch  1704 Mar  3 10:11 os2.key
-rw------- 1 opensearch opensearch  1602 Mar  3 10:11 os2.pem
-rw------- 1 opensearch opensearch  1704 Mar  3 10:11 root-ca.key
-rw------- 1 opensearch opensearch  1342 Mar  3 10:11 root-ca.pem

The openseach.yml file contains

cluster.name: "frontier"

node.name: "os2"

network.host: "xxx.xxx.xxx.xxx"   <- EDITED

http.port: 9200

bootstrap.memory_lock: true

discovery.seed_hosts: ["os1","os2","os3","os4"]


node.roles: [data,master]

If I add

plugins.security.ssl.transport.pemkey_filepath: os2.key 
plugins.security.ssl.transport.pemcert_filepath: os2.pem
plugins.security.ssl.transport.pemtrustedcas_filepath: root-ca.pem

to the config file and restart Opensearch with systemctl restart opensearch, it goes past the error.

Shouldn't the configuration have been created correctly by the playbook?

Thanks

[BUG] OpenSearch duplicate plugins and modules on upgrade

Describe the bug
I set up OpenSearch 1.2.0 via this playbook some days ago and I got it to work. Now I wanted to update the OpenSearch version to fix the Log4J security vulnerability but I ran into some problems. Via journalctl I could see that duplicate libs, plugins and modules prevented OpenSearch from starting.

To Reproduce
Steps to reproduce the behavior:

1. Install OpenSearch via this Playbook
2. Update OpenSearch via this Playbook by setting OS_VERSION variable to something newer (1.2.1 in my case)
3. Run the playbook again
4. See error

Expected behavior
Update should work in normal environments.

Playbook Name
Opensearch

Host/Environment (please complete the following information):

• Ansible Version: 2.9.27
• Playbook Version: 1.1.0

Additional context
I tried resolving the error by manually deleting old plugin jar files but new errors keep popping up on another plugin. And when all duplicate plugins where removed the error would pop up again with different modules.

Example logs from journal:

uncaught exception in thread [main]
java.lang.IllegalStateException: jar hell!
class: org.opensearch.cli.Command
jar1: /usr/share/opensearch/lib/opensearch-cli-1.2.1.jar
jar2: /usr/share/opensearch/lib/opensearch-cli-1.2.0.jar

uncaught exception in thread [main]
java.lang.IllegalStateException: jar hell!
class: org.apache.logging.log4j.core.appender.SocketAppender$Builder
jar1: /usr/share/opensearch/lib/log4j-core-2.11.1.jar
jar2: /usr/share/opensearch/lib/log4j-core-2.15.0.jar

java.lang.IllegalStateException: codebase property already set: codebase.protocol -> file:/usr/share/opensearch/plugins/opensearch-sql/protocol-1.2.1.0.jar, cannot set to file:/usr/share/opensearch/plugins/opensearch-sql/protocol-1.2.0.0.jar

java.lang.IllegalStateException: codebase property already set: codebase.reindex-client -> file:/usr/share/opensearch/plugins/opensearch-sql/reindex-client-1.2.0.jar, cannot set to file:/usr/share/opensearch/plugins/opensearch-sql/reindex-client-1.2.1.jar

java.lang.IllegalStateException: codebase property already set: codebase.repository-url -> file:/usr/share/opensearch/modules/repository-url/repository-url-1.2.1.jar, cannot set to file:/usr/share/opensearch/modules/repository-url/repository-url-1.2.0.jar

In the end the only way for me was to completely remove plugin and module directory (and create them again or the Playbook would fail). This should be handled by the playbook.

[CVE] CVE-2021-45046

[Enhancement] Support for Oracle linux

Is your feature request related to a problem? Please describe.
Add support for Oracle linux

[CVE] CVE-2021-45046

Baseline MAINTAINERS, CODEOWNERS, and external collaborator permissions

Follow opensearch-project/.github#125 to baseline MAINTAINERS, CODEOWNERS, and external collaborator permissions.

Close this issue when:

1. MAINTAINERS.md has the correct list of project maintainers.
2. CODEOWNERS exists and has the correct list of aliases.
3. Repo permissions only contain individual aliases as collaborators with maintain rights, admin, and triage teams.
4. All other teams are removed from repo permissions.

If this repo's permissions was already baselined, please confirm the above when closing this issue.

[BUG][Documentation] Missing logstash_password on ansible-playbook set up

Describe the bug
The logstash environment is missing on documentation

To Reproduce
Steps to reproduce the behavior:

1. Follow the documentation as is
2. See error

Expected behavior
The cluster should be successfully deployed.

Current behavior
The error bellow is being displayed to the user

fatal: [mpt-kpl01]: FAILED! => {
    "msg": "The task includes an option with an undefined variable. The error was: {'_meta': {'type': 'internalusers', 'config_version': 2}, 'admin': {'hash': '{{ admin_password }}', 'reserved': True, 'backend_roles': ['admin'], 'description': 'admin user'}, 'kibanaserver': {'hash': '{{ kibanaserver_password }}', 'reserved': True, 'description': 'kibanaserver user'}, 'logstash': {'hash': '{{ logstash_password }}', 'reserved': True, 'description': 'logstash user'}}: 'logstash_password' is undefined\n\nThe error appears to be in '/root/ansible-playbook/roles/linux/opensearch/tasks/security.yml': line 268, column 7, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n    # passwords will be written next\n    - name: Security Plugin configuration | Filter service keys from the list of users\n      ^ here\n"
}

Host/Environment (please complete the following information):

• Ansible Version: 2.13.3
• Playbook Version: 2.8.0

[Enhancement]Support for ubuntu 20.04

Support for recent ubuntu distro

Describe the solution you'd like
Be able to choose between different major linux distributions like ubuntu, centos7 also centos 8

Describe alternatives you've considered
Using centos7 is not possible for some environments

Additional context
Add any other context or screenshots about the feature request here.

CVE-2021-45046

[BUG]Log4j CVE-2021-44228 needs to update OpenSearch/Dashboards patch once it is up

Describe the bug

Related to this:
https://discuss.opendistrocommunity.dev/t/log4j-patch-for-cve-2021-44228/7950

OpenSearch is actively working on a patch, will need to change our Ansible playbook to it once it is ready.

To Reproduce
Steps to reproduce the behavior:

1. Go to '...'
2. Click on '....'
3. Scroll down to '....'
4. See error

Expected behavior
A clear and concise description of what you expected to happen.

Playbook Name
Specify the Playbook which is affected?

Screenshots
If applicable, add screenshots to help explain your problem.

Host/Environment (please complete the following information):

• Ansible Version: [e.g. 2.x.x]
• Playbook Version: [e.g. 1.x.x]

Additional context
Add any other context about the problem here.

[BUG][Security Plugin Configuration] securityadmin.sh execution fails

Describe the bug
New securityadmin.sh execution fails on task Security Plugin configuration | Initialize the opensearch security index in opensearch if copy_custom_security_configs is False.

To Reproduce
Steps to reproduce the behavior:

1. checkout the latest version from this repository
2. Apply the fix for #80
3. Apply the fix for #82 or workaround local become request
4. Apply the fix for #86
5. Execute the playbook as instructed on the README:

$ ansible-playbook -i inventories/opensearch/hosts opensearch.yml --extra-vars "admin_password=Test@123 kibanaserver_password=Test@6789" --become 

7. See error

{
  "changed": true,
  "cmd": "bash /usr/share/opensearch/plugins/opensearch-security/tools/securityadmin.sh -cacert /usr/share/opensearch/config/root-ca.pem -cert /usr/share/opensearch/config/admin.pem -key /usr/share/opensearch/config/admin.key -cd /usr/share/opensearch/plugins/opensearch-security/securityconfig -nhnv -icl -h 172.31.83.42
",
  "delta": "0:00:02.979015",
  "end": "2022-07-19 17:57:47.499792",
  "msg": "non-zero return code",
  "rc": 255,
  "start": "2022-07-19 17:57:44.520777",
  "stderr": "",
  "stderr_lines": [],
  "stdout": "**************************************************************************
** This tool will be deprecated in the next major release of OpenSearch **
** https://github.com/opensearch-project/security/issues/1755           **
**************************************************************************
Security Admin v7
Will connect to 172.31.83.42:9200 ... done
Connected as \"CN=admin.example.com,OU=Ops,O=example.com\\\\, Inc.,DC=example.com\"
OpenSearch Version: 2.1.0
Contacting opensearch cluster 'opensearch' and wait for YELLOW clusterstate ...
Clustername: development-cluster
Clusterstate: GREEN
Number of nodes: 1
Number of data nodes: 1
.opendistro_security index already exists, so we do not need to create one.
Legacy index '.opendistro_security' (ES 6) detected (or forced). You should migrate the configuration!
Populate config from /usr/share/opensearch/plugins/opensearch-security/securityconfig/
ERR: Seems /usr/share/opensearch/plugins/opensearch-security/securityconfig/config.yml is not in legacy format: java.io.FileNotFoundException: /usr/share/opensearch/plugins/opensearch-security/securityconfig/config.yml (No such file or directory)
ERR: Seems /usr/share/opensearch/plugins/opensearch-security/securityconfig/roles.yml is not in legacy format: java.io.FileNotFoundException: /usr/share/opensearch/plugins/opensearch-security/securityconfig/roles.yml (No such file or directory)
ERR: Seems /usr/share/opensearch/plugins/opensearch-security/securityconfig/roles_mapping.yml is not in legacy format: java.io.FileNotFoundException: /usr/share/opensearch/plugins/opensearch-security/securityconfig/roles_mapping.yml (No such file or directory)
Will update '/internalusers' with /usr/share/opensearch/plugins/opensearch-security/securityconfig/internal_users.yml (legacy mode)
   SUCC: Configuration for 'internalusers' created or updated
ERR: Seems /usr/share/opensearch/plugins/opensearch-security/securityconfig/action_groups.yml is not in legacy format: java.io.FileNotFoundException: /usr/share/opensearch/plugins/opensearch-security/securityconfig/action_groups.yml (No such file or directory)
ERR: Seems /usr/share/opensearch/plugins/opensearch-security/securityconfig/nodes_dn.yml is not in legacy format: java.io.FileNotFoundException: /usr/share/opensearch/plugins/opensearch-security/securityconfig/nodes_dn.yml (No such file or directory)
ERR: Seems /usr/share/opensearch/plugins/opensearch-security/securityconfig/whitelist.yml is not in legacy format: java.io.FileNotFoundException: /usr/share/opensearch/plugins/opensearch-security/securityconfig/whitelist.yml (No such file or directory)
ERR: cannot upload configuration, see errors above",
  "stdout_lines": [
    "**************************************************************************",
    "** This tool will be deprecated in the next major release of OpenSearch **",
    "** https://github.com/opensearch-project/security/issues/1755           **",
    "**************************************************************************",
    "Security Admin v7",
    "Will connect to 172.31.83.42:9200 ... done",
    "Connected as \"CN=admin.example.com,OU=Ops,O=example.com\\\\, Inc.,DC=example.com\"",
    "OpenSearch Version: 2.1.0",
    "Contacting opensearch cluster 'opensearch' and wait for YELLOW clusterstate ...",
    "Clustername: development-cluster",
    "Clusterstate: GREEN",
    "Number of nodes: 1",
    "Number of data nodes: 1",
    ".opendistro_security index already exists, so we do not need to create one.",
    "Legacy index '.opendistro_security' (ES 6) detected (or forced). You should migrate the configuration!",
    "Populate config from /usr/share/opensearch/plugins/opensearch-security/securityconfig/",
    "ERR: Seems /usr/share/opensearch/plugins/opensearch-security/securityconfig/config.yml is not in legacy format: java.io.FileNotFoundException: /usr/share/opensearch/plugins/opensearch-security/securityconfig/config.yml (No such file or directory)",
    "ERR: Seems /usr/share/opensearch/plugins/opensearch-security/securityconfig/roles.yml is not in legacy format: java.io.FileNotFoundException: /usr/share/opensearch/plugins/opensearch-security/securityconfig/roles.yml (No such file or directory)",
    "ERR: Seems /usr/share/opensearch/plugins/opensearch-security/securityconfig/roles_mapping.yml is not in legacy format: java.io.FileNotFoundException: /usr/share/opensearch/plugins/opensearch-security/securityconfig/roles_mapping.yml (No such file or directory)",
    "Will update '/internalusers' with /usr/share/opensearch/plugins/opensearch-security/securityconfig/internal_users.yml (legacy mode)",
    "   SUCC: Configuration for 'internalusers' created or updated",
    "ERR: Seems /usr/share/opensearch/plugins/opensearch-security/securityconfig/action_groups.yml is not in legacy format: java.io.FileNotFoundException: /usr/share/opensearch/plugins/opensearch-security/securityconfig/action_groups.yml (No such file or directory)",
    "ERR: Seems /usr/share/opensearch/plugins/opensearch-security/securityconfig/nodes_dn.yml is not in legacy format: java.io.FileNotFoundException: /usr/share/opensearch/plugins/opensearch-security/securityconfig/nodes_dn.yml (No such file or directory)",
    "ERR: Seems /usr/share/opensearch/plugins/opensearch-security/securityconfig/whitelist.yml is not in legacy format: java.io.FileNotFoundException: /usr/share/opensearch/plugins/opensearch-security/securityconfig/whitelist.yml (No such file or directory)",
    "ERR: cannot upload configuration, see errors above"
  ]
}

in case of multi-node deployment the service will not start: #83 (comment)

Host/Environment (please complete the following information):

• Ansible Version: 2.12.6
• Playbook Version: 2.1.0

[BUG][opensearch/security] Certificates can not be generated due to allegedly missing java

Describe the bug
When trying to bootstrap a new OpenSearch cluster on Ubuntu 20.04 using the latest main version of this repository, the process is stuck during the initial deployment:

TASK [linux/opensearch : Security Plugin configuration | Generate the node & admin certificates in local] ***
fatal: [os-master-1]: FAILED! => {"changed": true, "cmd": ["/tmp/opensearch-nodecerts/tools/sgtlstool.sh", "-c", "/tmp/opensearch-nodecerts/config/tlsconfig.yml", "-ca", "-crt", "-t", "/tmp/opensearch-nodecerts/config/"], "delta": "0:00:00.005591", "end": "2023-08-07 17:22:33.094310", "msg": "non-zero return code", "rc": 127, "start": "2023-08-07 17:22:33.088719", "stderr": "/tmp/opensearch-nodecerts/tools/sgtlstool.sh: line 11: /usr/lib/jvm/java-1.17.0-openjdk-amd64/bin/java: No such file or directory", "stderr_lines": ["/tmp/opensearch-nodecerts/tools/sgtlstool.sh: line 11: /usr/lib/jvm/java-1.17.0-openjdk-amd64/bin/java: No such file or directory"], "stdout": "", "stdout_lines": []}

As you can see, the script complains about not finding the java binary. However, when connecting to the server und running the java binary manually, everything works as intended:

os-master-1~ # echo $JAVA_HOME
/usr/lib/jvm/java-1.17.0-openjdk-amd64
os-master-1~ # /usr/lib/jvm/java-1.17.0-openjdk-amd64/bin/java --version
openjdk 17.0.8 2023-07-18
OpenJDK Runtime Environment (build 17.0.8+7-Ubuntu-120.04.2)
OpenJDK 64-Bit Server VM (build 17.0.8+7-Ubuntu-120.04.2, mixed mode, sharing)

To Reproduce
Steps to reproduce the behavior:

1. Install all required servers with a minimal Ubuntu 20.04
2. Install OpenJDK 17 (JRE/JDK) and make sure the JAVA_HOME is in the environment
3. Clone the latest version of opensearch-project/ansible-playbook and execute the playbook according to the README

Expected behavior
The error shouldn't occur as Java is not only properly installed but also the JAVA_HOME variable is properly set and the java binary works as intended.

Playbook Name
opensearch/tasks/security.yaml

Host/Environment (please complete the following information):

• Ansible Version: 2.10.8
• Playbook Version: 2.9.0
• Java Version: openjdk 17.0.8 2023-07-18
• Ubuntu Version: 20.04 LTS

Additional context
Executed command:

bastion~ # /opt/opensearch-ansible (main*=) # ansible-playbook -i inventories/opensearch/hosts opensearch.yml --extra-vars "admin_password=<PW> kibanaserver_password=<PW>"

Inventory:

os-master-1 ip=10.33.0.48 roles=data,master
os-master-2 ip=10.33.0.49 roles=data,master
os-master-3 ip=10.33.0.50 roles=data,master
os-data-1 ip=10.33.0.51 roles=data,ingest
os-data-2 ip=10.33.0.52 roles=data,ingest
os-data-3 ip=10.33.0.53 roles=data,ingest

# List all the nodes in the os cluster
[os-cluster]
os-master-1
os-master-2
os-master-3
os-data-1
os-data-2
os-data-3

# List all the Master eligible nodes under this group
[master]
os-master-1
os-master-2
os-master-3

[dashboards]
os-master-1

[Enhancement] Add Opensearch cluster status verification

Add Opensearch cluster status verification in ansible playbook as part of post installation step, and fail the deployment if the cluster health verification fails.

[Testing Confirmation] Confirm current testing requirements

As part of the discussion around implementing an organization-wide testing policy, I am visiting each repo to see what tests they currently perform. I am conducting this work on GitHub so that it is easy to reference.

Looking at the Ansible Playbook repository, it appears there is

Repository	Unit Tests	Integration Tests	Backwards Compatibility Tests	Additional Tests	Link
Ansible Playbook					#117

[BUG][Security Plugin Configuration] Unecessary user left on internal_users.yml

Describe the bug
An unnecessary user logstash was left on the files/internal_users.yml making the execution fail when following the instruction from the README.md

To Reproduce

1. Run this playbook as instructed on the README;

$ ansible-playbook -i inventories/opensearch/hosts opensearch.yml --extra-vars "admin_password=Test@123 kibanaserver_password=Test@6789" --become

2. See the error:

TASK [linux/opensearch : Security Plugin configuration | Filter service keys from the list of users] ****************************************************************************************
fatal: [os1]: FAILED! => {"msg": "The task includes an option with an undefined variable. The error was: {'_meta': {'type': 'internalusers', 'config_version': 2}, 'admin': {'hash': '{{ admin_password }}', 'reserved': True, 'backend_roles': ['admin'], 'description': 'admin user'}, 'kibanaserver': {'hash': '{{ kibanaserver_password }}', 'reserved': True, 'description': 'kibanaserver user'}, 'logstash': {'hash': '{{ logstash_password }}', 'reserved': True, 'description': 'logstash user'}}: 'logstash_password' is undefined\n\nThe error appears to be in '/Users/rodvill/code/git/github/rodolfovillordo/opensearch-playbook/roles/linux/opensearch/tasks/security.yml': line 259, column 7, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n    # passwords will be written next\n    - name: Security Plugin configuration | Filter service keys from the list of users\n      ^ here\n"}

PLAY RECAP **********************************************************************************************************************************************************************************
os1                        : ok=17   changed=4    unreachable=0    failed=1    skipped=27   rescued=0    ignored=0

Playbook Name
Specify the Playbook which is affected?

• roles/linux/opensearch/tasks/security.yml
• files/internal_users.yml

Screenshots
N/A

Host/Environment (please complete the following information):

• Ansible Version: 2.12.6
• Playbook Version: 2.1.0

Additional context
The logstash user is not needed in the default opensearch deployment.

[BUG][opensearch.yml]task failures with non-standard os_api_port

Describe the bug
If os_api_port is changed to a non-standard value (in a multi-node cluster), various tasks fail

To Reproduce

• Define an inventory for a multi-node OS cluster
• Define a custom os_api_port value other than 9200 in either the inventory or group_vars
• Run the the opensearch.yml playbook

Expected behavior
Multi-node OS cluster to be fully functional

Playbook Name
Playbook: opensearch.yml
Task files:

• roles/linux/opensearch/tasks/main.yml (both curl commands)
• roles/linux/opensearch/tasks/security.yml (both securityadmin.sh shell tasks)

Screenshots
N/A

Host/Environment (please complete the following information):

• Ansible Version: 2.9.27
• Playbook Version: current (last commit: 2022-12-13)

Additional context
I could try to create a patch if that helps.

[Enhancement]Test runs on Amazon Linux 2 Machines

Is your feature request related to a problem? Please describe.
We currently support RHEL/CentOS7 machines for this Ansible Playbook.
We also like to check on Amazon Linux 2 as it is the same flavor as RHEL/CentOS7.
It offers another distro for people who want to use this playbook on.

Describe the solution you'd like
A clear and concise description of what you want to happen.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

[BUG][security.yml] deploy custom security config is not supported if auth type is different than oidc

If copy_custom_security_configs is true custom file security_plugin_conf.yml is not deployed because the check is made on auth_type == 'oidc'.

- name: Security Plugin configuration | Copy the security configuration file 3 to cluster
  template:
    src: security_plugin_conf.yml
    dest: "{{ os_sec_plugin_conf_path }}/config.yml"
    backup: yes
    owner: "{{ os_user }}"
    group: "{{ os_user }}"
    mode: 0600
    force: yes
  when: auth_type == 'oidc'

So if you customised security_plugin_conf.yml to use ldap or anything but oidc then you cannot deploy the configuration.

I suggest an easy fix by adding an alternative check :

- name: Security Plugin configuration | Copy the security configuration file 3 to cluster
  template:
    src: security_plugin_conf.yml
    dest: "{{ os_sec_plugin_conf_path }}/config.yml"
    backup: yes
    owner: "{{ os_user }}"
    group: "{{ os_user }}"
    mode: 0600
    force: yes
  when: auth_type == 'oidc' or copy_custom_security_configs

This changed should not cause too much trouble.

[BUG][opensearch] sudo: a password is required

Describe the bug
sudo: a password is required

Run Playbook

git clone https://github.com/opensearch-project/ansible-playbook.git opensearch-project
ansible-playbook -i host.ini opensearch-project/opensearch.yml -b

Inventory

master0 ansible_host=xxxxx ip=192.168.10.34 roles=master,ingest
data0 ansible_host=xxxxx ip=192.168.10.26 roles=data
data1 ansible_host=xxxxx ip=192.168.10.19 roles=data
dashboard0 ansible_host=xxxxx ip=192.168.10.18

[os-cluster]
master0
data0
data1
dashboard0

[master]
master0

[dashboard]
dashboard0

[all:vars]
ansible_user=ubuntu
ansible_ssh_private_key_file=~/.ssh/id_rsa
domain_name=opensearch.local
os_download_url=https://artifacts.opensearch.org/releases/bundle/opensearch
os_version=1.3.0
os_user=opensearch
cluster_type=multi-node
os_cluster_name=opensearch
xms_value=8
xmx_value=8

error:

TASK [linux/opensearch : Security Plugin configuration | Create local temporary directory for certificates generation] *******************************************************************
Friday 01 April 2022  11:54:03 +0600 (0:00:01.821)       0:00:15.542 **********
fatal: [master0 -> localhost]: FAILED! => changed=false
  module_stderr: |-
    sudo: a password is required
  module_stdout: ''
  msg: |-
    MODULE FAILURE
    See stdout/stderr for the exact error
  rc: 1

Version

ansible [core 2.12.2]
  python version = 3.8.10 (default, Nov 26 2021, 20:14:08) [GCC 9.3.0]

Server where ansible

cat /etc/*release*
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=20.04
DISTRIB_CODENAME=focal
DISTRIB_DESCRIPTION="Ubuntu 20.04.3 LTS"
NAME="Ubuntu"
VERSION="20.04.3 LTS (Focal Fossa)"

Remote server

cat /etc/*release*
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=20.04
DISTRIB_CODENAME=focal
DISTRIB_DESCRIPTION="Ubuntu 20.04.3 LTS"
NAME="Ubuntu"
VERSION="20.04.3 LTS (Focal Fossa)"