Giter Site home page Giter Site logo

idealista / solr_role Goto Github PK

View Code? Open in Web Editor NEW
25.0 26.0 10.0 879 KB

Ansible role to install an Apache Solr (Cloud) server/cluster

License: Apache License 2.0

JavaScript 20.16% CSS 3.15% XSLT 31.55% YAML 19.31% Jinja 25.84%
solrcloud ansible debian apache-solr solrcloud-server solr-cloud solr-server

solr_role's Introduction

Logo

Build Status

Solr Ansible role

This ansible role installs a Solr server in a debian environment.

Getting Started

These instructions will get you a copy of the role for your ansible playbook. Once launched, it will install a Solr server in a Debian system.

This role is tested on:

  • Debian
    • stretch (java 8 and 11)
    • buster (java 11)
  • Ubuntu
    • focal (java 8 and 14)

Prerequisities

Ansible 2.8.8 version installed. Inventory destination should be a Debian environment.

For testing purposes, Molecule with Docker as driver.

Installing

Create or add to your roles dependency file (e.g requirements.yml):

- src: idealista.solr_role
  version: x.x.x
  name: solr

Install the role with ansible-galaxy command:

ansible-galaxy install -p roles -r requirements.yml -f

Use in a playbook:

---
- hosts: someserver
  roles:
    - { role: solr }

Playbook example below showing how to provision from scratch a SolrCloud cluster with two nodes plus create an example (and empty) collection called sample_techproducts_configs, using idealista java, zookeeper and solr roles:

Note: Assuming that 'solrcloud' group has two nodes (solrcloud1 and solrcloud2) as is declared in molecule.yml, collection will have two shards, one replica and one shard per node as is declared in group vars file called solrcloud.yml and configuration files are stored under directory called sample_techproducts_configs under template directory.

⚠️ Use the example below just as a reference, requires inventory host groups solr and zookeeper to be correctly defined

---

- hosts: zookeeper
  roles:
    - role: zookeeper
  pre_tasks:
    - name: installing required libs
      apt:
        pkg: "{{ item }}"
        state: present
      with_items:
        - net-tools
        - netcat

- hosts: solrcloud
  roles:
    - role: solr_role

Usage

Look to the defaults properties file to see the possible configuration properties.

Add JVM Agent to your installation

This role supports JVM agents (such as Newrelic, Datadog, etc.) to be used inside your installation. You can view an example with JVM Agents in the molecule tests section --> example_setup_with_agent.

Its very simple, must follow this steps (in this case we will add config for Newrelic's agent):

  • Just add the config in your group_vars.
    solr_agents_required_libs:
      - unzip
      - apt-transport-https
    solr_agents_config:
      - name: "newrelic"
        download_url: "http://download.newrelic.com/newrelic/java-agent/newrelic-agent/current/newrelic-java.zip"
        vm_opts:
          - '-javaagent:{{ solr_installation_dir }}/newrelic/newrelic.jar'
        configuration_files:
          - "newrelic.yml"
        params: {
          application_name: "application_sample_name",
          license_key: "your_license_key"
        }
  • Optional: Place the configuration files in the templates folder using this order "templates/{{ agent_name }}/{{ file names specified in solr_agents_config.configuration_files }}.j2. In this case we have the newrelic.yml.j2 in templates/agents/newrelic/newrelic.yml.j2.

Backup with S3

By default, backups are configured to a local directory. This role also supports S3 Backup, just add the next configuration to your variable file:

solr_backup_s3_repository:
  name: s3
  class: org.apache.solr.s3.S3BackupRepository
  default: "false"
  config:
    - name: "s3.bucket.name"
      value: "solr-backup"
      type: str
    - name: "s3.region"
      value: "eu-west-1"
      type: str

The backup bucket needs to be created.

When using IAM user, you can add credentials with these two extra variables (ansible-vault is recommended):

aws_access_key_id: "admin"
aws_secret_access_key: "adminpass"

Set up collections

In order to configure collections just put this config in yml like this example:

solr_mode: cloud
solr_collections:
  # Extracted from https://github.com/apache/lucene-solr/tree/master/solr/server/solr/configsets/sample_techproducts_configs/conf
  # Should have configuration files under "templates/collections/[collection_name]" directory
  sample_techproducts_configs:
    shards: 2
    replicas: 1
    shards_per_node: 1
    auto_add_replicas: false
  sample_techproducts_configs_2:
    shards: 2
    replicas: 1
    shards_per_node: 1

Set up cores

In order to configure cores just put this config in yml like this example:

solr_mode: standalone
solr_cores:
  # Extracted from https://github.com/apache/lucene-solr/tree/master/solr/server/solr/configsets/sample_techproducts_configs/conf
  # Should have configuration files under "templates/collections/[collection_name]" directory
  - mail

Prometheus Exporter

If you want to scrape metrics from Solr using Prometheus, you will need to configure a exporter. We have a Prometheus Solr Exporter role that will make configuration easier for you, just keep in mind that the variables solr_version and prometheus_solr_exporter_version must have the same value.

Testing

$ pipenv sync
$ pipenv shell

# This will execute tests but doesn't destroy created environment (because of --destroy=never)
$ molecule test --destroy=never -s setup_with_collections

Solr Admin UI should be accessible from docker container host at URL:

http://localhost:8983/solr/#/ (node: solrcloud1)

or

http://localhost:8984/solr/#/ (node: solrcloud2)

Solr Admin UI example

See molecule.yml to check possible testing platforms.

Built With

Ansible Molecule Goss

Versioning

For the versions available, see the tags on this repository.

Additionaly you can see what change in each version in the CHANGELOG.md file.

Authors

See also the list of contributors who participated in this project.

License

Apache 2.0 License

This project is licensed under the Apache 2.0 license - see the LICENSE file for details.

Contributing

Please read CONTRIBUTING.md for details on our code of conduct, and the process for submitting pull requests to us.

solr_role's People

Contributors

ademiguel avatar alizbeo-jiang avatar antonioarenzanapulido avatar devetonez avatar dortegau avatar ilorancab avatar jmonterrubio avatar marionv91 avatar santi-eidu avatar snebel29 avatar sorobon avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

solr_role's Issues

Add agents support

Prerequisites

Description

Add java agents support to this role. I want to use agents like newrelic or something else with this role.

Versions

All

Additional Information

Any additional information, configuration or data that might be necessary to reproduce the issue.

Upgrade to solr 8.2.0

Prerequisites

Description

This role is not compatible with the latests releases of solr (solr 8.x).
The role failed to config the new http2 support part and the installation is not able to handle query requests.

Steps to Reproduce

  1. Install 8.x version in a minimun cluster (2 instances)
  2. Create a collection
  3. Query it

Expected behavior: Query result as usually

Actual behavior: This error

Caused by: java.io.IOException: 6/invalid_frame_length
        at org.eclipse.jetty.http2.HTTP2Session.onConnectionFailure(HTTP2Session.java:539)

Reproduces how often: 100%

Versions

All versions

Support for Package Management

Prerequisites

Description

This role already supports copy external libs but in solrcloud mode the recommended way is to use Package Management.
Package Management supports multiple versions in parallel.

Expected behavior: Use same lib with different versions in collections.

Actual behavior: Not possible

Reproduces how often: 100%

Versions

All

Enable all the jetty HttpConfiguration parameters from ansible

Right now only a few parameters could be managed by ansible for the jetty HttpConfiguration. It would be great to add all of them. The list of parameters are:

  • secureScheme
  • securePort
  • outputBufferSize
  • outputAggregationSize
  • requestHeaderSize
  • responseHeaderSize
  • sendServerVersion
  • sendDateHeader
  • headerCacheSize
  • delayDispatchUntilContent

Logging is not working

Prerequisites

Description

Apache Solr changed the logging mechanism and is not working with properties files since the 7.4.0 version.
With the current behaviour of this role we wont have log files in the installation.

7.4.0 release notes

Steps to Reproduce

  1. [First Step]
  2. [Second Step]
  3. [and so on...]

Expected behavior: Log files filled by the application

Actual behavior: No log is registered and we cant see whats going on.

Reproduces how often: 100%

Versions

1.9.0+

Additional Information

Any additional information, configuration or data that might be necessary to reproduce the issue.

Unable to open env file: /etc/default/locale

Prerequisites

Description

/etc/default/locale seems not configured at least running molecule tests.

root@solrcloud1:/opt/solr/server# journalctl -u solr.service
-- Logs begin at Tue 2019-01-29 10:25:33 UTC, end at Tue 2019-01-29 10:53:48 UTC. --
Jan 29 10:29:13 solrcloud1 systemd[1]: Starting LSB: Controls Apache Solr as a Service...
Jan 29 10:29:13 solrcloud1 su[1505]: Successful su for solr by root
Jan 29 10:29:13 solrcloud1 su[1505]: + ??? root:solr
Jan 29 10:29:13 solrcloud1 su[1505]: pam_env(su:session): Unable to open env file: /etc/default/locale: No such file or directory

It would be nice to check if under common installation (not under this docker container) prints that error.

Steps to Reproduce

  1. Execute molecule tests
  2. See journalctl for that service: journalctl -u solr.service

Expected behavior: Not to see that error message

Actual behavior: Prints that error message

Reproduces how often: 100%

Versions

2.1.1

Fix Molecule tests

Prerequisites

Description

Due to problems with idealista/jdk:11.0.2-stretch-openjdk-headless Docker image, tests won't pass for those scenarios with this image.

Change "action" tasks to use the modules instead

Prerequisites

Description

There are three tasks that use action instead of uri. This makes the role incompatible with Ansible 2.8.

External libs support

Prerequisites

Description

We need support to add external libraries like jts to enable the use of spatial behaviour.
The role must run following the steps in [https://lucene.apache.org/solr/guide/7_6/spatial-search.html#jts-and-polygons-flat](this guide) to copy the external libs.

Steps to Reproduce

  1. [First Step]
  2. [Second Step]
  3. [and so on...]

Expected behavior: Add external libs support
Actual behavior: Manual steps required to do this.

Reproduces how often: Always

Versions

All

Additional Information

Any additional information, configuration or data that might be necessary to reproduce the issue.

Improve agents management

Prerequisites

Description

When you run the role twice it will download and configure the agent every run.

Expected behavior: Only do things when its necessary.

Actual behavior: Download and configure the agent every time.

Reproduces how often: Always

Versions

Since 2.4.0

Additional Information

Any additional information, configuration or data that might be necessary to reproduce the issue.

[BUG] cloud mode not working

It looks like servers are started as a standalone installation when deploying from scrap, while the mode in defaults is actually cloud.

Role fails when collections aren't provided

Prerequisites

Description

Role fails when collections aren't provided.

Steps to Reproduce

  1. Executing a molecule test with an empty template/collections folder

Expected behavior: SolrCloud standalone installation when collections are not provided

Actual behavior: It fails

Reproduces how often: 100%

Versions

2.0.0

Current build is failing

Prerequisites

Description

Current build is failing because the maven repository only listen secured requests.

Steps to Reproduce

  1. run molecule tests

Restart when new external lib is added

Prerequisites

Description

When a new external lib is added the service must be restarted to apply it.

Versions

Since external libs behaviour added.

Additional Information

Any additional information, configuration or data that might be necessary to reproduce the issue.

Update to solr 8.5.1

Prerequisites

Description

Default version is solr 8.3.0 and the current stable release is 8.5.1.
May be necessary to update solr.in.sh params.

Fix idempotence tests

There are some tests failing:

ERROR: * [solrcloud.vm] => solrcloud-role : SolrCloud | Extract SolrCloud installation script in the current directory
* [solrcloud.vm] => solrcloud-role : SolrCloud | Execute SolrCloud installation script
* [solrcloud.vm] => solrcloud-role : SolrCloud | Configuring service

Fix CI in Travis

CI fails in Java because Java Installation during ZooKeeper installation:

https://travis-ci.org/idealista/solrcloud-role/jobs/453844457#L784

fatal: [zookeeper.vm]: FAILED! => {"changed": false, "cmd": "/usr/bin/apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv EEA14886", "failed": true, "msg": "Error fetching key EEA14886 from keyserver: hkp://keyserver.ubuntu.com:80", "rc": 2, "stderr": "Warning: apt-key output should not be parsed (stdout is not a terminal)\ngpg: key 8F9293A1EEA14886: public key \"Totally Legit Signing Key <[email protected]>\" imported\ngpg: cannot open '/dev/tty': No such device or address\n", "stderr_lines": ["Warning: apt-key output should not be parsed (stdout is not a terminal)", "gpg: key 8F9293A1EEA14886: public key \"Totally Legit Signing Key <[email protected]>\" imported", "gpg: cannot open '/dev/tty': No such device or address"], "stdout": "Executing: /tmp/apt-key-gpghome.QXesoDqadE/gpg.1.sh --keyserver hkp://keyserver.ubuntu.com:80 --recv EEA14886\n", "stdout_lines": ["Executing: /tmp/apt-key-gpghome.QXesoDqadE/gpg.1.sh --keyserver hkp://keyserver.ubuntu.com:80 --recv EEA14886"]}

Create (and use 😜) an Ansible Solr module

It would be nice if an Ansible Solr Module could be used to:

  • Create/modify/drop collections
  • Create/split/delete shards
  • Create/modify/delete alias
    ...

Actually this role is being updated to manage collections using Solr Collections API but I think that readability could be improved creating an Ansible Module.

AFAIK there isn't a Ansible Module (official or not) to manage Solr collections using Collections API

Java 11 support

Prerequisites

Description

The vm options used in this role are incompatible with java 9+. Solr is fully compatible with java 11 since 7.4.x version.

Steps to Reproduce

  1. [First Step]
  2. [Second Step]
  3. [and so on...]

Expected behavior: Install solr with java8 or java9+

Actual behavior: With java 9+ this role fails.

Reproduces how often: Always when java 9+ is used.

Versions

The version/s you notice the behavior.

Additional Information

Any additional information, configuration or data that might be necessary to reproduce the issue.

Prometheus exporter installation support

Prerequisites

Description

Solr now provides a prometheus exporter to collect the metrics exposed.
Maybe a good idea to include this feature in this role (its similar to collections behaviour).

Steps to Reproduce

  1. [First Step]
  2. [Second Step]
  3. [and so on...]

Expected behavior: [What you expect to happen]

Actual behavior: [What actually happens]

Reproduces how often: [What percentage of the time does it reproduce?]

Versions

The version/s you notice the behavior.

Additional Information

Any additional information, configuration or data that might be necessary to reproduce the issue.

extra space in SOLR_ULIMIT_CHECKS var in solr.in.sh.j2

Prerequisites

Install role with ansible-galaxy install -r requirements.txt
$cat requirements.txt

  • src: idealista.solrcloud-role
    version: 2.4.0
    name: idealista.solrcloud-role

Description

There is an extra space in SOLR_ULIMIT_CHECKS var in solr.in.sh.j2

Steps to Reproduce

  1. grep SOLR_ULIMIT_CHECKS ./templates/solr.in.sh.j2

Expected behavior:
SOLR_ULIMIT_CHECKS="{{ solr_ulimit_checks }}"

Actual behavior:
SOLR_ULIMIT_CHECKS= "{{ solr_ulimit_checks }}"

Reproduces how often: 100%

Versions

2.4.0

Additional Information

I can push this small fix if needed, just dont have perms.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.