fdv/running-elasticsearch-fun-profit

A book about running Elasticsearch

WIP, COVERS ELASTICSEARCH 5.5.x, UPDATING TO ES 6.5.x

Operating Elasticsearch

for Fun and Profit

---

Fred de Villamil

---

Read online

---

Code of Conduct

• Behave like normal, friendly, welcoming human beings or get the hell out.
• Any reference to a non scientific, verifiable element is irrelevant.

---

TOC

• Getting Started with Elasticsearch

  • Prerequisites
  • Elasticsearch basic concepts
    • REST APIs
    • Open Source
    • Java
    • Distributed
    • Scalable
    • Fault tolerant
  • What's an Elasticsearch cluster?
    • Master node
    • Ingest nodes
    • Data Nodes
    • Tribe Nodes
    • A Minimal, Fault Tolerant Elasticsearch Cluster
  • What's an Elasticsearch index
  • Deploying your first Elasticsearch cluster
    • Deploying Elasticsearch on Debian
    • Deploying Elasticsearch on RHEL / CentOS
  • First step using Elasticsearch
  • Elasticsearch Configuration
  • Elasticsearch Plugins

• Elasticsearch and the Java Virtual Machine

  • Supported JVM and operating systems / distributions
    • Operating system matrix
    • Java Virtual Machine matrix
  • Memory management
  • Garbage collection
    • Concurrent Mark & Sweep Garbage Collector
    • Garbage First Garbage Collector

• A few things you need to know about Lucene

  • Lucene segments
  • Lucene deletes and updates

• Designing the Perfect Elasticsearch Cluster

  • Elasticsearch is elastic, for real
  • Design for failure
  • Hardware
    • CPU
    • Memory
    • Network
    • Storage
  • Software
    • The Linux (or FreeBSD) kernel
    • The Java Virtual Machine
    • The filesystem
  • Designing your indices
    • Sharding
    • Replication
  • Optimising allocation
  • Troubleshooting and scaling
    • CPU
    • Memory

• Design for Event Logging

  • Design of an event logging infrastructure cluster
    • Throughput: how many events per second (005-design-event-logging/005-design-event-logging.md//eps) are you going to collect?
    • Retention: how long do you want to keep your data, hot and cold?
    • Size: what is the average size of a collected event?
    • Fault tolerance: can you afford losing your indexed data?
    • Queries
  • Which hardware do I need?
  • How to design my indices?
  • What about some tuning?

• Operating Daily

  • Elasticsearch most common operations
    • Mass index deletion with pattern
    • Mass optimize, indexes with the most deleted docs first
    • Restart a cluster using rack awareness
    • Optimize your cluster restart
    • Remove data nodes from a cluster the safe way
  • Get useful information about your cluster
    • Nodes information
    • Monitor your search queues
    • Indices information
    • Shard allocation information
    • Recovery information
    • Segments information (006-operating-daily/006-operating-daily.md//can be extremely verbose)
    • Cluster stats
    • Nodes stats
    • Indice stats
    • Indice mapping
    • Indice settings
    • Cluster dynamic settings
    • All the cluster settings (006-operating-daily/006-operating-daily.md//can be extremely verbose)

• Monitoring Elasticsearch

  • Tools
  • Monitoring at the host level
  • Monitoring at the node level
  • Monitoring at the cluster level
  • Monitoring at the index level

• How we reindexed 36 billion documents in 5 days within the same Elasticsearch cluster

  • The "Blackhole" cluster
  • Elasticsearch configuration
  • Tuning the Java virtual machine
    • Blackhole Initial indexing
  • Blackhole initial migration
  • Blackhole reindexing
    • The reindexing process
    • Logstash configuration
    • Reindexing Elasticsearch configuration
    • Introducing Yoko and Moulinette
  • Reindexing in 5 days
  • Conclusion

• Use Case: Migrating a Cluster Across the Ocean Without Downtime

• Use Case: An Advanced Elasticsearch Architecture for High-volume Reindexing

  • A glimpse at our infrastructure
  • Using Elasticsearch for fun and profit
  • Conclusion

• Migrating a 130TB Cluster from Elasticsearch 2 to 5 in 20 Hours with 0 Downtime and a Rollback Strategy

  • Elasticsearch @Synthesio, November 2017
  • The Blackhole Cluster
  • Migration Strategies: Cluster restart VS Reindex API VS Logstash VS the Fun Way
    • The Cluster Restart Strategy
    • The Reindex API Strategy
    • The Logstash Strategy
    • The Fun Way
  • Migrating Blackhole for Real
    • Expanding Blackhole
    • Splitting Blackhole in 2
  • Conclusion

---

Styling

This is the Markdown styling used in this book. If you plan to contribute, please use it.

Chapter title

# This is a chapter title

Chapter part

---

## A chapter part title is preceded by an horizontal line

Chapter subpart

### A level 1 subpart
#### A level 2 subpart

Images

![An image should have an alt text](use/a/relative.link)

Code:

An `inline code block` goes like this

API calls go the Curl way

curl -X POST "localhost:9200/_search" -H 'Content-Type: application/json' -d'
{
    "query" : {
        "match_all" : {}
    },
    "stats" : ["group1", "group2"]
}
'

Yaml code is expanded for more readability

---
some:
  value:
    goes: "like this"

Links

[An internal link](has/a/relative.path)
[An external link](https://has.an.absolute/path)

Lists

Urdered lists:

Only one line break between a paragraph and

* An
* unordered
* list
	* with
	* subitems

Ordered lists:

1. An
2. Ordered
3. List
	1. With
	2. subitems

running-elasticsearch-fun-profit's Issues

Add missing First step using Elasticsearch

• Adding data
• First search
• Finding the configurations
• Installing / removing plugins

Add docsify static website generator

Docsify has good website UI. https://docsify.js.org/#/

Add missing "What's an Elasticsearch Index" part

Add new chapter: benchmarking your cluster

Add a code of conduct

Update the commands to ES 6.5.x

Add missing Elasticsearch basic concepts

Update to support 7.X.X? - Enhancement

Redo all the schema using draw.io for more consistency

Change the whole project structure

To make chapters insertion easier, the new structure will be:

root/
    001-chapter-title/
        content.md
        images/
          picture.svg

Pictures should not have alpha transparency

This makes the files bigger and the schemas unreadable on any non-white screen.

Move the use cases numbering from 100

Use cases come last in the book, and I don't want have to rename my chapters every time I want to insert a new one.

Add missing Deploying your Elasticsearch cluster on...

• Debian
• RHEL / CentOS
• FreeBSD

Move 003-cluster-design A few things you need to know about Lucene into its own chapter

Clarification in cluster design

In the following excerpt do the number defined to be read as 10k and 50k respectively ?

I try not to run more than 10.000 open indices and 50.000 primary shards on the same cluster.

Use Semantic Versioning so we can keep track of new contents

Great work @fdv
I am almost at the end of the "book".

What do you think of using [https://semver.org/](semantic versions) so we can keep up with your work. Thus readers could ask GitHub to notify when new versions are released.

Being the semantic version made up of at least three numbers (MAJOR.MINOR.PATCH), I thought of something like this:

• MAJOR version when you add a new chapter.
• MINOR version when you significantly change something in a way that those who had read the old content would have to read it again to fully understaind the subject.
• PATCH version when your changes does not affect the understanting of subject. Those who had read it does not need to read it again.

Add a part about Java memory management

Chinese Version Contribution

Hi all, I am an ES engineer in China and very interested in your project. I would like to make some contribution on this wonderful project by translating the docs into Chinese and make the book more popular in Chinese readers.

Missing image

https://github.com/fdv/running-elasticsearch-fun-profit/blame/master/004-cluster-design/004-cluster-design.md#L86 image2.svg is missing

Add a functional and technical description of CMS GC

Add missing cluster schema

Add a script to regenerate the TOC every time a file is updated

Make is a pre-commit git hook?

Fixe code styling

• define a global code styling
• apply everywhere

Add a functional and technical description of G1GC

image17.png in 103-use-case-migrating-130tb-cluster-without-downtime is still a png

what is http nodes？

In https://github.com/fdv/running-elasticsearch-fun-profit/blob/master/004-cluster-design/004-cluster-design.md， You mentioned http client 3 times.
But In elasticsearch concept, there is only master client, data client, ingest client, coordinate client, there is no http client.
So I want to know the meaning of http client?
I think the http client is coordinate client，I want to konw the real meaning
Thanks！