ritalwar/elastic-agent

Elastic Agent - single, unified way to add monitoring for logs, metrics, and other types of data to a host.

Elastic Agent

Developer docs

The source files for the general Elastic Agent documentation are currently stored in the observability-docs repo. The following docs are only focused on getting developers started building code for Elastic Agent.

Testing

Prerequisites:

• installed mage
• Docker
• X-pack to pre-exist in the parent folder of the local Git repository checkout
• elastic-agent-changelog-tool to add changelog fragments for changelog generation

If you are on a Mac with M1 chip, don't forget to export some docker variable to be able to build for AMD

export DOCKER_BUILDKIT=0
export COMPOSE_DOCKER_CLI_BUILD=0
export DOCKER_DEFAULT_PLATFORM=linux/amd64

In Linux operating systems that you can not run docker as a root user you need to follow linux-postinstall steps

Testing docker container

Running Elastic Agent in a docker container is a common use case. To build the Elastic Agent and create a docker image run the following command:

DEV=true SNAPSHOT=true PLATFORMS=linux/amd64 PACKAGES=docker mage package

If you are in the 7.13 branch, this will create the docker.elastic.co/beats/elastic-agent:7.13.0-SNAPSHOT image in your local environment. Now you can use this to for example test this container with the stack in elastic-package:

elastic-package stack up --version=7.13.0-SNAPSHOT -v

Please note that the docker container is built in both standard and 'complete' variants. The 'complete' variant contains extra files, like the chromium browser, that are too large for the standard variant.

Testing Elastic Agent on Kubernetes

Prerequisites

• create kubernetes cluster using kind, check here for details
• deploy kube-state-metrics, check here for details
• deploy required infrastructure:
  • for elastic agent in standalone mode: EK stack or use elastic cloud, check here for details
  • for managed mode: use elastic cloud or bring up the stack on docker and then connect docker network with kubernetes kind nodes:

  elastic-package stack up -d -v
  docker network connect elastic-package-stack_default <kind_container_id>
  

1. Build elastic-agent:

DEV=true PLATFORMS=linux/amd64 PACKAGES=docker mage package

Use environmental variables GOHOSTOS and GOHOSTARCH to specify PLATFORMS variable accordingly. eg.

❯ go env GOHOSTOS
darwin
❯ go env GOHOSTARCH
amd64

2. Build docker image:

cd build/package/elastic-agent/elastic-agent-linux-amd64.docker/docker-build
docker build -t custom-agent-image .

3. Load this image in your kind cluster:

kind load docker-image custom-agent-image:latest

4. Deploy agent with that image:

• download all-in-ome manifest for elastic-agent in standalone or managed mode, change version if needed

ELASTIC_AGENT_VERSION="8.0"
ELASTIC_AGENT_MODE="standalone"     # ELASTIC_AGENT_MODE="managed"
curl -L -O https://raw.githubusercontent.com/elastic/elastic-agent/${ELASTIC_AGENT_VERSION}/deploy/kubernetes/elastic-agent-${ELASTIC_AGENT_MODE}-kubernetes.yaml

• Modify downloaded manifest:

  • change image name to the one, that was created in the previous step and add imagePullPolicy: Never:

  containers:
    - name: elastic-agent
      image: custom-agent-image:latest
      imagePullPolicy: Never
  

  • set environment variables accordingly to the used setup.

  Elastic-agent in standalone mode: set ES_USERNAME, ES_PASSWORD,ES_HOST.

  Elastic-agent in managed mode: set FLEET_URL and FLEET_ENROLLMENT_TOKEN.

• create

kubectl apply -f elastic-agent-${ELASTIC_AGENT_MODE}-kubernetes.yaml

5. Check status of elastic-agent:

kubectl -n kube-system get pods -l app=elastic-agent

Updating dependencies/PRs

Even though we prefer mage to our automation, we still have some rules implemented on our Makefile as well as CI will use the Makefile. CI will run make check-ci, so make sure to run it locally before submitting any PRs to have a quicker feedback instead of waiting for a CI failure.

Generating the NOTICE.txt when updating/adding dependencies

To do so, just run make notice, this is also part of the make check-ci and is the same check our CI will do.

At some point we will migrate it to mage (see discussion on elastic#1108 and on elastic#1107). However until we have the mage automation sorted out, it has been removed to avoid confusion.