This project integrates JFrog logs into various log analytic providers through the use of fluentd as a common logging agent.
The goal of this project is to provide JFrog customers with robust log analytic solutions that they could use to monitor the JFrog unified platform microservices.
This project makes use of git submodules for this reason we recommend you clone with the --recursive flag to ensure all log vendor files are downloaded.
git clone https://github.com/jfrog/log-analytics.git --recursive
cd log-analytics
git submodule foreach git checkout master
git submodule foreach git pull origin masterFluentd is a required component to use this integration.
Fluentd has an logger agent called td-agent which will be required to be installed into each node you wish to monitor logs on.
For more details on how to install Fluentd into your environment please visit: [Fluentd installation guide] (https://docs.fluentd.org/installation) or read the steps provided in this document.
Due to the nature of customer installations varying we cannot account for all possible installations however to ensure our integration works with your installation please review:
JFrog Product Directory Structure guide
The environment variable JF_PRODUCT_DATA_INTERNAL must be defined to the correct location.
Helm based installs will already have this defined based upon the underlying docker images.
For non-k8s based installations below is a reference to the Docker image locations per product. Note these locations may be different based upon the installation location chosen.
Artifactory:
export JF_PRODUCT_DATA_INTERNAL=/var/opt/jfrog/artifactory/
Xray:
export JF_PRODUCT_DATA_INTERNAL=/var/opt/jfrog/xray/
Mission Control:
export JF_PRODUCT_DATA_INTERNAL=/var/opt/jfrog/mc/
Distribution:
export JF_PRODUCT_DATA_INTERNAL=/var/opt/jfrog/distribution/
Pipelines:
export JF_PRODUCT_DATA_INTERNAL=/opt/jfrog/pipelines/var/
Note if you are using Artifactory 6.x you will need to use the legacy environment variable ARTIFACTORY_HOME instead.
Recommended install is through fluentd's native OS based package installs:
| OS | Package Manager | Link |
|---|---|---|
| CentOS/RHEL | RPM (YUM) | https://docs.fluentd.org/installation/install-by-rpm |
| Debian/Ubuntu | APT | https://docs.fluentd.org/installation/install-by-deb |
| MacOS/Darwin | DMG | https://docs.fluentd.org/installation/install-by-dmg |
| Windows | MSI | https://docs.fluentd.org/installation/install-by-msi |
User installs can utilize the zip installer for Linux
| OS | Package Manager | Link |
|---|---|---|
| Linux (x86_64) | ZIP | https://github.com/jfrog/log-analytics/raw/master/fluentd-installer/fluentd-1.11.0-linux-x86_64.tar.gz |
Download it to a directory the user has permissions to write such as the $JF_PRODUCT_DATA_INTERNAL locations discussed above:
cd $JF_PRODUCT_DATA_INTERNAL
wget https://github.com/jfrog/log-analytics/raw/master/fluentd-installer/fluentd-1.11.0-linux-x86_64.tar.gz
Untar to create the folder:
tar -xvf fluentd-1.11.0-linux-x86_64.tar.gz
Move into the new folder:
cd fluentd-1.11.0-linux-x86_64
Run the fluentd wrapper with one argument pointed to the configuration file to load:
./fluentd test.conf
Next steps are to setup a fluentd.conf file using the relevant integrations for Splunk, DataDog, Elastic, or Prometheus.
Recommended install for Docker is to utilize the zip installer for Linux
| OS | Package Manager | Link |
|---|---|---|
| Linux (x86_64) | ZIP | https://github.com/jfrog/log-analytics/raw/master/fluentd-installer/fluentd-1.11.0-linux-x86_64.tar.gz |
Download it to a directory the user has permissions to write such as the $JF_PRODUCT_DATA_INTERNAL locations discussed above:
cd $JF_PRODUCT_DATA_INTERNAL
wget https://github.com/jfrog/log-analytics/raw/master/fluentd-installer/fluentd-1.11.0-linux-x86_64.tar.gz
Untar to create the folder:
tar -xvf fluentd-1.11.0-linux-x86_64.tar.gz
Move into the new folder:
cd fluentd-1.11.0-linux-x86_64
Run the fluentd wrapper with one argument pointed to the configuration file to load:
./fluentd test.conf
Next steps are to setup a fluentd.conf file using the relevant integrations for Splunk, DataDog, Elastic, or Prometheus.
Recommended install for Kubernetes is to utilize the helm chart with the associated values.yaml in this repo.
| Product | Example Values File |
|---|---|
| Artifactory | helm/artifactory-values.yaml |
| Artifactory HA | helm/artifactory-ha-values.yaml |
| Xray | helm/xray-values.yaml |
| Distribution | helm/distribution-values.yaml |
| Mission Control | helm/mission-control-values.yaml |
| Pipelines | helm/pipelines-values.yaml |
To modify existing Kubernetes based deployments without using Helm users can use the zip installer for Linux:
| OS | Package Manager | Link |
|---|---|---|
| Linux (x86_64) | ZIP | https://github.com/jfrog/log-analytics/raw/master/fluentd-installer/fluentd-1.11.0-linux-x86_64.tar.gz |
Download it to a directory the user has permissions to write such as the $JF_PRODUCT_DATA_INTERNAL locations discussed above:
cd $JF_PRODUCT_DATA_INTERNAL
wget https://github.com/jfrog/log-analytics/raw/master/fluentd-installer/fluentd-1.11.0-linux-x86_64.tar.gz
Untar to create the folder:
tar -xvf fluentd-1.11.0-linux-x86_64.tar.gz
Move into the new folder:
cd fluentd-1.11.0-linux-x86_64
Run the fluentd wrapper with one argument pointed to the configuration file to load:
./fluentd test.conf
Next steps are to setup a fluentd.conf file using the relevant integrations for Splunk, DataDog, Elastic, or Prometheus.
Non-root users to make life easier for we have provided a tar.gz containing everything you need to run fluentd.
Follow these steps:
-
Download the tar from this Github: fluentd-installer/fluentd-1.11.0-linux-x86_64.tar.gz
-
Explode the tar into $JF_PRODUCT_DATA_INTERNAL and run:
$JF_PRODUCT_DATA_INTERNAL/fluentd-1.11.0-linux-x86_64/fluentd <conf_file>
Updating fluentd to future releases is simple as well:
$JF_PRODUCT_DATA_INTERNAL/fluentd-1.11.0-linux-x86_64/lib/ruby/bin/gem install fluentd
Adding any fluentd plugins like Datadog as works in the same fashion:
$JF_PRODUCT_DATA_INTERNAL/fluentd-1.11.0-linux-x86_64/lib/ruby/bin/gem install fluent-plugin-datadog
- Package Manager installations of td-agent only.
The default configuration file for td-agent is located at:
/etc/td-agent/td-agent.conf
You should update this configuration file and run td-agent as a service.
If you wish to only run td-agent against a test configuration file you can also run:
td-agent -c fluentd.conf
Once td-agent has been installed on an Artifactory or Xray node you will also need to install the relevant plugin if you are using Splunk or Datadog:
Splunk:
td-agent-gem install fluent-plugin-splunk-enterprise
Datadog:
td-agent-gem install fluent-plugin-datadog
Elastic:
td-agent-gem install fluent-plugin-elasticsearch
It's also possible to use the shell script (bash) to install Fluentd as service or create a docker image. The script is limited to Linux (Centos, Red Hat, Amazon). The script performs the following tasks:
- Checks if the Fluentd requirements are met and updates the OS if needed.
- Installs/Updates Fluentd as a service depending on the Linux distro (Centos, Red Hat and Amazon is supported).
- Creates docker image based on the latest Fluentd docker image.
- Updates the JPD log files/folders permissions.
- Installs and configures Fluentd plugins (Splunk, Datadog).
- Starts and enables the Fluentd service.
[ALERT] Depending on the selected options SUDO might be required.
To download and run the script (Linux only):
wget https://raw.githubusercontent.com/jfrog/log-analytics/master/fluentd-installer/scripts/linux/fluentd-agent-installer.sh
chmod +x fluentd-agent-installer.sh
./fluentd-agent-installer.sh| OS | Package Manager | Link |
|---|---|---|
| Linux (x86_64) Centos/Red Hat/Amazon | N/A | https://raw.githubusercontent.com/jfrog/log-analytics/master/fluentd-installer/scripts/linux/fluentd-agent-installer.sh |
The following sections will cover how you can configure fluentd once it has been installed.
Fluentd requires configuration file to know which logs to tail and how to ship them to the correct log provider.
Our configurations are saved into each log provider's folder.
We will need to store these configurations into the correct location per our installer type.
The environment variable JF_PRODUCT_DATA_INTERNAL must be defined to the correct location.
Helm based installs will already have this defined based upon the underlying docker images.
For non-k8s based installations below is a reference to the Docker image locations per product. Note these locations may be different based upon the installation location chosen.
Artifactory:
export JF_PRODUCT_DATA_INTERNAL=/var/opt/jfrog/artifactory/
Xray:
export JF_PRODUCT_DATA_INTERNAL=/var/opt/jfrog/xray/
Mision Control:
export JF_PRODUCT_DATA_INTERNAL=/var/opt/jfrog/mc/
Distribution:
export JF_PRODUCT_DATA_INTERNAL=/var/opt/jfrog/distribution/
Pipelines:
export JF_PRODUCT_DATA_INTERNAL=/opt/jfrog/pipelines/var/
If you are running on RT 6.x you will need to ensure the ARTIFACTORY_HOME environment variable is set instead.
Additional information how to configure fluentd conf:
By default td-agent will run as the td-agent user however the JFrog logs folder only has file permissions for the artifactory or xray user.
- Fix the group and file permissions issue in Artifactory as root:
usermod -a -G artifactory td-agent
chmod 0770 $JF_PRODUCT_DATA_INTERNAL/log
chmod 0640 $JF_PRODUCT_DATA_INTERNAL/log/*.log
- Fix the group and file permissions issue in Xray as root:
usermod -a -G xray td-agent
chmod 0770 $JF_PRODUCT_DATA_INTERNAL/log
chmod 0640 $JF_PRODUCT_DATA_INTERNAL/log/*.log
- Run td-agent and check it's status
systemctl start td-agent
systemctl status td-agent
Using systemd:
- Create a service unit configuration file
mkdir -p ~/.config/systemd/user/
touch ~/.config/systemd/user/jfrogfluentd.service
- Copy paste below snippet, update the path to match $JF_PRODUCT_DATA_INTERNAL/ and fluentd configuration file location, and save into the file:
[Unit]
Description=JFrog_Fluentd
[Service]
ExecStart=/opt/jfrog/artifactory/var/fluentd-1.11.0-linux-x86_64/fluentd <conf_file>
Restart=always
[Install]
WantedBy=graphical.target
See man systemd.service and man systemd.unit for more options.
- Enable service in userspace
systemctl --user enable jfrogfluentd
- Start it and check it's status
systemctl --user start jfrogfluentd
systemctl --user status jfrogfluentd
- Enjoy!
We currently support Splunk, Datadog, Elastic, and Prometheus as log vendors. The relevant configuration files can be found in the related sub repo for each vendor.
We currently use JFrog's Xray SIEM plugin to observe vulnerabilities associated with artifacts
- Fluentd - Fluentd Logging Aggregator/Agent
- Splunk - Splunk Logging Platform
- Splunk HEC - Splunk HEC used to upload data into Splunk
- Elasticsearch - Elastic search log data platform
- Kibana - Elastic search visualization layer
- Prometheus - Prometheus metrics and monitoring
Please read CONTRIBUTING.md for details on our code of conduct, and the process for submitting pull requests to us.
We use SemVer for versioning. For the versions available, see the tags on this repository.
- Github issues.
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent