tilfin/fluent-plugin-bunyan-to-google-cloud-logging

Fluentd plugin to parse bunyan format logs and to transfer Google Cloud Logging.

fluent-plugin-bunyan-to-google-cloud-logging

Fluentd plugin to parse bunyan format logs and to transfer Google Cloud Logging.

Installation

RubyGems

$ gem install fluent-plugin-bunyan-to-google-cloud-logging

Bundler

Add following line to your Gemfile:

gem "fluent-plugin-bunyan-to-google-cloud-logging"

And then execute:

$ bundle

Configuration

Example 1. Tailing a log file and transfering to Google Cloud Logging

<source>
  @type tail
  path /tmp/myapp.log
  pos_file /tmp/myapp.log.pos
  tag myapp.trace
  <parse>
    @type json
    time_key time
    time_format %FT%T.%L%z
  </parse>
</source>

<match **>
  @type bunyan_to_google_cloud_logging
  project_id  'my-project-211117'
  keyfile /etc/td-agent/secret/keyfile.json
</match>

Example 2. Receiving docker container logs and transfering to Google Cloud Logging

/etc/td-agent/td-agent.conf

<source>
  @type forward
</source>

<filter docker.**>
  @type parser
  format json
  key_name log
  time_key time
  time_format %FT%T.%L%z
  reserve_data yes
</filter>

<filter docker.**>
  @type record_transformer
  remove_keys log
  <record>
    time ${time}
    app_tag ${tag_parts[1]}
  </record>
</filter>

<match docker.**>
  @type bunyan_to_google_cloud_logging
  project_id  'high-office-001'
  keyfile /etc/td-agent/secret/gcp-key.json
</match>

<label @ERROR>
  <match **>
    @type bunyan_to_google_cloud_logging
    project_id  'high-office-001'
    keyfile /etc/td-agent/secret/gcp-key.json
  </match>
</label>

docker-compose.yml

version: '3'
services:
  yourwebapp:
    image: registry.example.com/yourwebapp
    logging:
      driver: fluentd
      options:
        fluentd-address: "localhost:24224"
        tag: "docker.webapp"
    environment:
      NODE_ENV: production
    ports:
      - 8080:8080

Fluent::Plugin::BunyanToGoogleCloudLoggingOutput

project_id (string) (required)

your Project ID on Google Cloud Platform

keyfile (string) (required)

Specify the path of key json file https://cloud.google.com/iam/docs/creating-managing-service-account-keys

Copyright

• Copyright(c) 2018- Toshimitsu Takahashi
• License
  • Apache License, Version 2.0