elsesiy/fluent-plugin-azure-storage-append-blob-lts

Fluent Plugin Azure Storage Append Blob

Hi, I'm Jonas. I like 🧗 and 📚.

Currently, I work @Snowflake in the Platform team. In my spare time I 👨‍💻 stuff mostly in Go and contribute to all things Kubernetes.

You can follow me on GitHub (obviously), Twitter or bookmark my personal website 😉.

fluent-plugin-azure-storage-append-blob-lts's Issues

Install failing

I'm unable to install this plugin in the official Fluentd (v1.12 and v1.11) Docker image, I don't see any additional requirements listed. I've attached an example docker file to show the issue, could you take a look?

FROM fluent/fluentd:v1.12

# Use root account to use apk
USER root

# Add custom plugins
RUN set -eu; \
  apk add --no-cache --update --virtual .build-deps sudo build-base ruby-dev; \
  sudo gem install --debug --backtrace --verbose \
  fluent-plugin-azure-storage-append-blob-lts; \
  sudo gem sources --clear-all; \
  apk del .build-deps; \
  rm -rf /tmp/* /var/tmp/* /usr/lib/ruby/gems/*/cache/*.gem;

# Set back to non-root user
USER fluent

Add support for Gov cloud

The blob storage service needs to handle different storage endpoint suffixes. These are based on environment definitions documented here.

Add gzip support

See microsoft#27

Support chunk id in storage path

Currently, we are passing Metadata instead of Chunk to function extract_placeholders.

fluent-plugin-azure-storage-append-blob-lts/lib/fluent/plugin/out_azure-storage-append-blob.rb

Line 231 in eb39f3c

@azure_storage_path = extract_placeholders(storage_path, metadata)

This prevents users from using chunk_id as a part of the output path. You will get this error message if you pass ${chunk_id} to azure_object_key_format:

[warn]: #0 ${chunk_id} is not allowed in this plugin. Pass Chunk instead of metadata in extract_placeholders's 2nd argument.