spotify / ffwd Goto Github PK

View Code? Open in Web Editor NEW

78.0 26.0 33.0 2.03 MB

a flexible metric forwarding agent

Home Page: https://spotify.github.io/ffwd/

License: Apache License 2.0

Java 98.74% Shell 0.16% Kotlin 1.07% Dockerfile 0.03%

java monitoring metrics

ffwd's Introduction

DEPRECATION NOTICE

This repo is no longer actively maintained. While it should continue to work and there are no major known bugs, we will not be improving ffwd or releasing new versions.

ffwd »

ffwd is a flexible metric forwarding agent. It is intended to run locally on the system and receive metrics through a wide set of protocols and then forward them to your TSDB.

By running locally, it is easily available to receive pushed data from any application or service that is running on the same system.

ffwd decorates the received metrics with system-wide tags or attributes. By doing this, the application generating the data becomes simpler to build, maintain, and configure since it doesn't have to know where it is running. Only that ffwd is available on the loopback interface.

Head over to https://spotify.github.io/ffwd/ for documentation.

Quick Start

ffwd can be started quickly with docker. This can be useful to run locally when troubleshooting metrics with your service.

docker run -it -p 19091:19091/udp -p 19000:19000 -p 8080:8080 spotify/ffwd:latest

Production Debugging

If the debug port is enabled, metrics can be emited to a shell with netcat:

nc localhost 19001

Clients

Libraries

semantic-metrics (ffwd-reporter)

Developing

This project is built using Maven. The package phase will also build a debian package.

mvn package

You can run the client using tools/ffwd.

$> tools/ffwd agent/ffwd.yaml

Testing

We run unit tests with Maven:

$ mvn test

A more comprehensive test suite is enabled with the environment=test property.

$ mvn -D environment=test verify

This adds:

It is strongly recommended that you run the full test suite before setting up a pull request, otherwise it will be rejected by the CI system.

Code of Conduct

This project adheres to the Open Code of Conduct. By participating, you are expected to honor this code.

Releasing

Releasing is done via the maven-release-plugin.

To release, run:

mvn release:clean release:prepare -D autoVersionSubmodules=true

You will be prompted for the release version and the next development version.

Add a Github release based on the tag that was created from the above command with notes on what changed.

ffwd's People

Contributors

Stargazers

Watchers

ffwd's Issues

Document configuration

The documentation should have all of the configuration fields, and the ways to load them (yaml file, env vars, etc).

flushing output plugins should be able to limit by size of batch

To prevent infinite memory use flushes, FlushingPluginSink has to be able to support size limiting in its batches.

If a batch is in progress, the current batch should be dropped, and instrumented appropriately as dropped data.

Large UDP payloads can be silently dropped

Due to nature of UDP protocol data could be silently dropped if reached specific limits.
As the data/metrics are dropped silently usually we alerted by teams noticing missing metrics.

One idea discussed is to put some ignorable/optional metric indicating the number of bytes or metrics sent over the wire so FFWD agent could detect the loss and emit a metric to be alerted on.

Incorrect META-INF/services records in ffwd-http-client

Jar-file of ffwd-http-client contains incorrect records in META-INF/services.

$ jar -xf ffwd-http-client-0.4.1.jar
$ cat META-INF/services/com.fasterxml.jackson.databind.Module
com.spotify.ffwd.http.fasterxml.jackson.datatype.jdk8.Jdk8Module

Note the mismatch between com.fasterxml in the name of the file and com.spotify.ffwd.http.fasterxml inside it.

This breaks java service discovery. For example, having ffwd-http-client on classpath and doing new ObjectMapper().findAndRegisterModules() results in this error:

java.util.ServiceConfigurationError: com.fasterxml.jackson.databind.Module:
Provider com.spotify.ffwd.http.fasterxml.jackson.datatype.jdk8.Jdk8Module not a subtype

PubSub dependencies are broken

Introduced by #228. That PR added gRPC dependencies that are probably conflicting with PubSub. The failure only happens when trying to authenticate to PubSub, so the tests using the emulator still pass.

08:58:16.151 [main] ERROR com.spotify.ffwd.FastForwardAgent - Uncaught exception in thread main, exiting (status = 2)
java.lang.RuntimeException: Failed during initialization
	at com.spotify.ffwd.AgentCore.<init>(AgentCore.java:115)
	at com.spotify.ffwd.AgentCore.<init>(AgentCore.java:91)
	at com.spotify.ffwd.AgentCore$Builder.build(AgentCore.java:467)
	at com.spotify.ffwd.FastForwardAgent.setup(FastForwardAgent.java:106)
	at com.spotify.ffwd.FastForwardAgent.main(FastForwardAgent.java:54)
Caused by: com.google.inject.CreationException: Unable to create injector, see the following errors:

1) Error in custom provider, java.lang.IncompatibleClassChangeError: class io.grpc.auth.GoogleAuthLibraryCallCredentials can not implement io.grpc.CallCredentials, because it is not an interface (io.grpc.CallCredentials is in unnamed module of loader 'app')
  at com.spotify.ffwd.pubsub.PubsubOutputPlugin$1.publisher(PubsubOutputPlugin.java:132) (via modules: com.spotify.ffwd.output.OutputManagerModule$1 -> com.spotify.ffwd.pubsub.PubsubOutputPlugin$1)
  while locating com.google.cloud.pubsub.v1.Publisher
    for field at com.spotify.ffwd.pubsub.PubsubPluginSink.publisher(PubsubPluginSink.java:59)
  while locating com.spotify.ffwd.pubsub.PubsubPluginSink
  at com.spotify.ffwd.pubsub.PubsubOutputPlugin$1.configure(PubsubOutputPlugin.java:189) (via modules: com.spotify.ffwd.output.OutputManagerModule$1 -> com.spotify.ffwd.pubsub.PubsubOutputPlugin$1)
  while locating com.spotify.ffwd.pubsub.PubsubPluginSink annotated with @com.google.inject.name.Named(value="pubsubSink")
  while locating com.spotify.ffwd.output.BatchablePluginSink annotated with @com.spotify.ffwd.output.BatchingDelegate()
    for field at com.spotify.ffwd.output.BatchingPluginSink.sink(BatchingPluginSink.java:56)
  at com.spotify.ffwd.output.OutputPlugin$1.configure(OutputPlugin.java:98) (via modules: com.spotify.ffwd.output.OutputManagerModule$1 -> com.spotify.ffwd.pubsub.PubsubOutputPlugin$1 -> com.spotify.ffwd.output.OutputPlugin$1)

1 error
	at com.google.inject.internal.Errors.throwCreationExceptionIfErrorsExist(Errors.java:543)
	at com.google.inject.internal.InternalInjectorCreator.injectDynamically(InternalInjectorCreator.java:178)
	at com.google.inject.internal.InternalInjectorCreator.build(InternalInjectorCreator.java:109)
	at com.google.inject.internal.InjectorImpl.createChildInjector(InjectorImpl.java:237)
	at com.spotify.ffwd.AgentCore.setupPrimaryInjector(AgentCore.java:374)
	at com.spotify.ffwd.AgentCore.<init>(AgentCore.java:113)
	... 4 more
Caused by: java.lang.IncompatibleClassChangeError: class io.grpc.auth.GoogleAuthLibraryCallCredentials can not implement io.grpc.CallCredentials, because it is not an interface (io.grpc.CallCredentials is in unnamed module of loader 'app')
	at java.base/java.lang.ClassLoader.defineClass1(Native Method)
	at java.base/java.lang.ClassLoader.defineClass(ClassLoader.java:1017)
	at java.base/java.security.SecureClassLoader.defineClass(SecureClassLoader.java:174)
	at java.base/jdk.internal.loader.BuiltinClassLoader.defineClass(BuiltinClassLoader.java:800)
	at java.base/jdk.internal.loader.BuiltinClassLoader.findClassOnClassPathOrNull(BuiltinClassLoader.java:698)
	at java.base/jdk.internal.loader.BuiltinClassLoader.loadClassOrNull(BuiltinClassLoader.java:621)
	at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:579)
	at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178)
	at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522)
	at com.google.api.gax.grpc.GrpcCallContext.withCredentials(GrpcCallContext.java:126)
	at com.google.api.gax.grpc.GrpcCallContext.withCredentials(GrpcCallContext.java:64)
	at com.google.api.gax.rpc.ClientContext.create(ClientContext.java:159)
	at com.google.cloud.pubsub.v1.stub.GrpcPublisherStub.create(GrpcPublisherStub.java:161)
	at com.google.cloud.pubsub.v1.Publisher.<init>(Publisher.java:154)
	at com.google.cloud.pubsub.v1.Publisher.<init>(Publisher.java:83)
	at com.google.cloud.pubsub.v1.Publisher$Builder.build(Publisher.java:607)
	at com.spotify.ffwd.pubsub.PubsubOutputPlugin$1.publisher(PubsubOutputPlugin.java:157)
	at com.spotify.ffwd.pubsub.PubsubOutputPlugin$1$$FastClassByGuice$$5c879ecf.invoke(<generated>)
	at com.google.inject.internal.ProviderMethod$FastClassProviderMethod.doProvision(ProviderMethod.java:264)
	at com.google.inject.internal.ProviderMethod.doProvision(ProviderMethod.java:173)
	at com.google.inject.internal.InternalProviderInstanceBindingImpl$CyclicFactory.provision(InternalProviderInstanceBindingImpl.java:185)
	at com.google.inject.internal.InternalProviderInstanceBindingImpl$CyclicFactory.get(InternalProviderInstanceBindingImpl.java:162)
	at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
	at com.google.inject.internal.SingletonScope$1.get(SingletonScope.java:168)
	at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:39)
	at com.google.inject.internal.SingleFieldInjector.inject(SingleFieldInjector.java:52)
	at com.google.inject.internal.MembersInjectorImpl.injectMembers(MembersInjectorImpl.java:147)
	at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:124)
	at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:91)
	at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:306)
	at com.google.inject.internal.FactoryProxy.get(FactoryProxy.java:62)
	at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
	at com.google.inject.internal.SingletonScope$1.get(SingletonScope.java:168)
	at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:39)
	at com.google.inject.internal.FactoryProxy.get(FactoryProxy.java:62)
	at com.google.inject.internal.SingleFieldInjector.inject(SingleFieldInjector.java:52)
	at com.google.inject.internal.MembersInjectorImpl.injectMembers(MembersInjectorImpl.java:147)
	at com.google.inject.internal.MembersInjectorImpl.injectAndNotify(MembersInjectorImpl.java:101)
	at com.google.inject.internal.Initializer$InjectableReference.get(Initializer.java:245)
	at com.google.inject.internal.Initializer.injectAll(Initializer.java:140)
	at com.google.inject.internal.InternalInjectorCreator.injectDynamically(InternalInjectorCreator.java:176)
	... 8 more```

instrumentation support

The following instrumentation hooks are required:

On a per input plugin basis.
- Measure errors.
- Measure dropped events/metrics.
- Measure failed events/metrics.
- Measure received events/metrics.
On an application basis.
- Measure event rates.
- Measure metric rates.
- Measure failure rates.
On a per output plugin basis.
- Measure sent events/metrics.
- Measure dropped events/metrics.
- Measure queue time in ms for events/messages.
- Measure queue size and rate-of-growth for events/messages.

Input protocol: statsd

Would be nice to have something like:
https://github.com/spotify/ffwd/blob/master/plugins/ffwd-statsd/lib/ffwd/plugin/statsd/parser.rb

running ffwd locally needs to use the correct snapshot version

Currently the version is hardcoded which no longer works because ffwd is using semantic versioning.

ffwd/tools/ffwd

Line 11 in 95bc6c8

dist_jar="$dist/target/ffwd-agent-0.0.1-SNAPSHOT.jar"

input protocol: collectd

For a reference implementation, see: https://github.com/spotify/ffwd/blob/master/plugins/ffwd-collectd/lib/ffwd/plugin/collectd/parser.rb

Detect when time between data points of the same series is too small

Detect when time between data points of the same series is too small and implement a limit on this

High cardinality detection to FFWD

High cardinality metrics are hard to manage and create unnecessary load on backend systems.
Earlier detection and following filtering of such metrics would help to automatically manage system without need for user intervention.

Detect and drop metrics with row key length bigger than BT limit

This is related to this PR issue where metrics won't be ingested to Bigtable due to row key length is above the limit of 4096 bytes.
Preventing these kind of metrics earlier is more beneficial for our metics pipeline.

Distribution Documentation

We recently refactor ffwd to add Distribution. We need to update ffwd documentation to reflect the changes.

DoD

Update ffwd documentation
Document Distribution Implementation in ffwd

Enforce rate limiting

Having a per-host rate limit will protect the rest of the metric pipeline against someone adding a bad metric.

There should be a metric when things are being rate-limited that way it can be monitored for.

Can do this guava - https://google.github.io/guava/releases/19.0/api/docs/index.html?com/google/common/util/concurrent/RateLimiter.html.

The limiting can be added where the current filter logic is done or as another filter itself (i think the latter is better).

All travis builds failing due to cache validation

Example: https://travis-ci.org/spotify/ffwd/builds/567080672

[WARNING] Checksum validation failed, expected 5538dc0b8171a0b00bd43d8546898079845c9429 but is 2b2c8a5f74fe4eef0c0de608ed6385ac01d795b8 from google-maven-central for https://maven-central.storage-download.googleapis.com/repos/central/data/org/codehaus/mojo/findbugs-maven-plugin/2.5.4/findbugs-maven-plugin-2.5.4.pom
379
[WARNING] Could not validate integrity of download from https://maven-central.storage-download.googleapis.com/repos/central/data/org/codehaus/mojo/findbugs-maven-plugin/2.5.4/findbugs-maven-plugin-2.5.4.pom: Checksum validation failed, expected 5538dc0b8171a0b00bd43d8546898079845c9429 but is 2b2c8a5f74fe4eef0c0de608ed6385ac01d795b8
380
[WARNING] Checksum validation failed, expected 5538dc0b8171a0b00bd43d8546898079845c9429 but is 2b2c8a5f74fe4eef0c0de608ed6385ac01d795b8 from google-maven-central for https://maven-central.storage-download.googleapis.com/repos/central/data/org/codehaus/mojo/findbugs-maven-plugin/2.5.4/findbugs-maven-plugin-2.5.4.pom
381
[INFO] Downloaded from google-maven-central: https://maven-central.storage-download.googleapis.com/repos/central/data/org/codehaus/mojo/findbugs-maven-plugin/2.5.4/findbugs-maven-plugin-2.5.4.pom (19 kB at 77 kB/s)
382
[INFO] Downloading from google-maven-central: https://maven-central.storage-download.googleapis.com/repos/central/data/org/codehaus/mojo/mojo-parent/30/mojo-parent-30.pom
383
[WARNING] Checksum validation failed, expected d600de340e88292da3ffeb8f8d3eff8258568f0d but is be742febdaa3cc7a971dacdf8cdc7ebd94b5765b from google-maven-central for https://maven-central.storage-download.googleapis.com/repos/central/data/org/codehaus/mojo/mojo-parent/30/mojo-parent-30.pom
384
[WARNING] Could not validate integrity of download from https://maven-central.storage-download.googleapis.com/repos/central/data/org/codehaus/mojo/mojo-parent/30/mojo-parent-30.pom: Checksum validation failed, expected d600de340e88292da3ffeb8f8d3eff8258568f0d but is be742febdaa3cc7a971dacdf8cdc7ebd94b5765b
385
[WARNING] Checksum validation failed, expected d600de340e88292da3ffeb8f8d3eff8258568f0d but is be742febdaa3cc7a971dacdf8cdc7ebd94b5765b from google-maven-central for https://maven-central.storage-download.googleapis.com/repos/central/data/org/codehaus/mojo/mojo-parent/30/mojo-parent-30.pom

These errors don't happen locally, even when using the google maven mirror.

Add tracing to PubSub publishing

OpenCensus instrumentation for PubSub was just released. This would be really nice to have - we have tracing for the consumers in Heroic already, so having the publisher traced would give us full end-to-end visibility.

There are other parts of the system (core/filtering and ingestion) that would also be nice to have, but getting the publisher traced first would be the most useful IMO.

Support resource identifiers

Currently the only way resource identifiers can be sent into ffwd is via the http/batch endpoint.

At a minimum the JSON and proto inputs should accept resource identifiers.

#20 broken Riemann input module

#20 broke Riemann in a number of way. Currently ffwd-java throws exceptions like this on startup:

14:34:32.722 [main] ERROR com.spotify.ffwd.FastForwardAgent - Error in agent, exiting
com.google.inject.CreationException: Unable to create injector, see the following errors:

1) Could not find a suitable constructor in com.spotify.ffwd.riemann.RiemannSerialization. Classes must have either one (and only one) constructor annotated with @Inject or a zero-argument constructor that is not private.
  at com.spotify.ffwd.riemann.RiemannSerialization.class(RiemannSerialization.java:42)
  while locating com.spotify.ffwd.riemann.RiemannSerialization
    for field at com.spotify.ffwd.riemann.RiemannFrameDecoder.serializer(RiemannFrameDecoder.java:29)
  at com.spotify.ffwd.riemann.RiemannInputPlugin$1.configure(RiemannInputPlugin.java:77) (via modules: com.spotify.ffwd.input.InputManagerModule$1 -> com.spotify.ffwd.riemann.RiemannInputPlugin$1)

2) Could not find a suitable constructor in com.spotify.ffwd.riemann.RiemannSerialization. Classes must have either one (and only one) constructor annotated with @Inject or a zero-argument constructor that is not private.
  at com.spotify.ffwd.riemann.RiemannSerialization.class(RiemannSerialization.java:42)
  while locating com.spotify.ffwd.riemann.RiemannSerialization
    for field at com.spotify.ffwd.riemann.RiemannDatagramDecoder.serializer(RiemannDatagramDecoder.java:26)
  at com.spotify.ffwd.riemann.RiemannInputPlugin$1.configure(RiemannInputPlugin.java:79) (via modules: com.spotify.ffwd.input.InputManagerModule$1 -> com.spotify.ffwd.riemann.RiemannInputPlugin$1)

3) Could not find a suitable constructor in com.spotify.ffwd.riemann.RiemannSerialization. Classes must have either one (and only one) constructor annotated with @Inject or a zero-argument constructor that is not private.
  at com.spotify.ffwd.riemann.RiemannSerialization.class(RiemannSerialization.java:42)
  while locating com.spotify.ffwd.riemann.RiemannSerialization
    for field at com.spotify.ffwd.riemann.RiemannMessageDecoder.serializer(RiemannMessageDecoder.java:26)
  at com.spotify.ffwd.riemann.RiemannInputPlugin$1.configure(RiemannInputPlugin.java:80) (via modules: com.spotify.ffwd.input.InputManagerModule$1 -> com.spotify.ffwd.riemann.RiemannInputPlugin$1)

4) Could not find a suitable constructor in com.spotify.ffwd.riemann.RiemannSerialization. Classes must have either one (and only one) constructor annotated with @Inject or a zero-argument constructor that is not private.
  at com.spotify.ffwd.riemann.RiemannSerialization.class(RiemannSerialization.java:42)
  while locating com.spotify.ffwd.riemann.RiemannSerialization
    for field at com.spotify.ffwd.riemann.RiemannFrameDecoder.serializer(RiemannFrameDecoder.java:29)
  at com.spotify.ffwd.riemann.RiemannInputPlugin$1.configure(RiemannInputPlugin.java:77) (via modules: com.spotify.ffwd.input.InputManagerModule$1 -> com.spotify.ffwd.riemann.RiemannInputPlugin$1)

5) Could not find a suitable constructor in com.spotify.ffwd.riemann.RiemannSerialization. Classes must have either one (and only one) constructor annotated with @Inject or a zero-argument constructor that is not private.
  at com.spotify.ffwd.riemann.RiemannSerialization.class(RiemannSerialization.java:42)
  while locating com.spotify.ffwd.riemann.RiemannSerialization
    for field at com.spotify.ffwd.riemann.RiemannDatagramDecoder.serializer(RiemannDatagramDecoder.java:26)
  at com.spotify.ffwd.riemann.RiemannInputPlugin$1.configure(RiemannInputPlugin.java:79) (via modules: com.spotify.ffwd.input.InputManagerModule$1 -> com.spotify.ffwd.riemann.RiemannInputPlugin$1)

6) Could not find a suitable constructor in com.spotify.ffwd.riemann.RiemannSerialization. Classes must have either one (and only one) constructor annotated with @Inject or a zero-argument constructor that is not private.
  at com.spotify.ffwd.riemann.RiemannSerialization.class(RiemannSerialization.java:42)
  while locating com.spotify.ffwd.riemann.RiemannSerialization
    for field at com.spotify.ffwd.riemann.RiemannMessageDecoder.serializer(RiemannMessageDecoder.java:26)
  at com.spotify.ffwd.riemann.RiemannInputPlugin$1.configure(RiemannInputPlugin.java:80) (via modules: com.spotify.ffwd.input.InputManagerModule$1 -> com.spotify.ffwd.riemann.RiemannInputPlugin$1)

6 errors
        at com.google.inject.internal.Errors.throwCreationExceptionIfErrorsExist(Errors.java:448)
        at com.google.inject.internal.InternalInjectorCreator.initializeStatically(InternalInjectorCreator.java:155)
        at com.google.inject.internal.InternalInjectorCreator.build(InternalInjectorCreator.java:107)
        at com.google.inject.internal.InjectorImpl.createChildInjector(InjectorImpl.java:225)
        at com.spotify.ffwd.AgentCore.setupPrimaryInjector(AgentCore.java:346)
        at com.spotify.ffwd.AgentCore.run(AgentCore.java:107)
        at com.spotify.ffwd.FastForwardAgent.main(FastForwardAgent.java:82)

Use java.util.Optional instead of Guava Optional

Replace use of Guava Optional.

Introduce Jdk8Module so that @JsonCreator constructors can use java.util.Optional in the parameter to reduce clutter.

investigate performance regression

With the latest release ffwd seems to be consuming metrics more slowly over udp which caused the buffer to fill up and metrics to be dropped. Could this be related to the netty upgrade done easier?

There should also be a regression/benchmark test before releasing.

Unable to run findbugs on newer maven versions

Findbugs is broken with newer versions of maven. Since the project has been abandoned for quite a while there's pretty much no chance of it being fixed. We should switch to a different library.

Add an OpenTelemetry output plugin

Protobuf file: https://github.com/open-telemetry/opentelemetry-proto/blob/master/opentelemetry/proto/metrics/v1/metrics.proto

This will likely eventually supersede the OpenCensus plugin that was introducted for StackDriver metrics as OpenTelemetry is replacing OpenCensus/OpenTracing.

Implement filtering for Batch type

com.spotify.ffwd.filter.Filter#matchesBatch

ffwd version should be added as a tag on a metric

That way the version can be monitored when doing a deploy.

Document debugging process with debug port

Deprecate events api

Currently the event support of ffwd is not used internally within Spotify and there are no plans to. Does it make sense to continue supporting it before we cut a major release of ffwd?

Add a new Metric Data Model that can Support Distribution.

Heroic histogram data is currently computed locally.
It is practically impossible to aggregate percentile.
We are adding distribution to heroic to address that issue. Distribution will create data sketch that will be used upstream to compute percentile on the entire data distribution.

This task will add a point value of object type. Please refer to existing implementations for more details:

Json Metric : https://github.com/spotify/ffwd/tree/master/api/src/main/java/com/spotify/ffwd/model
Proto Metric : https://github.com/spotify/ffwd/blob/master/core/src/main/proto/spotify_100.proto
Proto Metric with distribution :https://github.com/spotify/ffwd-client-java/tree/master/ffwd-client/src/main/java/com/spotify/ffwd/v1

DoD : Spotify_100 and Spotify100ProtoSerializer.java serializers should support both the old and new metric formats.

Add tests to http module

Need tests around http module to avoid future regressions.

on-disk persistent queue to handle network weather

WIP implementation: QLogManager

See docs/on-disk-queue.md for the current draft specification of the queue.

debug protocol

Allow connections to 'sniff' what is going on internally, make sure to follow
the previous protocol to allow existing clients to keep working until
replacements can be built.

Message structure would be:

{"id": "identifier", "type": "'metric' or 'event'", "data": {}}

[k8s] running ffwd as a DaemonSet

Currently at Spotify a ffwd container is injected into each pod with an admission controller.

It has been slow and cumbersome to rollout new versions of ffwd since it requires recreating all the pods.

An alternative solution to the sidecar approach is to run ffwd as a demon set. Fluentd which ships logs off the GKE nodes is deployment in a similar way. However fluentd get's metadata around the logs based on filename (this was the case in 2018, it might be different now?).

This approach doesn't come without its own unique set of challenges some of which are outlined below.

Would need to map the incoming ip address to a pod to get metadata such as podname. IP addresses could move around quickly and this would need to be kept fresh. We could watch for pod change events and use that as a cache buster.
Does the UDP buffer need to be sized even higher? Currently each pod on a node get's his own ffwd/udp buffer.

Part of this issue should be doing the discovery work to see how feasible it would be.

input protocol: carbon

Implement resource identifier support in json module

Currently only the http module supports metric input with resource identifiers, it was never added to the json input:

ffwd/modules/json/src/main/java/com/spotify/ffwd/json/JsonObjectMapperDecoder.java

Line 111 in 049aa48

// TODO: support resource?

Json module should be updated to handle metrics with resource identifiers

Bonus: all input modules should handle resource identifiers to standardize it across inputs.

Add support for sending metrics to StackDriver

Initially, I'm planning on using OpenCensus.

port fwc

fwc is a small client utility available for ruby.
implementing it here would ease adoption.

[input plugin] prometheus

Add a prometheus plugin module which can scrape a list of endpoints for metrics to re-emit, with the normal tagging and filtering applied.

ffwd with OpenTSDB

I am looking for some documentation related to how would one go about publishing metrics to OpenTSDB from ffwd.

The http output plugin seems to be able to route metrics only to another ffwd agent. Is there a way to forward metrics to a generic HTTP end-point?

Carbon pickle support

It would be great if ffwd could support the pickle protocol (+ eventually parsing metrics into tags ;), so it could be used during a migration path from a carbon-relay graphite setup into kafka/heroic streams...

Patch output plugins to support filtering

Remaining output plugins to be patched: kafka, noop, riemann, template

ffwd issues with the agent

05:35:25.755 [main] ERROR com.spotify.ffwd.FastForwardAgent - Error in agent, exiting
java.util.concurrent.ExecutionException: java.lang.UnsupportedOperationException

Update kafka protocol to newer version

current version: 0.8.2.1

Setup system tests that run in CI/CD

Ideally these can be set up similar to those in Heroic via a docker full environment.

Move builds to Circle CI

Spotify is slowly moving over to Circl CI as the primary open source build tool. We should see if it makes sense for ffwd, and if so migrate over.

Move to a centralized approach with metric definition.

FFWD and its client have the dependency on the metric format. Currently we have many definitions of the metric that reside in different project. I suggest that we create a centralized location for all the metrics.
This will help with maintenance and provide better understanding of dependency.

Reference :

signalfx output plugin reports all metrics as gauges

Currently the signalfx output plugin reports all metrics as gauges instead of the appropriate metric type

ffwd/modules/signalfx/src/main/java/com/spotify/ffwd/signalfx/SignalFxPluginSink.java

Line 111 in c662849

.setMetricType(SignalFxProtocolBuffers.MetricType.GAUGE)

This causes default rollups and aggregations in Siganlfx to be incorrect for cumulative counters. And also produced negative values when instances are reset.

https://docs.signalfx.com/en/latest/charts/resolution-rollups.html#rollup-types

Rate/sec (default for counter and cumulative counter metrics): Divide the result of Sum (counter) or Delta (cumulative counter) by the number of seconds in the time interval.