kamon-io/kamon

Distributed Tracing, Metrics and Context Propagation for applications running on the JVM

Kamon Telemetry

Kamon Telemetry is a set of libraries for instrumenting applications running on the JVM. With Kamon Telemetry you can collect metrics, propagate context across threads and services, and get distributed traces automatically. The best way to get started is following our installation guides and taking it from there. Have fun with Kamon!

License

This software is licensed under the Apache 2 license, quoted below.

Copyright © 2013-2022 the kamon project https://kamon.io/

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

[http://www.apache.org/licenses/LICENSE-2.0]

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

kamon's Issues

Load all newrelic relevant information from the newrelic.yml file.

Thought of using SnakeYaml.

create KamonStandalone module/helper

The idea is to have a helper class that encapsulates a ActorSystem and provides a very simple API for metrics (and traces right after that). This way everyone that wants to benefit from the instruments and reportes we have can do so, no matter if they are building reactive applications or not.

Lost headers in spray request intrumentation

ie.
val apiRoute = pathPrefix("foo") {
path(PathEnd) {
redirect(Uri("/foo/bar/index.html"), StatusCodes.MovedPermanently)
}
}

curl -i "http://localhost:8000/foo/" return:
HTTP/1.1 301 Moved Permanently
Server: spray-can/1.2.0
Date: adate
X-Trace-Token: atoken
Content-Type: text/html
Content-Length: 96

and should return:

HTTP/1.1 301 Moved Permanently
Server: spray-can/1.2.0
Date: adate
X-Trace-Token: atoken
Location: /foo/bar/index.html
Content-Type: text/html
Content-Length: 96

NPE when application start with multiples Actor Systems

[ERROR][threads-redis-client-akka.actor.default-dispatcher-5] [akka://threads-redis-client/system/IO-TCP/selectors] null
akka.actor.ActorInitializationException: exception during creation
at akka.actor.ActorInitializationException$.apply(Actor.scala:164)
at akka.actor.ActorCell.create(ActorCell.scala:596)
at akka.actor.ActorCell.invokeAll$1_aroundBody2(ActorCell.scala:456)
at akka.actor.ActorCell$AjcClosure3.run(ActorCell.scala:1)
at org.aspectj.runtime.reflect.JoinPointImpl.proceed(JoinPointImpl.java:149)
at akka.instrumentation.ActorSystemMessagePassingTracing$$anonfun$aroundSystemMessageInvoke$1.apply(ActorSystemMessagePassingTracing.scala:61)
at kamon.trace.TraceRecorder$.withTraceContext(TraceRecorder.scala:66)
at akka.instrumentation.ActorSystemMessagePassingTracing.aroundSystemMessageInvoke(ActorSystemMessagePassingTracing.scala:61)
at akka.actor.ActorCell.invokeAll$1(ActorCell.scala:1)
at akka.actor.ActorCell.systemInvoke(ActorCell.scala:478)
at akka.dispatch.Mailbox.processAllSystemMessages(Mailbox.scala:263)
at akka.dispatch.Mailbox.run(Mailbox.scala:219)
at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:393)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at akka.util.Reflect$.instantiate(Reflect.scala:66)
at akka.actor.ArgsReflectConstructor.produce(Props.scala:349)
at akka.actor.Props.newActor(Props.scala:249)
at akka.actor.ActorCell.newActor(ActorCell.scala:552)
at akka.actor.ActorCell.create(ActorCell.scala:578)
... 15 more
Caused by: java.lang.NullPointerException
at akka.actor.ActorSystemImpl.findExtension(ActorSystem.scala:697)
at akka.actor.ActorSystemImpl.registerExtension(ActorSystem.scala:706)
at akka.actor.ExtensionId$class.apply(Extension.scala:79)
at kamon.metrics.Metrics$.apply(MetricsExtension.scala:97)
at kamon.Kamon$.apply(Kamon.scala:31)
at akka.instrumentation.DispatcherTracing.afterDispatcherStartup(DispatcherTracing.scala:55)
at akka.dispatch.Dispatcher$LazyExecutorServiceDelegate.executor$lzycompute(Dispatcher.scala:44)
at akka.dispatch.Dispatcher$LazyExecutorServiceDelegate.executor(Dispatcher.scala:44)
at akka.dispatch.ExecutorServiceDelegate$class.execute(ThreadPoolBuilder.scala:212)
at akka.dispatch.Dispatcher$LazyExecutorServiceDelegate.execute(Dispatcher.scala:43)
at akka.dispatch.Dispatcher.executeTask(Dispatcher.scala:76)
at akka.dispatch.MessageDispatcher.unbatchedExecute(AbstractDispatcher.scala:145)
at akka.dispatch.BatchingExecutor$class.execute(BatchingExecutor.scala:113)
...

Manually merged aop.xml needed when using sbt assembly?

Have I understood correctly that if I want to use "sbt assembly", I need to manually create my own aop.xml file, which combines the aop definitions from the individual kamon modules? Are you considering better solutions to this problem?

ensure that the datadog module works on Windows

Apparently the Windows agent does not support multiple metrics per packet to work properly, we might need to add a setting to define whether multiple metrics can be reported in a single packet or not. Issue #42 has more details on how this was initiated.

create kamon-statsd module

It seems like using statsd is becoming more and more popular and using the subscription protocol it should be really easy to get this integration working very quickly.

avoid reporting data to Newrelic if no metrics have been collected

If no data is available, avoid posting to Newrelic. Also a back-off strategy must be implemented for cases where collectors become unreachable or the POST requests fail.

Remove Highcharts dependency

we need change Highcharts dependency by other open source chart library

Actor treeview in the dashboard

Custom Action to rename current trace

An approach could be action composition.

http://www.playframework.com/documentation/2.2.x/ScalaActionsComposition

automatic trace token propagation does not work with chunked responses

Our current instrumentation is wrapping the call to DefaultOpenRequest.handleResponseEndAndReturnNextOpenRequest, which is OK in case of a regular HttpResponse, but when the response is sent in chunks by the time the ChunkedMessageEnd is received the headers were already sent to the client and setting the trace token header becomes useless.

create HTTP-Metrics for Play/Spray/Akka-Http

NPE when dispatcher shutdown

[ERROR] [services-scheduler-1] [akka.dispatch.Dispatcher] null
java.lang.NullPointerException
    at akka.instrumentation.DispatcherTracing.afterDispatcherShutdown(DispatcherTracing.scala:85)
    at akka.dispatch.Dispatcher.shutdown(Dispatcher.scala:101)
    at akka.dispatch.MessageDispatcher$$anon$2.run(AbstractDispatcher.scala:212)
    at akka.dispatch.MessageDispatcher$$anon$3.execute(AbstractDispatcher.scala:174)
    at akka.actor.LightArrayRevolverScheduler$TaskHolder.executeTask(Scheduler.scala:467)
    at akka.actor.LightArrayRevolverScheduler$$anon$8.executeBucket$1(Scheduler.scala:419)
    at akka.actor.LightArrayRevolverScheduler$$anon$8.nextTick(Scheduler.scala:423)
    at akka.actor.LightArrayRevolverScheduler$$anon$8.run(Scheduler.scala:375)
    at java.lang.Thread.run(Thread.java:744)

It's because we trying to remove from context a dispatcher that not it registered before.

NPE when shuting down dispatcher with Scala 2.11

The following exception is thrown, as first reported on #34:

[ERROR] [06/05/2014 15:36:21.550] [services-scheduler-1] [akka.dispatch.Dispatcher] null
java.lang.NullPointerException
    at akka.instrumentation.DispatcherTracing.afterDispatcherShutdown(DispatcherTracing.scala:85)
    at akka.dispatch.Dispatcher.shutdown(Dispatcher.scala:101)
    at akka.dispatch.MessageDispatcher$$anon$2.run(AbstractDispatcher.scala:212)
    at akka.dispatch.MessageDispatcher$$anon$3.execute(AbstractDispatcher.scala:174)
    at akka.actor.LightArrayRevolverScheduler$TaskHolder.executeTask(Scheduler.scala:467)
    at akka.actor.LightArrayRevolverScheduler$$anon$8.executeBucket$1(Scheduler.scala:419)
    at akka.actor.LightArrayRevolverScheduler$$anon$8.nextTick(Scheduler.scala:423)
    at akka.actor.LightArrayRevolverScheduler$$anon$8.run(Scheduler.scala:375)
    at java.lang.Thread.run(Thread.java:744)

@adrien-aubel, could you please describe a bit more how you found this issue? does it happen right after starting your app? did you notice any way to reproduce the issue that we might isolate and test?

scalatestplus as runtime dependency on kamon-play

Hi,

On our play project, using https://github.com/jrudolph/sbt-dependency-graph we notice that kamon-play have scalatestplus as a runtime dependency, and using plugins like sbt-pack or fatjar, this library (and its dependencies) are included on our package, can you look at this?

Support for REST style URLs (Spray)

I'm testing Kamon with NewRelic and Spray. My application uses REST style requests where the URL contains, for example, a unique DeviceID. Currently every request to the same resource type is reported separately because the unique identifier is included in the path.

Ideally Kamon could understand from the Spray path definitions how the requests should be grouped.

create a new kamon-system-metrics module

Create a simple module that registers a few gauges with information about CPU usage, garbage collection stats, disk IO usage and network traffic.

Increasing CPU usage

Hi,

I've been investigating lately on an issue that was happening on my application: the more the application is running, the higher is the CPU usage of this application.
It gets so high that it sometimes gets killed by the OS.

After like 24h the CPU usage starts to increase significantly, and I've attached a profiling on it. I've attached below two screenshots of a CPU call tree on JProfiler (same tree, just scrolling down):

Most of the CPU usage is related to Kamon IO. Let me know if you need more information. I'll need to safely restart the application soon, to avoid the app to get killed by the OS.

FYI, I'm using 2 JVMs on this server, that has 2x E5-2650v2 (16 cores / 32 threads)

explore inlining of Trace.withContext via macros

watch all actors that subscribe to metrics

Watch the actors and make sure that we unsubscribe them if they die.

alert when the AspectJ weaver is missing

Something as simple as having a class that provides an error log message that will be overwritten with a successful error message by an aspect. This should probably be logged when the Kamon(Metrics) extension is loaded.

create activator templates for the most common modules

We should have a simple tutorial per toolkit (Akka, Spray and Play) as an activator template that can be pointed out to when discussing issues or giving help to people.

automatic uow and traceName generation for Spray instrumentation

See comments in this PR: #13

max-packet-size in bytes

Why is it documentation that max-packet-size is in bytes?
Instead, the HOCON-supported file size format might be used to just do

max-packet-size=1024bytes

Instrument Play LoggerLike in order to add the properties of LocalStorage into the MDC

provide basic support for akka cluster and remoting

the basic support should at least allow:

• propagate the trace-token across the cluster
• start a trace in one machine and finish it in another

ClassNotFoundException when instrumenting Play! applications

During application startup several exceptions as the one bellow are logged, and after some time the application starts working normally. The help we got at the aspectj mailing list suggests that the problem is related to the perthis association + funky class loaders, will try avoiding perthis association to solve this issue.

org.aspectj.weaver.bcel.BcelWorld - Unable to find class 'akka.instrumentation.BehaviourInvokeTracing$ajcMightHaveAspect' in repository
java.lang.ClassNotFoundException: akka.instrumentation.BehaviourInvokeTracing$ajcMightHaveAspect not found - unable to determine URL
at org.aspectj.apache.bcel.util.ClassLoaderRepository.loadClass(ClassLoaderRepository.java:292) ~[aspectjweaver-1.7.4.jar:1.7.4]
at org.aspectj.weaver.bcel.BcelWorld.lookupJavaClass(BcelWorld.java:418) [aspectjweaver-1.7.4.jar:1.7.4]
at org.aspectj.weaver.bcel.BcelWorld.resolveDelegate(BcelWorld.java:392) [aspectjweaver-1.7.4.jar:1.7.4]
at org.aspectj.weaver.ltw.LTWWorld.resolveDelegate(LTWWorld.java:107) [aspectjweaver-1.7.4.jar:1.7.4]
at org.aspectj.weaver.World.resolveToReferenceType(World.java:477) [aspectjweaver-1.7.4.jar:1.7.4]
at org.aspectj.weaver.World.resolve(World.java:318) [aspectjweaver-1.7.4.jar:1.7.4]
at org.aspectj.weaver.World.resolve(World.java:228) [aspectjweaver-1.7.4.jar:1.7.4]
at org.aspectj.weaver.UnresolvedType.resolve(UnresolvedType.java:616) [aspectjweaver-1.7.4.jar:1.7.4]
at org.aspectj.weaver.bcel.BcelTypeMunger.mungePerObjectInterface(BcelTypeMunger.java:796) [aspectjweaver-1.7.4.jar:1.7.4]
at org.aspectj.weaver.bcel.BcelTypeMunger.munge(BcelTypeMunger.java:116) [aspectjweaver-1.7.4.jar:1.7.4]
at org.aspectj.weaver.bcel.BcelClassWeaver.weave(BcelClassWeaver.java:516) [aspectjweaver-1.7.4.jar:1.7.4]
at org.aspectj.weaver.bcel.BcelClassWeaver.weave(BcelClassWeaver.java:101) [aspectjweaver-1.7.4.jar:1.7.4]
at org.aspectj.weaver.bcel.BcelWeaver.weave(BcelWeaver.java:1691) [aspectjweaver-1.7.4.jar:1.7.4]
at org.aspectj.weaver.bcel.BcelWeaver.weaveWithoutDump(BcelWeaver.java:1635) [aspectjweaver-1.7.4.jar:1.7.4]
at org.aspectj.weaver.bcel.BcelWeaver.weaveAndNotify(BcelWeaver.java:1400) [aspectjweaver-1.7.4.jar:1.7.4]
at org.aspectj.weaver.bcel.BcelWeaver.weave(BcelWeaver.java:1186) [aspectjweaver-1.7.4.jar:1.7.4]
at org.aspectj.weaver.tools.WeavingAdaptor.getWovenBytes(WeavingAdaptor.java:527) [aspectjweaver-1.7.4.jar:1.7.4]
at org.aspectj.weaver.tools.WeavingAdaptor.weaveClass(WeavingAdaptor.java:363) [aspectjweaver-1.7.4.jar:1.7.4]
at org.aspectj.weaver.loadtime.Aj.preProcess(Aj.java:121) [aspectjweaver-1.7.4.jar:1.7.4]
at org.aspectj.weaver.loadtime.ClassPreProcessorAgentAdapter.transform(ClassPreProcessorAgentAdapter.java:54) [aspectjweaver-1.7.4.jar:1.7.4]
at sun.instrument.TransformerManager.transform(TransformerManager.java:188) [na:1.7.0_45]
at sun.instrument.InstrumentationImpl.transform(InstrumentationImpl.java:424) [na:1.7.0_45]
at java.lang.ClassLoader.defineClass1(Native Method) [na:1.7.0_45]
at java.lang.ClassLoader.defineClass(ClassLoader.java:800) [na:1.7.0_45]
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) [na:1.7.0_45]
at java.net.URLClassLoader.defineClass(URLClassLoader.java:449) [na:1.7.0_45]
at java.net.URLClassLoader.access$100(URLClassLoader.java:71) [na:1.7.0_45]
at java.net.URLClassLoader$1.run(URLClassLoader.java:361) [na:1.7.0_45]
at java.net.URLClassLoader$1.run(URLClassLoader.java:355) [na:1.7.0_45]
at java.security.AccessController.doPrivileged(Native Method) [na:1.7.0_45]
at java.net.URLClassLoader.findClass(URLClassLoader.java:354) [na:1.7.0_45]
at java.lang.ClassLoader.loadClass(ClassLoader.java:425) [na:1.7.0_45]
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) [na:1.7.0_45]
at java.lang.ClassLoader.loadClass(ClassLoader.java:358) [na:1.7.0_45]
at java.lang.Class.getDeclaredFields0(Native Method) [na:1.7.0_45]
at java.lang.Class.privateGetDeclaredFields(Class.java:2397) [na:1.7.0_45]
at java.lang.Class.getDeclaredField(Class.java:1946) [na:1.7.0_45]

Update the grafana-graphite docker image dashboard with best practices in mind

The dashboard that we are currently presenting just shows some data and expresses that it is possible to see metrics there, but lacks a few important bits:

• Ensure correctness of data: the current approach aggregates data from 10 seconds to 1 minute intervals after the first day and the way graphite aggregates percentiles is doing averages, which make it loose the charm. Probably the best choice is to avoid downsampling and only store data for a single day.
• Focus on important entities: Instead of having a single big dashboard with many data we could have one focused on traces, one on actors + dispatchers and one for user metrics.

Add support for spray-servlet

NoClassDefFoundError when starting Kamon without slf4j-api in classpath

The exception bellow is fired when no slf4j-api is available in the classpath.. we should either declare the dependency explicitly to make sure it is always there or eliminate the dependency.

[AppClassLoader@138297fe] error Cannot instantiate message handler kamon.weaver.logging.KamonWeaverMessageHandler -- (NoClassDefFoundError) org/slf4j/LoggerFactory
12:54:33 web.1   | org/slf4j/LoggerFactory
12:54:33 web.1   | java.lang.NoClassDefFoundError: org/slf4j/LoggerFactory
12:54:33 web.1   | at kamon.weaver.logging.KamonWeaverMessageHandler.<init>(KamonWeaverMessageHandler.scala:29)

Remove NewRelic Agent dependency

ClassCastException when using NewRelicErrorLogger without the aspectj weaver

When a application is configured to use the NewRelicErrorLogger but wasn't started with the aspectj weaver the class cast fails and the error loops back into the logger, producing a infinite failure loop that eventually kills the app.

When the application is not instrumented this logger should report the error anyway, missing the TraceToken information.

Exception:
flights-catalog java.lang.ClassCastException: akka.event.Logging$Error cannot be cast to kamon.trace.TraceContextAware
flights-catalog at kamon.newrelic.NewRelicErrorLogger.notifyError(NewRelicErrorLogger.scala:33) ~[kamon-newrelic-0.3.0.jar:0.3.0]

Refactor and clean of Dashboard view

kamon/util/GlobPathFilter : Unsupported major.minor version 51.0

Exception in thread "main" java.lang.UnsupportedClassVersionError: kamon/util/GlobPathFilter : Unsupported major.minor version 51.0
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClassCond(ClassLoader.java:632)
at java.lang.ClassLoader.defineClass(ClassLoader.java:616)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:141)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:283)
at java.net.URLClassLoader.access$000(URLlassLoader.java:58)
at java.net.URLClassLoader$1.run(URLClassLoader.java:197)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)

Kamon:

io.kamon
kamon-core_2.10
0.3.1


io.kamon
kamon-datadog_2.10
0.3.1

    <SCALA_MAJOR>2.10</SCALA_MAJOR>
    <SCALA_VERSION>2.10.3</SCALA_VERSION>
    <AKKA_VERSION>2.3.2</AKKA_VERSION>

JDK 1.6

Deadlock when starting Kamon with Spray and NewRelic

When I try to start my server, it encounters a Java level deadlock. Please take a look at the attached jstack output for more details. The JVM aguments are:

-Xms256m -Xmx1536m -javaagent:/opt/newrelic/java/newrelic.jar -javaagent:/opt/aspectj/aspectjweaver.jar

Found one Java-level deadlock:
=============================

"main":
  waiting to lock Monitor@0x00007fe3ec004ab8 (Object@0x000000009af590a8, a java/lang/Object),
  which is held by "New Relic Sampler Service"
"New Relic Sampler Service":
  waiting to lock Monitor@0x00007fe3ec006218 (Object@0x00000000baedc9f0, a sun/misc/Launcher$AppClassLoader),
  which is held by "main"

Found a total of 1 deadlock.

Thread 28160: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Interpreted frame)
 - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long) @bci=20, line=226 (Interpreted frame)
 - java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(long) @bci=68, line=2082 (Interpreted frame)
 - java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take() @bci=122, line=1090 (Interpreted frame)
 - java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take() @bci=1, line=807 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=156, line=1068 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=26, line=1130 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=615 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=744 (Interpreted frame)


Thread 28159: (state = BLOCKED)
 - org.aspectj.weaver.loadtime.Aj.preProcess(java.lang.String, byte[], java.lang.ClassLoader, java.security.ProtectionDomain) @bci=136, line=106 (Interpreted frame)
 - org.aspectj.weaver.loadtime.ClassPreProcessorAgentAdapter.transform(java.lang.ClassLoader, java.lang.String, java.lang.Class, java.security.ProtectionDomain, byte[]) @bci=43, line=54 (Interpreted frame)
 - sun.instrument.TransformerManager.transform(java.lang.ClassLoader, java.lang.String, java.lang.Class, java.security.ProtectionDomain, byte[]) @bci=50, line=188 (Compiled frame)
 - sun.instrument.InstrumentationImpl.transform(java.lang.ClassLoader, java.lang.String, java.lang.Class, java.security.ProtectionDomain, byte[], boolean) @bci=34, line=424 (Interpreted frame)
 - java.lang.ClassLoader.defineClass1(java.lang.String, byte[], int, int, java.security.ProtectionDomain, java.lang.String) @bci=0 (Interpreted frame)
 - java.lang.ClassLoader.defineClass(java.lang.String, byte[], int, int, java.security.ProtectionDomain) @bci=30, line=800 (Interpreted frame)
 - java.security.SecureClassLoader.defineClass(java.lang.String, byte[], int, int, java.security.CodeSource) @bci=12, line=142 (Interpreted frame)
 - java.net.URLClassLoader.defineClass(java.lang.String, sun.misc.Resource) @bci=220, line=449 (Interpreted frame)
 - java.net.URLClassLoader.access$100(java.net.URLClassLoader, java.lang.String, sun.misc.Resource) @bci=3, line=71 (Interpreted frame)
 - java.net.URLClassLoader$1.run() @bci=43, line=361 (Interpreted frame)
 - java.net.URLClassLoader$1.run() @bci=1, line=355 (Interpreted frame)
 - java.security.AccessController.doPrivileged(java.security.PrivilegedExceptionAction, java.security.AccessControlContext) @bci=0 (Interpreted frame)
 - java.net.URLClassLoader.findClass(java.lang.String) @bci=13, line=354 (Interpreted frame)
 - java.lang.ClassLoader.loadClass(java.lang.String, boolean) @bci=70, line=425 (Interpreted frame)
 - sun.misc.Launcher$AppClassLoader.loadClass(java.lang.String, boolean) @bci=36, line=308 (Interpreted frame)
 - java.lang.ClassLoader.loadClass(java.lang.String) @bci=3, line=358 (Interpreted frame)
 - com.newrelic.agent.stats.StatsEngineImpl.getStats(java.lang.String) @bci=2, line=46 (Interpreted frame)
 - com.newrelic.agent.samplers.MemorySampler$HeapAndNonHeapUsage.recordStats(com.newrelic.agent.stats.StatsEngine) @bci=3, line=96 (Interpreted frame)
 - com.newrelic.agent.samplers.MemorySampler.sampleMemory(com.newrelic.agent.stats.StatsEngine) @bci=15, line=42 (Interpreted frame)
 - com.newrelic.agent.samplers.MemorySampler.sample(com.newrelic.agent.stats.StatsEngine) @bci=2, line=35 (Interpreted frame)
 - com.newrelic.agent.samplers.SamplerServiceImpl.runSampler(com.newrelic.agent.samplers.MetricSampler) @bci=18, line=106 (Interpreted frame)
 - com.newrelic.agent.samplers.SamplerServiceImpl.access$000(com.newrelic.agent.samplers.SamplerServiceImpl, com.newrelic.agent.samplers.MetricSampler) @bci=2, line=31 (Interpreted frame)
 - com.newrelic.agent.samplers.SamplerServiceImpl$1.run() @bci=8, line=85 (Interpreted frame)
 - java.util.concurrent.Executors$RunnableAdapter.call() @bci=4, line=471 (Interpreted frame)
 - java.util.concurrent.FutureTask.runAndReset() @bci=47, line=304 (Interpreted frame)
 - java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask) @bci=1, line=178 (Interpreted frame)
 - java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run() @bci=37, line=293 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=95, line=1145 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=615 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=744 (Interpreted frame)


Thread 28158: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Interpreted frame)
 - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long) @bci=20, line=226 (Interpreted frame)
 - java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(long) @bci=68, line=2082 (Interpreted frame)
 - java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take() @bci=122, line=1090 (Interpreted frame)
 - java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take() @bci=1, line=807 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=156, line=1068 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=26, line=1130 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=615 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=744 (Interpreted frame)


Thread 28157: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Interpreted frame)
 - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long) @bci=20, line=226 (Interpreted frame)
 - java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(long) @bci=68, line=2082 (Interpreted frame)
 - java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take() @bci=122, line=1090 (Interpreted frame)
 - java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take() @bci=1, line=807 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor.getTask() @bci=156, line=1068 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=26, line=1130 (Interpreted frame)
 - java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=615 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=744 (Interpreted frame)


Thread 28156: (state = BLOCKED)


Thread 28155: (state = BLOCKED)
 - java.lang.Object.wait(long) @bci=0 (Interpreted frame)
 - java.lang.ref.ReferenceQueue.remove(long) @bci=44, line=135 (Interpreted frame)
 - java.lang.ref.ReferenceQueue.remove() @bci=2, line=151 (Interpreted frame)
 - java.lang.ref.Finalizer$FinalizerThread.run() @bci=16, line=189 (Interpreted frame)


Thread 28154: (state = BLOCKED)
 - java.lang.Object.wait(long) @bci=0 (Interpreted frame)
 - java.lang.Object.wait() @bci=2, line=503 (Interpreted frame)
 - java.lang.ref.Reference$ReferenceHandler.run() @bci=46, line=133 (Interpreted frame)


Thread 28152: (state = BLOCKED)
 - java.lang.ClassLoader.loadClass(java.lang.String, boolean) @bci=8, line=407 (Interpreted frame)
 - sun.misc.Launcher$AppClassLoader.loadClass(java.lang.String, boolean) @bci=36, line=308 (Interpreted frame)
 - java.lang.ClassLoader.loadClass(java.lang.String) @bci=3, line=358 (Interpreted frame)
 - com.newrelic.agent.stats.StatsEngineImpl.getStats(java.lang.String) @bci=2, line=46 (Interpreted frame)
 - com.newrelic.api.agent.RecordMetric.doWork(com.newrelic.agent.stats.StatsEngine) @bci=5, line=20 (Interpreted frame)
 - com.newrelic.agent.stats.StatsServiceImpl$StatsEngineQueue.doStatsWorkUnderLock(com.newrelic.agent.stats.StatsWork) @bci=34, line=137 (Interpreted frame)
 - com.newrelic.agent.stats.StatsServiceImpl$StatsEngineQueue.doStatsWork(com.newrelic.agent.stats.StatsWork) @bci=28, line=120 (Interpreted frame)
 - com.newrelic.agent.stats.StatsServiceImpl.doStatsWork(com.newrelic.agent.stats.StatsWork) @bci=19, line=57 (Interpreted frame)
 - com.newrelic.agent.instrumentation.AbstractTracingMethodAdapter.onMethodEnter() @bci=311, line=126 (Interpreted frame)
 - com.newrelic.deps.org.objectweb.asm.commons.AdviceAdapter.visitCode() @bci=45 (Interpreted frame)
 - com.newrelic.agent.instrumentation.AbstractTracingMethodAdapter.visitCode() @bci=1, line=170 (Interpreted frame)
 - com.newrelic.agent.instrumentation.InvocationHandlerTracingMethodAdapter.visitCode() @bci=1, line=16 (Interpreted frame)
 - com.newrelic.deps.org.objectweb.asm.ClassReader.accept(com.newrelic.deps.org.objectweb.asm.ClassVisitor, com.newrelic.deps.org.objectweb.asm.Attribute[], int) @bci=2343 (Interpreted frame)
 - com.newrelic.deps.org.objectweb.asm.ClassReader.accept(com.newrelic.deps.org.objectweb.asm.ClassVisitor, int) @bci=7 (Interpreted frame)
 - com.newrelic.agent.instrumentation.ClassTransformer$WeavingLoaderImpl.preProcess(java.lang.String, java.lang.Class, byte[]) @bci=285, line=366 (Interpreted frame)
 - com.newrelic.agent.instrumentation.ClassTransformer.transform(java.lang.ClassLoader, java.lang.String, java.lang.Class, java.security.ProtectionDomain, byte[]) @bci=98, line=198 (Interpreted frame)
 - sun.instrument.TransformerManager.transform(java.lang.ClassLoader, java.lang.String, java.lang.Class, java.security.ProtectionDomain, byte[]) @bci=50, line=188 (Compiled frame)
 - sun.instrument.InstrumentationImpl.transform(java.lang.ClassLoader, java.lang.String, java.lang.Class, java.security.ProtectionDomain, byte[], boolean) @bci=34, line=424 (Interpreted frame)
 - java.lang.ClassLoader.defineClass1(java.lang.String, byte[], int, int, java.security.ProtectionDomain, java.lang.String) @bci=0 (Interpreted frame)
 - java.lang.ClassLoader.defineClass(java.lang.String, byte[], int, int, java.security.ProtectionDomain) @bci=30, line=800 (Interpreted frame)
 - java.security.SecureClassLoader.defineClass(java.lang.String, byte[], int, int, java.security.CodeSource) @bci=12, line=142 (Interpreted frame)
 - java.net.URLClassLoader.defineClass(java.lang.String, sun.misc.Resource) @bci=220, line=449 (Interpreted frame)
 - java.net.URLClassLoader.access$100(java.net.URLClassLoader, java.lang.String, sun.misc.Resource) @bci=3, line=71 (Interpreted frame)
 - java.net.URLClassLoader$1.run() @bci=43, line=361 (Interpreted frame)
 - java.net.URLClassLoader$1.run() @bci=1, line=355 (Interpreted frame)
 - java.security.AccessController.doPrivileged(java.security.PrivilegedExceptionAction, java.security.AccessControlContext) @bci=0 (Interpreted frame)
 - java.net.URLClassLoader.findClass(java.lang.String) @bci=13, line=354 (Interpreted frame)
 - java.lang.ClassLoader.loadClass(java.lang.String, boolean) @bci=70, line=425 (Interpreted frame)
 - sun.misc.Launcher$AppClassLoader.loadClass(java.lang.String, boolean) @bci=36, line=308 (Interpreted frame)
 - java.lang.ClassLoader.loadClass(java.lang.String) @bci=3, line=358 (Interpreted frame)
 - com.typesafe.config.ConfigFactory.load(com.typesafe.config.ConfigParseOptions, com.typesafe.config.ConfigResolveOptions) @bci=13, line=375 (Interpreted frame)
 - com.typesafe.config.ConfigFactory.load(com.typesafe.config.ConfigParseOptions) @bci=4, line=299 (Interpreted frame)
 - com.typesafe.config.ConfigFactory.load() @bci=3, line=288 (Interpreted frame)
 - kamon.weaver.logging.KamonWeaverMessageHandler.<init>() @bci=14, line=30 (Interpreted frame)
 - sun.reflect.NativeConstructorAccessorImpl.newInstance0(java.lang.reflect.Constructor, java.lang.Object[]) @bci=0 (Interpreted frame)
 - sun.reflect.NativeConstructorAccessorImpl.newInstance(java.lang.Object[]) @bci=72, line=57 (Interpreted frame)
 - sun.reflect.DelegatingConstructorAccessorImpl.newInstance(java.lang.Object[]) @bci=5, line=45 (Interpreted frame)
 - java.lang.reflect.Constructor.newInstance(java.lang.Object[]) @bci=79, line=526 (Interpreted frame)
 - java.lang.Class.newInstance() @bci=132, line=374 (Interpreted frame)
 - org.aspectj.weaver.loadtime.Options.parse(java.lang.String, java.lang.ClassLoader, org.aspectj.bridge.IMessageHandler) @bci=112, line=70 (Interpreted frame)
 - org.aspectj.weaver.loadtime.ClassLoaderWeavingAdaptor.registerOptions(org.aspectj.weaver.bcel.BcelWeaver, java.lang.ClassLoader, java.util.List) @bci=68, line=337 (Interpreted frame)
 - org.aspectj.weaver.loadtime.ClassLoaderWeavingAdaptor.registerDefinitions(org.aspectj.weaver.bcel.BcelWeaver, java.lang.ClassLoader, java.util.List) @bci=30, line=301 (Interpreted frame)
 - org.aspectj.weaver.loadtime.ClassLoaderWeavingAdaptor.initialize(java.lang.ClassLoader, org.aspectj.weaver.loadtime.IWeavingContext) @bci=146, line=171 (Interpreted frame)
 - org.aspectj.weaver.loadtime.Aj$ExplicitlyInitializedClassLoaderWeavingAdaptor.initialize(java.lang.ClassLoader, org.aspectj.weaver.loadtime.IWeavingContext) @bci=18, line=339 (Interpreted frame)
 - org.aspectj.weaver.loadtime.Aj$ExplicitlyInitializedClassLoaderWeavingAdaptor.getWeavingAdaptor(java.lang.ClassLoader, org.aspectj.weaver.loadtime.IWeavingContext) @bci=3, line=344 (Interpreted frame)
 - org.aspectj.weaver.loadtime.Aj$WeaverContainer.getWeaver(java.lang.ClassLoader, org.aspectj.weaver.loadtime.IWeavingContext) @bci=134, line=318 (Interpreted frame)
 - org.aspectj.weaver.loadtime.Aj.preProcess(java.lang.String, byte[], java.lang.ClassLoader, java.security.ProtectionDomain) @bci=178, line=113 (Interpreted frame)
 - org.aspectj.weaver.loadtime.ClassPreProcessorAgentAdapter.transform(java.lang.ClassLoader, java.lang.String, java.lang.Class, java.security.ProtectionDomain, byte[]) @bci=43, line=54 (Interpreted frame)
 - sun.instrument.TransformerManager.transform(java.lang.ClassLoader, java.lang.String, java.lang.Class, java.security.ProtectionDomain, byte[]) @bci=50, line=188 (Interpreted frame)
 - sun.instrument.InstrumentationImpl.transform(java.lang.ClassLoader, java.lang.String, java.lang.Class, java.security.ProtectionDomain, byte[], boolean) @bci=34, line=424 (Interpreted frame)
 - java.lang.ClassLoader.defineClass1(java.lang.String, byte[], int, int, java.security.ProtectionDomain, java.lang.String) @bci=0 (Interpreted frame)
 - java.lang.ClassLoader.defineClass(java.lang.String, byte[], int, int, java.security.ProtectionDomain) @bci=30, line=800 (Interpreted frame)
 - java.security.SecureClassLoader.defineClass(java.lang.String, byte[], int, int, java.security.CodeSource) @bci=12, line=142 (Interpreted frame)
 - java.net.URLClassLoader.defineClass(java.lang.String, sun.misc.Resource) @bci=220, line=449 (Interpreted frame)
 - java.net.URLClassLoader.access$100(java.net.URLClassLoader, java.lang.String, sun.misc.Resource) @bci=3, line=71 (Interpreted frame)
 - java.net.URLClassLoader$1.run() @bci=43, line=361 (Interpreted frame)
 - java.net.URLClassLoader$1.run() @bci=1, line=355 (Interpreted frame)
 - java.security.AccessController.doPrivileged(java.security.PrivilegedExceptionAction, java.security.AccessControlContext) @bci=0 (Interpreted frame)
 - java.net.URLClassLoader.findClass(java.lang.String) @bci=13, line=354 (Interpreted frame)
 - java.lang.ClassLoader.loadClass(java.lang.String, boolean) @bci=70, line=425 (Interpreted frame)
 - sun.misc.Launcher$AppClassLoader.loadClass(java.lang.String, boolean) @bci=36, line=308 (Interpreted frame)
 - java.lang.ClassLoader.loadClass(java.lang.String) @bci=3, line=358 (Interpreted frame)
 - sun.launcher.LauncherHelper.checkAndLoadMain(boolean, int, java.lang.String) @bci=114, line=482 (Interpreted frame)

Attach AspectJ LTW at runtime

site: Ensure that preview-site can watch for changes

Currently we either start/stop the preview-site task to see the changes or we run jekyll serve --watch directly in the jekyll folder.. ideally, sbt should take care of watching the changes and building again.

will this work for akka 2.3 and scala 2.11?

When can we expect the project work for akka 2.3 and scala 2.11?

TraceContext is not propagated when call WS outside an Action

Implement error handling with NewRelic Agent

We should report data every minute and in case a request fails (for network issues, for example) we should wait until the next piece of data is available, merge both sets of values and report the merged values.

Make sure ask pattern tracing is optional

store in TraceLocal useful data to diagnose errors

some additional diagnostic data should be available when logging errors, specifically when doing HTTP the request URI might be a very useful information, probably putting it in a TraceLocal variable would be the best way to go.

Remove js dependencies in dashboard.

Add X-TRACE-TOKEN for play applications

http://www.playframework.com/documentation/2.1.3/ScalaHttpFilters

upgrade the kamon-newrelic module to our latest metrics scheme

After the huge changes introduced in #43 we need to upgrade the module and make sure that it correctly reports the data that was previously sent.

provide metrics for routers

Provide a way of monitoring a router with aggregate metrics from all of its routees, except for mailbox size which doesn't make sense. Maybe a configuration setting to determine whether you want router, routees or both metrics will be useful.

See more details here: https://groups.google.com/forum/#!searchin/kamon-user/router$20metrics/kamon-user/GC8g-UKEwV0/F68dqWpvRFsJ

make sure that automatically generated trace names are StatsD friendly

The default trace names generated by Kamon contain spaces and colons, which makes them fail when using Kamon/Spray/StatsD right out of the box.

create a Console Reporter(something like Metrics)

write tests for the subscriptions protocol

So far we just know that it works, but I'm not sure of the behavior when one subscribe multiple times or to multiple types of metrics, should we receive various messages or only one per tick?, let's define and document it with tests.