kamon-io / kamon Goto Github PK
View Code? Open in Web Editor NEWDistributed Tracing, Metrics and Context Propagation for applications running on the JVM
Home Page: https://kamon.io
License: Other
Distributed Tracing, Metrics and Context Propagation for applications running on the JVM
Home Page: https://kamon.io
License: Other
The following exception is thrown, as first reported on #34:
[ERROR] [06/05/2014 15:36:21.550] [services-scheduler-1] [akka.dispatch.Dispatcher] null
java.lang.NullPointerException
at akka.instrumentation.DispatcherTracing.afterDispatcherShutdown(DispatcherTracing.scala:85)
at akka.dispatch.Dispatcher.shutdown(Dispatcher.scala:101)
at akka.dispatch.MessageDispatcher$$anon$2.run(AbstractDispatcher.scala:212)
at akka.dispatch.MessageDispatcher$$anon$3.execute(AbstractDispatcher.scala:174)
at akka.actor.LightArrayRevolverScheduler$TaskHolder.executeTask(Scheduler.scala:467)
at akka.actor.LightArrayRevolverScheduler$$anon$8.executeBucket$1(Scheduler.scala:419)
at akka.actor.LightArrayRevolverScheduler$$anon$8.nextTick(Scheduler.scala:423)
at akka.actor.LightArrayRevolverScheduler$$anon$8.run(Scheduler.scala:375)
at java.lang.Thread.run(Thread.java:744)
@adrien-aubel, could you please describe a bit more how you found this issue? does it happen right after starting your app? did you notice any way to reproduce the issue that we might isolate and test?
we need change Highcharts dependency by other open source chart library
[ERROR] [services-scheduler-1] [akka.dispatch.Dispatcher] null
java.lang.NullPointerException
at akka.instrumentation.DispatcherTracing.afterDispatcherShutdown(DispatcherTracing.scala:85)
at akka.dispatch.Dispatcher.shutdown(Dispatcher.scala:101)
at akka.dispatch.MessageDispatcher$$anon$2.run(AbstractDispatcher.scala:212)
at akka.dispatch.MessageDispatcher$$anon$3.execute(AbstractDispatcher.scala:174)
at akka.actor.LightArrayRevolverScheduler$TaskHolder.executeTask(Scheduler.scala:467)
at akka.actor.LightArrayRevolverScheduler$$anon$8.executeBucket$1(Scheduler.scala:419)
at akka.actor.LightArrayRevolverScheduler$$anon$8.nextTick(Scheduler.scala:423)
at akka.actor.LightArrayRevolverScheduler$$anon$8.run(Scheduler.scala:375)
at java.lang.Thread.run(Thread.java:744)
It's because we trying to remove from context a dispatcher that not it registered before.
When I try to start my server, it encounters a Java level deadlock. Please take a look at the attached jstack output for more details. The JVM aguments are:
-Xms256m -Xmx1536m -javaagent:/opt/newrelic/java/newrelic.jar -javaagent:/opt/aspectj/aspectjweaver.jar
Found one Java-level deadlock:
=============================
"main":
waiting to lock Monitor@0x00007fe3ec004ab8 (Object@0x000000009af590a8, a java/lang/Object),
which is held by "New Relic Sampler Service"
"New Relic Sampler Service":
waiting to lock Monitor@0x00007fe3ec006218 (Object@0x00000000baedc9f0, a sun/misc/Launcher$AppClassLoader),
which is held by "main"
Found a total of 1 deadlock.
Thread 28160: (state = BLOCKED)
- sun.misc.Unsafe.park(boolean, long) @bci=0 (Interpreted frame)
- java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long) @bci=20, line=226 (Interpreted frame)
- java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(long) @bci=68, line=2082 (Interpreted frame)
- java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take() @bci=122, line=1090 (Interpreted frame)
- java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take() @bci=1, line=807 (Interpreted frame)
- java.util.concurrent.ThreadPoolExecutor.getTask() @bci=156, line=1068 (Interpreted frame)
- java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=26, line=1130 (Interpreted frame)
- java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=615 (Interpreted frame)
- java.lang.Thread.run() @bci=11, line=744 (Interpreted frame)
Thread 28159: (state = BLOCKED)
- org.aspectj.weaver.loadtime.Aj.preProcess(java.lang.String, byte[], java.lang.ClassLoader, java.security.ProtectionDomain) @bci=136, line=106 (Interpreted frame)
- org.aspectj.weaver.loadtime.ClassPreProcessorAgentAdapter.transform(java.lang.ClassLoader, java.lang.String, java.lang.Class, java.security.ProtectionDomain, byte[]) @bci=43, line=54 (Interpreted frame)
- sun.instrument.TransformerManager.transform(java.lang.ClassLoader, java.lang.String, java.lang.Class, java.security.ProtectionDomain, byte[]) @bci=50, line=188 (Compiled frame)
- sun.instrument.InstrumentationImpl.transform(java.lang.ClassLoader, java.lang.String, java.lang.Class, java.security.ProtectionDomain, byte[], boolean) @bci=34, line=424 (Interpreted frame)
- java.lang.ClassLoader.defineClass1(java.lang.String, byte[], int, int, java.security.ProtectionDomain, java.lang.String) @bci=0 (Interpreted frame)
- java.lang.ClassLoader.defineClass(java.lang.String, byte[], int, int, java.security.ProtectionDomain) @bci=30, line=800 (Interpreted frame)
- java.security.SecureClassLoader.defineClass(java.lang.String, byte[], int, int, java.security.CodeSource) @bci=12, line=142 (Interpreted frame)
- java.net.URLClassLoader.defineClass(java.lang.String, sun.misc.Resource) @bci=220, line=449 (Interpreted frame)
- java.net.URLClassLoader.access$100(java.net.URLClassLoader, java.lang.String, sun.misc.Resource) @bci=3, line=71 (Interpreted frame)
- java.net.URLClassLoader$1.run() @bci=43, line=361 (Interpreted frame)
- java.net.URLClassLoader$1.run() @bci=1, line=355 (Interpreted frame)
- java.security.AccessController.doPrivileged(java.security.PrivilegedExceptionAction, java.security.AccessControlContext) @bci=0 (Interpreted frame)
- java.net.URLClassLoader.findClass(java.lang.String) @bci=13, line=354 (Interpreted frame)
- java.lang.ClassLoader.loadClass(java.lang.String, boolean) @bci=70, line=425 (Interpreted frame)
- sun.misc.Launcher$AppClassLoader.loadClass(java.lang.String, boolean) @bci=36, line=308 (Interpreted frame)
- java.lang.ClassLoader.loadClass(java.lang.String) @bci=3, line=358 (Interpreted frame)
- com.newrelic.agent.stats.StatsEngineImpl.getStats(java.lang.String) @bci=2, line=46 (Interpreted frame)
- com.newrelic.agent.samplers.MemorySampler$HeapAndNonHeapUsage.recordStats(com.newrelic.agent.stats.StatsEngine) @bci=3, line=96 (Interpreted frame)
- com.newrelic.agent.samplers.MemorySampler.sampleMemory(com.newrelic.agent.stats.StatsEngine) @bci=15, line=42 (Interpreted frame)
- com.newrelic.agent.samplers.MemorySampler.sample(com.newrelic.agent.stats.StatsEngine) @bci=2, line=35 (Interpreted frame)
- com.newrelic.agent.samplers.SamplerServiceImpl.runSampler(com.newrelic.agent.samplers.MetricSampler) @bci=18, line=106 (Interpreted frame)
- com.newrelic.agent.samplers.SamplerServiceImpl.access$000(com.newrelic.agent.samplers.SamplerServiceImpl, com.newrelic.agent.samplers.MetricSampler) @bci=2, line=31 (Interpreted frame)
- com.newrelic.agent.samplers.SamplerServiceImpl$1.run() @bci=8, line=85 (Interpreted frame)
- java.util.concurrent.Executors$RunnableAdapter.call() @bci=4, line=471 (Interpreted frame)
- java.util.concurrent.FutureTask.runAndReset() @bci=47, line=304 (Interpreted frame)
- java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask) @bci=1, line=178 (Interpreted frame)
- java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run() @bci=37, line=293 (Interpreted frame)
- java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=95, line=1145 (Interpreted frame)
- java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=615 (Interpreted frame)
- java.lang.Thread.run() @bci=11, line=744 (Interpreted frame)
Thread 28158: (state = BLOCKED)
- sun.misc.Unsafe.park(boolean, long) @bci=0 (Interpreted frame)
- java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long) @bci=20, line=226 (Interpreted frame)
- java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(long) @bci=68, line=2082 (Interpreted frame)
- java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take() @bci=122, line=1090 (Interpreted frame)
- java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take() @bci=1, line=807 (Interpreted frame)
- java.util.concurrent.ThreadPoolExecutor.getTask() @bci=156, line=1068 (Interpreted frame)
- java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=26, line=1130 (Interpreted frame)
- java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=615 (Interpreted frame)
- java.lang.Thread.run() @bci=11, line=744 (Interpreted frame)
Thread 28157: (state = BLOCKED)
- sun.misc.Unsafe.park(boolean, long) @bci=0 (Interpreted frame)
- java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long) @bci=20, line=226 (Interpreted frame)
- java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(long) @bci=68, line=2082 (Interpreted frame)
- java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take() @bci=122, line=1090 (Interpreted frame)
- java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take() @bci=1, line=807 (Interpreted frame)
- java.util.concurrent.ThreadPoolExecutor.getTask() @bci=156, line=1068 (Interpreted frame)
- java.util.concurrent.ThreadPoolExecutor.runWorker(java.util.concurrent.ThreadPoolExecutor$Worker) @bci=26, line=1130 (Interpreted frame)
- java.util.concurrent.ThreadPoolExecutor$Worker.run() @bci=5, line=615 (Interpreted frame)
- java.lang.Thread.run() @bci=11, line=744 (Interpreted frame)
Thread 28156: (state = BLOCKED)
Thread 28155: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Interpreted frame)
- java.lang.ref.ReferenceQueue.remove(long) @bci=44, line=135 (Interpreted frame)
- java.lang.ref.ReferenceQueue.remove() @bci=2, line=151 (Interpreted frame)
- java.lang.ref.Finalizer$FinalizerThread.run() @bci=16, line=189 (Interpreted frame)
Thread 28154: (state = BLOCKED)
- java.lang.Object.wait(long) @bci=0 (Interpreted frame)
- java.lang.Object.wait() @bci=2, line=503 (Interpreted frame)
- java.lang.ref.Reference$ReferenceHandler.run() @bci=46, line=133 (Interpreted frame)
Thread 28152: (state = BLOCKED)
- java.lang.ClassLoader.loadClass(java.lang.String, boolean) @bci=8, line=407 (Interpreted frame)
- sun.misc.Launcher$AppClassLoader.loadClass(java.lang.String, boolean) @bci=36, line=308 (Interpreted frame)
- java.lang.ClassLoader.loadClass(java.lang.String) @bci=3, line=358 (Interpreted frame)
- com.newrelic.agent.stats.StatsEngineImpl.getStats(java.lang.String) @bci=2, line=46 (Interpreted frame)
- com.newrelic.api.agent.RecordMetric.doWork(com.newrelic.agent.stats.StatsEngine) @bci=5, line=20 (Interpreted frame)
- com.newrelic.agent.stats.StatsServiceImpl$StatsEngineQueue.doStatsWorkUnderLock(com.newrelic.agent.stats.StatsWork) @bci=34, line=137 (Interpreted frame)
- com.newrelic.agent.stats.StatsServiceImpl$StatsEngineQueue.doStatsWork(com.newrelic.agent.stats.StatsWork) @bci=28, line=120 (Interpreted frame)
- com.newrelic.agent.stats.StatsServiceImpl.doStatsWork(com.newrelic.agent.stats.StatsWork) @bci=19, line=57 (Interpreted frame)
- com.newrelic.agent.instrumentation.AbstractTracingMethodAdapter.onMethodEnter() @bci=311, line=126 (Interpreted frame)
- com.newrelic.deps.org.objectweb.asm.commons.AdviceAdapter.visitCode() @bci=45 (Interpreted frame)
- com.newrelic.agent.instrumentation.AbstractTracingMethodAdapter.visitCode() @bci=1, line=170 (Interpreted frame)
- com.newrelic.agent.instrumentation.InvocationHandlerTracingMethodAdapter.visitCode() @bci=1, line=16 (Interpreted frame)
- com.newrelic.deps.org.objectweb.asm.ClassReader.accept(com.newrelic.deps.org.objectweb.asm.ClassVisitor, com.newrelic.deps.org.objectweb.asm.Attribute[], int) @bci=2343 (Interpreted frame)
- com.newrelic.deps.org.objectweb.asm.ClassReader.accept(com.newrelic.deps.org.objectweb.asm.ClassVisitor, int) @bci=7 (Interpreted frame)
- com.newrelic.agent.instrumentation.ClassTransformer$WeavingLoaderImpl.preProcess(java.lang.String, java.lang.Class, byte[]) @bci=285, line=366 (Interpreted frame)
- com.newrelic.agent.instrumentation.ClassTransformer.transform(java.lang.ClassLoader, java.lang.String, java.lang.Class, java.security.ProtectionDomain, byte[]) @bci=98, line=198 (Interpreted frame)
- sun.instrument.TransformerManager.transform(java.lang.ClassLoader, java.lang.String, java.lang.Class, java.security.ProtectionDomain, byte[]) @bci=50, line=188 (Compiled frame)
- sun.instrument.InstrumentationImpl.transform(java.lang.ClassLoader, java.lang.String, java.lang.Class, java.security.ProtectionDomain, byte[], boolean) @bci=34, line=424 (Interpreted frame)
- java.lang.ClassLoader.defineClass1(java.lang.String, byte[], int, int, java.security.ProtectionDomain, java.lang.String) @bci=0 (Interpreted frame)
- java.lang.ClassLoader.defineClass(java.lang.String, byte[], int, int, java.security.ProtectionDomain) @bci=30, line=800 (Interpreted frame)
- java.security.SecureClassLoader.defineClass(java.lang.String, byte[], int, int, java.security.CodeSource) @bci=12, line=142 (Interpreted frame)
- java.net.URLClassLoader.defineClass(java.lang.String, sun.misc.Resource) @bci=220, line=449 (Interpreted frame)
- java.net.URLClassLoader.access$100(java.net.URLClassLoader, java.lang.String, sun.misc.Resource) @bci=3, line=71 (Interpreted frame)
- java.net.URLClassLoader$1.run() @bci=43, line=361 (Interpreted frame)
- java.net.URLClassLoader$1.run() @bci=1, line=355 (Interpreted frame)
- java.security.AccessController.doPrivileged(java.security.PrivilegedExceptionAction, java.security.AccessControlContext) @bci=0 (Interpreted frame)
- java.net.URLClassLoader.findClass(java.lang.String) @bci=13, line=354 (Interpreted frame)
- java.lang.ClassLoader.loadClass(java.lang.String, boolean) @bci=70, line=425 (Interpreted frame)
- sun.misc.Launcher$AppClassLoader.loadClass(java.lang.String, boolean) @bci=36, line=308 (Interpreted frame)
- java.lang.ClassLoader.loadClass(java.lang.String) @bci=3, line=358 (Interpreted frame)
- com.typesafe.config.ConfigFactory.load(com.typesafe.config.ConfigParseOptions, com.typesafe.config.ConfigResolveOptions) @bci=13, line=375 (Interpreted frame)
- com.typesafe.config.ConfigFactory.load(com.typesafe.config.ConfigParseOptions) @bci=4, line=299 (Interpreted frame)
- com.typesafe.config.ConfigFactory.load() @bci=3, line=288 (Interpreted frame)
- kamon.weaver.logging.KamonWeaverMessageHandler.<init>() @bci=14, line=30 (Interpreted frame)
- sun.reflect.NativeConstructorAccessorImpl.newInstance0(java.lang.reflect.Constructor, java.lang.Object[]) @bci=0 (Interpreted frame)
- sun.reflect.NativeConstructorAccessorImpl.newInstance(java.lang.Object[]) @bci=72, line=57 (Interpreted frame)
- sun.reflect.DelegatingConstructorAccessorImpl.newInstance(java.lang.Object[]) @bci=5, line=45 (Interpreted frame)
- java.lang.reflect.Constructor.newInstance(java.lang.Object[]) @bci=79, line=526 (Interpreted frame)
- java.lang.Class.newInstance() @bci=132, line=374 (Interpreted frame)
- org.aspectj.weaver.loadtime.Options.parse(java.lang.String, java.lang.ClassLoader, org.aspectj.bridge.IMessageHandler) @bci=112, line=70 (Interpreted frame)
- org.aspectj.weaver.loadtime.ClassLoaderWeavingAdaptor.registerOptions(org.aspectj.weaver.bcel.BcelWeaver, java.lang.ClassLoader, java.util.List) @bci=68, line=337 (Interpreted frame)
- org.aspectj.weaver.loadtime.ClassLoaderWeavingAdaptor.registerDefinitions(org.aspectj.weaver.bcel.BcelWeaver, java.lang.ClassLoader, java.util.List) @bci=30, line=301 (Interpreted frame)
- org.aspectj.weaver.loadtime.ClassLoaderWeavingAdaptor.initialize(java.lang.ClassLoader, org.aspectj.weaver.loadtime.IWeavingContext) @bci=146, line=171 (Interpreted frame)
- org.aspectj.weaver.loadtime.Aj$ExplicitlyInitializedClassLoaderWeavingAdaptor.initialize(java.lang.ClassLoader, org.aspectj.weaver.loadtime.IWeavingContext) @bci=18, line=339 (Interpreted frame)
- org.aspectj.weaver.loadtime.Aj$ExplicitlyInitializedClassLoaderWeavingAdaptor.getWeavingAdaptor(java.lang.ClassLoader, org.aspectj.weaver.loadtime.IWeavingContext) @bci=3, line=344 (Interpreted frame)
- org.aspectj.weaver.loadtime.Aj$WeaverContainer.getWeaver(java.lang.ClassLoader, org.aspectj.weaver.loadtime.IWeavingContext) @bci=134, line=318 (Interpreted frame)
- org.aspectj.weaver.loadtime.Aj.preProcess(java.lang.String, byte[], java.lang.ClassLoader, java.security.ProtectionDomain) @bci=178, line=113 (Interpreted frame)
- org.aspectj.weaver.loadtime.ClassPreProcessorAgentAdapter.transform(java.lang.ClassLoader, java.lang.String, java.lang.Class, java.security.ProtectionDomain, byte[]) @bci=43, line=54 (Interpreted frame)
- sun.instrument.TransformerManager.transform(java.lang.ClassLoader, java.lang.String, java.lang.Class, java.security.ProtectionDomain, byte[]) @bci=50, line=188 (Interpreted frame)
- sun.instrument.InstrumentationImpl.transform(java.lang.ClassLoader, java.lang.String, java.lang.Class, java.security.ProtectionDomain, byte[], boolean) @bci=34, line=424 (Interpreted frame)
- java.lang.ClassLoader.defineClass1(java.lang.String, byte[], int, int, java.security.ProtectionDomain, java.lang.String) @bci=0 (Interpreted frame)
- java.lang.ClassLoader.defineClass(java.lang.String, byte[], int, int, java.security.ProtectionDomain) @bci=30, line=800 (Interpreted frame)
- java.security.SecureClassLoader.defineClass(java.lang.String, byte[], int, int, java.security.CodeSource) @bci=12, line=142 (Interpreted frame)
- java.net.URLClassLoader.defineClass(java.lang.String, sun.misc.Resource) @bci=220, line=449 (Interpreted frame)
- java.net.URLClassLoader.access$100(java.net.URLClassLoader, java.lang.String, sun.misc.Resource) @bci=3, line=71 (Interpreted frame)
- java.net.URLClassLoader$1.run() @bci=43, line=361 (Interpreted frame)
- java.net.URLClassLoader$1.run() @bci=1, line=355 (Interpreted frame)
- java.security.AccessController.doPrivileged(java.security.PrivilegedExceptionAction, java.security.AccessControlContext) @bci=0 (Interpreted frame)
- java.net.URLClassLoader.findClass(java.lang.String) @bci=13, line=354 (Interpreted frame)
- java.lang.ClassLoader.loadClass(java.lang.String, boolean) @bci=70, line=425 (Interpreted frame)
- sun.misc.Launcher$AppClassLoader.loadClass(java.lang.String, boolean) @bci=36, line=308 (Interpreted frame)
- java.lang.ClassLoader.loadClass(java.lang.String) @bci=3, line=358 (Interpreted frame)
- sun.launcher.LauncherHelper.checkAndLoadMain(boolean, int, java.lang.String) @bci=114, line=482 (Interpreted frame)
I'm testing Kamon with NewRelic and Spray. My application uses REST style requests where the URL contains, for example, a unique DeviceID. Currently every request to the same resource type is reported separately because the unique identifier is included in the path.
Ideally Kamon could understand from the Spray path definitions how the requests should be grouped.
The dashboard that we are currently presenting just shows some data and expresses that it is possible to see metrics there, but lacks a few important bits:
Have I understood correctly that if I want to use "sbt assembly", I need to manually create my own aop.xml file, which combines the aop definitions from the individual kamon modules? Are you considering better solutions to this problem?
When a application is configured to use the NewRelicErrorLogger but wasn't started with the aspectj weaver the class cast fails and the error loops back into the logger, producing a infinite failure loop that eventually kills the app.
When the application is not instrumented this logger should report the error anyway, missing the TraceToken information.
Exception:
flights-catalog java.lang.ClassCastException: akka.event.Logging$Error cannot be cast to kamon.trace.TraceContextAware
flights-catalog at kamon.newrelic.NewRelicErrorLogger.notifyError(NewRelicErrorLogger.scala:33) ~[kamon-newrelic-0.3.0.jar:0.3.0]
[ERROR][threads-redis-client-akka.actor.default-dispatcher-5] [akka://threads-redis-client/system/IO-TCP/selectors] null
akka.actor.ActorInitializationException: exception during creation
at akka.actor.ActorInitializationException$.apply(Actor.scala:164)
at akka.actor.ActorCell.create(ActorCell.scala:596)
at akka.actor.ActorCell.invokeAll$1_aroundBody2(ActorCell.scala:456)
at akka.actor.ActorCell$AjcClosure3.run(ActorCell.scala:1)
at org.aspectj.runtime.reflect.JoinPointImpl.proceed(JoinPointImpl.java:149)
at akka.instrumentation.ActorSystemMessagePassingTracing$$anonfun$aroundSystemMessageInvoke$1.apply(ActorSystemMessagePassingTracing.scala:61)
at kamon.trace.TraceRecorder$.withTraceContext(TraceRecorder.scala:66)
at akka.instrumentation.ActorSystemMessagePassingTracing.aroundSystemMessageInvoke(ActorSystemMessagePassingTracing.scala:61)
at akka.actor.ActorCell.invokeAll$1(ActorCell.scala:1)
at akka.actor.ActorCell.systemInvoke(ActorCell.scala:478)
at akka.dispatch.Mailbox.processAllSystemMessages(Mailbox.scala:263)
at akka.dispatch.Mailbox.run(Mailbox.scala:219)
at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:393)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at akka.util.Reflect$.instantiate(Reflect.scala:66)
at akka.actor.ArgsReflectConstructor.produce(Props.scala:349)
at akka.actor.Props.newActor(Props.scala:249)
at akka.actor.ActorCell.newActor(ActorCell.scala:552)
at akka.actor.ActorCell.create(ActorCell.scala:578)
... 15 more
Caused by: java.lang.NullPointerException
at akka.actor.ActorSystemImpl.findExtension(ActorSystem.scala:697)
at akka.actor.ActorSystemImpl.registerExtension(ActorSystem.scala:706)
at akka.actor.ExtensionId$class.apply(Extension.scala:79)
at kamon.metrics.Metrics$.apply(MetricsExtension.scala:97)
at kamon.Kamon$.apply(Kamon.scala:31)
at akka.instrumentation.DispatcherTracing.afterDispatcherStartup(DispatcherTracing.scala:55)
at akka.dispatch.Dispatcher$LazyExecutorServiceDelegate.executor$lzycompute(Dispatcher.scala:44)
at akka.dispatch.Dispatcher$LazyExecutorServiceDelegate.executor(Dispatcher.scala:44)
at akka.dispatch.ExecutorServiceDelegate$class.execute(ThreadPoolBuilder.scala:212)
at akka.dispatch.Dispatcher$LazyExecutorServiceDelegate.execute(Dispatcher.scala:43)
at akka.dispatch.Dispatcher.executeTask(Dispatcher.scala:76)
at akka.dispatch.MessageDispatcher.unbatchedExecute(AbstractDispatcher.scala:145)
at akka.dispatch.BatchingExecutor$class.execute(BatchingExecutor.scala:113)
...
It seems like using statsd is becoming more and more popular and using the subscription protocol it should be really easy to get this integration working very quickly.
The default trace names generated by Kamon contain spaces and colons, which makes them fail when using Kamon/Spray/StatsD right out of the box.
So far we just know that it works, but I'm not sure of the behavior when one subscribe multiple times or to multiple types of metrics, should we receive various messages or only one per tick?, let's define and document it with tests.
We should have a simple tutorial per toolkit (Akka, Spray and Play) as an activator template that can be pointed out to when discussing issues or giving help to people.
We should report data every minute and in case a request fails (for network issues, for example) we should wait until the next piece of data is available, merge both sets of values and report the merged values.
During application startup several exceptions as the one bellow are logged, and after some time the application starts working normally. The help we got at the aspectj mailing list suggests that the problem is related to the perthis association + funky class loaders, will try avoiding perthis association to solve this issue.
org.aspectj.weaver.bcel.BcelWorld - Unable to find class 'akka.instrumentation.BehaviourInvokeTracing$ajcMightHaveAspect' in repository
java.lang.ClassNotFoundException: akka.instrumentation.BehaviourInvokeTracing$ajcMightHaveAspect not found - unable to determine URL
at org.aspectj.apache.bcel.util.ClassLoaderRepository.loadClass(ClassLoaderRepository.java:292) ~[aspectjweaver-1.7.4.jar:1.7.4]
at org.aspectj.weaver.bcel.BcelWorld.lookupJavaClass(BcelWorld.java:418) [aspectjweaver-1.7.4.jar:1.7.4]
at org.aspectj.weaver.bcel.BcelWorld.resolveDelegate(BcelWorld.java:392) [aspectjweaver-1.7.4.jar:1.7.4]
at org.aspectj.weaver.ltw.LTWWorld.resolveDelegate(LTWWorld.java:107) [aspectjweaver-1.7.4.jar:1.7.4]
at org.aspectj.weaver.World.resolveToReferenceType(World.java:477) [aspectjweaver-1.7.4.jar:1.7.4]
at org.aspectj.weaver.World.resolve(World.java:318) [aspectjweaver-1.7.4.jar:1.7.4]
at org.aspectj.weaver.World.resolve(World.java:228) [aspectjweaver-1.7.4.jar:1.7.4]
at org.aspectj.weaver.UnresolvedType.resolve(UnresolvedType.java:616) [aspectjweaver-1.7.4.jar:1.7.4]
at org.aspectj.weaver.bcel.BcelTypeMunger.mungePerObjectInterface(BcelTypeMunger.java:796) [aspectjweaver-1.7.4.jar:1.7.4]
at org.aspectj.weaver.bcel.BcelTypeMunger.munge(BcelTypeMunger.java:116) [aspectjweaver-1.7.4.jar:1.7.4]
at org.aspectj.weaver.bcel.BcelClassWeaver.weave(BcelClassWeaver.java:516) [aspectjweaver-1.7.4.jar:1.7.4]
at org.aspectj.weaver.bcel.BcelClassWeaver.weave(BcelClassWeaver.java:101) [aspectjweaver-1.7.4.jar:1.7.4]
at org.aspectj.weaver.bcel.BcelWeaver.weave(BcelWeaver.java:1691) [aspectjweaver-1.7.4.jar:1.7.4]
at org.aspectj.weaver.bcel.BcelWeaver.weaveWithoutDump(BcelWeaver.java:1635) [aspectjweaver-1.7.4.jar:1.7.4]
at org.aspectj.weaver.bcel.BcelWeaver.weaveAndNotify(BcelWeaver.java:1400) [aspectjweaver-1.7.4.jar:1.7.4]
at org.aspectj.weaver.bcel.BcelWeaver.weave(BcelWeaver.java:1186) [aspectjweaver-1.7.4.jar:1.7.4]
at org.aspectj.weaver.tools.WeavingAdaptor.getWovenBytes(WeavingAdaptor.java:527) [aspectjweaver-1.7.4.jar:1.7.4]
at org.aspectj.weaver.tools.WeavingAdaptor.weaveClass(WeavingAdaptor.java:363) [aspectjweaver-1.7.4.jar:1.7.4]
at org.aspectj.weaver.loadtime.Aj.preProcess(Aj.java:121) [aspectjweaver-1.7.4.jar:1.7.4]
at org.aspectj.weaver.loadtime.ClassPreProcessorAgentAdapter.transform(ClassPreProcessorAgentAdapter.java:54) [aspectjweaver-1.7.4.jar:1.7.4]
at sun.instrument.TransformerManager.transform(TransformerManager.java:188) [na:1.7.0_45]
at sun.instrument.InstrumentationImpl.transform(InstrumentationImpl.java:424) [na:1.7.0_45]
at java.lang.ClassLoader.defineClass1(Native Method) [na:1.7.0_45]
at java.lang.ClassLoader.defineClass(ClassLoader.java:800) [na:1.7.0_45]
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) [na:1.7.0_45]
at java.net.URLClassLoader.defineClass(URLClassLoader.java:449) [na:1.7.0_45]
at java.net.URLClassLoader.access$100(URLClassLoader.java:71) [na:1.7.0_45]
at java.net.URLClassLoader$1.run(URLClassLoader.java:361) [na:1.7.0_45]
at java.net.URLClassLoader$1.run(URLClassLoader.java:355) [na:1.7.0_45]
at java.security.AccessController.doPrivileged(Native Method) [na:1.7.0_45]
at java.net.URLClassLoader.findClass(URLClassLoader.java:354) [na:1.7.0_45]
at java.lang.ClassLoader.loadClass(ClassLoader.java:425) [na:1.7.0_45]
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) [na:1.7.0_45]
at java.lang.ClassLoader.loadClass(ClassLoader.java:358) [na:1.7.0_45]
at java.lang.Class.getDeclaredFields0(Native Method) [na:1.7.0_45]
at java.lang.Class.privateGetDeclaredFields(Class.java:2397) [na:1.7.0_45]
at java.lang.Class.getDeclaredField(Class.java:1946) [na:1.7.0_45]
If no data is available, avoid posting to Newrelic. Also a back-off strategy must be implemented for cases where collectors become unreachable or the POST requests fail.
After the huge changes introduced in #43 we need to upgrade the module and make sure that it correctly reports the data that was previously sent.
Hi,
On our play project, using https://github.com/jrudolph/sbt-dependency-graph we notice that kamon-play have scalatestplus as a runtime dependency, and using plugins like sbt-pack or fatjar, this library (and its dependencies) are included on our package, can you look at this?
the basic support should at least allow:
Apparently the Windows agent does not support multiple metrics per packet to work properly, we might need to add a setting to define whether multiple metrics can be reported in a single packet or not. Issue #42 has more details on how this was initiated.
Watch the actors and make sure that we unsubscribe them if they die.
Why is it documentation that max-packet-size
is in bytes?
Instead, the HOCON-supported file size format might be used to just do
max-packet-size=1024bytes
some additional diagnostic data should be available when logging errors, specifically when doing HTTP the request URI might be a very useful information, probably putting it in a TraceLocal variable would be the best way to go.
Currently we either start/stop the preview-site
task to see the changes or we run jekyll serve --watch
directly in the jekyll folder.. ideally, sbt should take care of watching the changes and building again.
When can we expect the project work for akka 2.3 and scala 2.11?
Hi,
I've been investigating lately on an issue that was happening on my application: the more the application is running, the higher is the CPU usage of this application.
It gets so high that it sometimes gets killed by the OS.
After like 24h the CPU usage starts to increase significantly, and I've attached a profiling on it. I've attached below two screenshots of a CPU call tree on JProfiler (same tree, just scrolling down):
Most of the CPU usage is related to Kamon IO. Let me know if you need more information. I'll need to safely restart the application soon, to avoid the app to get killed by the OS.
FYI, I'm using 2 JVMs on this server, that has 2x E5-2650v2 (16 cores / 32 threads)
Provide a way of monitoring a router with aggregate metrics from all of its routees, except for mailbox size which doesn't make sense. Maybe a configuration setting to determine whether you want router, routees or both metrics will be useful.
See more details here: https://groups.google.com/forum/#!searchin/kamon-user/router$20metrics/kamon-user/GC8g-UKEwV0/F68dqWpvRFsJ
The idea is to have a helper class that encapsulates a ActorSystem and provides a very simple API for metrics (and traces right after that). This way everyone that wants to benefit from the instruments and reportes we have can do so, no matter if they are building reactive applications or not.
The exception bellow is fired when no slf4j-api is available in the classpath.. we should either declare the dependency explicitly to make sure it is always there or eliminate the dependency.
[AppClassLoader@138297fe] error Cannot instantiate message handler kamon.weaver.logging.KamonWeaverMessageHandler -- (NoClassDefFoundError) org/slf4j/LoggerFactory
12:54:33 web.1 | org/slf4j/LoggerFactory
12:54:33 web.1 | java.lang.NoClassDefFoundError: org/slf4j/LoggerFactory
12:54:33 web.1 | at kamon.weaver.logging.KamonWeaverMessageHandler.<init>(KamonWeaverMessageHandler.scala:29)
See comments in this PR: #13
Thought of using SnakeYaml.
Create a simple module that registers a few gauges with information about CPU usage, garbage collection stats, disk IO usage and network traffic.
Our current instrumentation is wrapping the call to DefaultOpenRequest.handleResponseEndAndReturnNextOpenRequest
, which is OK in case of a regular HttpResponse, but when the response is sent in chunks by the time the ChunkedMessageEnd
is received the headers were already sent to the client and setting the trace token header becomes useless.
Something as simple as having a class that provides an error log message that will be overwritten with a successful error message by an aspect. This should probably be logged when the Kamon(Metrics) extension is loaded.
Exception in thread "main" java.lang.UnsupportedClassVersionError: kamon/util/GlobPathFilter : Unsupported major.minor version 51.0
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClassCond(ClassLoader.java:632)
at java.lang.ClassLoader.defineClass(ClassLoader.java:616)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:141)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:283)
at java.net.URLClassLoader.access$000(URLlassLoader.java:58)
at java.net.URLClassLoader$1.run(URLClassLoader.java:197)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
Kamon:
io.kamon
kamon-core_2.10
0.3.1
io.kamon
kamon-datadog_2.10
0.3.1
<SCALA_MAJOR>2.10</SCALA_MAJOR>
<SCALA_VERSION>2.10.3</SCALA_VERSION>
<AKKA_VERSION>2.3.2</AKKA_VERSION>
JDK 1.6
An approach could be action composition.
http://www.playframework.com/documentation/2.2.x/ScalaActionsComposition
ie.
val apiRoute = pathPrefix("foo") {
path(PathEnd) {
redirect(Uri("/foo/bar/index.html"), StatusCodes.MovedPermanently)
}
}
curl -i "http://localhost:8000/foo/" return:
HTTP/1.1 301 Moved Permanently
Server: spray-can/1.2.0
Date: adate
X-Trace-Token: atoken
Content-Type: text/html
Content-Length: 96
and should return:
HTTP/1.1 301 Moved Permanently
Server: spray-can/1.2.0
Date: adate
X-Trace-Token: atoken
Location: /foo/bar/index.html
Content-Type: text/html
Content-Length: 96
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.