mirkosertic/flight-recorder-starter

This is a Spring Boot 2/3 Starter exposing the JDK Flight Recorder as a Spring Boot Actuator Endpoint.

Hello, my name is Mirko Sertic. Software Engineer 👋.

What I do

I provide Software and Support in the field of electronic data processing. As a consultant and software craftsman I have supported a number of major players in projects of all sizes in Germany and Switzerland. My experience as a developer and architect makes informed decisions, sustainable architectures and sound operation of all kinds of software.

flight-recorder-starter's Issues

Suggestion about the loading of properties sources

Currently, we have the @propertysource and the definition of @beans in the same configuration. That situation doesn't allow easily to extend the starter and to load custom defined values before the standard values already predefined in the starter code.

I think that it should be a great idea if we could separate them into two different configuration classes, one for property source and another for beans.

[bug] Error on startup when the property flightrecorder.enabled has the value false

Currently, by default the starter is up for all configuration classes, but when we mark as false the property flightrecorder.enabled the Autoconfiguration FlightRecorderAutoConfiguration is not loaded, but yes the ManagementConfiguration.

In that case, ManagementConfiguration need a @bean that should load into context the class FlightRecorderAutoConfiguration. In fact, that has no sense because the management configuration class should only be loaded when the whole starter is enabled.

Suggestion about MicrometerAdapter with more than one MeterRegistry

Sometimes, there are more than one MeterRegistry object bean in the project so it could be interesting if we can redefine the bean MicrometerAdapter or define for which meter registry we want to apply which expression

Suggestion about parameters for the recording

It could be useful if we can set not just only the duration of the recording but the map with settings to configure de JFR api with more settings like delay, maxSize, maxAge, etc...

JFR File Download does not use transfer-encoding/chunked method

I faced an issue using this excellent starter (thank you). Our service was running behind Google Cloud Run which defines a hard limit of 32MB response unless the

from: https://cloud.google.com/run/quotas#cloud_run_limits

Maximum HTTP/1 response size | 32 MiB if not using Transfer-Encoding: chunked or streaming mechanisms

The Download endpoint sets the headers for file transfer, but doesn't specifically return a 'friendly' streaming response (at least not friendly to Google Cloud Run):

flight-recorder-starter/src/main/java/de/mirkosertic/flightrecorderstarter/actuator/FlightRecorderEndpoint.java

Line 121 in c47fe6f

return ResponseEntity.ok()

I suspect this method would need to change to something like a StreamingResponseBody, possibly?

Issue with trigger metrics with Prometheus Registry

If you use prometheus registry, the metrics are never reset, so when a threshold is reached, the condition will be always matched so a new recording starts each time the trigger checker is launched.

Maybe, it should be a great idea if we can note that in documentation. Just only "step" registries works propertly with that functionality.

Explanation of autodeletion of files in documentation

Currently, the deletion process is executed via an scheduled operation that is not explained in documentation. It could be a great idea if we can explain it at documentation.

Webflux support for actuator endpoints

Currently, the starter is providing webmvc by default. That mean that a project with webflux run tomcat in addition of netty server.

The starter now should not include webmvc neither webflux, and delegates to the developer to choose the stack he prefers

got exception during startup process

I use a simple spring boot app which expose rest endpoint. If I use the following configuration in application.yml
flightrecorder:
enabled: true # is this starter active?

I've got the following exception. Please see log below.

And everything is fine if I use the following configuration

flightrecorder:
enabled: true # is this starter active?
recordingCleanupInterval: 5000 # try to cleanup old recordings every 5 seconds
triggerCheckInterval: 10000 # evaluate trigger expressions every 10 seconds
trigger:
- expression: meter('jvm.memory.used').tag('area','nonheap').tag('id','Metaspace').measurement('value') > 100
startRecordingCommand:
duration: 60
timeUnit: SECONDS

C:\Users\evgen\jfr_test\jfrdemo1>docker run --name jfrtestimage_c -p 8080:8080 --rm -i -t jfrtestimage1 bash

. ____ _ __ _ _
/\ / ' __ _ () __ __ _ \ \ \
( ( )__ | '_ | '| | ' / ` | \ \ \
\/ )| |)| | | | | || (| | ) ) ) )
' || .__|| ||| |_, | / / / /
=========||==============|/=////
:: Spring Boot :: (v2.4.0)

2020-12-09 09:28:45.547 INFO 1 --- [ main] com.example.jfrdemo.JfrdemoApplication : Starting JfrdemoApplication v0.0.1-SNAPSHOT using Java 11.0.7 on d7e2cec6ca38 with PID 1 (/app.jar started by root in /)
2020-12-09 09:28:45.554 INFO 1 --- [ main] com.example.jfrdemo.JfrdemoApplication : No active profile set, falling back to default profiles: default
2020-12-09 09:28:47.351 INFO 1 --- [ main] o.s.b.w.embedded.tomcat.TomcatWebServer : Tomcat initialized with port(s): 8080 (http)
2020-12-09 09:28:47.365 INFO 1 --- [ main] o.apache.catalina.core.StandardService : Starting service [Tomcat]
2020-12-09 09:28:47.366 INFO 1 --- [ main] org.apache.catalina.core.StandardEngine : Starting Servlet engine: [Apache Tomcat/9.0.39]
2020-12-09 09:28:47.439 INFO 1 --- [ main] o.a.c.c.C.[Tomcat].[localhost].[/] : Initializing Spring embedded WebApplicationContext
2020-12-09 09:28:47.440 INFO 1 --- [ main] w.s.c.ServletWebServerApplicationContext : Root WebApplicationContext: initialization completed in 1746 ms
2020-12-09 09:28:47.990 INFO 1 --- [ main] o.s.s.concurrent.ThreadPoolTaskExecutor : Initializing ExecutorService 'applicationTaskExecutor'
2020-12-09 09:28:48.259 WARN 1 --- [ main] ConfigServletWebServerApplicationContext : Exception encountered during context initialization - cancelling refresh attempt: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'triggerChecker' defined in class path resource [de/mirkosertic/flightrecorderstarter/FlightRecorderConfiguration.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [de.mirkosertic.flightrecorderstarter.TriggerChecker]: Factory method 'triggerChecker' threw exception; nested exception is java.lang.NullPointerException
2020-12-09 09:28:48.260 INFO 1 --- [ main] o.s.s.concurrent.ThreadPoolTaskExecutor : Shutting down ExecutorService 'applicationTaskExecutor'
2020-12-09 09:28:48.264 INFO 1 --- [ main] o.apache.catalina.core.StandardService : Stopping service [Tomcat]
2020-12-09 09:28:48.292 INFO 1 --- [ main] ConditionEvaluationReportLoggingListener :

Error starting ApplicationContext. To display the conditions report re-run your application with 'debug' enabled.
2020-12-09 09:28:48.313 ERROR 1 --- [ main] o.s.boot.SpringApplication : Application run failed

org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'triggerChecker' defined in class path resource [de/mirkosertic/flightrecorderstarter/FlightRecorderConfiguration.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [de.mirkosertic.flightrecorderstarter.TriggerChecker]: Factory method 'triggerChecker' threw exception; nested exception is java.lang.NullPointerException
at org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:656) ~[spring-beans-5.3.1.jar!/:5.3.1]
at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:636) ~[spring-beans-5.3.1.jar!/:5.3.1]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1336) ~[spring-beans-5.3.1.jar!/:5.3.1]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1179) ~[spring-beans-5.3.1.jar!/:5.3.1]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:571) ~[spring-beans-5.3.1.jar!/:5.3.1]
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:531) ~[spring-beans-5.3.1.jar!/:5.3.1]
at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:335) ~[spring-beans-5.3.1.jar!/:5.3.1]
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234) ~[spring-beans-5.3.1.jar!/:5.3.1]
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:333) ~[spring-beans-5.3.1.jar!/:5.3.1]
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:208) ~[spring-beans-5.3.1.jar!/:5.3.1]
at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:944) ~[spring-beans-5.3.1.jar!/:5.3.1]
at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:925) ~[spring-context-5.3.1.jar!/:5.3.1]
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:588) ~[spring-context-5.3.1.jar!/:5.3.1]
at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:144) ~[spring-boot-2.4.0.jar!/:2.4.0]
at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:767) ~[spring-boot-2.4.0.jar!/:2.4.0]
at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:759) ~[spring-boot-2.4.0.jar!/:2.4.0]
at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:426) ~[spring-boot-2.4.0.jar!/:2.4.0]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:326) ~[spring-boot-2.4.0.jar!/:2.4.0]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1309) ~[spring-boot-2.4.0.jar!/:2.4.0]
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1298) ~[spring-boot-2.4.0.jar!/:2.4.0]
at com.example.jfrdemo.JfrdemoApplication.main(JfrdemoApplication.java:10) ~[classes!/:0.0.1-SNAPSHOT]
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:na]
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:na]
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:na]
at java.base/java.lang.reflect.Method.invoke(Method.java:566) ~[na:na]
at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:49) ~[app.jar:0.0.1-SNAPSHOT]
at org.springframework.boot.loader.Launcher.launch(Launcher.java:107) ~[app.jar:0.0.1-SNAPSHOT]
at org.springframework.boot.loader.Launcher.launch(Launcher.java:58) ~[app.jar:0.0.1-SNAPSHOT]
at org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:88) ~[app.jar:0.0.1-SNAPSHOT]
Caused by: org.springframework.beans.BeanInstantiationException: Failed to instantiate [de.mirkosertic.flightrecorderstarter.TriggerChecker]: Factory method 'triggerChecker' threw exception; nested exception is java.lang.NullPointerException
at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:185) ~[spring-beans-5.3.1.jar!/:5.3.1]
at org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:651) ~[spring-beans-5.3.1.jar!/:5.3.1]
... 28 common frames omitted
Caused by: java.lang.NullPointerException: null
at de.mirkosertic.flightrecorderstarter.TriggerChecker.(TriggerChecker.java:70) ~[flight-recorder-starter-2.0.0.jar!/:2.0.0]
at de.mirkosertic.flightrecorderstarter.FlightRecorderConfiguration.triggerChecker(FlightRecorderConfiguration.java:49) ~[flight-recorder-starter-2.0.0.jar!/:2.0.0]
at de.mirkosertic.flightrecorderstarter.FlightRecorderConfiguration$$EnhancerBySpringCGLIB$$3320d9b9.CGLIB$triggerChecker$3() ~[flight-recorder-starter-2.0.0.jar!/:2.0.0]
at de.mirkosertic.flightrecorderstarter.FlightRecorderConfiguration$$EnhancerBySpringCGLIB$$3320d9b9$$FastClassBySpringCGLIB$$d14052db.invoke() ~[flight-recorder-starter-2.0.0.jar!/:2.0.0]
at org.springframework.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:244) ~[spring-core-5.3.1.jar!/:5.3.1]
at org.springframework.context.annotation.ConfigurationClassEnhancer$BeanMethodInterceptor.intercept(ConfigurationClassEnhancer.java:331) ~[spring-context-5.3.1.jar!/:5.3.1]
at de.mirkosertic.flightrecorderstarter.FlightRecorderConfiguration$$EnhancerBySpringCGLIB$$3320d9b9.triggerChecker() ~[flight-recorder-starter-2.0.0.jar!/:2.0.0]
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:na]
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:na]
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:na]
at java.base/java.lang.reflect.Method.invoke(Method.java:566) ~[na:na]
at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:154) ~[spring-beans-5.3.1.jar!/:5.3.1]
... 29 common frames omitted

Issue: physical deletion of recordings

Currently, the stopping/deletion endpoint just only delete the files in memory (in order to avoid show in the list) but the file is still remaining in the disk.

IMHO, it could be better if we delete from disk too.

Emphatize in documentation that the annotation @EnableScheduling should be present in the main application.

Update the doc to empathize that the scheduler processes (deletion and triggers) only works if you enable the scheduling in our Spring Boot app (@EnableScheduling).

java.lang.IllegalStateException: Can't stop an already stopped recording.

Steps to reproduce

• run command curl -i -X PUT -H "Content-Type: application/json" -d '{"duration": "60","timeUnit":"SECONDS"}' http://localhost:8080/actuator/flightrecorder
• after 65 seconds run command curl --output recording.jfr http://localhost:8080/actuator/flightrecorder/1

Output

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   156    0   156    0     0   1974      0 --:--:-- --:--:-- --:--:--  1974

{
	"timestamp": "2020-11-04T13:16:28.303+00:00",
	"status": 500,
	"error": "Internal Server Error",
	"message": "",
	"path": "/actuator/flightrecorder/1"
}

Log

java.lang.IllegalStateException: Can't stop an already stopped recording.
	at jdk.jfr/jdk.jfr.internal.PlatformRecorder.stop(PlatformRecorder.java:262)
	at jdk.jfr/jdk.jfr.internal.PlatformRecording.stop(PlatformRecording.java:157)
	at jdk.jfr/jdk.jfr.Recording.stop(Recording.java:210)
	at de.mirkosertic.flightrecorderstarter.FlightRecorder.stopRecording(FlightRecorder.java:68)
	at de.mirkosertic.flightrecorderstarter.FlightRecorderEndpoint.downloadRecording(FlightRecorderEndpoint.java:271)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)

Cause

• before stopping recording recording.stop()
• you need to check recording state
• if recording is already done then IllegalStateException is thrown according to the documentation
• and it is not possible to download JFR file

    public File stopRecording(final long recordingId) {
        final Recording recording = recordings.get(recordingId);
        if (recording != null) {
              //
              // IllegalStateException if already stopped - check state before calling this method
              //
            recording.stop();
            return recording.getDestination().toFile();
        } else {
            LOGGER.log(Level.WARNING, "No recording with id {0} found", recordingId);
            return null;
        }
    }

Workaround

• call curl --output recording.jfr http://localhost:8080/actuator/flightrecorder/1 during recording
• before elapsed time defined by period {"duration": "60","timeUnit":"SECONDS"}

---

Thank you for creating this piece of code. It is exactly what I need.

Separate in two different controller the static resources and actuator operations

Currently, the "FlightRecorderEndpoint.java" file contains both the actuator operations and methods to get static resources.

It could be a great idea if we can separate them. I can help you if you are interested.

Basically, the solution is to create a RestController just in the management context.

Ability to specify the JFR base configuration to be used

Currently, the system just only works with the base configuration called "profile". Sometimes, the teams needs to define a new specific configuration in their JVM and use it to their recordings. It could be a great idea if we can parametrize via property entry the name of custom configuration to be used. The default value should be "profile" as now.

The actuator endpoint need @Selector to be parsed by swagger-ui

It's required that the endpoint methods params are annotated with @selector to be parsed by swagger-ui and have the possibility to execute them through swagger-ui web interface.

Ability to specify the folder for the temporal JFR file

Currently, the system always creates the temporal files into /tmp folder.

It could be a great idea if we can determine via property the base path to create the files in.

Generate continuous recording without time fixed

Currently, the startRecording handler method of the FlightRecorderEndpoint rest controller is validating if the input request contains the duration and timeUnit fields in the body.

I understand this is mandatory when the request represents the start of a time fixed recording. However, in case we want to start a continuous recording, these fields may not be needed. In fact, when the jfr file is imported in JMC, the Duration field value is 0 s in Event Browser view if the original request contains maxSize or maxAgeDuration fields.

I think it could be nice if the startRecording handler method will validate the request but in another way. Checking if the request contains the duration and timeUnit fields in case maxSize or maxAgeDuration (and maxAgeUnit) do not appear.

On this way, three possible scenarios will be available:

1. Time fixed recording -> Input request contains the duration but not the maxAgeDuration or maxSize fields
2. Continuous + Time fixed recording -> Input request contains the duration and the maxAgeDuration and/or maxSize fields
3. [new] Continuous recording -> Input request does not contain the duration but the maxAgeDuration and/or maxSize fields must appear. These recordings must be stopped manually using the API.

What do you think? I can submit a Pull Request so you can have a look to my implementation. The idea of this change is have a similar behavior of Java Mission Control.

Property to disable the triggers

It could be interesting to have the ability to disable, at least, the triggers engine so we can enabled the starter but disabled the trigger cause we realized that the triggers don't work propertly for all cases.

Suggestion about stop and delete endpoint actuator

Separate the endpoints actuator in two different operations: one to stop the flight recorder and another different to delete some recording

Problem with triggers expressions that matches with more than one Meter

I did a test with this query "meter('http.server.requests').tag('method','GET').tag('status','200').measurement('count') > 2" . This expression matches with more than one meter if you do requests for different uris. The expression parser in your starter just only takes into account the first meter in the collection so it's not right. It should group all the meters that match the query, like the actuator metrics endpoint does (http://localhost:8080/actuator/metrics/http.server.requests?tag=method:GET&tag=status:200). Maybe the problem is the RequiredSearch java class from micrometer library.

The times returned by the endpoints are not normalized to GMT

When I invoke the endpoint to list the recordings, the time are not normalized to GMT or similar, but it's an Instance java object that doesn't have any mark regarding the time zone used. We should be aware that internal JFR api works with Instance and the cleanUp scheduled job uses these fields to work.