Comments (13)
Please focus analysis on Problem Suspect 3 of undertow. We thing it this item.
from spring-boot.
Thanks for the report. Unfortunately, without some more information we won't be able to diagnose the problem. It looks like your application contains a memory leak. I find that the best way to analyse a memory leak is by taking a series of heap dumps and then compare them looking for an increasing number of a particular type of object being retained in the heap.
Due to the risk of a heap dump containing secrets, such analysis is often best done by you. Alternatively, if you're confident that the heap does not contain any information that should not be public, you could share them with us here and we can take a look.
from spring-boot.
Thanks for the answer.
We are trying to find more detailed information about this issue.
from spring-boot.
Regarding what I said about AWS, we noticed that only in apis we use spring-cloud-aws (latest version 3.1.0) this problem occurred with the springboot update.
from spring-boot.
Hello!
I came back here with more details.
Under analysis with the new relic observability tool:
The class with the possible problem was identified:
with the help of Heapdump tool we compared both versions and found this scenario:
using eclipse memory analyser:
from spring-boot.
Thanks for the additional details. Spring Boot 3.2.3 upgraded to Undertow 2.3.12.Final. I suspect that it contains a regression that is causing the memory leak. Please try using Spring Boot 3.2.3 with Undertow downgraded to 2.3.10.Final which is the version used by Spring Boot 3.2.2. You may also want to try Undertow 2.3.11.Final to narrow things down further.
from spring-boot.
Hi @clucca-gb, are you passing in an executor to the Undertow servlet like this?
@Bean
public UndertowDeploymentInfoCustomizer undertowDeploymentInfoCustomizer() {
return deploymentInfo -> deploymentInfo.setExecutor(new CustomExecutor());
}
Using Undertow in the default configuration did not cause problems for me until I switched to using a different executor.
from spring-boot.
In our tests it appears to be a problem related to aws.
What do you mean? Do you have data showing more memory consumed by Spring Boot classes? Did you attach a memory profiler to the application?
Please provide more information - right now this report isn't actionable for us.
from spring-boot.
@Tythor Hello!
Yeah, it is!
I'll try to understand if it's possible to remove it.
from spring-boot.
By the way, we changed Undertown versions to the suggested ones and Memory Leak no longer occurred anymore!
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-undertow</artifactId>
</dependency>
<dependency>
<groupId>io.undertow</groupId>
<artifactId>undertow-core</artifactId>
<version>2.3.10.Final</version>
</dependency>
<dependency>
<groupId>io.undertow</groupId>
<artifactId>undertow-servlet</artifactId>
<version>2.3.10.Final</version>
</dependency>
<dependency>
<groupId>io.undertow</groupId>
<artifactId>undertow-websockets-jsr</artifactId>
<version>2.3.10.Final</version>
</dependency>
<dependency>
from spring-boot.
Hi,
I would like to add on my finding from my tech stack.
- leaking happened to server threads, as many as 200 (display one hundred, and boots default allow 200 as max). these server threads are managing the POST requests “/instances” from API services.
- these leaking threads lock memory and shown as memory leaks but not root cause indeed, if you see this leaking memory such as Byte objects leak(biggest), HttpClient leak, NioEndpoint leak.
- Use Dynatrace profiling tool to check memory allocation, first check ThreadPoolExecutor pool, look for CloseableHTTPClient.execute, I could not find them visually
- continue to check FastThreadLocalRunnable which is referenced by each worker in ThreadPoolExcutor above, I could not find CloseableHTTPClient.execute either.
- Given FastThreadLocalRunnable worker to manage all concurrent threads and the parallel threads are self-managed, who are going to manage the server threads in ThreadPoolExecutor? Given the finding that no CloseableHTTPClient is implemented, any leftover in thread local could make the server threads leak. Or If CloseableHTTPClient is implemented, how to manage reference from server worker to FastThreadLocalRunnable pool? Seems the thread leaks involved a dilemma in design and not resolved and leave it as leak hole.
Hope these feedbacks help designers and help to resolve the leak quicker.
I intend to disable POST on boots server threads to stop thread leaking as temporary solution. I would like to hear your opinions.
Regards
from spring-boot.
That's good news. Thanks, @clucca-gb. Please open an Undertow issue so that they can investigate why configuring a custom executor as you have done now causes a memory leak. I'll close this issue for now as I don't think there's anything we can do about this in Spring Boot.
from spring-boot.
@zhlian94 your problem appears to be unrelated. As far as I tell you're using Tomcat, not Undertow, and are perhaps using a Reactor Netty based WebClient too. If you'd like some help investigating why your application appears to be leaking memory, please ask on Stack Overflow to begin with.
from spring-boot.
Related Issues (20)
- spring-boot-dependencies cannot be used with repositories that ban com.oracle.database.jdbc:ojdbc-bom
- spring-boot-dependencies cannot be used with repositories that ban com.oracle.database.jdbc:ojdbc-bom
- Upgrade to Asciidoctor Gradle Plugin 4.0.2
- Avoid calling getProject() during execution of StarterMetadata task
- Avoid calling getProject() during execution of StarterMetadata task
- Avoid calling getProject() during execution of StarterMetadata task
- Avoid calling getProject() during execution of AutoConfigurationMetadata task
- Avoid calling getProject() during execution of AutoConfigurationMetadata task
- Avoid calling getProject() during execution of AutoConfigurationMetadata task
- Avoid calling getProject() during execution of publishMavenPublicationToProjectRepository
- Avoid calling getProject() during execution of publishMavenPublicationToProjectRepository
- Avoid calling getProject() during execution of publishMavenPublicationToProjectRepository
- Avoid calling getProject() when checking configuration metadata
- Avoid calling getProject() when checking configuration metadata
- Avoid calling getProject() when checking configuration metadata
- IllegalArgumentException can be thrown when running an uber jar on a shared drive HOT 2
- Avoid calling getProject() during execution of extractVersionConstraints
- Avoid calling getProject() during execution of extractVersionConstraints
- Using prometheus 0.x with distribution percentiles lead to "Counts in ClassicHistogramBuckets cannot be negative."
- LocalContainerEntityManagerFactoryBean : Closing JPA EntityManagerFactory for persistence unit 'default'
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from spring-boot.