Memory Leak related to io.netty.util.concurrent.FastThreadLocal in Vertx code

[bolbat]

Memory Leak related to io.netty.util.concurrent.FastThreadLocal in Vertx code

[bolbat]

And example how OldGen heap looks like with ZGC on Java 21 with this problem

[franz1981]

@bolbat And I was very wrong, it seems there's no way to prevent indexes to keep on growing there, so, the fix made by @vietj should be the one to solve AND improve performance in one go (as the original Pr I sent was meant to do, sorry for that :"( )

[bolbat]

Confirming that problem was fixed in 4.5.2-SNAPSHOT
Heap looks stable, InternalThreadLocalMap map index and size is near 50 elements, WebClient works fine.

Heap OldGen graph for last 2 days from one node:

[bolbat]

If someone would need code to debug this situation I will share my as example
Code is very ad hock, just for fast debugging

Add this dependencies to ByteBuddy library

		<dependency>
			<groupId>net.bytebuddy</groupId>
			<artifactId>byte-buddy</artifactId>
		</dependency>
		<dependency>
			<groupId>net.bytebuddy</groupId>
			<artifactId>byte-buddy-agent</artifactId>
		</dependency>

Instrument related code with something like this
Call instrumentForDebug on some early application startup stage

	public static void instrumentForDebug() {
		ByteBuddyAgent.install();
		new ByteBuddy()
				.redefine(FastThreadLocal.class)
				.visit(Advice.to(FastThreadLocalInstantiationInterceptor.class).on(ElementMatchers.isDefaultConstructor()))
				.make()
				.load(InternalThreadLocalMap.class.getClassLoader(), ClassReloadingStrategy.fromInstalledAgent())
				.getLoaded();
	}

	public static final class FastThreadLocalInstantiationInterceptor {

		public static final Map<String, AtomicInteger> STATS = Maps.mutable.empty();

		@OnMethodEnter
		public static void intercept() {
			final String stackTraceString = ExceptionUtils.getStackTrace(new Throwable());

			final AtomicInteger existing = STATS.get(stackTraceString);
			if (existing == null) {
				STATS.put(stackTraceString, new AtomicInteger(1));
			} else {
				existing.incrementAndGet();
			}
		}

	}

Prepare some debug information and log it or get by some debug API like I did

	private void debugFastThreadLocalInstantiationStats(final StringBuilder sb, final int minInstantiations) {
		line(sb.append("FastThreadLocal Instantiation Stats"));

		FastThreadLocalInstantiationInterceptor.STATS.forEach((k, v) -> {
			final int instantiations = v.get();
			if (instantiations >= minInstantiations) {
				line(sb.append("Instantiation[").append(instantiations).append("] ")
						.append("StackTrace: ").append(k));
				line(sb);
			}
		});

	}

	private void line(final StringBuilder sb) {
		sb.append(System.lineSeparator());
	}

[vietj]

thanks @bolbat

[franz1981]

FYI, I think you could the the leak detection of a sync profiler (adding --live to the cmd line options) to detect this or just allocation profiling.

Thanks for the analysis, although by reading this I don't see the problem in the fast thread local, but instead in the high number of combiner executors created: any idea why @vietj ?

Said that, fast thread locals could be "smarter" in resizing the array, if it knows that a combiner is no longer reachable, and could do a better job.
This means making it autocloseable and handle thread local dispose somehow and we should have a clean life cycle for it already, just need to use it to enforce thread local disposal