flyway connection error for r2dbc in testcontainers in Spring boot versions >= 3.x.x

[sakthi-logn]

flyway connection error for r2dbc in testcontainers in Spring boot versions >= 3.x.x

[wilkinsona]

Can you please share a minimal example that reproduces the behaviour that you have described? Unless the failure is specific to Kotlin, that example should be written in Java in order to minimise the number of moving parts.

[sakthi-logn]

Can you please share a minimal example that reproduces the behaviour that you have described? Unless the failure is specific to Kotlin, that example should be written in Java in order to minimise the number of moving parts.

sure. I will share a minimal example written in Java.

Just reiterating to emphasize that the testcontainers r2dbc test code written in kotlin works well for Spring Boot 2.7.18 but throws flyway connection error for any version of Spring Boot 3.x.x

[sakthi-logn]

This public git repo contains the minimal java code showing the problem with flyway, r2dbc, testcontainers and Spring Boot versions >= 3.0.0.

https://github.com/sakthi-logn/spring-boot-flyway-r2dbc-testcontainer.git

Please do the following steps to reproduce the problem:

1. install docker on your machine because testcontainers needs to connect to the docker daemon.
2. clone the repo
3. change directory to the root of the repo and run ./gradlew clean build test --info to see that there are no errors with Spring boot 2.7.18
4. In the build.gradle.kts file, uncomment the line number 3 which containsid("org.springframework.boot") version "3.2.4"
   and comment line number 4 which contains id("org.springframework.boot") version "2.7.18"
5. rerun step 3 to see the same flyway connection error shown in my earlier comments. The only change done was the upgrade to spring boot 3.2.4

I also tried overriding flyway version to the latest using maven BOM overrides in the kotlin gradle.build.kts file but the same error occurred again. So i am not sure if the problem is with flyway.

Please let me know if you need more information.

[wilkinsona]

Thanks for the sample. Unfortunately, it doesn't reproduce the problem with UserRepositoryImplTest passing using Spring Boot 3.2.4:

20:18:39.954 [main] INFO org.testcontainers.images.PullPolicy -- Image pull policy will be performed by: DefaultPullPolicy()
20:18:39.958 [main] INFO org.testcontainers.utility.ImageNameSubstitutor -- Image name substitution will be performed by: DefaultImageNameSubstitutor (composite of 'ConfigurationFileImageNameSubstitutor' and 'PrefixingImageNameSubstitutor')
20:18:40.145 [main] INFO org.testcontainers.dockerclient.DockerClientProviderStrategy -- Loaded org.testcontainers.dockerclient.UnixSocketClientProviderStrategy from ~/.testcontainers.properties, will try it first
20:18:40.157 [main] WARN org.testcontainers.dockerclient.DockerClientProviderStrategy -- DOCKER_HOST tcp://127.0.0.1:53248 is not listening
20:18:40.362 [main] INFO org.testcontainers.dockerclient.DockerClientProviderStrategy -- Found Docker environment with local Unix socket (unix:///var/run/docker.sock)
20:18:40.364 [main] INFO org.testcontainers.DockerClientFactory -- Docker host IP address is localhost
20:18:40.390 [main] INFO org.testcontainers.DockerClientFactory -- Connected to docker: 
  Server Version: 20.10.24
  API Version: 1.41
  Operating System: Docker Desktop
  Total Memory: 16010 MB
20:18:40.487 [main] INFO tc.testcontainers/ryuk:0.6.0 -- Creating container for image: testcontainers/ryuk:0.6.0
20:18:41.226 [main] INFO tc.testcontainers/ryuk:0.6.0 -- Container testcontainers/ryuk:0.6.0 is starting: 397df0afad39464ec8a9763eb67c5380da4ab8a5fd197cb0760ffa8433402595
20:18:41.682 [main] INFO tc.testcontainers/ryuk:0.6.0 -- Container testcontainers/ryuk:0.6.0 started in PT1.195098S
20:18:41.687 [main] INFO org.testcontainers.utility.RyukResourceReaper -- Ryuk started - will monitor and terminate Testcontainers containers on JVM exit
20:18:41.687 [main] INFO org.testcontainers.DockerClientFactory -- Checking the system...
20:18:41.687 [main] INFO org.testcontainers.DockerClientFactory -- ✔︎ Docker server version should be at least 1.6.0
20:18:41.688 [main] INFO tc.postgres:12 -- Creating container for image: postgres:12
20:18:41.689 [main] WARN tc.postgres:12 -- Reuse was requested but the environment does not support the reuse of containers
To enable reuse of containers, you must set 'testcontainers.reuse.enable=true' in a file located at /Users/awilkinson/.testcontainers.properties
20:18:41.739 [main] INFO tc.postgres:12 -- Container postgres:12 is starting: f03afe05b62787c5201c871073a37923fd285d3b1a7adc137d4d435590b314c1
20:18:43.477 [main] INFO tc.postgres:12 -- Container postgres:12 started in PT1.789419S
20:18:43.478 [main] INFO tc.postgres:12 -- Container is started (JDBC URL: jdbc:postgresql://localhost:59760/test?loggerLevel=OFF)

  .   ____          _            __ _ _
 /\\ / ___'_ __ _ _(_)_ __  __ _ \ \ \ \
( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
 \\/  ___)| |_)| | | | | || (_| |  ) ) ) )
  '  |____| .__|_| |_|_| |_\__, | / / / /
 =========|_|==============|___/=/_/_/_/
 :: Spring Boot ::                (v3.2.4)

20:18:43.842 [main] INFO  c.t.t.p.p.u.u.a.r.UserRepositoryImplTest - Starting UserRepositoryImplTest using Java 17.0.1 with PID 14001 (started by awilkinson in /Users/awilkinson/dev/temp/spring-boot-flyway-r2dbc-testcontainer)
20:18:43.843 [main] INFO  c.t.t.p.p.u.u.a.r.UserRepositoryImplTest - The following 1 profile is active: "local"
20:18:44.277 [main] INFO  o.s.d.r.c.RepositoryConfigurationDelegate - Bootstrapping Spring Data R2DBC repositories in DEFAULT mode.
20:18:44.300 [main] INFO  o.s.d.r.c.RepositoryConfigurationDelegate - Finished Spring Data repository scanning in 18 ms. Found 0 R2DBC repository interfaces.
20:18:45.266 [main] INFO  o.f.c.i.license.VersionPrinter - Flyway Community Edition 9.22.3 by Redgate
20:18:45.266 [main] INFO  o.f.c.i.license.VersionPrinter - See release notes here: https://rd.gt/416ObMi
20:18:45.266 [main] INFO  o.f.c.i.license.VersionPrinter - 
20:18:45.298 [main] INFO  org.flywaydb.core.FlywayExecutor - Database: jdbc:postgresql://localhost:59760/test (PostgreSQL 12.18)
20:18:45.333 [main] INFO  o.f.c.internal.database.base.Schema - Creating schema "service_one" ...
20:18:45.340 [main] INFO  o.f.c.i.s.JdbcTableSchemaHistory - Creating Schema History table "service_one"."flyway_schema_history" ...
20:18:45.389 [main] INFO  o.f.core.internal.command.DbMigrate - Current version of schema "service_one": null
20:18:45.397 [main] INFO  o.f.core.internal.command.DbMigrate - Migrating schema "service_one" to version "1.0.0 - create user"
20:18:45.416 [main] INFO  o.f.core.internal.command.DbMigrate - Successfully applied 1 migration to schema "service_one", now at version v1.0.0 (execution time 00:00.008s)
20:18:45.569 [main] INFO  o.s.b.w.e.netty.NettyWebServer - Netty started on port 59766
20:18:45.576 [main] INFO  c.t.t.p.p.u.u.a.r.UserRepositoryImplTest - Started UserRepositoryImplTest in 2.07 seconds (process running for 6.697)
Java HotSpot(TM) 64-Bit Server VM warning: Sharing is only supported for boot loader classes because bootstrap classpath has been appended
20:18:46.496 [main] INFO  c.t.t.p.p.u.u.a.r.UserRepositoryImplTest - before test, the number of rows deleted in 'user' table: 0
20:18:46.573 [main] INFO  c.t.t.p.p.u.u.a.r.UserRepositoryImplTest - before test, the number of rows deleted in 'user' table: 0
20:18:46.580 [reactor-tcp-nio-6] INFO  c.t.t.p.p.u.u.a.r.UserRepositoryImpl - user with id 'user-1' and version '0' does not exist for update

Looking more closely at the failure above, I can see that the Postgres database is available on localhost: 33728:

23:05:14.090 [Test worker] INFO tc.postgres:12 -- Container is started (JDBC URL: jdbc:postgresql://localhost:33728/test?loggerLevel=OFF)

I can also see that Flyway is trying to connect to the same:

 Connection to localhost:33728 refused

From Spring Boot's perspective, everything is behaving as it should here with Flyway being configured to use the JDBC URL of the container.

I suspect that there's a problem specific to your environment or Docker networking. The Ryuk problem may also be evidence of this:

23:05:13.090 [testcontainers-ryuk] WARN org.testcontainers.utility.RyukResourceReaper -- Can not connect to Ryuk at localhost:33727

I'm going to close this one, for now at least, as I cannot see any evidence of a problem with Spring Boot. I suspect it's an environmental problem that perhaps only occurs with Spring Boot 3.2's version of Testcontainers.

[sakthi-logn]

Yes. I found out that the problem does not occur in Intel Macbooks but occurred only in Apple M3 laptops.
I used colima to install Docker in my apple M3 laptop.
Flyway was not able to connect to the docker container runtime inside aarch64 architecture Linux VM started by colima.

So i stopped that VM instance and started a new x86_64 arch Linux VM to run the docker daemon and it worked !!
Flyway was able to connect to testcontainer and the tests ran without any failures.

So, the problem seems to have occurred because of the way docker was setup on the apple M3 arm64 arch laptop.