[BUG] Netty HTTP Server exception for There is a blocking call in the CosmosClientAsync

[alejojarahi]

[BUG] Netty HTTP Server exception for There is a blocking call in the CosmosClientAsync

[alzimmermsft]

thanks @alejojarahi for creating this issue, looks like a genuine problem in case of async client creation. @alzimmermsft I am curious, if this is the same case in azure-core as well and have we solved it? I would like to keep the solutions consistent if possible. Referencing the PR where we added this, will be good to re-iterate our approach to warming up http clients in case of async clients.

azure-core-http-netty doesn't perform a warmup of the Netty HttpClient

[github-actions]

Thanks for the feedback! We are routing this to the appropriate team for follow-up. cc @kushagraThapar @pjohari-ms @TheovanKraay.

[kushagraThapar]

thanks @alejojarahi for creating this issue, looks like a genuine problem in case of async client creation. @alzimmermsft I am curious, if this is the same case in azure-core as well and have we solved it? I would like to keep the solutions consistent if possible. Referencing the PR where we added this, will be good to re-iterate our approach to warming up http clients in case of async clients.

[kushagraThapar]

IMO, we should remove the blocking warmup of the http client.
cc: @xinlian12 / @FabianMeiswinkel

[kushagraThapar]

thanks @alzimmermsft for the confirmation.
@alejojarahi - actually the issue here is not the warmup, rather the way you are creating the cosmos client. The code snippet you pasted in the issue description is creating multiple cosmos clients (per database) which is not recommended. We strongly recommend to only create a singleton client.

azure-cosmos client or any other client if needs to warmup, they will need to call certain APIs that will allow the client to warmup. Ideally, you would want to have a warmed-up client and then only enable the traffic on it (through nio threads or whatever). You would want to warmup the client on the main thread before opening it up for traffic.

If your ask is to make this configurable, we can think about doing so.

[alejojarahi]

Hi @kushagraThapar. Thanks for the reply.

This error also occurs in the scenario that CosmosAsyncClient is instantiated only onces. Example in Springboot 3

import com.azure.cosmos.CosmosAsyncClient;
import com.azure.cosmos.CosmosClientBuilder;
import reactor.core.publisher.Flux;

public class UserRepository {
    private static final String DATABASE_NAME = "test";

    CosmosAsyncClient cosmosAsyncClient = null;

    public CosmosAsyncClient getInstance() {
        if(cosmosAsyncClient == null) {
            cosmosAsyncClient = new CosmosClientBuilder()
                    .endpoint("ENDPOINT")
                    .key("KEY")
                    .buildAsyncClient();
        }
        return cosmosAsyncClient;
    }

    public Flux<User> getAll() {
        return getInstance().getDatabase(DATABASE_NAME)
                            .getContainer("users")
                            .queryItems("SELECT * FROM users", User.class);
    }
}

17:44:54.516 Data 06-may.-2024 | ERROR | Thread [reactor-http-nio-3] | c.a.c.i.RxDocumentClientImpl | unexpected failure in initializing client.
java.lang.RuntimeException: java.lang.IllegalStateException: block()/blockFirst()/blockLast() are blocking, which is not supported in thread reactor-http-nio-3
	at com.azure.cosmos.implementation.http.ReactorNettyClient.attemptToWarmupHttpClient(ReactorNettyClient.java:116)
	at com.azure.cosmos.implementation.http.ReactorNettyClient.createWithConnectionProvider(ReactorNettyClient.java:96)
	at com.azure.cosmos.implementation.http.HttpClient.createFixed(HttpClient.java:62)
	at com.azure.cosmos.implementation.RxDocumentClientImpl.httpClient(RxDocumentClientImpl.java:744)
	at com.azure.cosmos.implementation.RxDocumentClientImpl.<init>(RxDocumentClientImpl.java:508)
	at com.azure.cosmos.implementation.RxDocumentClientImpl.<init>(RxDocumentClientImpl.java:335)
	at com.azure.cosmos.implementation.RxDocumentClientImpl.<init>(RxDocumentClientImpl.java:295)
	at com.azure.cosmos.implementation.AsyncDocumentClient$Builder.build(AsyncDocumentClient.java:275)
	at com.azure.cosmos.CosmosAsyncClient.<init>(CosmosAsyncClient.java:170)
	at com.azure.cosmos.CosmosClientBuilder.buildAsyncClient(CosmosClientBuilder.java:1084)
	at com.azure.cosmos.CosmosClientBuilder.buildAsyncClient(CosmosClientBuilder.java:1071)
	at co.com.springexample.entrypoint.UserRepository.getInstance(UserRepository.java:19)

If the reactive client is subject to only being able to start by on the main thread, this can limit or affect the architecture of a project. For example this use case https://stackoverflow.com/questions/73412719/create-recreate-azure-cosmosdb-async-client-from-java-reactor-context.

If the team does not consider it a bug, it should be a configurable functionality

[kushagraThapar]

@alejojarahi the stack overflow code snippet you mentioned is also not a valid use case of the cosmos client, because the customer is trying to create a new cosmos client every time they need to rotate their keys.

SecretAsyncClient secretAsyncClient = new SecretClientBuilder().buildAsyncClient()
...
Mono<CosmosAsyncClient> client = secretAsyncClient.getSecret(KEY_NAME).map(
  s -> s.getValue()
).map(
  key -> new CosmosClientBuilder()
                        .endpoint(HOST)
                        .key(key)
                        .buildAsyncClient()
);
return client.flatMap(
   ...
);

Instead they should use a key rotation pattern like some form of KeyCredential object which has an update() API to update the key on the fly.

Creating clients randomly and during the execution of the application is never the right approach, as client creation is considered a heavy operation, specially in case of Cosmos SDK where it needs to get Database account, metadata, fill the caches, create TCP / HTTP connections, etc. when the client boots up.
Any application which is doing this on nio thread is basically blocking their I/O resources for unnecessary client creation.