RST_STREAM 1 error with GRPC and ESP

[gombasp]

RST_STREAM 1 error with GRPC and ESP

[qiwzhang]

Yes, ESP indeed pass content-type to Grpc server.

[gombasp]

@qiwzhang just to be sure: we are talking about the content-length, not the content-type.

[qiwzhang]

Sorry, it was a typo. I mean "content-length" too. Here is the code of filtering the headers from client to grpc server. It did not remove content-length.

[gombasp]

@qiwzhang im wondering when can be expect a bug fix? Currently this makes us unable upgrade the grpc libs - however there are newer versions already (not urgent, but after a point it would make sense). Can you give us any info?

[qiwzhang]

Let me summarize the issue: grpc should not use "content-length" in its header, but ESPv1 sending it. Java grpc-server starts to enforce this in the version 1.44.0, the requests proxyed by ESPv1 starts to fail if Java gRPC server upgrades to new version.

Is that correct?

We are estimating the effort to fix this. Will update you.

[qiwzhang]

BTW, is it possible to upgrade your system to use ESPv2?

[gombasp]

@qiwzhang your summary is correct. My only small remark is that it fails already from grpc version 1.42.0.

About ESPv2: we will consider if this would be possible. Are you recommending it because the bug is already solved in v2?

[gombasp]

@qiwzhang i tested my original scenario with espv2. Unfortunately i only get a http 415 in response, regardless of the grpc lib i use. Maybe i need to add some extra options in case of esp v2? I started it locally similar to this:

sudo docker run
--detach
--name="esp_hello12"
--net="host"
--volume=/home/user/projects/:/esp
--publish=8087
gcr.io/endpoints-release/endpoints-runtime:2
--service=helloservice.endpoints.projectx.cloud.goog
--rollout_strategy=managed
--http_port=8087
--backend=grpc://localhost:8089
--service_account_key=/esp/hello.json
--non_gcp

Basically i added the non_gcp flag to the example what i found on the page: https://cloud.google.com/endpoints/docs/openapi/running-esp-localdev
Note that the documentation doesnt say its needed, it just states that v1 and v2 works the same in this regards, however the v2 does not work without the --non_gcp. Just a remark from my side, maybe add to the doc.

After starting it like above and sending the same json to v2 as i did with v1 i see the following log in backend:

ápr. 21, 2022 11:14:09 DE. io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2FrameLogger logSettings
FINE: [id: 0x25d37ba8, L:/[0:0:0:0:0:0:0:1%0]:8089 - R:/[0:0:0:0:0:0:0:1%0]:50958] OUTBOUND SETTINGS: ack=false settings={MAX_CONCURRENT_STREAMS=2147483647, INITIAL_WINDOW_SIZE=1048576, MAX_HEADER_LIST_SIZE=8192}
ápr. 21, 2022 11:14:09 DE. io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2FrameLogger logWindowsUpdate
FINE: [id: 0x25d37ba8, L:/[0:0:0:0:0:0:0:1%0]:8089 - R:/[0:0:0:0:0:0:0:1%0]:50958] OUTBOUND WINDOW_UPDATE: streamId=0 windowSizeIncrement=983041
ápr. 21, 2022 11:14:09 DE. io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2FrameLogger logSettings
FINE: [id: 0x25d37ba8, L:/[0:0:0:0:0:0:0:1%0]:8089 - R:/[0:0:0:0:0:0:0:1%0]:50958] INBOUND SETTINGS: ack=false settings={HEADER_TABLE_SIZE=4096, ENABLE_PUSH=0, MAX_CONCURRENT_STREAMS=2147483647, INITIAL_WINDOW_SIZE=268435456,=0}
ápr. 21, 2022 11:14:09 DE. io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2FrameLogger logSettingsAck
FINE: [id: 0x25d37ba8, L:/[0:0:0:0:0:0:0:1%0]:8089 - R:/[0:0:0:0:0:0:0:1%0]:50958] OUTBOUND SETTINGS: ack=true
ápr. 21, 2022 11:14:09 DE. io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2FrameLogger logWindowsUpdate
FINE: [id: 0x25d37ba8, L:/[0:0:0:0:0:0:0:1%0]:8089 - R:/[0:0:0:0:0:0:0:1%0]:50958] INBOUND WINDOW_UPDATE: streamId=0 windowSizeIncrement=268369921
ápr. 21, 2022 11:14:09 DE. io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2FrameLogger logHeaders
FINE: [id: 0x25d37ba8, L:/[0:0:0:0:0:0:0:1%0]:8089 - R:/[0:0:0:0:0:0:0:1%0]:50958] INBOUND HEADERS: streamId=1 headers=GrpcHttp2RequestHeaders[:path: /helloservice/hello, :authority: localhost:8087, :method: POST, :scheme: http, x-api-key: AIzaSyCvhWZ2ixEhuBV96VcOiXPUFYx7WTYCV1Y, content-type: application/json, user-agent: PostmanRuntime/7.29.0, accept: /, postman-token: 2bc9b280-e18a-485c-aeba-80944121cbca, accept-encoding: gzip, deflate, br, content-length: 45, x-forwarded-proto: http, x-request-id: 3b613dd4-3d38-4198-b6de-63f9b438905e, x-endpoint-api-consumer-type: PROJECT, x-endpoint-api-consumer-number: 247678761469] padding=0 endStream=false
ápr. 21, 2022 11:14:09 DE. io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2FrameLogger logHeaders
FINE: [id: 0x25d37ba8, L:/[0:0:0:0:0:0:0:1%0]:8089 - R:/[0:0:0:0:0:0:0:1%0]:50958] OUTBOUND HEADERS: streamId=1 headers=DefaultHttp2Headers[:status: 415, content-type: text/plain; charset=utf-8, grpc-status: 13, grpc-message: Content-Type 'application/json' is not supported] padding=0 endStream=false
ápr. 21, 2022 11:14:09 DE. io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2FrameLogger logData
FINE: [id: 0x25d37ba8, L:/[0:0:0:0:0:0:0:1%0]:8089 - R:/[0:0:0:0:0:0:0:1%0]:50958] INBOUND DATA: streamId=1 padding=0 endStream=true length=45 bytes=7b2275736572223a7b2266697273744e616d65223a20226161222c226c6173744e616d65223a20226262227d7d
ápr. 21, 2022 11:14:09 DE. io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2FrameLogger logPing
FINE: [id: 0x25d37ba8, L:/[0:0:0:0:0:0:0:1%0]:8089 - R:/[0:0:0:0:0:0:0:1%0]:50958] OUTBOUND PING: ack=false bytes=1234
ápr. 21, 2022 11:14:09 DE. io.grpc.netty.shaded.io.grpc.netty.NettyServerHandler onDataRead
WARNING: Exception in onDataRead()
java.lang.NullPointerException: Cannot invoke "io.grpc.netty.shaded.io.grpc.netty.NettyServerStream$TransportState.tag()" because "stream" is null
at io.grpc.netty.shaded.io.grpc.netty.NettyServerHandler.onDataRead(NettyServerHandler.java:506)
at io.grpc.netty.shaded.io.grpc.netty.NettyServerHandler.access$900(NettyServerHandler.java:106)
at io.grpc.netty.shaded.io.grpc.netty.NettyServerHandler$FrameListener.onDataRead(NettyServerHandler.java:840)
at io.grpc.netty.shaded.io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder$FrameReadListener.onDataRead(DefaultHttp2ConnectionDecoder.java:307)
at io.grpc.netty.shaded.io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder$PrefaceFrameListener.onDataRead(DefaultHttp2ConnectionDecoder.java:691)
at io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2InboundFrameLogger$1.onDataRead(Http2InboundFrameLogger.java:48)
at io.grpc.netty.shaded.io.netty.handler.codec.http2.DefaultHttp2FrameReader.readDataFrame(DefaultHttp2FrameReader.java:415)
at io.grpc.netty.shaded.io.netty.handler.codec.http2.DefaultHttp2FrameReader.processPayloadState(DefaultHttp2FrameReader.java:250)
at io.grpc.netty.shaded.io.netty.handler.codec.http2.DefaultHttp2FrameReader.readFrame(DefaultHttp2FrameReader.java:159)
at io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2InboundFrameLogger.readFrame(Http2InboundFrameLogger.java:41)
at io.grpc.netty.shaded.io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder.decodeFrame(DefaultHttp2ConnectionDecoder.java:173)
at io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2ConnectionHandler$FrameDecoder.decode(Http2ConnectionHandler.java:378)
at io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2ConnectionHandler$PrefaceDecoder.decode(Http2ConnectionHandler.java:242)
at io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2ConnectionHandler.decode(Http2ConnectionHandler.java:438)
at io.grpc.netty.shaded.io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:507)
at io.grpc.netty.shaded.io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:446)
at io.grpc.netty.shaded.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276)
at io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
at io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
at io.grpc.netty.shaded.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
at io.grpc.netty.shaded.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.grpc.netty.shaded.io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:795)
at io.grpc.netty.shaded.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:480)
at io.grpc.netty.shaded.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378)
at io.grpc.netty.shaded.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
at io.grpc.netty.shaded.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.grpc.netty.shaded.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Thread.java:833)

ápr. 21, 2022 11:14:09 DE. io.grpc.netty.shaded.io.grpc.netty.NettyServerHandler onStreamError
WARNING: Stream Error
io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2Exception$StreamException: Cannot invoke "io.grpc.netty.shaded.io.grpc.netty.NettyServerStream$TransportState.tag()" because "stream" is null
at io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2Exception.streamError(Http2Exception.java:172)
at io.grpc.netty.shaded.io.grpc.netty.NettyServerHandler.newStreamException(NettyServerHandler.java:817)
at io.grpc.netty.shaded.io.grpc.netty.NettyServerHandler.onDataRead(NettyServerHandler.java:515)
at io.grpc.netty.shaded.io.grpc.netty.NettyServerHandler.access$900(NettyServerHandler.java:106)
at io.grpc.netty.shaded.io.grpc.netty.NettyServerHandler$FrameListener.onDataRead(NettyServerHandler.java:840)
at io.grpc.netty.shaded.io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder$FrameReadListener.onDataRead(DefaultHttp2ConnectionDecoder.java:307)
at io.grpc.netty.shaded.io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder$PrefaceFrameListener.onDataRead(DefaultHttp2ConnectionDecoder.java:691)
at io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2InboundFrameLogger$1.onDataRead(Http2InboundFrameLogger.java:48)
at io.grpc.netty.shaded.io.netty.handler.codec.http2.DefaultHttp2FrameReader.readDataFrame(DefaultHttp2FrameReader.java:415)
at io.grpc.netty.shaded.io.netty.handler.codec.http2.DefaultHttp2FrameReader.processPayloadState(DefaultHttp2FrameReader.java:250)
at io.grpc.netty.shaded.io.netty.handler.codec.http2.DefaultHttp2FrameReader.readFrame(DefaultHttp2FrameReader.java:159)
at io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2InboundFrameLogger.readFrame(Http2InboundFrameLogger.java:41)
at io.grpc.netty.shaded.io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder.decodeFrame(DefaultHttp2ConnectionDecoder.java:173)
at io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2ConnectionHandler$FrameDecoder.decode(Http2ConnectionHandler.java:378)
at io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2ConnectionHandler$PrefaceDecoder.decode(Http2ConnectionHandler.java:242)
at io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2ConnectionHandler.decode(Http2ConnectionHandler.java:438)
at io.grpc.netty.shaded.io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:507)
at io.grpc.netty.shaded.io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:446)
at io.grpc.netty.shaded.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276)
at io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
at io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
at io.grpc.netty.shaded.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
at io.grpc.netty.shaded.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.grpc.netty.shaded.io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:795)
at io.grpc.netty.shaded.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:480)
at io.grpc.netty.shaded.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378)
at io.grpc.netty.shaded.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
at io.grpc.netty.shaded.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.grpc.netty.shaded.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: java.lang.NullPointerException: Cannot invoke "io.grpc.netty.shaded.io.grpc.netty.NettyServerStream$TransportState.tag()" because "stream" is null
at io.grpc.netty.shaded.io.grpc.netty.NettyServerHandler.onDataRead(NettyServerHandler.java:506)
... 30 more

ápr. 21, 2022 11:14:09 DE. io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2FrameLogger logRstStream
FINE: [id: 0x25d37ba8, L:/[0:0:0:0:0:0:0:1%0]:8089 - R:/[0:0:0:0:0:0:0:1%0]:50958] OUTBOUND RST_STREAM: streamId=1 errorCode=2
ápr. 21, 2022 11:14:09 DE. io.grpc.netty.shaded.io.grpc.netty.NettyServerHandler onStreamError
FINE: Stream Error
io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2Exception$StreamException: Stream closed before write could take place
at io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2Exception.streamError(Http2Exception.java:172)
at io.grpc.netty.shaded.io.netty.handler.codec.http2.DefaultHttp2RemoteFlowController$FlowState.cancel(DefaultHttp2RemoteFlowController.java:481)
at io.grpc.netty.shaded.io.netty.handler.codec.http2.DefaultHttp2RemoteFlowController$1.onStreamClosed(DefaultHttp2RemoteFlowController.java:105)
at io.grpc.netty.shaded.io.netty.handler.codec.http2.DefaultHttp2Connection.notifyClosed(DefaultHttp2Connection.java:357)
at io.grpc.netty.shaded.io.netty.handler.codec.http2.DefaultHttp2Connection$ActiveStreams.removeFromActiveStreams(DefaultHttp2Connection.java:1007)
at io.grpc.netty.shaded.io.netty.handler.codec.http2.DefaultHttp2Connection$ActiveStreams.deactivate(DefaultHttp2Connection.java:963)
at io.grpc.netty.shaded.io.netty.handler.codec.http2.DefaultHttp2Connection$DefaultStream.close(DefaultHttp2Connection.java:515)
at io.grpc.netty.shaded.io.netty.handler.codec.http2.DefaultHttp2Connection$DefaultStream.close(DefaultHttp2Connection.java:521)
at io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2ConnectionHandler.closeStream(Http2ConnectionHandler.java:613)
at io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2ConnectionHandler.processRstStreamWriteResult(Http2ConnectionHandler.java:894)
at io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2ConnectionHandler.access$1000(Http2ConnectionHandler.java:66)
at io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2ConnectionHandler$4.operationComplete(Http2ConnectionHandler.java:809)
at io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2ConnectionHandler$4.operationComplete(Http2ConnectionHandler.java:806)
at io.grpc.netty.shaded.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:578)
at io.grpc.netty.shaded.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:552)
at io.grpc.netty.shaded.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:491)
at io.grpc.netty.shaded.io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:616)
at io.grpc.netty.shaded.io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:605)
at io.grpc.netty.shaded.io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:104)
at io.grpc.netty.shaded.io.netty.util.internal.PromiseNotificationUtil.trySuccess(PromiseNotificationUtil.java:48)
at io.grpc.netty.shaded.io.netty.channel.ChannelOutboundBuffer.safeSuccess(ChannelOutboundBuffer.java:717)
at io.grpc.netty.shaded.io.netty.channel.ChannelOutboundBuffer.remove(ChannelOutboundBuffer.java:272)
at io.grpc.netty.shaded.io.netty.channel.ChannelOutboundBuffer.removeBytes(ChannelOutboundBuffer.java:352)
at io.grpc.netty.shaded.io.netty.channel.epoll.AbstractEpollStreamChannel.writeBytesMultiple(AbstractEpollStreamChannel.java:305)
at io.grpc.netty.shaded.io.netty.channel.epoll.AbstractEpollStreamChannel.doWriteMultiple(AbstractEpollStreamChannel.java:510)
at io.grpc.netty.shaded.io.netty.channel.epoll.AbstractEpollStreamChannel.doWrite(AbstractEpollStreamChannel.java:422)
at io.grpc.netty.shaded.io.netty.channel.AbstractChannel$AbstractUnsafe.flush0(AbstractChannel.java:949)
at io.grpc.netty.shaded.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.flush0(AbstractEpollChannel.java:557)
at io.grpc.netty.shaded.io.netty.channel.AbstractChannel$AbstractUnsafe.flush(AbstractChannel.java:913)
at io.grpc.netty.shaded.io.netty.channel.DefaultChannelPipeline$HeadContext.flush(DefaultChannelPipeline.java:1372)
at io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeFlush0(AbstractChannelHandlerContext.java:750)
at io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeFlush(AbstractChannelHandlerContext.java:742)
at io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.flush(AbstractChannelHandlerContext.java:728)
at io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2ConnectionHandler.onError(Http2ConnectionHandler.java:643)
at io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2ConnectionHandler$FrameDecoder.decode(Http2ConnectionHandler.java:380)
at io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2ConnectionHandler$PrefaceDecoder.decode(Http2ConnectionHandler.java:242)
at io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2ConnectionHandler.decode(Http2ConnectionHandler.java:438)
at io.grpc.netty.shaded.io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:507)
at io.grpc.netty.shaded.io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:446)
at io.grpc.netty.shaded.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276)
at io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
at io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
at io.grpc.netty.shaded.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
at io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
at io.grpc.netty.shaded.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.grpc.netty.shaded.io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:795)
at io.grpc.netty.shaded.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:480)
at io.grpc.netty.shaded.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378)
at io.grpc.netty.shaded.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
at io.grpc.netty.shaded.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.grpc.netty.shaded.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Thread.java:833)

ápr. 21, 2022 11:14:09 DE. io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2FrameLogger logSettingsAck
FINE: [id: 0x25d37ba8, L:/[0:0:0:0:0:0:0:1%0]:8089 - R:/[0:0:0:0:0:0:0:1%0]:50958] INBOUND SETTINGS: ack=true
ápr. 21, 2022 11:14:09 DE. io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2FrameLogger logPingAck
FINE: [id: 0x25d37ba8, L:/[0:0:0:0:0:0:0:1%0]:8089 - R:/[0:0:0:0:0:0:0:1%0]:50958] INBOUND PING: ack=true bytes=1234
ápr. 21, 2022 11:14:09 DE. io.grpc.netty.shaded.io.grpc.netty.NettyServerHandler$FrameListener onPingAckRead
FINE: Window: 1048576

Do i need some extra setting to get it work or is this some kind of a bug? As said i tested with libs 1.41.0 and 1.44.0, and it did not work with either of them.

[qiwzhang]

Since you are running in localhost, it doesn't GCP metadata server, yes, "--non_gcp" is required.

Could you raise this issue in ESPv2 repo with following info:

• add --enable_debug flag in ESPv2 container arguments
• get the debug log from the ESPv2 container

[qiwzhang]

BTW, ESPv2 is based on Envoy, it should NOT have this issue. It is currently supported, if there are any bugs, we will fix it.

[gombasp]

Hi,

any news on this topic? Earlier you wrote about espv2 that "if there are any bugs, we will fix it". Is this not the case for espv1?? We use it still extensively in our projects so a bugfix would be nice to have.
Thanks,

Peter

[qiwzhang]

Unfortunately, many ESPv1 release components, e.g. CI/CT are broken. Most likely they are outdated. It will take some effort to restore them. We don't have resource to make new releases for ESPv1 for now.

[gombasp]

I tested espv2 and it seems to be working without the above mentioned issue. So we will consider the upgrade. Thank you.