dcache/dcache-cta

dCache driver for CTA

dcache-cta's Issues

Provide performance metricks

To identify bottlenecks the following performance metrics are required:

• Requests per seconds
• Disk IO
• Network performance
• CTA gRPC request processing time

Plugin needs a percistent storage for pending requests

As CTA has it's own job queue, if dCache is restarted after request submition we might get an IO request and/or status update for a request, that is not in the dCache's flush/stage pool's queue. Thus we need a persistent store for all on-the-flight requests.

The hsm type and name should be included in URI

The HSM URI generated by the plugin must include configure hsm name and type

Update gRPC channel to support lazy connection

The currently used ManagedChannel is connected at configure stage and plugin fails to start if connection can't be created. more over, the re-connect on failure as well as error handling are missing.

Switch to release version of NearlineStorage API

The functionality of the driver depends on new NearlineStorage API (dCache/dcache#6141).

Add/test HSM cleaner functionality

Handle is closed exception during flush

04 Jan 2022 11:36:58 [nioEventLoopGroup-3-8] [] Opening /osmcache/dcache/dcache-tpm103/data/0000F3B008DCC88C45028E0CDDA9A5ED004B for reading.
04 Jan 2022 11:37:26 [nioEventLoopGroup-3-8] [] xrootd server error while processing org.dcache.xrootd.protocol.messages.CloseRequest@362106c3 (please report this to [email protected])java.lang.IllegalStateException: Handle is closed
	at org.dcache.pool.repository.v5.ReadHandleImpl.getReplicaFile(ReadHandleImpl.java:102)
	at org.dcache.pool.nearline.NearlineStorageHandler$FlushRequestImpl.getReplicaUri(NearlineStorageHandler.java:969)
	at org.dcache.nearline.cta.ForwardingFlushRequest.getReplicaUri(ForwardingFlushRequest.java:26)
	at org.dcache.nearline.cta.xrootd.DataServerHandler.getFile(DataServerHandler.java:490)
	at org.dcache.nearline.cta.xrootd.DataServerHandler.doOnClose(DataServerHandler.java:333)
	at org.dcache.nearline.cta.xrootd.DataServerHandler.doOnClose(DataServerHandler.java:84)
	at org.dcache.xrootd.core.XrootdRequestHandler.requestReceived(XrootdRequestHandler.java:186)
	at org.dcache.xrootd.core.XrootdRequestHandler.channelRead(XrootdRequestHandler.java:99)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
	at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
	at org.dcache.xrootd.core.XrootdAuthenticationHandler.channelRead(XrootdAuthenticationHandler.java:216)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
	at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:324)
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:296)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:719)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:655)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:581)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.base/java.lang.Thread.run(Thread.java:829)

The stage request must succseed when file is closed

The CTA doesn't have a special success even on stage.Thus closing the destination file should trigger StageRequest#complete

Better test coverage of IO handling

The IO testing is based on manual store/restore with integration of running dCache. A simpler unit-test based alternative is desired.

the pendig request queue is not cleaned up

When a submitted request completes (successfully or unsuccessfully) it stays in the queue and never removed, which will endup with OOM at some point.

it's inpossible to puase flushing

in some situations we want driver to pause until admin enables it again.