bindings/java: read and write big files through streams

[maiorBoltach]

bindings/java: read and write big files through streams

[tisonkun]

Perhaps we should support InputStream and OutputStream related APIs in BlockingOperator.

The async version should be something like ReactiveStream but it can be over complicated here for now.

[Xuanwo]

Are there any solutions or ideas how we can deal with big files?

We can add a stream or reader in Java, but we're unsure which abstraction to use on the Java side. Can you describe which interface you expect in java? Does java has Reader or Stream concepts like in rust or python?

Also cc owner of java binding: @tisonkun @G-XD for ideas.

[tisonkun]

@Xuanwo What does the Rust's design for streaming read/write look like?

Java has both Reader or Stream, but their API may not be mapped naturally. That is, we still need some work to bridge the API.

[Xuanwo]

What does the Rust's design for streaming read/write look like?

OpenDAL provides three styles of API as listed:

• opendal::Reader: fn read(&self, offset, limit) -> Result<Buffer>
• opendal::FuturesAsyncReader: fn read(&mut self, buf: &mut [u8]) -> Result<size>
• opendal::FuturesBytesStream: fn next(&mut self) -> Option<Result<Bytes>>

[tisonkun]

Seems we can implement something like public class OperatorInputStream extends InputStream based on FuturesBytesStream.

What about the write APIs?

[Xuanwo]

What about the write APIs?

write supoorts fn write(&mut self, bs) -> Result<size>

[tisonkun]

@Xuanwo I guess what I'm looking for is FuturesBytesSink.

But InputStream and OutputStream are all blocking APIs. Although they are streaming read/write bytes instead of loading everything into the memory. Let me consider a little bit about what the API should be like.

[tisonkun]

#4626 Here is the read part (InputStream) without any optimizations. But it should implement the streaming read feature.

@G-XD you may implement the write part following the similar structure.

@maiorBoltach Welcome to try out this version and see if it meets your requirements.