nejckorasa/s3-stream-unzip

Lightweight Java library to manage unzipping of large files and data in AWS S3 without knowing the size beforehand and without keeping it all in memory or writing to disk.

s3-stream-unzip's Issues

pipe to unzip to s3?

What if you have this zip locally (or at least a stream for it) and want to unzip it into an s3 bucket? How would we do this?

Add Javadoc and improve documentation

You cannot unzip an archive if it contains folders with files

When unzipping an archive in which there are packs with files or subfolders with files, I encounter the following error:

Failed initialising multipart upload with uploadId null
Aborting [S3MultipartUpload uploading to <folder>/, with uploadId null due to error: {}
com.amazonaws.services.s3.model.AmazonS3Exception: Object name contains unsupported characters. (Service: Amazon S3; Status Code: 400)

Can you please tell me if the library can unzip the archive if it contains folders?

Add SplitUnzipStrategy

Add a strategy that allows larger files to be split to into multiple shards/files that are uploaded to S3.

• it should handle possible headers (e.g. csv files)

Compatibility Issue between s3-stream-unzip and MinIO

Description

I am encountering a compatibility issue between the s3-stream-unzip library and the latest version of MinIO (dated 9/19/2023). When attempting to use s3-stream-unzip with the latest version of MinIO,

• MinIO version (latest): 9/19/2023
• MinIO version (working): 1/4/2023 that is compatible]

Publish the code to maven central

What happens if submit task fails with exception due to bounded queues & maximumPoolSize.

Hi
I was trying port code to C#, during prototype I come across the situation where submitting a task fails because of bounded queues & maximumPoolSize . It looks this exception case is not handled. Executor Service can throw RejectedExecutionException
Corresponding code

s3-stream-unzip/src/main/java/io/github/nejckorasa/s3/upload/S3MultipartUpload.java

Line 148 in b751ffd

submitTask(() -> {

Suggestion: you could AWS transfer utilitie which does multiple upload of given stream under the hood.

My minimal C# implementation of the same here

https://github.com/madhub/S3StreamUnzip/blob/main/README.md