apoorvam/aws-s3-multipart-upload

Example AWS S3 Multipart upload with aws-sdk for Go - Retries for failing parts

aws-s3-multipart-upload's Issues

added goroutine

Hi there, I'm new to S3 api and golang and came across this script
I added goroutine to upload part so that it uploads in parallel (where I monitor my network bandwidth it should be)
feel free to take a look to my repo
thanks!

fatal error: runtime: out of memory

I'm trying to perform a multipart upload of a file that is 5.5 GB large, with a memory of around 4 GB.
In line 42 of the code, we have to create a slice of bytes (buffer := make([]byte, size). So in my case, I'd be creating a buffer whose size is 5.5 GB, which is breaking the code since it's running out of memory.

Is there a workaround you might suggest to solve this problem?

Update: The solution I'd be ideally looking for is
1- Being able to make my slice of bytes in chunks, and then somehow glue things together with the multipart upload
2- Being able to store my slice of bytes in storage rather than RAM.

Error: BatchedUploadIncomplete: some objects have failed to upload

I'm trying to upload the content of a whole directory. The content of the directory is never static, sometimes I have 20 files, some other time I've got 25 files, etc.

The code is always failing at the last file with the following stack trace:

Error: BatchedUploadIncomplete: some objects have failed to upload.
caused by: failed to perform batch operation on "path/to/file/file" to "bucket-name":
MultipartUpload: upload multipart failed
	upload id: random_file_id
caused by: BodyHashError: failed to compute body hashes
caused by: read path/to/directory/of/files: is a directory

any idea what might be going on?

Note: It is always failing on the last file

maxPartSize const error

First, thanks for sharing your code it was really helpful!
It appears to cause an "EntityTooSmall" error from S3. As currently stated in AWS docs the minimum part size is 5MB except for the last part.
In the implementation now maxPartSize is set to int64(512 * 10000), whereas it should be int64(5 * 1024 * 1024)
Replacing the const value to the one I wrote here solved the problem

while file gt 1024Mb，file content is 000000

`

buffer := make([]byte, maxPartSize)

for curr = 0; remaining != 0; curr += partLength {
	if remaining < maxPartSize {
		partLength = remaining
	} else {
		partLength = maxPartSize
	}

	nr, err := file.Read(buffer[:partLength])
	if err != nil {
		fmt.Println(err)
		return "", err
	}
	defer file.Close()

	if nr >0 {
		completedPart, err := uploadPart(svc, resp, buffer[:partLength], partNumber)
		if err != nil {
			fmt.Println("=========================",err.Error())
			err := abortMultipartUpload(svc, resp)
			if err != nil {
				fmt.Println(err.Error())
			}
			return "", err
		}
		remaining -= partLength
		partNumber++
		completedParts = append(completedParts, completedPart)
	}

}

`

curiosity

hello,
do you think its possible to change your sample in order to allow to upload from any available connection on the machine at the same time?
eg tethered mobile internet , and ethernet

awsS3-multipart-upload-script - Suggestion.

I've created a script awsS3-multipart-upload-script to upload multipart files to S3 standard storage using the S3API, and thought about sharing it here, so, if in the future more people like me end up in this Repo while searching how to do it, can easily find a solution. Thanks!

Repository: awsS3-multipart-upload-script