fclairamb/afero-s3

afero-s3's Issues

AWS Go SDK v2?

Is upgrading to the AWS Go SDK v2 on the roadmap?

Guard against Windows file paths (backslash delimited)

Hi @fclairamb! Thanks for this package :)

I am running into an issue when combining s3.Fs with afero.BasePathFs on Windows systems.

My issue is that S3 doesn't like backslash delimited paths (such as Windows file paths).

When nesting an S3 filesystem inside afero.BasePathFs, the paths are first processed using filepath package which uses os.PathSeparator, which is backslash (\) on Windows.

Therefore when you call afero.BasePathFs.MkdirAll the path it provides to s3.Fs.MkdirAll is a Windows path. It uploads "successfully" to S3, however S3 treats the path differently in such a way that makes uploading files from Windows use a different S3 path than from MacOS or Linux.

This is undesirable, at least in my case. The paths, no matter which OS they are created from should be consistent and valid S3 paths using forward slash delimiter.

So I can see two options:

1. inside s3.Fs process the input path and replace all backslashes with forward slashes, at least when runtime.GOOS == "windows"
2. change afero.BasePathFs to handle overriding the path separator manually

For the moment I figure option 1 is more correct since S3 dictates that all paths should be forward slash paths.

Do you have thoughts on this? I can create a PR for this if you agree on the desired behavior.

bug: File.Readdir appends trailing slash even if there is already one present

afero-s3/s3_file.go

Line 74 in a24edce

if name != "" {

If I call f.Readdir where f.name is /Path/ then it appends another slash to make it /Path// which fails
in s3/minio/etc.

cd and other command not working

Hi, we have been using this library at my job for managing s3 buckets as an FTP client.
We are having some issues concerning the cd command as it throws a 550 error whenever our application calls the fs.

550 CD issue: stat /<directory_name>: InternalServerError: Internal Server Error\n	status code: 500

After investigation, we concluded this error was originating from this library.
We do know that s3 does not support directories but we expected the naming standard /directory/file.ex to handle the navigation feature.

Is this a known issue and can you provide any support about this?

Thank you

Default ACL when writing files

Would it be possible to add a way to set the ACL on newly created objects? For example, I am using Digital Ocean Spaces and I am uploading files that should be accessible by the general public. By default, those files have the ACL set to Private, while I would benefit from having this set to Public (by setting the appropriate AWS S3 setting for the ACL that DO understands since DO Spaces follows the AWS S3 api).
I propose that during the Fs initialization we could set the default ACL that would get applied when writing to a file. I am open to suggestions.

If that sounds OK, I'd be more than happy to implement this if you're busy.

Open should return afero.ErrFileNotFound, but doesn't

When you open a file that doesn't exist, you get a generic error with. a "state : file does not exist" message rather than an afero.ErrFileNotFound. This makes it difficult to check for not found errors.

All files uploaded to S3 with Content-Type binary/octet-stream

All files are uploaded with a Content-Type of binary/octetstream. This is a problem for our use case where we're uploading to a bucket which is used for hosting files for a website. I believe there is functionality in the SDK to auto-detect the Content-Type based on the file extension which might be a better default.

Would be good to be able to set a Cache-Control header

As with my other issue, it would be good to be able to upload with a Cache-Control header in metadata for usage with S3 buckets used to serve web resources.

Fix stat for bucket root

Issue : Cant list the root ("/") directory of the s3 bucket folder as it is recognized as file rather than a directory

Can someone please take a look at #393 as we can see the issue and the fix for it has been open for a while .

Thanks in advance

Server-Side Encryption

Hi

I'd like to have the library to support SSE: https://docs.aws.amazon.com/sdk-for-go/v1/developer-guide/s3-example-server-side-encryption-with-kms.html

As far as I can tell, the way to go would be to add new public fields to the UploadedFileProperties type, like this:

// UploadedFileProperties defines all the set properties applied to future files
type UploadedFileProperties struct {
	ACL          *string // ACL defines the right to apply
	CacheControl *string // CacheControl defines the Cache-Control header
	ContentType  *string // ContentType define the Content-Type header
+	// Parameters below mirror the ones from s3.PutObjectInput type from AWS SDK
+	SSEKMSEncryptionContext *string
+	SSEKMSKeyId *string
+	ServerSideEncryption *string
}

And then copy them to the struct passed to AWS SDK in applyFileCreateProps and in applyFileWriteProps:

func applyFileCreateProps(req *s3.PutObjectInput, p *UploadedFileProperties) {
	if p.ACL != nil {
		req.ACL = p.ACL
	}

	if p.CacheControl != nil {
		req.CacheControl = p.CacheControl
	}

	if p.ContentType != nil {
		req.ContentType = p.ContentType
	}

+	if p.SSEKMSEncryptionContext != nil {
+		req.SSEKMSEncryptionContext = p.SSEKMSEncryptionContext
+	}
+
+	if p.SSEKMSKeyId != nil {
+		req.SSEKMSKeyId = p.SSEKMSKeyId
+	}
+
+	if p.ServerSideEncryption != nil {
+		req.ServerSideEncryption = p.ServerSideEncryption
+	}
}

(same code in the other function)

Couple of questions:

• Is this approach okay with you? If so, I'd go ahead and create a PR. Although, I'm not sure how to go about testing it. Minio, which you seem to be using to mock the real S3, supports SSE with their own KMS implementation.
• The seemingly public type seems to be undocumented. IMO, it exposes a very useful functionality. Maybe it's worth putting into examples in README?

Thanks,
Alexei

No error thrown with bad connection

Hello,

The following code does not works (nothing written on the bucket), but doesn't give any errors :

package main

import (
	"fmt"
	"os"

	"github.com/aws/aws-sdk-go/aws"
	"github.com/aws/aws-sdk-go/aws/credentials"
	"github.com/aws/aws-sdk-go/aws/session"

	s3 "github.com/fclairamb/afero-s3"
)

func main() {

	// Configure to use MinIO Server
	s3Config := &aws.Config{
		Credentials:      credentials.NewStaticCredentials("***", "****", ""),
		Endpoint:         aws.String("https://s3.fr-par.scw.cloud"),
		Region:           aws.String(""),
		S3ForcePathStyle: aws.Bool(true),
	}

	fmt.Printf("Starting....\n")

	// You create a session
	sess, err := session.NewSession(s3Config)
	if err != nil {
		fmt.Printf("error: %v\n", err)
	}

	// Initialize the file system
	s3Fs := s3.NewFs("my-bucket", sess)

	// And do your thing
	file, err := s3Fs.OpenFile("my_new_file.txt", os.O_WRONLY, 0777)
	if err != nil {
		fmt.Printf("error: %v\n", err)
	}
	file.WriteString("Hello world !")
	file.Close()
}

It is actually the Region that is not correct (works with fr-par), but an error should be thrown ?!

Directory stat is broken on windows

statDirectory fails because filepath.Clean is platform dependent. On windows \ is used as directory separator instead of '/'.
Instead use path.Clean, which always uses unix style path on all OSes.

https://golang.org/pkg/path/

afero-s3/s3_fs.go

Lines 254 to 255 in 5eb0a32

	func (fs Fs) statDirectory(name string) (os.FileInfo, error) {
	nameClean := filepath.Clean(name)

Action Required: Fix Renovate Configuration

There is an error with this repository's Renovate configuration that needs to be fixed. As a precaution, Renovate will stop PRs until it is resolved.

Error type: undefined. Note: this is a nested preset so please contact the preset author if you are unable to fix it yourself.

Subdir in S3?

Is there any way to create a FS that accesses only a subdir in S3? (basically a prefix/ to everything)

Dependency Dashboard

This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.

Open

These updates have all been created already. Click a checkbox below to force a retry/rebase of any.

• Update actions/checkout action to v4
• Update actions/setup-go action to v5
• Update dependency ubuntu to v22
• Update golangci/golangci-lint-action action to v4
• Click on this checkbox to rebase all open PRs at once

Detected dependencies

github-actions

.github/workflows/build.yml

• actions/checkout v3
• golangci/golangci-lint-action v3
• actions/setup-go v4
• fclairamb/minio-github-actions v0.2
• ubuntu 20.04

gomod

go.mod

• go 1.19
• github.com/aws/aws-sdk-go v1.51.18
• github.com/spf13/afero v1.11.0
• github.com/stretchr/testify v1.9.0

---

• Check this box to trigger a request for Renovate to run again on this repository