Comments (13)
ok with s3-multipart i managed to reproduce the problem. I think what's going on is:
createPresignedPost
needs endpoint: https://bucket.s3-accelerate.amazonaws.com
because it doesn't support useAccelerateEndpoint: true
however all other S3 methods do NOT support endpoint
when useAccelerateEndpoint
is set to true
, and because we share the s3 client between all s3 functionality, it creates an impossibility.
from uppy.
Hi. i'm not sure how to reproduce this problem. when I set COMPANION_AWS_USE_ACCELERATE_ENDPOINT=true
companion starts up without any error. i can even upload files
from uppy.
are you using an old bucket with the old behaviour? (bucket created 10+ years ago)
from uppy.
The bucket was created last week, here's my env:
COMPANION_HIDE_WELCOME=true
COMPANION_DOMAINS=['domain.com','(\\w+).domain.com']
COMPANION_UPLOAD_URLS=['https://upload.domain.com/']
COMPANION_PATH=/
COMPANION_DATADIR=./output
COMPANION_DOMAIN=upload.domain.com
COMPANION_PROTOCOL=https
COMPANION_PORT=3020
COMPANION_PREAUTH_SECRET="sdfadsffasdfads"
COMPANION_ALLOW_LOCAL_URLS=false
COMPANION_AWS_KEY="ALALALALALLALALALLA"
COMPANION_AWS_SECRET_FILE="/home/ubuntu/aws_s3_secret"
COMPANION_AWS_BUCKET="bucket-name"
COMPANION_AWS_REGION="eu-west-1"
# COMPANION_AWS_PREFIX="OPTIONAL PREFIX"
COMPANION_AWS_USE_ACCELERATE_ENDPOINT=true
COMPANION_AWS_EXPIRES="13800"
The error shows only when trying to upload a file.
from uppy.
how's you frontend code look like? any codesandbox or similar to reproduce the problem? we need some steps to reproduce the problem. "trying to upload a file" can be done in many ways using uppy.
from uppy.
Good point, sorry I wasn't clear about it.
I'm using Uppy dashboard + '@uppy/aws-s3-multipart' inside a Vue component, roughly:
uppyOptions: {
id: 'uppy',
autoProceed: true,
allowMultipleUploadBatches: false,
debug: true,
restrictions: {
maxFileSize: 2 * 1024 * 1024 * 1024, // 2Gb
allowedFileTypes: [
'video/mp4',
'video/mpeg',
'application/x-mpegURL'
],
requiredMetaFields: [],
},
meta: {},
infoTimeout: 10_000,
}
this.uppy = new Uppy({...this.uppyOptions})
this.uppy.use(this.AwsS3Multipart, {
shouldUseMultipart: (file) => file.size > 100 * 2 ** 20,
companionUrl: 'https://upload.domain.com/',
getTemporarySecurityCredentials: true})
I will try to create a repro later, as it might be tricky
from uppy.
Hi. i'm not sure how to reproduce this problem. when I set
COMPANION_AWS_USE_ACCELERATE_ENDPOINT=true
companion starts up without any error. i can even upload files
I wanted to clarify, do you receive an accelerated S3 endpoint in such case?
from uppy.
Thanks so much for looking into this, I was bracing to spend my Saturday on an investigation. Please let me know if I can help with tests.
from uppy.
Sure, feel free to test the pr linked above if youre able to
from uppy.
I can see now that s3ClientOptions
inside
...s3-accelerate.amazonaws.com/'
, yet in browser I still can see upload going to a regional bucket URL. Trying to find where it get's overwrited down the line.from uppy.
After some more debugging I found out that inside server-side Companion an endpoint is formed and passed around correctly as accelerated, but on client-side the actual endpoint is formed by AwsS3Multipart
plugin:
despite companionUrl
in its init config.
Moreover, I checked network exchange between browser and companion and I cannot see endpoint passed anywhere, means AwsS3Multipart
rightfully generates one on its own.
Bucket name and region are returned on /sts
route, but there's nothing about accelerated attribute.
{
"credentials": {
"AccessKeyId": "....",
"SecretAccessKey": "....",
"SessionToken": "...=",
"Expiration": "2023-12-09T17:40:03.000Z"
},
"bucket": "bucket-name",
"region": "eu-west-1"
}
So, my question is: is it a bug in Companion that the actual endpoint is never passed to the client or I missed something about plugin or Companion configuration?
I've updated @uppy/aws-s3-multipart/lib/createSignedURL.js
with
const host = `${bucketName}.s3-accelerate.amazonaws.com`
And everything works as expected now. So seems like accelerated endpoint feature was never a part of @uppy/aws-s3-multipart/
, unless I missed some config option to enforce it.
from uppy.
We support signing on the server using aws-s3
(which calls createPresignedPost
in companion) using accelerated endpoints. I tested that and it works. Companion returns:
GET s3/params
returns
{
...
"url": "https://***.s3-accelerate.amazonaws.com/",
...
}
We also support signing on the server using aws-s3-multipart
(calls getSignedUrl(client, new UploadPartCommand({...})
in companion). Companion returns:
GET s3/multipart/uploadId
{
"url": "https://***.s3-accelerate.amazonaws.com/my-prefix/...",
"expires": 800
}
I believe what you are asking for is the /sts
endpoint (signing on the client using an AWS STS temporary security credential). Support for that was recently added in #4519 where we implemented our own client signing code (createSignedURL
) for the aws-s3-multipart
plugin. But we did not think about accelerated endpoints. @aduh95 do you know if it would be easy to adopt createSignedURL
/ getTemporarySecurityCredentials
to accomodate an accelerated s3 endpoint instead of the hardcoded URL?
from uppy.
I'll create a new issue for that #4822
from uppy.
Related Issues (20)
- IMPORTANT: macOS Safari does not work with Uppy HOT 2
- @uppy/core and @uppy/dashboard 4.0.0-beta-3 versions contain references to React packages and JSX files in an Angular 17 app HOT 3
- `DashboardMiscOptions.trigger` requires `string`, but `findAllDOMElements` accepts `Node` as well
- have maxFileSize check the file after compression/resizing via Compressor plugin HOT 2
- Rewrite all Companion validation errors to respond with a status code and message
- Using uppy.setOptions({locale}) won't update locale strings in Restricter instance
- Incomplete type definitions for `BasePlugin` HOT 1
- XHRUpload stuck at 100% despite 2xx response code. HOT 12
- [AWS S3 Uploader] Allow retrieving Object Version ID HOT 3
- Dashboard - status bar upload button visible after restoring files, despite hideUploadButton being true.
- Custom Uploader add list of necessary events
- @transloadit/prettier-bytes 0.3.3 not working with webpack HOT 7
- uppy pauseResume throw unhandled error HOT 2
- Disable browse file / drag and drop on uppy dashboard HOT 2
- Can't resolve '@uppy/core/lib/UploaderPlugin.js HOT 10
- Errors in TypeScript definitions when installed with pnpm HOT 5
- Latest version of @uppy/xhr-upload doesn't support PATCH method HOT 11
- uppy-multipart + golden retriever HOT 3
- Event listener that fires before file upload process starts HOT 4
- Drag & Drop not working HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from uppy.