erikdahlstrand/shrine-rails-example

Example application using the Shrine gem

Shrine Rails demo

This is a Rails demo for Shrine. It allows the user to create albums and attach images. The demo shows an advanced workflow:

Uploading:

1. User selects one or more files
2. The files get asynchronously uploaded directly to S3 and a progress bar is displayed
3. The cached file data gets written to the hidden fields
4. Once the form is submitted, background jobs are kicked off to process the images
5. The records are saved with cached files, which are shown as fallback
6. Once background jobs are finished, records are updated with processed attachment data

Deleting:

1. User marks photos for deletion and submits
2. Deletion starts in background, and form submits instantly
3. Background job finishes deleting

This asynchronicity generally provides an ideal user experience, because the user doesn't have to wait for processing or deleting, and due to fallbacks they can be unaware of background jobs.

Direct uploads and backgrounding also have performance advantages, since your app doesn't have to receive file uploads (as files are uploaded directly to S3), and the web workers aren't blocked by processing, storing or deleting.

Implementation

The demo can upload files directly to S3 (default in production), or they can be uploaded to the app on stored on disk (default in development and test environment). See "Upload server modes" below for more info.

The demo features both single and multiple uploads.

On the client side Uppy is used for handling file uploads. The complete JavaScript implementation for the demo can be found in application.js.

Requirements

To run the app you need to setup the following things:

• Install ImageMagick:

  $ brew install imagemagick

• Install the gems:

  $ bundle install

• Have SQLite installed and run the migrations:

  $ rake db:migrate

• If you'll be using Amazon S3, run rails credentials:edit and put your S3 credentials, and setup CORS:

  s3:
    access_key_id: "..."
    secret_access_key: "..."
    region: "..."
    bucket: "..."

Once you have all of these things set up, you can run the app:

$ rails server

Upload server modes

This demo app is capable of uploading files directly to S3 (using straight upload or S3 multipart upload), or of uploading to an application action and storing on local disk.

In all three modes, the file selected in the browser is immediately uploaded by Javascript to some storage location ("JS direct upload"), and then on form submit a shrine-compatible hash describing the already-stored file is sent to the Rails app. Using shrine cached_attachment_data and restore_cached_data plugins. The difference is in where the Javascript sends the file, and how.

You can choose which upload server mode to by setting the UPLOAD_SERVER env variable. Otherwise, the default is s3 in production, and app in test and development.

• s3
  • Shrine storages are set to S3.
  • The Uppy AwsS3 plugin is used to upload files directly to the S3 cache storage.
  • The shrine presign_endpoint plugin is used to support Uppy AwsS3 plugin, providing authorized signed S3 URL for upload.
  • This is the default in Rails production environment.
• app
  • Shrine storages are set to local filesystem, in ./public.
  • The Uppy XHRUpload plugin is used to submit files directly to Rails app.
  • The Shrine upload_endpoint plugin is used to provide a local app HTTP action to accept the uploads.
  • This is the default in non-production Rails environments (development and test).
• s3_multipart
  • Shrine storages are set to S3.
  • The Uppy AwsS3Multipart plugin is used to upload files directly to the S3 cache storage, using S3's multipart upload strategy. This allows files larger than 5GB to be uploaded to S3, and can have other reliability and performance advantages from uploading in multiple smaller requests, especially for large files even if less than 5GB.
  • The uppy-s3_multipart gem, supporting the shrine uppy_s3_multipart plugin, are used to provide endpoints for the AwsS3Multipart Uppy plugin.
  • This is never the default, but you can have the app use it by setting an ENV variable.

So if you would like to use the app with the S3 multipart upload server strategy, launch the rails app with:

$ UPLOAD_SERVER=s3_multipart rails server

Consider access control

In a real apps, if you only want logged-in users to be able to upload files directly to your cache storage, you will want to limit access to the signing and/or file-receiving endpoints in routes.rb. For example, if using devise one way to do this is:

authenticate :user do
  mount Shrine.upload_endpoint(:cache) => "/upload"
end

References

• Shrine docs: Direct Uploads to S3
• Shrine wiki: Adding Direct App Uploads
• Shrine wiki: Adding Direct S3 Uploads
• uppy-s3_multipart gem
• Janko's Blog: Better File Uploads with Shrine: Direct Uploads

shrine-rails-example's Issues

Getting Webpacker error: Webpacker can't find application.css

I was having trouble with .mjs files in the Uppy packages in my application so i tried cloning this repo and running it fresh. When I run rails s i get the following error:

ActionView::Template::Error (Webpacker can't find application.css in ~/code/shrine-rails-example/public/packs/manifest.json. Possible causes:
1. You want to set webpacker.yml value of compile to true for your environment
   unless you are using the `webpack -w` or the webpack-dev-server.
2. webpack has not yet re-run to reflect updates.
3. You have misconfigured Webpacker's config/webpacker.yml file.
4. Your webpack configuration is not creating a manifest.
Your manifest contains:
{
}
):
     5:     <%= csrf_meta_tags %>
     6:     <%= csp_meta_tag %>
     7:
     8:     <%= stylesheet_pack_tag 'application' %>
     9:     <%= javascript_pack_tag 'application' %>
    10:   </head>
    11:

Adding a default image

Is it possible to add a default_url when s3 is not reachable?

undefined method `mime_type' (when removing image)

I have a User model that has_one Profile model, with the Profile model having the image_data attribute. Everything is working fine when uploading the image, but when I check the 'remove attachment' checkbox and submit the form, I get the following error:
undefined method 'mime_type' for #<Hash:0x007fed4367c608>
and references this line in the IMageUploader file:
validate_mime_type_inclusion ['image/jpeg', 'image/png', 'image/gif']

Again, this only happens when removing the image from my Profile model. Any ideas on where I might start looking to troubleshoot this issue?

NOTE: I should also mention that the image does in fact get removed from the model, it just errors on that validation, which seems weird.

Shrine 3 and Backgrounding

I am setting up a process to direct upload excel file to S# bucket.

File makes it to S3 cache directory. No background job starts to allow me move the cached file to store, and I cannot then open the file and process it.

What am I doing wrong? I have set the Rails 6 App to use Sidekiq to process jobs, Redis and Sidekiq are all good. My foreman Procfile.dev I use has a bundle exec sidekiq...

So the backgrounding has me stumped. Not sure how to test this, or get it going...

My shrine.rb is right out of the example:

# delay promoting and deleting files to a background job (`backgrounding` plugin)
Shrine.plugin :backgrounding
Shrine::Attacher.promote_block do
   puts "Promote Block is going now..."
   Attachment::PromoteJob.perform_in(3, self.class.name, record.class.name, record.id, name, file_data)
 end
 Shrine::Attacher.destroy_block do
    Attachment::DestroyJob.perform_later(self.class.name, data)
 end

Unsupported argument type: Symbol

rails 5.2
shrine 3.3
sidekiq 5.2

rails/rails#25993

I think best to use String instead Symbol in promote block:

Shrine::Attacher.promote_block { Attachment::PromoteJob.perform_later(record, name.to_s, file_data) }

Branches demo

Is it possible to have 2 working branches following direct S3 upload walk through, one with stimulus.js and the other without stimulus.js?

Trailing Comma.

Regard this line
why this comma - at the end of that line - specifically putted there ? is it some convention ?

S3 Objects are not deleted

After deleting an image the object in S3 (store prefix) is not being deleted.

I thought it could be a permission problem, but I manually deleted the object using the rails c.

Testing

It could be a good idea to provide the tests for this shrine example.

regards,

Would love to see this extended to show polymorphic upload handling

I haven't been able to find an example of shrine with ImageUploader and VideoUploader ::Shrine classes used with a single parent model. Would be great to see something like this added to the example to understand the recommendations and the best practices around a scenario where we want to handle video and image uploads for a gallery as well as validation, processing, and display of those uploads with distinct logic per upload type.

• STI, MTI, Join Table?
• Separate uploader classes and direct use of Attacher logic or Single uploader class and conditional logic with dynamic use of Attacher via Attachment inclusion?

Is there any particular reason on moving to suckerpunch from sidekiq??

I was just wondering Is there any particular reason on moving to suckerpunch from sidekiq which was on the previous one?

By the way good to see the progress bar implemented in example app.

Another Module build failed

Cloned the example to see what it would do.

I have node v14.10.1. Don't know much about how all this stuff works, but got the below error when running webpack-dev-server and have no idea how to fix it.

Module build failed (from ./node_modules/babel-loader/lib/index.js):
Error [ERR_PACKAGE_PATH_NOT_EXPORTED]: No "exports" main defined in /Users/salex/Work/rf/shrine-rails-example/node_modules/@babel/helper-compilation-targets/package.json
    at throwExportsNotFound (internal/modules/esm/resolve.js:285:9)
    at packageExportsResolve (internal/modules/esm/resolve.js:491:3)
    at resolveExports (internal/modules/cjs/loader.js:450:36)
    at Function.Module._findPath (internal/modules/cjs/loader.js:490:31)
    at Function.Module._resolveFilename (internal/modules/cjs/loader.js:880:27)
    at Function.Module._load (internal/modules/cjs/loader.js:743:27)
    at Module.require (internal/modules/cjs/loader.js:965:19)
    at require (internal/modules/cjs/helpers.js:88:18)
    at Object.<anonymous> (/Users/salex/Work/rf/shrine-rails-example/node_modules/@babel/preset-env/lib/debug.js:8:33)
    at Module._compile (internal/modules/cjs/loader.js:1076:30)

Validate Dimensions Range Mis-Typed

The dimensions validation range on this line: https://github.com/erikdahlstrand/shrine-rails-example/blob/master/app/uploaders/image_uploader.rb#L18

should be:

validate_dimensions [0..5000, 0..5000]

instead of

validate_dimensions [0..0, 5000..5000]

Full url for an image-path for displaying the image

The normal rails image_tag works fine. But how can I get the full image url without using the rails helpers so that I can use in my angular app?

Missing something?

I've setup everything as the Readme instructed (Bundled gems, setup .env, and migrated), but when i run the app and try to upload an image, it does nothing? I'm not getting JS errors in the console either. I just click 'choose files' select a .jpg image, and then nothing. (below is an image of what I see).

I doubt it's related, but I am running Ruby 2.3.0?

Pic:

Preflight requests issue on Safari

Hi @erikdahlstrand !

1st: Thanks for this example on how to implement shrine presigned uploads to S3!

I've followed on a app that I'm developing and it works very good. But after a few tests on Safari I went back to your example to check "what I did wrong" and the same error occurs on your example also.

Well maybe you know what's going on:

I'm running your example on Safari Version 10.0.1 (12602.2.14.0.7) / Mac OS 10.12.1

When I try to upload files I've got the following errors:

Failed to load resource: the server responded with a status of 403 (Forbidden)
Failed to load resource: Preflight response is not successful
XMLHttpRequest cannot load https://my-development-bucket.s3.amazonaws.com/. Preflight response is not successful

Currently my CORS configurations on S3 are:

<?xml version="1.0" encoding="UTF-8"?>
<CORSConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
    <CORSRule>
        <AllowedOrigin>*</AllowedOrigin>
        <AllowedOrigin>http://localhost:3000</AllowedOrigin>
        <AllowedMethod>GET</AllowedMethod>
        <AllowedMethod>POST</AllowedMethod>
        <MaxAgeSeconds>3000</MaxAgeSeconds>
        <ExposeHeader>Accept-Ranges</ExposeHeader>
        <ExposeHeader>Content-Range</ExposeHeader>
        <ExposeHeader>Content-Encoding</ExposeHeader>
        <ExposeHeader>Content-Length</ExposeHeader>
        <AllowedHeader>Range</AllowedHeader>
        <AllowedHeader>Authorization</AllowedHeader>
        <AllowedHeader>Origin</AllowedHeader>
        <AllowedHeader>Content-Type</AllowedHeader>
        <AllowedHeader>Range</AllowedHeader>
        <AllowedHeader>X-Requested-With</AllowedHeader>
    </CORSRule>
</CORSConfiguration>

I'm trying to debug the preflight (which I think Is the same presign request) to check the headers and the full HTTP request so I can understand what may cause this issue.

I'll be very thankful If you also know what is causing this..

POST error

I can get this example app to work, and am trying to set a similar file upload up in another app. I'm getting a 400 Bad Request with this response.

<Error>
<Code>InvalidArgument</Code>
<Message>Malformed Unicode code sequence in the field.</Message>
<ArgumentName>formField</ArgumentName>
<RequestId>6D3FCAB1348B56CB</RequestId>
<HostId>LzZXcHD2st8YRUBpw6qptqRCRQ4Pj6QKMT1+as6zdvmyL8l/mAEt6xvxknhvnomb26aum/mdGzs=</HostId>
</Error>

Any suggestions on what might be triggering this Malformed Unicode message? Thanks!

Update it please it not working properly and make it some better example thanks

Module build fails

I tried following the instructions to get this to build, but have the following error:

bootstrap:83 Uncaught Error: Module build failed (from ./node_modules/babel-loader/lib/index.js):
Error: Package exports for '/home/matthew/projects/shrine-rails-example/node_modules/@babel/helper-compilation-targets' do not define a '.' subpath

no implicit conversion of nil into String on Albums Show _photo

Cloned repo and got it up and running on local linux wsl. Somewhat problematic due to Shrine secret_key and prefix issues. No real documentation about that.

However after working on getting it working 100%. I cannot get it to render the _show.html.erb file. I get

ActionView::Template::Error (no implicit conversion of nil into String):

on the

<%= photo.image.derivation_url(:thumbnail, 300, 300).to_s,
width: 150,
class: "img-thumbnail mr-3" %>

Can you please explain why this is happening. I have tried to change it several times but have not gotten anywhere.

Thanks in advance and pretty excited about this uppy integration

Will progress bar work with this kind of approach?

I am trying to add a progress bar on top of this file upload example. But cant figure out with the example setup

$(function() {
  $('[type=file]').fileupload({
    add: function(e, data) {
      $.getJSON('/attachments/images/cache/presign', function(result) {
        data.formData = result['fields'];
        data.url = result['url'];
        data.submit();
      });
    },

    done: function(e, data) {
      var image = {
        id: data.formData.key.match(/\w+$/)[0],
        storage: 'cache',
        metadata: {
          size:      data.files[0].size,
          filename:  data.files[0].name,
          mime_type: data.files[0].type
        }
      }

      data = {photo: {image: JSON.stringify(image)}}
      $.ajax("#{@listing.id}/photos", {method: 'POST', data: data, dataType: 'script'})
    }
  });
});

As all here is the done callback I cant figure out how to get the progress event and error events?

Rack::QueryParser::InvalidParameterError: invalid %-encoding

We get these errors:

ArgumentError: invalid %-encoding (cwAj48LEStGei5i+tR%YzQ.jpg)
Rack::QueryParser::InvalidParameterError: invalid %-encoding

/presign?filename=cwAj48LEStGei5i%20tR%25YzQ.jpg
/presign?filename=4TpA5aMGRMu%25AXwpAXDK2Q.jpg

rails 5.1.6
rack 2.0.5
shrine 2.12

Any blog post about this ?

is there any blog post or article about this example ?