rails/activestorage

Store files in Rails applications

Active Storage has been merged into Rails!

You can find the latest version of the source code inside the Rails repository, where we’ll continue to work on it.

Please report any bugs to the Rails issue tracker.

An archive of the source code before the merge is available here.

activestorage's Issues

Incompatible with Rails 5.1.2

In VerifiedKeyWithExpiration, class_attribute use default as an option, which is a new API. default option does not appear in activesupport 5.1.2, but there is a line of s.add_dependency "activesupport", ">= 5.1" in gemspec.

I think we should add some documents to indicate this incompatibility.

https://github.com/rails/activestorage/blob/master/lib/active_storage/verified_key_with_expiration.rb#L2
rails/rails#29270

Fix variant-related tests

They fail consistently in CI and on my machine, presumably due to platform differences. (MiniMagick produces different images in CI, on my machine, and on David’s machine.) I’ve temporarily skipped them in master.

/cc @dhh

Add support for hosted S3 object storage

I currently do a lot of work with OpenStack and we run a Ceph cluster which can be accessed via an S3 compatibility layer using our own host name.

Since S3 is a published protocol, it doesn't necessarily follow that an Active Storage user will always want to use S3 storage against Amazon's cloud.

The Amazon SDK being used by Active Storage supports an additional endpoint parameter to enable support for hosted S3. This is the path that I take directly in my Rails apps at present - I would prefer to go via the ActiveStorage route in future since it'll simplify my code.

I'm going to be working on this change here: #48

If anyone on the Rails team has any thoughts on this change then please let me know 😄

Change blob URL?

Is there a way to change the URL of a blob beyond the default /rails/blobs/<encoded-key> pattern? It would make it less obvious that it is a Rails app. A config setting like url_pattern would be nice.

Add direct upload file_field using JavaScript

Through the DirectUploadsController we support generating blobs prior to direct upload, but we don't yet provide any convenience to complete the full story with the required JavaScript needed to actually perform the direct upload. We should.

Here's the broken JS I've been using to test that this approach actually works:

window.fileUpload = function(file) {
  // FIXME: Block the form from submitting while we are doing the direct upload

  let data = new FormData()
  data.append("blob[filename]", file.name)
  data.append("blob[byte_size]", file.size)
  data.append("blob[content_type]", file.type)
  
  // FIXME: Compute proper md5 in base64
  data.append("blob[checksum]", "xxx23423")

  fetch("/rails/active_storage/direct_uploads", { method: "POST", body: data }).then(r => r.json()).then(function(directUploadDetails) {
    const reader = new FileReader()

    reader.onload = function(event) {
      fetch(directUploadDetails["url"], { method: "PUT", body: event.currentTarget.result}).then(function() {
        // FIXME: Set a hidden field on the form with the value of directUploadDetails["sgid"] 
        // to reference the uploaded file
        console.log(directUploadDetails)
      })
    }

    reader.readAsBinaryString(file)
  })
  
  // FIXME: Release the form for submission now that the files are done uploading
  return true
}

What I'd like to see is something like this:

<%= form_with(model: Message.new) do |form| %>
  <%= form.file_field :images, multiple: true, 'data-direct-upload': true %>
<% end %>

This should do the following:

1. When one or more files are selected, we call /rails/active_storage/direct_uploads for each with the details ala the broken JS above. This gives us a set of details for each.
2. We then disable the submit button on the form to prevent it from being sent before we're done upload.
3. Once each file is done uploading, we add a hidden field that provides the sgids and we reenable the submit button.

When this form is submitted, we'll verified the sgids in #attach and just associate them as though they were finally uploaded blobs.

Get more informations on direct_uploads

Currently from the DirectUploadsController we return {:url, :sgid} which works fine with S3 and GCS since they sign all required fields in query_string.
But, Azure require some informations to be passed in headers, currently that header is
x-ms-blob-type: "BlockBlob".

Can we return fatter response from the controller that contains url, sgid, headers and use him on the fontend somethig like this

  fetch("/rails/active_storage/direct_uploads", { method: "POST", body: data }).then(r => r.json()).then(function(directUploadDetails) {
    const reader = new FileReader()

    reader.onload = function(event) {
      fetch(directUploadDetails["url"], { 
        method: "PUT",
        headers: directUploadDetails.headers, // something like this
        body: event.currentTarget.result}).then(function() {
          ...

      })
    }

    reader.readAsBinaryString(file)
  })
 

Also, I suppose there are other storage providers in the wild that are going to need something like this

MirrorService needs work

There are two problems with ActiveStorage::Service::MirrorService in the current code:

1. The generated sample config is wrong. Looking at the service code, it should be something like this instead:

mirror:
  service: Mirror
  services: [ local, amazon, google ]

2. Even with that change, the service still doesn't work, because the services attribute gets initialized to an array of strings, when it really needs to be an array of other Services. Quick and dirty fix is to edit the initializer:

  # def initialize(services:)
  #   @services = services
  # end
  def initialize(services:)
    service_array = []
    config_file   = Pathname.new(Rails.root.join("config/storage_services.yml"))
    configs = YAML.load(ERB.new(config_file.read).result)
    services.each do |service|
      if service_configuration = configs[service.to_s].symbolize_keys
        service_name = service_configuration.delete(:service)
        service_array << ActiveStorage::Service.configure(service_name, service_configuration)
      end
    end

    @services = service_array
  end

I'm not putting in a PR at the moment because it's entirely possible the current code authors have a better fix in mind and just haven't gotten around to implementing it yet. Mostly just noting this for anyone else playing with early versions.

File types / sizes / location

Thank you guys, for working on this thing. I'm really excited about it. I'm wondering how to put some control on the uploads:

1. How to control which file extensions are allowed or not? I've just uploaded a pdf with this gem, and I don't want to allow this on my app.
2. How to control allowed file sizes and image dimensions?
3. Is there a way to upload the files to a folder in the AWS S3 bucket, and not to the root of the bucket?

Add direct upload handling to disk controller

This is a bit more involved than with the cloud services. We need a new controller that'll actually accept an upload and write it to disk. Needs to generate signing, expiring direct upload URLs to this controller.

See the S3 implementation and remember to broaden the DirectUploadsControllerTest.

Shortcut for downsampling

I like what David have done with Variants, but I would like to still have a shortcut like

<%= image_tag magazine.cover_photo.url(size: '600x400') %>

Because downsampling is the most common use-case. More common than making no transformations at all.


But maybe you have something else planned? Like extending image_tag to accept a Variant instance?

<%= image_tag magazine.cover_photo.variant(resize: '600x400') %>

Can we get a Backblaze adapter out of the box?

Not sure if this is the correct spot for this but I noticed you don't have an adapter for Backblaze.

They are a very attractive competitor to S3. For example they are 4x cheaper than S3 in storage and 2.5x cheaper for downloads. They've been around for a long time and are rock solid.

Pricing details on how they compare to S3 and Google Cloud:
https://www.backblaze.com/b2/cloud-storage-pricing.html

General details on their S3-like service:
https://www.backblaze.com/b2/cloud-storage.html

I know there's the whole "but if we support this, then where do we draw the line?" argument for supporting services but I really think these guys are popular enough to have it officially supported. It could also be a good excuse to write a really nice Backblaze connectivity gem because that's currently lacking.

VerifiedKeyWithExpiration.verifier is nil

Installed ActiveStorage, configured according to README, successfully uploaded a photo via MirrorService (local + amazon). Now,

2.3.4 :003 > ActiveStorage::VerifiedKeyWithExpiration.verifier
=> nil

even though

2.3.4 :004 > defined?(Rails) ? Rails.application.message_verifier("ActiveStorage") : nil
 => #<ActiveSupport::MessageVerifier:0x007f9ea69a3818 @secret="\x8C\eA>\x96\xC2\x91c\xBAP\xAB$%\xB5|\xBA\x9E\xD9\x8E\xB8=o\xB9\x8E\x88aI\x85g\"\xCE\x7FL+>l|\xF8S#?\xD2\xE8\xBE[g\xD8\xF7\"\x0FR\xEDBe\xE5\xE6\xB3\x9EF~\xB2\xEBJ\x14", @digest="SHA1", @serializer=Marshal>

As a result,

2.3.4 :005 > jacket.photos.first.url
  Disk Storage (9.5ms) Generated URL for file at key: TzDZEQAJdY1HBUg5Hym16oPd ()
NoMethodError: undefined method `generate' for nil:NilClass
        from (irb):5

Any help is welcome 🤔

Support for S3 server_side_encryption

Thank you guys for the all work on this, it looks great so far!
It would be nice if we could set the server_side_encryption option on S3 uploads, ideally, on a per model basis.

object_for(key).put(body: io, content_md5: checksum, server_side_encryption: 'AES256')

I'm happy to open a PR but I want to get some feedback on the best way to implement this or maybe you guys already have thought about this? I can see others wanting to pass additional options to aws calls in the future, as well.

Thanks!

Invalid S3 Checksum

Using the etag of an object is not a valid checksum of an S3 object if the object was multipart uploaded.

If an object is created by either the Multipart Upload or Part Copy operation, the ETag is not an MD5 digest, regardless of the method of encryption.

http://docs.aws.amazon.com/AmazonS3/latest/API/RESTCommonResponseHeaders.html

This also raises another question in my mind. Should we be checking the checksum of a file before it gets uploaded to ensure that the file was not corrupted in transit? It is recommended and supported by S3 and GCS.

Supporting Additional Services

Rather than clog #36 with an off topic discussion I have started a separate issue regarding the usage of gems to add support for additional services.

Having used thoughtbot/administrate that uses a similar approach for adding extra functionality to the dashboards, I think this approach works well for adding features. The only struggle I had with the approach was when they broke out a feature into a gem that really should have been in core. In their case, the image thumbnail view.

As long as the services that cover the majority of cases (as is the Rails way), I think you're golden. Like ActiveRecord, I'd like to see support out of the box for the most "popular services" So it just works. Wanna do something weird, you'll need to write your own service.

I think supporting S3, AWS, Azure, and OpenStack give you great coverage. OpenStack can provide the bridge to ton of niche services. The fog gem can be used for this.

With this position, I think we should also try to write a guide on how to implement your own service. As would be first-time rails contributor, I can take a stab at this as I attempt the OpenStack service. My extra green eyes might help bring the right things to the surface.

Amend image_tag to resolve Blobs and Variants

These should just work:

  <%= image_tag Current.user.avatar %>
  <%= image_tag Current.user.avatar.variant(resize: "100x100") %>

Right now you have to manually call url_for to resolve ActiveStorage::Blob and ActiveStorage::Variant into strings that image_tag will take. Need to amend image_tag to take these classes.

Direct upload checksums aren't verified

Tested using AWS:

>> blob
=> #<ActiveStorage::Blob id: 19, key: "jwh9haVATotjTm3pjyoL4io4", filename: "image.png", content_type: "image/png", metadata: {}, byte_size: 305132, checksum: "abc123", created_at: "2017-07-24 15:48:18">

>> blob.checksum
=> "abc123"

>> blob.url_for_direct_upload
  S3 Storage (1.1ms) Generated URL for file at key: jwh9haVATotjTm3pjyoL4io4 
=> "https://mybucket.s3.amazonaws.com/jwh9haVATotjTm3pjyoL4io4?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAJFN4P4CD7L4DB7VQ%2F20170724%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20170724T161105Z&X-Amz-Expires=300&X-Amz-SignedHeaders=content-type%3Bhost&X-Amz-Signature=[SNIP]"

$ curl -s --data-binary @image.png -H "Content-Type: image/png" -X PUT "[DIRECT UPLOAD URL HERE]"

Upload succeeds

Add direct upload URL generation to GCS service

See the S3 implementation and remember to broaden the DirectUploadsControllerTest.

Eager loading Attachments?

Since this library associates attachments to records without using ActiveRecord associations, eager loading with includes does not work:

  test "eager load attachments" do
    @second_user = User.create!(name: 'Jason')

    assert User.includes(:attachment).to_a
  end

Result:

  1) Error:
ActiveStorage::AttachmentsTest#test_eager_load_attachments:
ActiveRecord::AssociationNotFoundError: Association named 'attachment' was not found on User; perhaps you misspelled it?

A use case for this would be a Users#index page that lists all users in a table and shows their avatar next to the user. The current approach would result an "N + 1" query problem as the Attachment for each User would be queried individually from the database.

Install error in new project (not migrated)

I create a new rails project to try Active Storage. Followed the install steps in README. But it gave error in step 2. Because the project was untouched , so there is not any migrations and db/migrate not exist.

For trying to copy a file to non-existent directory, ruby gave Errno::ENOENT: No such file or directory error.

Full text of the error:

Made storage and tmp/storage directories for development and testing
Copied default configuration to config/storage_services.yml
rails aborted!
Errno::ENOENT: No such file or directory @ rb_sysopen - /home/u/asd/db/migrate/20170721060652_active_storage_create_tables.rb
/home/u/asd/bin/rails:9:in 'require'
/home/u/asd/bin/rails:9:in '<top (required)>'
/home/u/asd/bin/spring:15:in '<top (required)>'
bin/rails:3:in 'load'
bin/rails:3:in '<main>'
Tasks: TOP => activestorage:install
(See full trace by running task with --trace)

Problem with config.active_storage.service in environment files

Hello, could anyone help me with the following problem?

i configure a simple rails app:

Ruby 2.4.1
Rails 5.1.2

Added the gem "activestorage" in Gemfile:

gem 'activestorage', github: 'rails/activestorage'

Added require "active_storage" to config/application.rb, after require "rails/all" line.

Runned rails activestorage:install;

I tried Configure the storage service in config/environments/* with config.active_storage.service = :local. When i start the server, dont work.

/Users/rafaeldl/.rbenv/versions/2.4.1/bin/ruby -e at_exit{sleep(1)};$stdout.sync=true;$stderr.sync=true;load($0=ARGV.shift) /Users/rafaeldl/.rbenv/versions/2.4.1/lib/ruby/gems/2.4.0/gems/ruby-debug-ide-0.6.1.beta4/bin/rdebug-ide --disable-int-handler --evaluation-timeout 10 --rubymine-protocol-extensions --port 55802 --host 0.0.0.0 --dispatcher-port 55803 -- /Users/rafaeldl/Projetos/babytube-web/bin/rails db:migrate Fast Debugger (ruby-debug-ide 0.6.1.beta4, debase 0.2.2.beta10, file filtering is supported) listens on 0.0.0.0:55802 rails aborted! NoMethodError: undefined method[]' for nil:NilClass
(erb):12:in <main>' /Users/rafaeldl/Projetos/babytube-web/config/environment.rb:5:in <top (required)>'
/Users/rafaeldl/Projetos/babytube-web/bin/rails:9:in require' /Users/rafaeldl/Projetos/babytube-web/bin/rails:9:in <top (required)>'
-e:1:in load' -e:1:in

'
Tasks: TOP => db:migrate => environment
(See full trace by running task with --trace)
`

Variants not working with local storage

Using on-disk storage, variants are ending up at locations like:

storage/va/ri/variants/q6wpQprqsgMjwsqEB2updXup/
BAh7BjoLcmVzaXplSSIMMTI4eDEyOAY6BkVU--d700c46c867cdefe2d3b138990d4b24913667441/headshot.png

But Rails is trying to reload them without the /va/ri when it uses url_for(...variant...)

Looks like the bug is somewhere in the interaction between the variant keys and the special case code in disk_service.rb to avoid dropping too many files in one directory, but I haven't sussed out the fix yet.