Giter Site home page Giter Site logo

carrierwave-aliyun's Introduction

CarrierWave for Aliyun OSS

This gem adds support for Aliyun OSS to CarrierWave

Gem Version build

NOTE: 此 Gem 是一个 CarrierWave 的组件,你需要配合 CarrierWave 一起使用,如果你需要直接用 Aliyun OSS,可以尝试用 aliyun-sdk 这个 Gem。

NOTE: This gem is a extends for CarrierWave for allow it support use Alicloud OSS as storage backend, if you wants use Alicloud OSS directly, please visit aliyun-sdk.

Using Bundler

gem 'carrierwave-aliyun'

Configuration

You need a config/initializers/carrierwave.rb for initialize, and update your configurations:

CarrierWave.configure do |config|
  config.storage           = :aliyun
  config.aliyun_access_key_id  = "xxxxxx"
  config.aliyun_access_key_secret = 'xxxxxx'
  # 你需要在 Aliyun OSS 上面提前创建一个 Bucket
  # You must create a Bucket on Alicloud OSS first.
  config.aliyun_bucket     = "simple"
  # 是否使用内部连接,true - 使用 Aliyun 主机内部局域网的方式访问  false - 外部网络访问
  # When your app server wants deployment in Alicloud internal network, enable this option can speed up uploading by using internal networking. otherwice you must disable it.
  config.aliyun_internal   = true
  # 配置存储的地区数据中心,默认: "cn-hangzhou"
  # Which region of your Bucket.
  # config.aliyun_region     = "cn-hangzhou"
  # 使用自定义域名,设定此项,carrierwave 返回的 URL 将会用自定义域名
  # 自定义域名请 CNAME 到 you_bucket_name.oss-cn-hangzhou.aliyuncs.com (you_bucket_name 是你的 bucket 的名称)
  # aliyun_host allow you config a custom host for your Alicloud Bucket, and you also need config that on Alicloud.
  config.aliyun_host       = "https://foo.bar.com"
  # Bucket 为私有读取请设置 true,默认 false,以便得到的 URL 是能带有 private 空间访问权限的逻辑
  # Tell SDK the privacy of you Bucket, if private CarrierWave xxx.url will generate URL with a expires parameter, default: :public.
  # config.aliyun_mode = :private
end

阿里云 OSS 图片缩略图 / About the image Thumb service for Alicloud OSS

NOTE: 此方法同样支持 Private 的 Bucket 哦!

NOTE: Private Bucket also support this feature!

关于阿里云 OSS 图片缩略图的详细文档,请仔细阅读:Aliyun OSS 接入图片服务

The details of the Alicoud OSS image thumb service, please visit Alicloud OSS - Image Processing / Resize images

irb> User.last.avatar.url(thumb: '?x-oss-process=image/resize,h_100')
"https://simple.oss-cn-hangzhou.aliyuncs.com/users/avatar/12.png?x-oss-process=image/resize,h_100"
irb> User.last.avatar.url(thumb: '?x-oss-process=image/resize,h_100,w_100')
"https://simple.oss-cn-hangzhou.aliyuncs.com/users/avatar/12.png?x-oss-process=image/resize,h_100,w_100"

增对文件设置 Content-Disposition / Customize the Content-Disposition

在文件上传的场景(非图片),你可能需要给上传的文件设置 Content-Disposition 以便于用户直接访问 URL 的时候能够用你期望的文件名或原文件名来下载并保存。

In some case, you may need change the Content-Disposition for your uploaded files for allow users visit URL with direct download, and get the original filename.

这个时候你需要给 Uploader 实现 content_disposition 函数,例如:

So, you need implement a content_disposition method for your CarrierWave Uploader, for example:

# app/uploaders/attachment_uploader.rb
class AttachmentUploader < CarrierWave::Uploader::Base
  def content_disposition
    # Only for non-image files
    unless file.extension.downcase.in?(%w(jpg jpeg gif png svg))
      "attachment;filename=#{file.original_filename}"
    end
  end
end

启用全球传输加速

阿里云允许我们通过 oss-accelerate.aliyuncs.com 的节点来实现全球的传输加速,如果你的需要在境外的服务器传输到国内,或许需要开启这个功能。

你只需要将 CarrierWave Aliyun 的 aliyun_region 配置为 accelerate 即可。

config.aliyun_region = "accelerate"

异常解析

错误:OSS Transfer Acceleration is not configured on this bucket. 确保有开启 传输加速,进入 Bucket / 传输管理 / 传输加速 / 开启传输加速。

额外注意:Aliyun OSS 开启传输加速后需要 30 分钟内全网生效

carrierwave-aliyun's People

Contributors

aligo avatar eric-guo avatar giaogiaocat avatar hayeah avatar huacnlee avatar lord63 avatar poiyzy avatar qwlong avatar sishen avatar sunfjun avatar victorteokw avatar wxianfeng avatar yoogoc avatar zhangalex avatar zhufenggood avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

carrierwave-aliyun's Issues

调用 update_column 更新 image 后,url 却没有变化

华顺你好:

0.8.1 版本调用 update_column 更新 image 后,url 却没有变化,老版本 gem 应该是会有变化的。
请看看是否需要可以 Fix。

[19] pry(main)> st.image.url
=> "https://morespace-dev-public.oss-cn-beijing.aliyuncs.com/huaba/special_topic/image/4149bde7-c02a-48c8-825d-2298619863e5-1495092777.jpg_lg"
[20] pry(main)> st.update_column(:image, 'test.jpg')
  SQL (5.7ms)  UPDATE `special_topics` SET `special_topics`.`image` = 'test.jpg' WHERE `special_topics`.`id` = 1
=> true
[21] pry(main)> st.image.url
=> "https://morespace-dev-public.oss-cn-beijing.aliyuncs.com/huaba/special_topic/image/4149bde7-c02a-48c8-825d-2298619863e5-1495092777.jpg_lg"
[22] pry(main)> st.image.url

缩略图功能和github上的carrierwave不兼容

Hello, carrierwave-aliyun上的缩略图功能:
avatar.url(thumb: '@!avatar')
在github版本的carrierwave不能用

Gemfile文件:
gem 'carrierwave', github: 'carrierwaveuploader/carrierwave'

错误信息
ArgumentError: wrong number of arguments (1 for 0)

README 是否需要细化?

我希望在自己的项目中使用 云存储, 但是看到这么简单的东西,不敢用啊.

最好把一步步的操作都写上. 不知道可以吗?

Thumb not correctly parsing!

Please don't use following code to decode thumb value in bucket.rb

thumb = thumb.gsub('?', '')
parameters = Hash[*thumb.split('=')]

Allow hash for thumb, or use Rack::Utils.parse_nested_query instead.

To apply respond-content-dispotion for attachment file name, I have to do workaround as below in my uploader.

  def with_download_name(file_name)
    with_params({'response-content-disposition'=>"attachment;filename=#{file_name}"})
  end

  def with_params(params)
    @bucket ||= CarrierWave::Aliyun::Bucket.new(self)
    @client ||= ::Aliyun::OSS::Client.new(endpoint: @bucket.endpoint, access_key_id: @bucket.access_key_id, access_key_secret: @bucket.access_key_secret).get_bucket(@bucket.bucket)
    @client.object_url(self.path, true, 15.minutes, params)
  end

carrierwave 2.0 兼容依然有问题

这次测试,新增的时候,能获取到 #{model.id} 了。但还是有个小的问题:

当验证失败的时候,例如文章标题没有填写,依然会上传文件到oss

image

切换到 carrierwave 1.3.1, 问题消失

单元测试提示DEPRECATION WARNING

DEPRECATION WARNING: #tables currently returns both tables and views. 

This behavior is deprecated and will be changed with Rails 5.1 to only return tables. Use #data_sources instead. 

应当是这里报的:

  def drop_db
    ActiveRecord::Base.connection.tables.each do |table|
      ActiveRecord::Base.connection.drop_table(table)
    end
  end

可以参考这里进行修改:
DatabaseCleaner/database_cleaner#412
https://github.com/DatabaseCleaner/database_cleaner/pull/415/files

含中文的文件名会上传失败

因为 carrierwave-aliyun 不支持设置 Content-Disposition 的,但是又想在下载的时候直接使用原文件名。
Carrierwave 默认 会过滤掉非拉丁字符的,按照官方的文档说法是,加上这一句,可以关闭过滤

CarrierWave::SanitizedFile.sanitize_regexp = /[^[:word:]\.\-\+]/

关闭过滤之后,文件名没问题,但是上传会失败,错误信息是:

Aliyun::Oss::RequestError (The request signature we calculated does not match the signature you provided. Check your key and signing method.)

我调试了一下发现,是 URI.encode(path) 导致的,如果不 encode 的话就没问题

服务器端上传报错 Put file failed: undefined method `seek' for @buffer: ""...0 bytes:Aliyun::OSS::HTTP::StreamWriter

class Api::UsersController < Api::BaseController
  def upload_avatar
    current_user.update(avatar: params[:avatar])
    render json: current_user.reload
  end
end

baseuploader

class BaseUploader < CarrierWave::Uploader::Base
  # Include RMagick or MiniMagick support:
  # include CarrierWave::RMagick
  include CarrierWave::MiniMagick

  # Choose what kind of storage to use for this uploader:
  if Rails.env.production?
    storage :aliyun
  else
    # storage :file
    storage :aliyun
  end

  def filename
    if original_filename
      file_type = File.extname(original_filename.downcase)
      file_name = Digest::MD5.hexdigest(original_filename.downcase)
      "#{Time.now.strftime('%Y%m%d%H')}#{file_name}#{file_type}"
    end
  end
end

在本地可以,两台不同的服务器上测试不ok

重载filename有问题

重载了filename方法:

def filename
"#{DateTime.now().strftime("%Y%m%d")}-#{SecureRandom.hex.first(12)}" if original_filename
end

然后最终上传到阿里云的文件名和保存到数据库的是完全的两个名称。并不一样

string contains null byte

def aliyun_upload file,target
    _connection = CarrierWave::Storage::Aliyun::Connection.new({
      :aliyun_access_id=> Rails.application.config.aliyun_access_id,
      :aliyun_access_key=> Rails.application.config.aliyun_access_key,
      :aliyun_bucket=> Rails.application.config.aliyun_bucket,
      :aliyun_area=> Rails.application.config.aliyun_area,
      :aliyun_upload_host=> Rails.application.config.aliyun_upload_host
    })
    _connection.put(target, file)
  end

  def upload_remote(remote_src,filename,dir)
    require 'open-uri' 
    web_contents  = open(remote_src).read
    aliyun_upload web_contents,"#{dir}/#{filename}"
  end

当这里的 remote_src 传入一个网络图片地址的时候就报这个错了:

string contains null byte

No implicit conversion of nil into String

Thanks for your brilliant gem package.

When I was uploading file, I got an error from this gem package.

no implicit conversion of nil into String

carrierwave/storage/aliyun.rb:136:in `digest'
carrierwave/storage/aliyun.rb:136:in `sign'
carrierwave/storage/aliyun.rb:55:in `put'
carrierwave/storage/aliyun.rb:200:in `store'
carrierwave/storage/aliyun.rb:232:in `store!'
carrierwave (0.10.0) lib/carrierwave/uploader/store.rb:59:in `block in store!'

My config is totally correct, and rigidly followed the instruction described in README.md

CarrierWave.configure do |config|
  config.storage = :aliyun
  config.aliyun_access_id = "my_access_id"
  config.aliyun_access_key = "my_access_key"
  config.aliyun_bucket = "my_bucket"
end

And my uploader:

class ImageUploader < CarrierWave::Uploader::Base
  storage :aliyun
end

#security# Attacker upload extension whitelist file with content type `text/html` 文件头安全漏洞

content_type: new_file.content_type,

通过该行代码,覆盖了OSS原生的自动识别content-type功能。
攻击者通过修改上传数据的content-type上传了后缀为.png的白名单后缀文件,简单利用curl即可。

--data-raw $'------WebKitFormBoundaryKYyQo1yApqB1pdPN\r\nContent-Disposition: form-data; name="image_file"; filename="attack.png"\r\nContent-Type: text/html\r\n\r\n\r\n------WebKitFormBoundaryKYyQo1yApqB1pdPN--\r\n' \

0.1.5有点问题

CarrierWave.configure do |config|
config.storage = :aliyun
config.aliyun_access_id = "xxxxxx"
config.aliyun_access_key = 'xxxxxx'

你需要在 Aliyum OSS 上面提前创建一个 Bucket

config.aliyun_bucket = "1chi"

是否使用内部连接,true - 使用 Aliyun 局域网的方式访问 false - 外部网络访问

config.aliyun_internal = true

使用自定义域名,设定此项,carrierwave 返回的 URL 将会用自定义域名

自定于域名请 CNAME 到 you_bucket_name.oss.aliyuncs.com (you_bucket_name 是你的 bucket 的名称)

config.aliyun_host = "1�chi.oss.aliyuncs.com"
end
我按照这么设置,图片的链接地址就不对,需要onfig.aliyun_bucket = "" 才能访问,但是这样又会导致不能上传图片

Cannot generate uploader

bundle exec rails generate uploader Image
/usr/local/lib/ruby/gems/2.2.0/gems/carrierwave-aliyun-0.3.3/lib/carrierwave/storage/aliyun.rb:3:in `require': cannot load such file -- digest/hmac (LoadError)
    from /usr/local/lib/ruby/gems/2.2.0/gems/carrierwave-aliyun-0.3.3/lib/carrierwave/storage/aliyun.rb:3:in `<top (required)>'
    from /usr/local/lib/ruby/gems/2.2.0/gems/carrierwave-aliyun-0.3.3/lib/carrierwave-aliyun.rb:1:in `require'
    from /usr/local/lib/ruby/gems/2.2.0/gems/carrierwave-aliyun-0.3.3/lib/carrierwave-aliyun.rb:1:in `<top (required)>'
    from /usr/local/lib/ruby/gems/2.2.0/gems/bundler-1.7.12/lib/bundler/runtime.rb:76:in `require'
    from /usr/local/lib/ruby/gems/2.2.0/gems/bundler-1.7.12/lib/bundler/runtime.rb:76:in `block (2 levels) in require'
    from /usr/local/lib/ruby/gems/2.2.0/gems/bundler-1.7.12/lib/bundler/runtime.rb:72:in `each'
    from /usr/local/lib/ruby/gems/2.2.0/gems/bundler-1.7.12/lib/bundler/runtime.rb:72:in `block in require'
    from /usr/local/lib/ruby/gems/2.2.0/gems/bundler-1.7.12/lib/bundler/runtime.rb:61:in `each'
    from /usr/local/lib/ruby/gems/2.2.0/gems/bundler-1.7.12/lib/bundler/runtime.rb:61:in `require'
    from /usr/local/lib/ruby/gems/2.2.0/gems/bundler-1.7.12/lib/bundler.rb:134:in `require'
    from /path/to/my/project/config/application.rb:7:in `<top (required)>'
    from /usr/local/lib/ruby/gems/2.2.0/gems/spring-1.2.0/lib/spring/application.rb:82:in `require'
    from /usr/local/lib/ruby/gems/2.2.0/gems/spring-1.2.0/lib/spring/application.rb:82:in `preload'
    from /usr/local/lib/ruby/gems/2.2.0/gems/spring-1.2.0/lib/spring/application.rb:143:in `serve'
    from /usr/local/lib/ruby/gems/2.2.0/gems/spring-1.2.0/lib/spring/application.rb:131:in `block in run'
    from /usr/local/lib/ruby/gems/2.2.0/gems/spring-1.2.0/lib/spring/application.rb:125:in `loop'
    from /usr/local/lib/ruby/gems/2.2.0/gems/spring-1.2.0/lib/spring/application.rb:125:in `run'
    from /usr/local/lib/ruby/gems/2.2.0/gems/spring-1.2.0/lib/spring/application/boot.rb:18:in `<top (required)>'
    from /usr/local/lib/ruby/2.2.0/rubygems/core_ext/kernel_require.rb:54:in `require'
    from /usr/local/lib/ruby/2.2.0/rubygems/core_ext/kernel_require.rb:54:in `require'
    from -e:1:in `<main>'

See this:
mailgun/mailgun-ruby#15

Private Bucket OSS Image Process Host Incorrect

According to readme, I get a image process url for a private bucket as following.

irb(main):076:0> private_uploader.url
=> "https://example.oss-cn-hangzhou.aliyuncs.com/tmp/b38106444a722146b05e0758dd0824c620200724.jpeg?Expires=1595819193&OSSAccessKeyId=3FtQWy4i5PErTleA&Signature=L0dUDTX0%2BP1Ko%2FO6VlevAA9Nhck%3D"
irb(main):077:0> private_uploader.url(thumb: '?x-oss-process=image/resize,h_100')
=> "https://example.img-cn-hangzhou.aliyuncs.com/tmp/b38106444a722146b05e0758dd0824c620200724.jpeg?x-oss-process=image%2Fresize%2Ch_100&Expires=1595820767&OSSAccessKeyId=3FtQWy4i5PErTleA&Signature=aE57PJo6CGCwDdSvq8ho6H4FQN4%3D"

The url is correct to download the image, but the resize image url is incorrect.
433 port for img-cn-hangzhou.aliyuncs.com is out of touch according to curl -v response.

* TCP_NODELAY set
* Connection failed
* connect to 112.124.225.142 port 443 failed: Operation timed out
* Failed to connect to example.img-cn-hangzhou.aliyuncs.com port 443: Operation timed out
* Closing connection 0

It seem like current latest version of OSS image process not support old version host https access.

Original code use hard code to set the image process host here.
https://github.com/huacnlee/carrierwave-aliyun/blob/master/lib/carrierwave/aliyun/bucket.rb#L49

Pls add a new config to support old image process, and keep new version as default.
Something like aliyun_img_process_host_prefix: 'img'

@endpoint = "https://oss-#{self.region}.aliyuncs.com"
@upload_endpoint = uploader.aliyun_internal == true ? "https://oss-#{self.region}-internal.aliyuncs.com" : @endpoint
@img_endpoint =
        if uploader.aliyun_img_process_host_prefix.blank?
            @endpoint
        else
            "https://#{uploader.aliyun_img_process_host_prefix}-#{self.region}.aliyuncs.com"
        end

判断是否文件存

比如有的时候我们判断用户头像是否存在,之前经常用 user.avatar?, 现在总是返回 false

这前中文上传下载一直都没问题。今天发现不行

上传一个文件,model里用 model.file.url和到阿里云管理平台看到的url不一样。。。。

比如上传一个"比赛顺序“

model.file.url为:

***** %E6%AF%94%E8%B5%9B%E9%A1%BA%E5%BA%8F.pdf

而阿里的url是:
***** %25E6%25AF%2594%25E8%25B5%259B%25E9%25A1%25BA%25E5%25BA%258F.pdf

carrierwave 2.0.0 support

After upgrade to carrierwave 2.0.0, meet below errors from log:

NotImplementedError (Need to implement #cache! if you want to use CarrierWave::Storage::Aliyun as a cache storage.):

version 1.3.1 works well.

上传到阿里云后,avatar?方法返回false

如果是上传到本地(storage :file),
@user.avatar? 返回 true

如果上传到阿里云的话(storage :aliyun),
在OSS的bucket里能看到图片,但是
@user.avatar? 返回 false

我现在暂时靠avatar.url来判断有无图片。

请问可能是什么原因导致的?

读取文件的时候报错

[36] pry(main)> maf.file.read
RestClient::Forbidden: 403 Forbidden
from /Users/zhanghuan/.rvm/gems/ruby-2.1.5@kt-mfame/gems/rest-client-1.8.0/lib/restclient/abstract_response.rb:74:in `return!'

可以上传和删除文件,就是读取文件内容时候报错。

rspec failures

  1. upload image时有MiniMagick错误
  2. get uploaded file时报403错误(private bucket)
  3. assert应该用expect语法,类似:expect(@photo.save).to be true

warning in ruby 2.7.0

/Users/xxx/.gem/ruby/2.7.0/gems/carrierwave-aliyun-1.1.1/lib/carrierwave/storage/aliyun_file.rb:12: warning: URI.escape is obsolete

carrierwave 2.0 兼容有问题

carrierwave-aliyun (1.1.0) 搭配使用 gem 'carrierwave', '~> 2.0'

出现两个严重问题。

1、新增数据的时候

  def store_dir
    "#{model.class.to_s.underscore}/#{mounted_as}/#{model.id}"
  end

这里使用 model.id,会直接是 ,拿不到数据

2、验证失败,照样上传

如果表单验证失败,例如文章标题没有填写之类的。
在阿里OSS中,照样能看到上传了图片上去了。

一些尝试

  • 测试过,使用 storage :file,并没有这些问题发生
  • gem 'carrierwave', '~> 1.0',改为 1.0版本后,这些问题也消失了,一切正常了。
    于是我现在使用的是 carrierwave 1.0 搭配 carrierwave-aliyun 1.1.0

支持自定义 File object Headers.

有些时候,有自定义文件头的需求,例如目前已经支持的对Content-Disposition自定义的支持,但如果要定义其他头就做不到:
screenshot

如果能提供一个比较通用的方式来自定义文件,就不会有任何限制了。
可以考虑在配置carrierwave的时候来处理文件头的自定义操作:

CarrierWave.configure do |config|
  config.custom_headers    = -> (uploader) {
    file = uploader.file
    {}.tap do |headers|
     # OSS 支持`application/rss+xml`的 gzip,但是`application/xml` 却不行
      headers[:content_type] = 'application/rss+xml' if file.content_type == 'application/xml'
      unless file.extension.downcase.in?(%w(jpg jpeg gif png svg))
        headers[:endcontent_disposition] = "attachment;filename=#{file.original_filename}"
      end
    end
  }
end

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.