Missing uploaded files, maybe because of too many files open

[matismasters]

Missing uploaded files, maybe because of too many files open

[qrush]

Anything and everything you can post will help with debugging this issue. http://gist.github.com

[ashleym1972]

Any movement on this issue? I'm seeing this behavior in my production ENV where we will have 60,000+ open files, most of them RackMultipart temp files that are days old.

I'm using nginx + passenger, but given the RackMultipart file handling I would expect this problem anywhere.

[pcg79]

I recently added paperclip to add a file (poster) to my model (Movie). I have a script that loops over all my movies and sets the movie's poster to a file I download from the net. Everything worked fine in dev but in prod it fails after maybe 1000 movies and I lose all the files.

Here's a gist w/, I think, all the relevant info. Is this my fault - do I need to close something?

Paperclip version = 2.3.1.1
Rails version = 2.3.4
http://gist.github.com/216541

[ashleym1972]

I'm really hoping that Rails 2.3.4 solved this for us. It included a close of the multipart files that seemed to be causing our problems.

[sikachu]

I'm closing this as it has been a long while after you reported in, and we didn't have any debugging information to solve the problem.

I think the problems we see here is the Rack bug, and it's hard to fix on the Paperclip side as we're only the middleman which handles the file from Rack to your storage.

[jlecour]

I regularly have this issue, with Paperclip 2.4.2 on Rails 2.3.14

I don't know how I can help debug this.

[jlecour]

Let's say I have a Picture model with an :image attachment

In a console session, If I do this :

Picture.create(:image => File.open('/Users/jlecour/Pictures/bilde.jpeg'))

With lsof in another terminal, I can see 2 open files :

ruby      97169 jlecour    7r     REG       14,2     35308   651662 /Users/jlecour/Pictures/bilde.jpeg
ruby      97169 jlecour    9u     REG       14,2     35308 16757967 /private/var/folders/9X/9XWGo2YVHP0iDllOTi886E+++TI/-Tmp-/stream20111007-97169-78vibh-0.jpeg

The Picture has been created, without errors, but the files handlers are not closed until I end my console (IRB) session.

That's why whis a long running rake task which deals with hundreds of files, I have the "Too many open files" error pretty fast.

I continue to dig and keep you posted

[jlecour]

I've cloned Paperclip and made my app using this local clone (master points now to #f7fdc9f13e348a760ca6ad3b068a66ce3773cb83). I've put some traces and I can see that some of the temp files are well closed (and destroyed), but the original file (the one passed in the create method) and another temp file are not closed at all.

[jlecour]

The temp file that is not closed and destroyed is the one created by IOStream#stream_to, called from IOStream#to_tempfile.

It comes from Attachment#assign where @queued_for_write[:original] and uploaded_file are the 2 files that are never closed.

In the ensure part of this method there is a close call on uploaded_file but only if the variable close_uploaded_fileevaluates to true. And in my case, this is never set to true.

At this point I've identified 2 problems :

1. The original file is never closed if it is a local file (as opposed to a remote file or an uploaded file) and not already an Attachment instance. If I remove the if in the ensure part, it is properly closed, but the test suite break with 2 failures and 66 errors (many "IOError: closed stream" errors)

2. If I have a style named "original", a temp file is created and never closed/destroyed.

You'll ask why I'd like to have an "original" that is not the original file. That's because some files can be really big (>1024x768), or with too high a resolution (> 72 dpi), and I don't want to store them. I've found this trick (to name a style "original"), but clearly there is an issue. Do you have any advice on that ?

[sikachu]

Reopened as request. I'll take a look at this soon.

[ikonyshev]

+1, I've got the same issue in my project

[jlecour]

Hi,

I'm sorry to push a little on this. Paperclip is an awesome project and I really appreciate the amount of work you guys put in it, but it's a real problem for me.

I know that it's an open-source project and I should submit a patch instead of complaining, but I've done the best I could.

As I've explain in my previous comment, the issue has two parts : a problem with how local files are treated (I guess it can be solved in Paperclip) and a problem with the way I'm using Paperclip to keep an altered original.

Please give me your opinion on both of these parts.

Thanks in advance.

[jyurek]

@jlecour: I'll be taking a look at this today. Thanks very much for your debugging efforts. The first part is getting a failing test so we can isolate the issue, and you've helped a lot there.

@ikonyshev: Are you also on a Rails 2 project? I wonder if the problem is related to that somehow.

[jyurek]

@jlecour: I'm trying to force a large number of open files, but even opening 100,000, I can only manage lsof to return that about 500 are open. What version of ruby are you running, and do you have any GC tuning enabled?

[jyurek]

Ah, nevermind. I spoke too soon.

[jyurek]

Doing a bit more research, if the lifecycle of the objects runs as expected, the GC should clear out any lingering objects, which will close their files. The to_tempfile'd file in assign is unset when @queued_for_write is cleared at the end of flush_writes. As long as the GC runs, those files will be closed automatically.

So, I have to ask... the files in question are all being saved, right? Do you simply have thousands up uploads at the same time? Or (to jump back a bit) do you tweak your GC settings (REE, jruby, etc)?

[jlecour]

Hi @jyurek

Thank you very much for looking at this.

I'm using REE (2011.03) on a Mac and MRI 1.8.7 on a Debian. I don't think I've tuned the GC in any of those environments. I do some tuning some times to run the GC manually in a big loop, but not here.

I don't have any uploads at all. I have some images that are downloaded and then given to Paperclip (as an IO) as if they were uploaded files.

Have you tried to reproduce the issue with the steps i've put in this comment #44 (comment) ?
If you do this and see that a file descriptor is not freed, then we are on the same track.

Feel free to ping me on twitter (@jlecour) if you want us to meet in an IRC chatroom or whatever. It might be easier to answer your questions and give you quicker feedback.

Thanks again.

[twmills]

Wow, I just hit this today after upgrading to Rails 3.1 (and upgrading all my gems in the process). Glad to see someone is actively looking into this.

[twmills]

FYI

I rolled back to Paperclip 2.3.8 and the problem went away.

[ikonyshev]

I've got the problem on rails 3.1, ruby 1.9.2 installed via rvm on linux and paperclip 2.3.16. Also I use Mongoid for storage, but I don't think that this is a problem.
The problem occurs with the rake task that crawles another site, downloads images and stores them locally. There are about 50 000 images

[ikonyshev]

After 3100 downloads there are 830 opened files.
Model has few styles defined:

has_mongoid_attached_file :image, styles: { 
                thumb: "87x100>", 
                normal: "348x400>",
                small: "174x200" },
                path: ':rails_root/public/system/:attachment/:id_partition/:style_:filename',
                url: '/system/:attachment/:id_partition/:style_:filename'

Part of lsof list:

makefashion@neuron:~/www/rails_apps/makefashion$ lsof -c rake | grep stream
rake 10851 makefashion 6u REG 251,2 2612 3275 /tmp/stream20111106-10851-ykkvdb20111106-10851-t6ro2w
rake 10851 makefashion 7u REG 251,2 14874 3296 /tmp/stream20111106-10851-1dfxoq720111106-10851-16ncmdh
rake 10851 makefashion 8u REG 251,2 858 3266 /tmp/stream20111106-10851-1cnn3s4.jpg
rake 10851 makefashion 9u REG 251,2 35324 3274 /tmp/stream20111106-10851-ykkvdb.jpg
rake 10851 makefashion 10u REG 251,2 51051 3306 /tmp/stream20111106-10851-12c07gz.jpg
rake 10851 makefashion 11u REG 251,2 330788 3269 /tmp/stream20111106-10851-1xu03oc.jpg
rake 10851 makefashion 12u REG 251,2 9471 3309 /tmp/stream20111106-10851-12c07gz20111106-10851-bpiqzk
rake 10851 makefashion 13u REG 251,2 11692 1228 /tmp/stream20111106-10851-1xu03oc20111106-10851-1h34e0a
rake 10851 makefashion 14u REG 251,2 55448 3270 /tmp/stream20111106-10851-1xu03oc20111106-10851-1pmbss4
rake 10851 makefashion 15u REG 251,2 21523 3272 /tmp/stream20111106-10851-1xu03oc20111106-10851-i9wh7e

[jyurek]

Ok I've looked into this some more, and after running some tests, I've determined that if I save my attachments, then what's choking my open files isn't Paperclip, it's the failure to close the files that are passed in to Paperclip. I won't close those for you, as I that would be presumptuous and an unexpected side effect.

If you don't save your attachments, you should be, as the Tempfile is kept around until the model is saved, whereupon it's closed when it's GC'ed.

I'm not seeing that this is a Paperclip problem at the moment. Can someone show me differently?

[jlecour]

@jyurek AFAICT You're absolutely right about this. I've looked into my code, and I definitely open a file and never close it. But I honestly don't know how I can close it.

Here is part of my code :

class Picture < ActiveRecord::Base

  attr_accessor :image_url

  before_validation :download_remote_image, :if => :image_url_provided?

  validates_presence_of :image_remote_url, :if => :image_url_provided?, :message => 'is invalid or inaccessible'
  validates_presence_of :image_file_name, :if => :image_url_provided?, :message => 'is invalid or inaccessible'

  has_attached_file :image

  private

  def image_url_provided?
    !self.image_url.blank?
  end

  def download_remote_image
    if io = do_download_remote_image
      self.image = io
      self.image_remote_url = image_url
    end
  end

  def do_download_remote_image
    require 'open-uri'

    io = open(URI.parse(image_url))
    def io.original_filename
      base_uri.path.split('/').last
    end

    io.original_filename.blank? ? nil : io
  rescue # catch url errors with validations instead of exceptions (Errno::ENOENT, OpenURI::HTTPError, etc...)
  end

end

As you can see, if I provide an image_url parameter, it is downloaded and processed as a regular uploaded file.
The file is opened with OpenURI as an "io" object, passed to Paperclip.

I'd like to close that file in a after_post_process callback, but I don't know how to access to this file handler anymore

Do you have an idea?

Thanks for your great and invaluable help.

[jyurek]

Well, the good news is that because Paperclip makes its own copy, you can assign it and then close it. So this might just be all you need:

  def download_remote_image
    if io = do_download_remote_image
      self.image = io
      self.image_remote_url = image_url
      io.close # Already copied to a Tempfile.
    end
  end

[jlecour]

Thanks @jyurek, I've put some io.close in my code and I don't have the first issue anymore :

class Picture < ActiveRecord::Base

  attr_accessor :image_url

  before_validation :download_remote_image, :if => :image_url_provided?

  validates_presence_of :image_remote_url, :if => :image_url_provided?, :message => 'is invalid or inaccessible'
  validates_presence_of :image_file_name, :if => :image_url_provided?, :message => 'is invalid or inaccessible'

  has_attached_file :image,

  private

  def image_url_provided?
    !self.image_url.blank?
  end

  def download_remote_image
    if io = do_download_remote_image
      self.image = io
      self.image_remote_url = image_url
      self.image_url = nil
      io.close if io.respond_to?(:close)
    end
  end

  def do_download_remote_image
    require 'open-uri'

    io = open(image_url)
    def io.original_filename
      if self.respond_to?(:base_uri)
        self.base_uri
      else
        self
      end.path.split('/').last
    end

    if io.original_filename.blank?
      io.close if io.respond_to?(:close)
      nil
    else
      io
    end
  rescue => ex # catch url errors with validations instead of exceptions (Errno::ENOENT, OpenURI::HTTPError, etc...)
    io.close if io.respond_to?(:close)
  end

end

Here is the diff :

--- old
+++ new
@@ -19,19 +19,31 @@
     if io = do_download_remote_image
       self.image = io
       self.image_remote_url = image_url
+      self.image_url = nil
+      io.close if io.respond_to?(:close)
     end
   end

   def do_download_remote_image
     require 'open-uri'

-    io = open(URI.parse(image_url))
+    io = open(image_url)
     def io.original_filename
-      base_uri.path.split('/').last
+      if self.respond_to?(:base_uri)
+        self.base_uri
+      else
+        self
+      end.path.split('/').last
     end

-    io.original_filename.blank? ? nil : io
-  rescue # catch url errors with validations instead of exceptions (Errno::ENOENT, OpenURI::HTTPError, etc...)
+    if io.original_filename.blank?
+      io.close if io.respond_to?(:close)
+      nil
+    else
+      io
+    end
+  rescue => ex # catch url errors with validations instead of exceptions (Errno::ENOENT, OpenURI::HTTPError, etc...)
+    io.close if io.respond_to?(:close)
   end

 end
\ No newline at end of file

[jlecour]

@jyurek Now that you've saved me once, I'm a little embarrassed to ask for your help on the second part of my issue.

The problem is that I'd like to keep a quite large version of image attachments, but not the original. For example, I sometimes get very big images, with a resolution higher than 72dpi, bigger than a regular screen, and sometimes in a CMYK mode.

That's why I use this : :convert_options => { :all => '-density 72x72 -strip -colorspace RGB -resample 72' } for my attachments.
I'd like to have a 1024x768px / 72dpi to become the original and destroy the provided picture.

It's working quite well, but I still have some file left open since what Paperclip think is the original is not really the original.

I'd be more than happy to write a use-case in the FAQ if there is a good solution for this.

Thanks.

[jyurek]

So, if I understand you right... you're overwriting the :original style, and Paperclip is holding the actual original file open as a result?

[jlecour]

@jyurek Absolutely

[jyurek]

Hmm, yeah. It would get overwritten during the processing phase. That's an issue I don't know the best solution for. I think that'll require a code change to detect if the original is being overwritten.

[jlecour]

OK, thank you very much @jyurek, you've been a great help for me.

[apolzon]

Was a fix for this ever merged into paperclip master? Or are users expected to handle the io.close themselves when they encounter this issue?

Running into this on paperclip 2.5.2 and Rails 3.0.10. Rolling back to 2.3 to see if the issue resolves. (We had encountered it occasionally on 2.3, but on 2.5 it happens every time we run a full rake -- we have integration tests that attach files, but they do not fail when run on their own).

[twmills]

This is still an issue for me on paperclip 2.7.0 when running cucumber tests. This same code works fine in paperclip 2.3.8.