Async conversion to webp fails

[mtgrosser]

Async conversion to webp fails

[mtippmann]

probably filesystem related issue - replacing

    new_file = file.sub(/\.(jpg|png)\z/i, "-#{Time.now.to_i}.webp")

with

    new_file = file.sub(/\.(jpg|png)\z/i, "-#{('a'..'z').to_a.shuffle[0..10].join}.webp")

creates all files - the machine is too fast and overwrites files due to the same timestamp and this likely causes the 0-byte issue. using Time.now.to_i creates less files on my machine.

[mtgrosser]

Could be a threading problem caused by imlib2 internals. Maybe the implementation of the webp loader is not thread safe?

Have you tried wrapping everything that happens inside perform with a Mutex? Based on the guess that ConvertJob is an ActiveJob::Base, something along the lines of

class ConvertJob < ApplicationJob
  MUTEX = Mutex.new

  def perform(*args)
    MUTEX.synchronize do
      load_resize_and_save_your_image
    end
  end
end

[jewe]

Thank you!
Unfortunately that doesn't work. Same behaviour: the file is created but has a size of zero. No errors...
In another test with delayed_jobs instead of sidekiq everything worked async...

[mtgrosser]

Tested with the following snippet and imlib2 v1.7.0-8 and could not reproduce the problem:

# test.rb
require 'sidekiq'
require 'rszr'

# Start up sidekiq via
# sidekiq -r ./test.rb
# and then you can open up an IRB session like so:
# irb -r ./test.rb
# where you can then say
# 10.times { ResizeWorker.perform_async "test.png" }

class ResizeWorker
  include Sidekiq::Worker

  def perform(file)
    image = Rszr::Image.load(file)
    new_file = file.sub(/\.(jpg|png)\z/i, "-#{Time.now.to_i}.webp")
    image.save(new_file)
    puts "Saved #{new_file}"
  end
end

Which OS, Ruby and imlib2 versions are you using?

[jewe]

Same result with your snippet.

dpkg -l | grep imlib2
ii  libimlib2:amd64                        1.6.1-1ubuntu0.1                              amd64        
ii  libimlib2-dev                          1.6.1-1ubuntu0.1                              amd64       

ruby -v
ruby 2.7.1p83 (2020-03-31 revision a0c7c23c9c) [x86_64-linux]

lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 20.04.3 LTS
Release:	20.04
Codename:	focal

[mtgrosser]

Same result meaning it works or it doesn't work on your setup?

[jewe]

Sorry - no, it doesn't work.
But in my local development environment everything works.
It is almost the same setup as vm:

dpkg -l | grep imlib2
ii  libimlib2:amd64                      1.6.1-1ubuntu0.1                  amd64       
ii  libimlib2-dev                        1.6.1-1ubuntu0.1                  amd64

ruby -v
ruby 2.7.1p83 (2020-03-31 revision a0c7c23c9c) [x86_64-linux]

lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 20.04 LTS
Release:        20.04
Codename:       focal

[mtgrosser]

Which are the versions of libwebp?

[jewe]

not working:

dpkg -l | grep libwebp
ii  libwebp-dev:amd64                      0.6.1-2ubuntu0.20.04.1                        amd64        
ii  libwebp6:amd64                         0.6.1-2ubuntu0.20.04.1                        amd64        
ii  libwebpdemux2:amd64                    0.6.1-2ubuntu0.20.04.1                        amd64        
ii  libwebpmux3:amd64                      0.6.1-2ubuntu0.20.04.1                        amd64        

But same lib versions on my dev system.

[mtgrosser]

Imlib2's webp loader has been refactored alot since your version of imlib2 1.6.1. You could try and build a more recent version of the library to see if the problem persists.

[mtgrosser]

The file I used was sufficiently large for the resizing to take several seconds, but you're right, to_i in my test snippet was sort of a bad idea for a fast resizer.

[jewe]

Sorry, i have determined that the problem was caused by a misconfiguration of sidekiq. There was a timeout set, which stopped the job prematurely. So the problem was not in the rszr library.
Sorry for the confusion and thank you for the support!