pageflow doesn't "find" "zencoded" content

[codevise]

pageflow doesn't "find" "zencoded" content

[kioopi]

Hi,

the PollZencoderJob tries to fetch the thumbnail and poster images that Zencoder put into S3 using the s3_host_alias. (s3_host_alias, old friend)
If this is configured incorrectly or the bucket doesn't allow website hosting, this will fail and be repeated.

You can see which URL expected by opening a rails console:
# rails c
and looking at your VideoFile directly:
Pageflow::VideoFile.find(1).zencoder_thumbnail.url
Try to open this URL in a browser to confirm that it is possible.

Greetings

[digitalkram]

Hi

the host alias must be fine because (besides the additional -out) it is the same for the mainbucket and also zencoding and S3 upload and output uploading to S3 works. Static website hosting is enabled.

The URL which is returned in rails console when I execute the command you suggested above is working. When opened in a browser I can see the image.

Thanks. Do you need more info?

Cheers

[kioopi]

Hi,
very hard to tell.

There are several ways PollZencoderJob can stay pending. The call throw(:halt, :pending) puts the job in the pending state to be re-tried later.

You could try use the console to re-trace what this job does and see if there are any errors thrown:

file = Pageflow::VideoFile.find(1)
api = Pageflow::ZencoderApi.instance

info = api.get_info(file.job_id)
info[:finished]

URI.parse(file.zencoder_thumbnail.url)
URI.parse(file.zencoder_poster.url)

api.get_input_details(file.job_id)

Post the results if you're unclear about it.
Greetings

[digitalkram]

Hi

Thanks for your response. I did the checks on the console as you suggested. For me as a naive end-user it looks good.

The two URI parse results are working and are accessible in a browser. Zencoder is done. The input details are correct.

I did some obfuscation out the URLs. But they work. Here is the log:

Loading development environment (Rails 4.0.2)
irb(main):001:0> file = Pageflow::VideoFile.find(1)
  Pageflow::VideoFile Load (12.3ms)  SELECT "pageflow_video_files".* FROM "pa
geflow_video_files" WHERE "pageflow_video_files"."id" = ? LIMIT 1  [["id", 1]
]
=> #<Pageflow::VideoFile id: 1, entry_id: 1, uploader_id: nil, attachment_on_
filesystem_file_name: nil, attachment_on_filesystem_content_type: nil, attach
ment_on_filesystem_file_size: nil, attachment_on_filesystem_updated_at: nil,
attachment_on_s3_file_name: "Test.mov", attachment_on_s3_content_type: "video
/quicktime", attachment_on_s3_file_size: 6764988, attachment_on_s3_updated_at
: "2014-05-12 18:34:30", job_id: 93530574, state: "encoding", encoding_progre
ss: nil, created_at: "2014-05-12 18:34:25", updated_at: "2014-05-12 18:34:43"
, encoding_error_message: nil, thumbnail_file_name: nil, width: nil, height:
nil, duration_in_ms: nil, format: nil, poster_file_name: nil, poster_content_
type: nil, thumbnail_content_type: nil, rights: "">



 irb(main):002:0> api = Pageflow::ZencoderApi.instance=> #<Pageflow::ZencoderApi:0x00000003afxxxx>



 irb(main):003:0>irb(main):004:0*
irb(main):005:0* info = api.get_info(file.job_id)
=> {:state=>"finished", :progress=>nil, :finished=>true}



 irb(main):009:0* URI.parse(file.zencoder_thumbnail.url)=> #<URI::HTTP:0x000000039e09a0 URL:http://de-xxxx-pageflow-development-
out.s3-website-eu-west-1.amazonaws.com/v1/xxxxx/pageflow/video_files/000/
000/001/thumbnail-0.jpg>
irb(main):010:0>
irb(main):011:0*

irb(main):010:0>irb(main):011:0*
irb(main):012:0* URI.parse(file.zencoder_poster.url)
=> #<URI::HTTP:0x000000039f0698 URL:http://de-xxxxxx-pageflow-development-
out.s3-website-eu-west-1.amazonaws.com/v1/xxxxxx/pageflow/video_files/000/
000/001/poster-0.jpg>
irb(main):013:0>
irb(main):014:0*
irb(main):015:0*

irb(main):015:0*
irb(main):016:0* api.get_input_details(file.job_id)
=> {:format=>"mpeg4", :duration_in_ms=>4767, :width=>1920, :height=>1080}
irb(main):017:0>
irb(main):018:0*

[digitalkram]

Only thing that looks suspicious is

state: "encoding"

In output of first command. Which is not true as Zencoder is successfully done..

[kioopi]

This is correct. When your VideoFile entered the state "encoding" the PollZencoderJob was triggered. When the job is successful the state will be "encoded". If it returns "pending" it will be retried.

You can see the definition of that in EncodedFileStateMachine.

The relevant part:

job PollZencoderJob do
  on_enter 'encoding'
  result :pending, :retry_after => 2.seconds
  result :ok => 'encoded'
  result :error => 'encoding_failed'
end

So, everything looks well, i'm at a loss, now. Have you tried restarting your Rails server and the Resque processes?

[digitalkram]

Restart didn't help. Tried deletion and RE upload as well. Anything I can do to debug the PollZencoderJob? Running it's steps manually or something? Or some log to check / share?

[digitalkram]

I tried not just a restart by now but also a fresh setup hoping that I messed something up during my error-prone setup. So I deleted all disk content, everything on S3 and uninstalled the pageflow gems (from x3 to x5). After re-setup I validated that picture processing did work. Trying an audio file I end up with the same problem.

Looking at the job status in zencoder I see "finished"

Input:

http://de-xxx-pageflow-development.s3.amazonaws.com/xxx/audio_files/attachment_on_s3s/000/000/001/original/Tocas_Miracle.mp3

Output:

http://de-xxx-pageflow-development-out.s3.amazonaws.com/v1/xxx/pageflow/audio_files/000/000/001/audio.m4a
http://de-xxx-pageflow-development-out.s3.amazonaws.com/v1/xxx/pageflow/audio_files/000/000/001/audio.mp3
http://de-xxx-pageflow-development-out.s3.amazonaws.com/v1/xxx/pageflow/audio_files/000/000/001/audio.ogg

Those files are accessible and valid (content-wise).

Is there more output from zencoder expected?

Doing the steps on the console again I am wondering whether the semicolon at the end:
file.encoding_progress = info[:finished] ? 100 : info[:progress]; is okay?

When doing in on console it shows nothing. Without the semicolon the progress is set to 100 which is expected as finished is true.

I checked pageflow / app / jobs / pageflow / poll_zencoder_job.rb where you obviously copied the commands from that you suggested and looking at the output it should be really end in ":ok" and thus it should result in "encoded" state.

Is there the possibility to enable some tracing to get some more output? Any suggestions?

[tf]

The semicolon is not needed, still I don't think it's the root of the issue. Console only prints the output of the last command which is the empty command if you append ;. Still variables are assigned properly.

You could try to run the PollZencoderJob manually. In rails console run:

video_file = Pageflow::VideoFile.find(1)
Pageflow::PollZencoderJob.perform_with_result(video_file, {})

If the result also is :pending, you'll have to continue running the single steps of the job manually to see which condition is not met for the file to be ready.

If the result is :ok, it could mean that the Pageflow configuration visible in your resque worker processes is not correct for some reason.

[digitalkram]

Hi tf,

perform_with_result finishes like this:

[...]
   (162.2ms)  commit transaction
=> :ok
irb(main):003:0>

So if I read this right the result is :ok, right?

So how to check on "the Pageflow configuration visible in my resque worker processes"?

Thanks!

Cheers,

[tf]

You could amend lib/tasks/resque.rake to include debug output:

namespace :resque do
  task :setup => :environment do
    file = Pageflow::VideoFile.find(1)
    p(api.get_info(file.job_id))
    p(file.zencoder_thumbnail.url)
    # etc
  end
end

Then run bundle exec rake resque:work. it will die, because you did not specify the QUEUE, but the
debug output should be printed.

Is there something about your pageflow initializer that could make it work differently in the context of the resque worker than within the rails server process?

[digitalkram]

This is how lib/tasks/resque.rake looks now:

require 'resque/tasks'
require 'resque_scheduler/tasks'

namespace :resque do
  # Tell resque to initialize the rails app in worker processes
  task :setup => :environment
  file = Pageflow::VideoFile.find(1)
  p(api.get_info(file.job_id))
  p(file.zencoder_thumbnail.url)
end

Result:

root@hostname[/mnt/data/playground]# bundle exec rake resque:work
rake aborted!
NameError: uninitialized constant Pageflow::VideoFile
/mnt/data/playground/lib/tasks/resque.rake:7:in `block in <top (required)>'
/mnt/data/playground/lib/tasks/resque.rake:4:in `<top (required)>'
/mnt/data/playground/Rakefile:6:in `<top (required)>'
(See full trace by running task with --trace)
root@hostname[/mnt/data/playground]#

[tf]

Please have another look at my example above. The code needs to be wrapped in task :setup ... do <code> end

[digitalkram]

I am very sorry, corrected. Had to add:

api = Pageflow::ZencoderApi.instance

as well to get round " undefined local variable or method `api' ", afterwards I get:

root@hostname[/mnt/data/playground]# bundle exec rake resque:work
{:state=>"finished", :progress=>nil, :finished=>true}
"http://de-xxx-pageflow-development-out.s3-website-eu-west-1.amazonaws.com/v1/xxx/pageflow/video_files/000/000/001/thumbnail-0.jpg"
set QUEUE env var, e.g. $ QUEUE=critical,high rake resque:work
root@hostname[/mnt/data/playground]#

EDIT: I just now realized that "etc" was meant to be completed with the rest of the code from the job, right? Will do that.

EDIT2:
This is how lib/tasks/resque.rake looks now:

require 'resque/tasks'
require 'resque_scheduler/tasks'

namespace :resque do
  # Tell resque to initialize the rails app in worker processes
  task :setup => :environment do
  api = Pageflow::ZencoderApi.instance
  file = Pageflow::VideoFile.find(1)
  p(info = api.get_info(file.job_id))
  p(file.zencoder_thumbnail.url)
  p(file.encoding_progress = info[:finished] ? 100 : info[:progress])
  p(file.meta_data_attributes = api.get_input_details(file.job_id))
  p(file.thumbnail = URI.parse(file.zencoder_thumbnail.url))
  p(file.poster = URI.parse(file.zencoder_poster.url))
  end
end

Output:

root@hostname[/mnt/data/playground]# bundle exec rake resque:work
{:state=>"finished", :progress=>nil, :finished=>true}
"http://de-xxx-pageflow-development-out.s3-website-eu-west-1.amazonaws.com/v1/xxx/pageflow/video_files/000/000/001/thumbnail-0.jpg"
100
{:format=>"mpeg4", :duration_in_ms=>48321, :width=>320, :height=>240}
#<URI::HTTP:0x00000005092f90 URL:http://de-xxx-pageflow-development-out.s3-website-eu-west-1.amazonaws.com/v1/xxx/pageflow/video_files/000/000/001/thumbnail-0.jpg>
#<URI::HTTP:0x000000046e7108 URL:http://de-xxx-pageflow-development-out.s3-website-eu-west-1.amazonaws.com/v1/xxx/pageflow/video_files/000/000/001/poster-0.jpg>
set QUEUE env var, e.g. $ QUEUE=critical,high rake resque:work
root@hostname[/mnt/data/playground]#

EDIT3:
Q:
Is there something about your pageflow initializer that could make it work differently in the context of the resque worker than within the rails server process?

A:

root@hostname[/mnt/data/playground]# grep -v \#  config/initializers/pageflow.rb
ActiveAdmin.application.load_paths.unshift(Dir[Pageflow::Engine.root.join('admins')])

Pageflow.configure do |config|
  config.register_page_type(Pageflow::BuiltInPageType.background_image)
  config.register_page_type(Pageflow::BuiltInPageType.background_video)
  config.register_page_type(Pageflow::BuiltInPageType.video)
  config.register_page_type(Pageflow::BuiltInPageType.audio)
  config.register_page_type(Pageflow::BuiltInPageType.internal_links)

  config.paperclip_attachments_version = 'v1'

  config.paperclip_filesystem_root = 'tmp/attachments/production'


  config.paperclip_s3_default_options.merge!(
    :s3_credentials => {
      :bucket => 'de-xxx-pageflow-development',
      :access_key_id => 'xxx',
      :secret_access_key => 'xxx',
      :s3_host_name => 's3-eu-west-1.amazonaws.com'
    },
    :s3_host_alias => 'de-xxx-pageflow-development.s3-website-eu-west-1.amazonaws.com',
    :s3_protocol => 'http'
  )

  config.zencoder_options.merge!(
    :api_key => 'xxx',
    :output_bucket => 'de-xxx-pageflow-development-out',
    :s3_host_alias => 'de-xxx-pageflow-development-out.s3-website-eu-west-1.amazonaws.com',
    :s3_protocol => 'http',
    :attachments_version => 'v1'
  )
end

Pageflow.finalize!
root@hostname[/mnt/data/playground]#

That's pretty much the default file (with hardcoded value for xxx) in contrast to what the comments suggest. But this is just to proof that it could work and it will be re-done the right way once we managed to get this playground running.

[kioopi]

Another possibility for debugging:
You could clone codevise/pageflow before creating your Rails-app and use your local copy as the gem for Pageflow in your rails app.

If your directory looks like this:

── parent_dir
     ├── pageflow
     └── rails_app
             └── Gemfile

Your Gemfile would include the line:

gem 'pageflow', path: '../pageflow'

After that you'd run the generator, etc...

This gives you the possibility to edit the code in Pageflow to add logging output and see it when running your app.

[tf]

Actually you do not even have to rerun the generator. Simply change your
gemfile.
Am 14.05.2014 17:37 schrieb "Vangelis Tsoumenis" [email protected]:

Another possibility for debugging:

You could clone codevise/pageflow before creating your Rails-app and use
your local copy as the gem for Pageflow in your rails app.

If your directory looks like this:

── parent_dir
├── pageflow
└── rails_app
└── Gemfile

Your Gemfile would include the line:

gem 'pageflow', path: '../pageflow'

After that you'd run the generator, etc...

This gives you the possibility to edit the code in Pageflow to add logging
output and see it when running your app.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/13#issuecomment-43096853
.

[digitalkram]

HI,

thanks. Did so. Sadly I am "just" a Java/C#/PHP/bash developer and totally blank in terms of Ruby and the way pageflow is working. So it is very unlikely that'll have success trying to debug your app.

It starts with the question how I could basically print some lines on STDOUT or to a file/the log from ../pageflow-master/app/jobs/pageflow/poll_zencoder_job.rb

Tried: print "lalal", that doesn't appear on STDOUT/rails output.

[digitalkram]

I meanwhile tried a lot of stuff based on google research on how to debug ruby - followed three differnt guides. Either it didn't work at all or had no effect/output. I can't produce any debugging output from the App itself. I can only provoke errors by ruining the file which at least proofs that I am messing with the correct file. Running the Job from Console returns ":ok" so I need to debug it in-app but I am not capable to so obviously. I am totally lost.

TF - you wrote above:
If the result is :ok, it could mean that the Pageflow configuration visible in your resque worker processes is not correct for some reason.

The result is okay, so is there a way to see that "pageflow configuration" that is visible in the resque worker processes without debugging?

Could you you maybe link a tutorial that is working (the ones I found did not) or give a few examples what I can do within the zencoder pull job code to output the contents of variables (this .inspect?) to the development.log, please?

[tf]

Hi,

this is really appears to be a hard to catch issue. But I'm sure we'll figure it out eventually. :)

Here's what I suggest: Later today I will have to run through the process of configuring Zencoder in an app using Pageflow myself. Maybe I'll hit the issue and can track it down. Either way, I will add some more logging to the PollZencoderJob to make it more verbose about exactly why it thinks a file is still pending. Maybe this will narrow things down in your case, and will be helpful for every one after you struggling with the Zencoder setup.

If you would like to continue your own investigation in the meantime: Have you tried adding lines like logger.info("...") to the job? Those should end up in the log/jobs/development/<queue>.log files.

Cheers

[digitalkram]

hi!

thanks for that. Yes I tried logger.info in poll_zencoder_job.rb. But no output in the log. That is why I wasn't sure if that file is even in use. But when moving it away while rails is running I see in the logs:
NameError (uninitialized constant Pageflow::EncodedFileStateMachine::PollZencoderJob)

So in general that file is used.

root@hostname [/mnt/data/playground]# grep -C 2 ententuempel ../pageflow-master/app/jobs/pageflow/poll_zencoder_job.rb
      result = catch(:halt) do
        poll_zencoder(file, api)
        logger.info("ententuempel")
        fetch_input_details(file, api)
        fetch_thumbnail(file)
root@hostname [/mnt/data/playground]# grep ententuempel log/development.log
root@hostname [/mnt/data/playground]#

Cheers,

[digitalkram]

Re-reading your latest post I realized that i checked the wrong log.

Checking on the job logs I found out the following:

1. All job logs do not get updated regularly (are they expected to do this as long as a result is still pending?). This means the log is created once per audio/video file upload and that is it. During the first run the result is for sure pending in log/jobs/development/resizing.log
2. Also during that first run the logger.info message is not shown in the log.

Are the logs meant to be updated with the result after each try (or when the state changed at least)? If yes, that would mean that the job is polling only once after upload and not regularly (in my case)..!? Could this be the problem?

[tf]

The PollZencoderJob is expected to log into resizing.log regularly until the result is no longer :pending. Maybe you don't have the rake resque:scheduler task running!? That would explain why jobs are only run once.

[digitalkram]

resque:work was running - resque:scheduler not... argh!! :)

I am terribly sorry.. And very happy that it finally works. I think the wiki could use some update on how to schedule thsoe jobs properly.

BTW, is it possible to make them start with the application/rails directly? For now i background them after starting the following commands:
QUEUE=* bundle exec rake resque:scheduler
QUEUE=* bundle exec rake resque:work

Some other info that would enhance the doc (and that I am wondering about):

1. How to install pageflow into different environments in parallel (where is the switch to tell which environment we actually run)?
2. How to migrate/refresh from dev=>test=>prod? To check if the content/story will look the same after upgrading the gem(s)? So basically how to copy a PROD instance back to DEV or TEST (including all stories/users) to play around with it? I mean besides bucket contents...

[tf]

For development mode you can have a look at the foreman gem. It supplies an easy way to start multiple processes defined in a so called Procfile.

We'll be sure to add some more explanation about the role of the different resque rake tasks to the getting started guide and link that up more prominently.

[digitalkram]

Thanks a lot for your support!

[tf]

Glad, we found the issue!

[gpsloco]

thy1985 – may I ask you about: "After some fiddling with images which works fine now. We tried to upload some video files."

What did you do to get the image problems solved? I have aws, zencoder etc running, incl. videos, but I don't see any images, poster images etc.

Thank you

[digitalkram]

With that sentence I wanted to express that after I solved the problems I had when opening issue #11 (uploaded images were not progressed (Zencoder not invovled)) I was now trying to get audio/video file processing via Zencoder to work.