iron-io/iron_worker_ruby_ng

Next Gen Ruby gem for IronWorker

DEPRECATED

This gem is deprecated! Please find the new tools at the links below.

• Use the new Docker Workflow for creating/packaging your workers.
• Use the new IronCLI for command line tools.
• Use the iron_worker gem in your Ruby code.

Please read above for changes to the Iron.io tooling, only use the below documentation if you need it for legacy apps

Introduction

To run your code in cloud you need to do three things:

• Create code package
• Upload code package
• Queue or schedule tasks for execution

While you can use REST APIs for that, it's easier to use an IronWorker library created specifically for your language of choice, such as this gem, IronWorkerNG.

Preparing Your Environment

You'll need to register at http://iron.io/ and get your credentials to use IronWorkerNG. Each account can have an unlimited number of projects, so take advantage of it by creating separate projects for development, testing and production. Each project is identified by a unique project ID and requires your access token before it will perform any action, like uploading or queuing workers.

Also, you'll need a Ruby 1.9 interpreter and the IronWorkerNG gem. Install it using following command.

gem install iron_worker_ng

Creating A Worker

Each IronWorkerNG Ruby worker is just Ruby code. It can be as simple or as complex as you want. For example, the following is an acceptable worker:

puts "Hello Worker!"
puts "My task_id is #{@iron_task_id}"
puts "I got '#{params}' parameters"

All output to STDOUT will be logged and available for your review when your worker finishes execution.

Creating The Code Package

Before you can use IronWorker, be sure you've created a free account with Iron.io and setup your Iron.io credentials on your system (either in a json file or using ENV variables). You only need to do that once for your machine. If you've done that, then you can continue.

Since our worker will be executed in the cloud, you'll need to bundle all the necessary gems, supplementary data, and other dependencies with it. .worker files make it easy to define your worker.

# define the runtime language, this can be ruby, java, node, php, go, etc.
# also you could set version of language(check of available versions via iron_worker stacks)
runtime "ruby","2.1"
# exec is the file that will be executed:
exec "hello_worker.rb"

You can read more about .worker files here: http://dev.iron.io/worker/reference/dotworker/

Uploading the Code Package

If your .worker file is called hello.worker, then run:

iron_worker upload hello

This will upload your worker with the name "hello" so you can reference it like that when queuing up tasks for it.

Queue Up a Task for your Worker

You can quicky queue up a task for your worker from the command line using:

iron_worker queue hello

Use the -p parameter to pass in a payload:

iron_worker queue hello -p "{\"hi\": \"world\"}"

Use the --wait parameter to queue a task, wait for it to complete and print the log.

iron_worker queue hello -p "{\"hi\": \"world\"}" --wait

Queue up a task from code

Most commonly you'll be queuing up tasks from code though, so you can do this:

require "iron_worker_ng"
client = IronWorkerNG::Client.new
100.times do
   client.tasks.create("hello", "foo"=>"bar")
end

Setting Task Priority

You can specify priority of the task using --priority parameter:

iron_worker queue hello --priority 0 # default value, lowest priority
iron_worker queue hello --priority 1 --label 'medium priority task' # medium priority

Value of priority parameter means the priority queue to run the task in. Valid values are 0, 1, and 2. 0 is the default.

From code you can set the priority like it done in snippet below:

client.tasks.create("hello", some_params, priority: 2) # highest priority

Setting additional Options

You can specify not only priority:

• priority: Setting the priority of your job. Valid values are 0, 1, and 2. The default is 0.
• timeout: The maximum runtime of your task in seconds. No task can exceed 3600 seconds (60 minutes). The default is 3600 but can be set to a shorter duration.
• delay: The number of seconds to delay before actually queuing the task. Default is 0.
• label: Optional text label for your task.
• cluster: cluster name ex: "high-mem" or "dedicated". If not set default is set to "default" which is the public IronWorker cluster.

Get task status

When you call iron_worker queue X, you'll see the task ID in the output which you can use to get the status.

iron_worker info task 5032f7360a4681382838e082

Get task log

Similar to getting status, get the task ID in the queue command output, then:

iron_worker log 5032f7360a4681382838e082 --wait

Retry a Task

You can retry task by id using same payload and options:

iron_worker retry 5032f7360a4681382838e082

or

client.tasks.retry('5032f7360a4681382838e082', :delay => 10)

## Pause or Resume task processing

You can temporarily pause or resume queued and scheduled tasks processing by code name:

    iron_worker pause hello

    iron_worker resume hello

or by code:
Pause or resume for the code package specified by `code_id`.

```ruby
response = client.codes.pause_task_queue('1234567890')
response = client.codes.resume_task_queue('1234567890')

Debugging

To get a bunch of extra output to debug things, turn it on using:

IronCore::Logger.logger.level = ::Logger::DEBUG

IronWorkerNG::Code::Base API

The IronWorkerNG::Code::Base class will help you package your code for upload, but to upload it to the cloud, you'll need to use the IronWorkerNG::Client class.

initialize(*args)

Create new code package with the specified args.

code_from_workerfile = IronWorkerNG::Code::Base.new(:workerfile => 'example.worker')
code_with_name = IronWorkerNG::Code::Base.new(:exec => 'example.rb', :name => 'Example')
code_with_guessed_name = IronWorkerNG::Code::Base.new(:exec => 'example.rb')
code = IronWorkerNG::Code::Base.new

runtime()

Return the code package's runtime.

puts code.runtime

runtime=(runtime)

Sets the code package's runtime. If no runtime provided it defaults to 'ruby'.

code.runtime = 'ruby'

name()

Return the code package's name.

puts code.name

name=(name)

Sets the code package's name.

code.name = 'CoolWorker'

remote_build_command(cmd)

build(cmd)

Command which will be executed once (on worker upload). Can be used for heavy tasks like building your worker from sources. Check https://github.com/iron-io/iron_worker_examples/tree/master/binary/phantomjs for real world example.

code.remote_build_command('curl http://www.kernel.org/pub/linux/kernel/v3.0/linux-3.4.6.tar.bz2 -o linux-3.4.6.tar.bz2 && tar xf linux-3.4.6.tar.bz2')

full_remote_build(activate)

If set to true, activates full remote build mode. In this mode iron_worker will try to resolve as much things as possible at build step. For example, all gems will be installed at build step, which will allow you to use gems with native extensions.

remote

Alias for full_remote_build(true).

run()

Runs code package on your local box. Can be useful for testing.

code.run

merge_file(path, dest = '')

file(path, dest = '')

Merge the file located at path into the code package. If dest is set, it will be used as the path to a directory within the zip, into which the file will be merged. If the directory does not exist, it will be automatically created.

code.merge_file '../config/database.yml' # will be in the same directory as worker
code.merge_file 'clients.csv', 'information/clients' # will be in information/clients subdirectory

merge_dir(path, dest = '')

dir(path, dest = '')

Recursively merge the directory located at path into the code package. If dest is set, it will be used as the path to a directory within the zip, into which the directory specified by path will be merged. If dest is set but does not exist, it will be automatically created.

code.merge_dir '../config' # will be in the same directory as worker
code.merge_dir 'lib', 'utils' # will be in utils subdirectory, accessible as utils/lib

merge_deb(path)

deb(path)

Merges provided deb package into your worker. Please note that it should be x86-64 deb and we don't do any dependencies resolving. It might not work for some packages which expects to find things it predefined place (e.g. imagemagick looks for codecs in /usr/lib/ImageMagick-X.X.X/codecs). If you are uploading from non-debian OS, just use full remote build, so deb manipulations will be done on IronWorker servers. Following example brings power of pdftk to your worker.

code.merge_deb 'http://mirror.pnl.gov/ubuntu/pool/universe/p/pdftk/pdftk_1.44-3_amd64.deb'
code.merge_deb 'http://mirror.pnl.gov/ubuntu/pool/main/g/gcj-4.6/libgcj12_4.6.1-4ubuntu2_amd64.deb'

IronWorkerNG::Code::Ruby API

Specific methods for ruby runtime.

merge_exec(path, klass = nil)

exec(path, klass = nil)

Merge the exec located at path. If klass is provided, it'll try to instantiate it, set attrs from params and fire up run method when executed.

code.merge_exec 'my_worker.rb'

merge_gem(name, version = '>= 0')

gem(name, version = '>= 0')

Merge a gem with dependencies. Gems with native extensions will not be merged by default, switching to full remote build should fix this. You can use version constrains if you need a specific gem version. Please note that git and path gems aren't supported yet.

code.merge_gem 'activerecord'
code.merge_gem 'paperclip', '< 3.0.0,>= 2.1.0'

merge_gemfile(path, *groups)

gemfile(path, *groups)

Merge all gems from specified the groups in a Gemfile. Please note that this will not auto-require the gems when executing the worker.

code.merge_gemfile '../Gemfile', 'common', 'worker' # merges gems from common and worker groups

IronWorkerNG::Code::Binary API

Specific methods for binary (freeform) runtime.

merge_exec(path)

exec(path)

Merge the exec located at path.

code.merge_exec 'my_worker.sh'

IronWorkerNG::Code::Go API

Specific methods for go runtime. It'll run provided exec via 'go run'.

merge_exec(path)

exec(path)

Merge the exec located at path.

code.merge_exec 'my_worker.go'

IronWorkerNG::Code::Java API

Specific methods for java runtime.

merge_exec(path, klass = nil)

exec(path, klass = nil)

Merge the exec located at path. If class isn't provided, it'll relay on jar's manifest.

code.merge_exec 'my_worker.jar'

merge_jar(path)

jar(path)

Merge the jar located at path. It'll be added to classpath when executing your worker.

code.merge_jar 'xerces.jar'

IronWorkerNG::Code::Mono API

Specific methods for mono (.net) runtime.

merge_exec(path)

exec(path)

Merge the exec located at path.

code.merge_exec 'my_worker.exe'

IronWorkerNG::Code::Node API

Specific methods for node runtime.

merge_exec(path)

exec(path)

Merge the exec located at path.

code.merge_exec 'my_worker.js'

IronWorkerNG::Code::Perl API

Specific methods for perl runtime.

merge_exec(path)

exec(path)

Merge the exec located at path.

code.merge_exec 'my_worker.pl'

IronWorkerNG::Code::PHP API

Specific methods for PHP runtime.

merge_exec(path)

exec(path)

Merge the exec located at path.

code.merge_exec 'my_worker.php'

IronWorkerNG::Code::Python API

Specific methods for python runtime.

merge_exec(path)

exec(path)

Merge the exec located at path.

code.merge_exec 'my_worker.py'

merge_pip(name, version = '')

pip(name, version = '')

Merge a pip package with dependencies. If any pip package contains native extensions, switch to full remote build. You can use version constrains if you need a specific pip package version.

code.merge_pip 'iron_mq'
code.merge_pip 'iron_worker', '==0.2'

merge_requirements(path)

requirements(path)

Merge all libraries from a python standard requirements.txt. The requirements.txt example file after freezing may look like as follows:

Flask==0.8
Jinja2==2.6
Werkzeug==0.8.3
certifi==0.0.8
chardet==1.0.

code.merge_requirements '../requirements.txt'

Upload Your Worker

When you have your code package, you are ready to upload and run it on the IronWorker cloud.

# Initialize the client
client = IronWorkerNG::Client.new(:token => 'IRON_IO_TOKEN', :project_id => 'IRON_IO_PROJECT_ID')
# Upload the code
client.codes.create(code)

NOTE: You only need to call client.codes.create(code) once for each time your code changes.

Queue Up Tasks for Your Worker

Now that the code is uploaded, we can create/queue up tasks. You can call this over and over for as many tasks as you want.

client.tasks.create('MyWorker', {:client => 'Joe'})

Stacks

Stack is a docker image based on ubuntu + some custom software(ruby/python/mono/java...) You can use different stacks to launch your tasks. To get list of available stacks you could do:

client.stacks_list

or using cli

iron_worker stacks

And to specify stack add following line in your .worker file

runtime 'ruby', '2.1'

Or

exec 'java.sh'
runtime 'binary'
stack 'java-1.7'
exec 'java.sh'

The Rest of the IronWorker API

IronWorker::Client

You can use the IronWorkerNG::Client class to upload code packages, queue tasks, create schedules, and more.

initialize(options = {})

Create a client object used for all your interactions with the IronWorker cloud.

client = IronWorkerNG::Client.new(:token => 'IRON_IO_TOKEN', :project_id => 'IRON_IO_PROJECT_ID')

codes.list(options = {})

Return an array of information about uploaded code packages. Visit http://dev.iron.io/worker/reference/api/#list_code_packages for more information about the available options and the code package object format.

client.codes.list.each do |code|
  puts code.inspect
end

codes.get(code_id)

Return information about an uploaded code package with the specified ID. Visit http://dev.iron.io/worker/reference/api/#get_info_about_a_code_package for more information about the code package object format.

puts client.codes.get('1234567890').name

codes.create(code)

Upload an IronWorkerNG::Code::Ruby object to the IronWorker cloud.

client.codes.create(code)

codes.delete(code_id)

Delete the code package specified by code_id from the IronWorker cloud.

client.codes.delete('1234567890')

codes.revisions(code_id, options = {})

Get an array of revision information for the code package specified by code_id. Visit http://dev.iron.io/worker/reference/api/#list_code_package_revisions for more information about the available options and the revision objects.

client.codes.revisions('1234567890').each do |revision|
  puts revision.inspect
end

codes.download(code_id, options = {})

Download the code package specified by code_id and return it as an array of bytes. Visit http://dev.iron.io/worker/reference/api/#download_a_code_package for more information about the available options.

data = client.codes.download('1234567890')

tasks.list(options = {})

Retrieve an array of information about your workers' tasks. Visit http://dev.iron.io/worker/reference/api/#list_tasks for more information about the available options and the task object format.

client.tasks.list.each do |task|
  puts task.inspect
end

tasks.get(task_id)

Return information about the task specified by task_id. Visit http://dev.iron.io/worker/reference/api/#get_info_about_a_task for more information about the task object format.

puts client.tasks.get('1234567890').code_name

tasks.create(code_name, params = {}, options = {})

Queue a new task for the code package specified by code_name, passing the params hash to it as a payload and returning a task object with only the id field filled. Visit http://dev.iron.io/worker/reference/api/#queue_a_task for more information about the available options.

task = client.tasks.create('MyWorker', {:client => 'Joe'}, {:delay => 180})
puts task.id

tasks.cancel(task_id)

Cancel the task specified by task_id.

client.tasks.cancel('1234567890')

tasks.cancel_all(code_id)

Cancel all tasks for the code package specified by code_id.

client.tasks.cancel_all('1234567890')

tasks.log(task_id)

Retrieve the full task log for the task specified by task_id. Please note that log is available only after the task has completed execution. The log will include any output to STDOUT.

puts client.tasks.log('1234567890')

tasks.set_progress(task_id, options = {})

Set the progress information for the task specified by task_id. This should be used from within workers to inform you about worker execution status, which you can retrieve with a tasks.get call. Visit http://dev.iron.io/worker/reference/api/#set_a_tasks_progress for more information about the available options.

client.tasks.set_progress('1234567890', {:msg => 'Still running...'})

tasks.wait_for(task_id, options = {})

Wait (block) while the task specified by task_id executes. Options can contain a :sleep parameter used to modify the delay between API invocations; the default is 5 seconds. If a block is provided (as in the example below), it will be called after each API call with the task object as parameter.

client.tasks.wait_for('1234567890') do |task|
  puts task.msg
end

schedules.list(options = {})

Return an array of scheduled tasks. Visit http://dev.iron.io/worker/reference/api/#list_scheduled_tasks for more information about the available options and the scheduled task object format.

client.schedules.list.each do |schedule|
  puts schedule.inspect
end

schedules.get(schedule_id)

Return information about the scheduled task specified by schedule_id. Visit http://dev.iron.io/worker/reference/api/#get_info_about_a_scheduled_task for more information about the scheduled task object format.

puts client.schedules.get('1234567890').last_run_time

schedules.create(code_name, params = {}, options = {})

Create a new scheduled task for the code package specified by code_name, passing the params hash to it as a data payload and returning a scheduled task object with only the id field filled. Visit http://dev.iron.io/worker/reference/api/#schedule_a_task for more information about the available options.

schedule = client.schedules.create('MyWorker', {:client => 'Joe'}, {:start_at => Time.now + 3600, :run_every =>60, :priority => 0, :run_times => 100, :end_at: Time.now + 2592000, Time.now + 84600})
puts schedule.id

Scheduling Options

• run_every: The amount of time, in seconds, between runs. By default, the task will only run once. run_every will return a 400 error if it is set to less than 60.
• end_at: The time tasks will stop being queued.
• run_times: The number of times a task will run.
• priority: Setting the priority of your job. Valid values are 0, 1, and 2. The default is 0. Higher values means tasks spend less time in the queue once they come off the schedule.
• start_at: The time the scheduled task should first be run.
• timeout: The maximum runtime of your task in seconds. No task can exceed 3600 seconds (60 minutes). The default is 3600 but can be set to a shorter duration.
• label: Optional label for adding custom labels to scheduled tasks.
• cluster: cluster name ex: "high-mem" or "dedicated". This is a premium feature for customers to have access to more powerful or custom built worker solutions. Dedicated worker clusters exist for users who want to reserve a set number of workers just for their queued tasks. If not set default is set to "default" which is the public IronWorker cluster.

schedules.update(schedule_id, options = {})

Update a scheduled task specified by id

client.schedules.update('545b3cb829acd33ea10016e4', {label: 'new_label'})

Or you can update a scheduled task for your worker from the command line using:

iron_worker update schedule 545b3cb829acd33ea10016e4 -s '{"label": "new_label"}'

schedules.cancel(schedule_id)

Cancel the scheduled task specified by schedule_id.

client.schedules.cancel('1234567890')

patch your worker using cli

If you have an uploaded worker named super_code with files qux.rb, bar.rb, etc. and want to replace the content of bar.rb with a local file foo.rb, qux.rb with baz.rb just run a command:

iron_worker patch super_code -p 'foo.rb=bar.rb,baz.rb=lib/qux.rb.rb,foo.rb,foo2.rb'

No need to pass the same two file names foo.rb=foo.rb, only one foo.rb would be enough. Normally the patched version is put in place of the originals.

iron_worker_ruby_ng's Issues

Show warnings when files are included but can't be found

add way to run test suite for all backends of 'rest' gem

Output the task id on iron_worker queue

@Gissues:{"order":50,"status":"inprogress"}

document :workerfile option in README

eg: IronWorkerNG::Code::Base.new(workerfile: "workers/sleepy.worker")

Schedule update

Is there a quick and elegant way to reschedule (update) workers without having to unschedule the old one and schedule the new one?

Beautify CLI output

AFTER

iron_worker upload hello

------> Detecting Configuration
Found configuration in working directory
Working with project 'Test Project' (project_id: '4e25e1d25c0dd27801000283')
------> Discovering .worker file
Found .worker file 'hello.worker'
[Ruby | PHP | .NET | Go | Binary] worker detected with path = 'hello_worker.rb'
Class name has been set to 'MyTestWorker'
------> Creating worker package
Worker upload started
------> Worker deployed with version r15
https://hud.iron.io/tq/projects/4e25e1d25c0dd27801000283/code/4fc7e44451ef0d2807087604

BEFORE

iron_worker upload hello

Working with project 'Test Project' ('4e25e1d25c0dd27801000283')
Using workerfile 'hello.worker'
Merging ruby exec with path='hello_worker.rb' and class=''
Worker 'hello' upload started
Worker 'hello' uploaded

CLI: Output the project_id when uploading/queuing

We've had several people that don't know what's going on, but turns out it's picking up the wrong credentials. It should say: "Uploading to project: #{project_id}..." or something.

Re: #63

Add a project_id flag

It feels a lot more natural to have a flag to set it and it's a very common thing. Don't need one for token though.

Example:

iron_worker upload -project_id MYID my_worker

Something like that.

@Gissues:{"order":50,"status":"notstarted"}

Add a logger method to access logger or change log level

Not through core, but something like:

IronWorkerNG.logger.level = Logger::DEBUG

Full remote build given worker url

ie: https://github.com/iron-io/iron_worker_ruby_ng/pull/64/files#L9R46

Install required dependencies based on .worker file

eg:

• clone a repo
• iron_worker install hello which will read file and install dependencies (ie: gem's for a ruby worker).

Better Documentation for limit errors

I tried scheduling a task to run every 30 seconds and API returned

400 Bad Request (Rest::Wrappers::RestClientExceptionWrapper)

Took me 15 minutes to figure out that run_every's minimum is 60 seconds. Needs to be handled here as well as in our docs.

README update: split apart upload from task creation so people don't upload and queue every time

Make it two separate boxes or something.

Add more info to codes.create command line output

Current:

treeder@ubuntu:~/dev/tmp/myapp$ iron_worker upload hello
I, [2012-06-06T13:47:29.499810 #14685]  INFO -- IronWorkerNG: Merging ruby exec with path='hello_worker.rb' and class=''
I, [2012-06-06T13:47:29.499953 #14685]  INFO -- IronWorkerNG: Fixating gems dependencies

• First, get rid of the cruft like other ticket says.
• Should say the following steps:
  • "Building package..."
  • "Uploading package to project id #{ID} (X bytes)..."
  • "Upload successful. Code name: {NAME} -- Code id: {ID}..."

@Gissues:{"order":87.5,"status":"notstarted"}

ruby_ng README: include passing args in in-code example

In README document how to pass in an API argument such as priority. Current not shown here.

Needs retry / exponential backoff

lib API change

Library API was changed by commit romand@b0d4801

The way code objects are created have been changed: instead of

  code = Code::Ruby.new(:workerfile => 'common.worker') do
    name 'asdf'
    exec 'worker.rb'
  end

or

  code = Code::Creator.create do
    runtime 'ruby'
    ...
  end

now we write

  code = Code.new(:runtime => 'ruby') do
    ...
  end

or

  code = Code.new do
    runtime 'ruby'
    ...
  end

or

  code = Code.new
  code.runtime 'ruby'
  ...

or

  code = Code.new(:workerfile => 'cool.worker') # with runtime set in workerfile

Before the change, block passed to Code::Creator.create was executed twice:

• at first, to detect runtime and determine corresponding subclass (Code::Ruby or Code::Binary or ...)
• second, within that subclass constructor (Code::Ruby.new or Code::Binary.new or ...)

Workerfiles content was executed twice, too.

In order to eliminate such unexpected and buggy behavior, code of subclasses was moved to modules with corresponding names, and those modules are mixed in code object when 'runtime' method is called.

As the result,

• the only 1 code class remains (Code) instead of hierarchy (Code::Base, Code::Ruby, ...)
• Code.new is the single code objects creation facility, with many good ways to specify runtime
• bug with duplicate block and workerfile evaluation is fixed
• library API is changed (have been cut)

The one way to protect customers from suddenly broken code, we can

  class Code
    class Ruby
      def initialize(*args, &block)
        IronCore::Logger.warn 'Code::Ruby.new is deprecated, call Code.new(:runtime => "ruby") instead'
        Code.new(...)
      end
    end
  end

Another way is just to document this change (in changelog, readme ...) and let customers make (quick!) fix in their code.

@iced @treeder please discuss

CLI --help shouldn't require pid/token

[/Users/chad/workspace/iron/hud/workers] # iron_worker queue --help

Using typhoeus gem.
E, [#330] ERROR -- IronWorkerNG: Both token and project_id must be specified
/usr/local/lib/ruby/gems/1.9.1/gems/iron_worker_ng-0.7.4/lib/iron_worker_ng/api_client.rb:21:in `initialize': Both token and project_id must be specified (IronCore::IronError)

On error responses, try to get "msg" value from body and display it to user

Impossible for user to debug otherwise.

Document CLI tool and .worker files

In README, bring the core methods to the top

ie:

• upload
• queue
• schedule
• log

It's hard to find what's important in the readme.

@Gissues:{"order":85.71428571428572,"status":"backlog"}

Full remote build usable example

A follow up to #79 is to get this example: https://github.com/iron-io/iron_worker_examples/tree/master/ruby_ng/image_processing

working like this:

• iron_worker upload https://github.com/iron-io/iron_worker_examples/blob/master/ruby_ng/image_processing/image_processor.worker
• iron_worker queue image_processor -p "{"image_url": ......., all the aws configs, etc from the config_example.yml .... }"

Extra points (optional):

• Pass in which image operations should be performed and the values required to perform them, for eg:

  [ "generate_thumb": {"width": 200, "height": 200, "format": "jpg"}, "operation2":....]

Much better error output when things dont' go right

for example, when I forget to add code.merge_worker 'some_worker' I get:

[/Users/chad/workspace/iron/test/ng] # ruby upload.rb
/usr/local/lib/ruby/gems/1.9.1/gems/rest-client-1.6.7/lib/restclient/abstract_response.rb:48:in return!': 400 Bad Request (RestClient::BadRequest) from /usr/local/lib/ruby/gems/1.9.1/gems/rest-client-1.6.7/lib/restclient/request.rb:230:inprocess_result'
from /usr/local/lib/ruby/gems/1.9.1/gems/rest-client-1.6.7/lib/restclient/request.rb:178:in block in transmit' from /usr/local/lib/ruby/1.9.1/net/http.rb:627:instart'

Would be MUCH better to have output that tried to understand what the problem is and said "You have not included a worker file with your code package"

or something liek that

Get log prints the log, it should return the log

I see double log output with this code:

p client.tasks.log(task_id)

Add a run_local CLI command that will package things up and run a worker locally

For instance, let's say my worker file (my_worker.worker) has:

exec 'my_worker.rb'
file '../models/my_model.rb'

And my worker has:

require 'my_model'

Running ruby my_worker.rb obviously won't work.

iron_worker run_local my_worker

would package it up as if it were about to be sent to IronWorker, then runs it.

@Gissues:{"order":64.2857142857143,"status":"backlog"}

calling upload() with a directory appears successful but doesn't work

iron_worker upload some_dir/my_worker

goes through the upload and looks like it works, but the worker is never actually updated.

cd some_dir
iron_worker upload my_worker

this obviously works.

support merge_gem(:path => ...)

very handy, too

Unwanted output to console

I see that you modified puts calls to IronIO::Logger. Well, while this is better than writing to the console unconditionally, there is still a lot of undesirable output going to the console through the info call. For instance,

Calling tasks.create with code_name=...

Can you please make sure there is no output at all coming from the gem unless I explicitly tell it to be verbose? Thanks.

Use same contructor format as other gems

eg:

 iron_mq = IronMQ::Client.new('token' => @config["iron"]["token"], 'project_id' => @config["iron"]["project_id"])

Can you change order or token and project_id in constructor?

Token should go first because project_id could become optional for certain things like getting a list of projects.

Is there a way to get the log?

If so, can you add to examples to wait then get log?

If I upload from a different directory, it uses the name of the full directory

For instance, if I have script/pop_reader.worker and I run iron_worker upload script/pop_reader, it will name the worker "script/pop_reader". It should be named pop_reader.

@Gissues:{"order":42.85714285714287,"status":"backlog"}

support merge_gem(:git => ...)

already wanted by customers

Thoughts:

• When uploading/queueing, has to display in the console what's going on.
• After uploading package, return to CLI: link to project (or code package to download), revision number of package
• create a way to get log from CLI (or results of last run? or all runs last 24 hours?) this would be bad ass.

(more to come)

Add something that will wait for job to finish

Like our current wait_until_complete.

Worker Source Handling

I ran into a problem earlier where heroku was trying to cache my app directory, which threw an error because of the worker I had in there. We had a chat about how workers may be uploaded outside of the app.

2 possible solutions brought up were:

1. Move workers out of app sources and upload workers with git hook
2. Ruby deploy script that
   • Uploads workers
   • Pushes sources to heroku

I suggested that it might be a good idea to wrap this inside the ng gems and see how well it works.

Thanks!

tests are broken

again
@iced could you please check they are not broken before commit?

remove the cruft from the CLI output

eg. ("I, [2012-05-31T16:47:23.065196 #997] INFO --")
@Gissues:{"order":50,"status":"done"}

Typo causing problems with named params?

https://github.com/iron-io/iron_worker_ruby_ng/blob/master/lib/iron_worker_ng/code/base.rb#L48

Seems like worke should be worker, which might explain why I was having some trouble passing :worker => 'test_linux' :name="GoTest".

run test suite in separate clean project

otherwise previous runs may screw up the results

Using -n flag throws error

eg:

iron_worker upload script/pop_reader -n pop_reader

Returns:

E, [#19693] ERROR -- IronWorkerNG: No exec specified
/usr/lib/ruby/gems/1.9.1/gems/iron_worker_ng-0.6.3/lib/iron_worker_ng/code/base.rb:94:in `create_zip': No exec specified (IronCore::IronError)
    from /usr/lib/ruby/gems/1.9.1/gems/iron_worker_ng-0.6.3/lib/iron_worker_ng/client.rb:59:in `codes_create'
    from /usr/lib/ruby/gems/1.9.1/gems/iron_worker_ng-0.6.3/lib/iron_worker_ng/client.rb:16:in `method_missing'
    from /usr/lib/ruby/gems/1.9.1/gems/iron_worker_ng-0.6.3

@Gissues:{"order":7.142857142857152,"status":"backlog"}

Check parameters for client and raise if the required ones aren't provided

I get a 301 response if I create the client like this:

client = IronWorkerNG::Client.new()

no error message when merging file as directory and vise versa

test_wrong_merges test case from test_common_merges addresses the problem

create separate project for tests run

Move examples out of this client lib and into iron_worker_examples repo

So they are consolidated in one place... or move themout of iron_worker_examples. Either way they should nto be duplicated.
@Gissues:{"order":71.42857142857144,"status":"backlog"}

Better error reporting on incorrect token/project_id

When trying to upload a code package using an incorrect token/project_id, user simply sees a 401 Rest error

/usr/local/lib/ruby/gems/1.9.1/gems/rest-client-1.6.7/lib/restclient/abstract_response.rb:48:in `return!': 401 Unauthorized (RestClient::Unauthorized)

Should return something much more meaningful like

401 Unauthorized Error: This may be due to an incorrect token or project_id

@Gissues:{"order":75,"status":"inprogress"}

CLI Tool: iron_worker MyWorker webhook

Returns the full URL to the webhook

this would be wicked cool. good/bad idea?

Point to dev center for how to create and upload workers with .worker files in README

@Gissues:{"order":78.57142857142858,"status":"backlog"}

stop supporting .workerfile and Workerfile, use just .worker

by numerous requests

Update rubygems info to point to this repo and docs, etc

https://rubygems.org/gems/iron_worker_ng