iron-io/iron_worker_ruby_ng

Next Gen Ruby gem for IronWorker

iron_worker_ruby_ng's Issues

CLI --help shouldn't require pid/token

[/Users/chad/workspace/iron/hud/workers] # iron_worker queue --help

Using typhoeus gem.
E, [#330] ERROR -- IronWorkerNG: Both token and project_id must be specified
/usr/local/lib/ruby/gems/1.9.1/gems/iron_worker_ng-0.7.4/lib/iron_worker_ng/api_client.rb:21:in `initialize': Both token and project_id must be specified (IronCore::IronError)

add way to run test suite for all backends of 'rest' gem

Get log prints the log, it should return the log

I see double log output with this code:

p client.tasks.log(task_id)

Add something that will wait for job to finish

Like our current wait_until_complete.

run test suite in separate clean project

otherwise previous runs may screw up the results

If I upload from a different directory, it uses the name of the full directory

For instance, if I have script/pop_reader.worker and I run iron_worker upload script/pop_reader, it will name the worker "script/pop_reader". It should be named pop_reader.

@Gissues:{"order":42.85714285714287,"status":"backlog"}

ruby_ng README: include passing args in in-code example

In README document how to pass in an API argument such as priority. Current not shown here.

document :workerfile option in README

eg: IronWorkerNG::Code::Base.new(workerfile: "workers/sleepy.worker")

Full remote build given worker url

ie: https://github.com/iron-io/iron_worker_ruby_ng/pull/64/files#L9R46

remove the cruft from the CLI output

eg. ("I, [2012-05-31T16:47:23.065196 #997] INFO --")
@Gissues:{"order":50,"status":"done"}

CLI: Output the project_id when uploading/queuing

We've had several people that don't know what's going on, but turns out it's picking up the wrong credentials. It should say: "Uploading to project: #{project_id}..." or something.

Re: #63

Better error reporting on incorrect token/project_id

When trying to upload a code package using an incorrect token/project_id, user simply sees a 401 Rest error

/usr/local/lib/ruby/gems/1.9.1/gems/rest-client-1.6.7/lib/restclient/abstract_response.rb:48:in `return!': 401 Unauthorized (RestClient::Unauthorized)

Should return something much more meaningful like

401 Unauthorized Error: This may be due to an incorrect token or project_id

@Gissues:{"order":75,"status":"inprogress"}

README update: split apart upload from task creation so people don't upload and queue every time

Make it two separate boxes or something.

On error responses, try to get "msg" value from body and display it to user

Impossible for user to debug otherwise.

Better Documentation for limit errors

I tried scheduling a task to run every 30 seconds and API returned

400 Bad Request (Rest::Wrappers::RestClientExceptionWrapper)

Took me 15 minutes to figure out that run_every's minimum is 60 seconds. Needs to be handled here as well as in our docs.

Using -n flag throws error

eg:

iron_worker upload script/pop_reader -n pop_reader

Returns:

E, [#19693] ERROR -- IronWorkerNG: No exec specified
/usr/lib/ruby/gems/1.9.1/gems/iron_worker_ng-0.6.3/lib/iron_worker_ng/code/base.rb:94:in `create_zip': No exec specified (IronCore::IronError)
    from /usr/lib/ruby/gems/1.9.1/gems/iron_worker_ng-0.6.3/lib/iron_worker_ng/client.rb:59:in `codes_create'
    from /usr/lib/ruby/gems/1.9.1/gems/iron_worker_ng-0.6.3/lib/iron_worker_ng/client.rb:16:in `method_missing'
    from /usr/lib/ruby/gems/1.9.1/gems/iron_worker_ng-0.6.3

@Gissues:{"order":7.142857142857152,"status":"backlog"}

Unwanted output to console

I see that you modified puts calls to IronIO::Logger. Well, while this is better than writing to the console unconditionally, there is still a lot of undesirable output going to the console through the info call. For instance,

Calling tasks.create with code_name=...

Can you please make sure there is no output at all coming from the gem unless I explicitly tell it to be verbose? Thanks.

Needs retry / exponential backoff

Install required dependencies based on .worker file

eg:

• clone a repo
• iron_worker install hello which will read file and install dependencies (ie: gem's for a ruby worker).

Output the task id on iron_worker queue

@Gissues:{"order":50,"status":"inprogress"}

Point to dev center for how to create and upload workers with .worker files in README

@Gissues:{"order":78.57142857142858,"status":"backlog"}

Worker Source Handling

I ran into a problem earlier where heroku was trying to cache my app directory, which threw an error because of the worker I had in there. We had a chat about how workers may be uploaded outside of the app.

2 possible solutions brought up were:

1. Move workers out of app sources and upload workers with git hook
2. Ruby deploy script that
   • Uploads workers
   • Pushes sources to heroku

I suggested that it might be a good idea to wrap this inside the ng gems and see how well it works.

Thanks!

Add a project_id flag

It feels a lot more natural to have a flag to set it and it's a very common thing. Don't need one for token though.

Example:

iron_worker upload -project_id MYID my_worker

Something like that.

@Gissues:{"order":50,"status":"notstarted"}

Add more info to codes.create command line output

Current:

treeder@ubuntu:~/dev/tmp/myapp$ iron_worker upload hello
I, [2012-06-06T13:47:29.499810 #14685]  INFO -- IronWorkerNG: Merging ruby exec with path='hello_worker.rb' and class=''
I, [2012-06-06T13:47:29.499953 #14685]  INFO -- IronWorkerNG: Fixating gems dependencies

• First, get rid of the cruft like other ticket says.
• Should say the following steps:
  • "Building package..."
  • "Uploading package to project id #{ID} (X bytes)..."
  • "Upload successful. Code name: {NAME} -- Code id: {ID}..."

@Gissues:{"order":87.5,"status":"notstarted"}

Add a logger method to access logger or change log level

Not through core, but something like:

IronWorkerNG.logger.level = Logger::DEBUG

Add a run_local CLI command that will package things up and run a worker locally

For instance, let's say my worker file (my_worker.worker) has:

exec 'my_worker.rb'
file '../models/my_model.rb'

And my worker has:

require 'my_model'

Running ruby my_worker.rb obviously won't work.

iron_worker run_local my_worker

would package it up as if it were about to be sent to IronWorker, then runs it.

@Gissues:{"order":64.2857142857143,"status":"backlog"}

no error message when merging file as directory and vise versa

test_wrong_merges test case from test_common_merges addresses the problem

In README, bring the core methods to the top

ie:

• upload
• queue
• schedule
• log

It's hard to find what's important in the readme.

@Gissues:{"order":85.71428571428572,"status":"backlog"}

Check parameters for client and raise if the required ones aren't provided

I get a 301 response if I create the client like this:

client = IronWorkerNG::Client.new()

tests are broken

again
@iced could you please check they are not broken before commit?

calling upload() with a directory appears successful but doesn't work

iron_worker upload some_dir/my_worker

goes through the upload and looks like it works, but the worker is never actually updated.

cd some_dir
iron_worker upload my_worker

this obviously works.

Move examples out of this client lib and into iron_worker_examples repo

So they are consolidated in one place... or move themout of iron_worker_examples. Either way they should nto be duplicated.
@Gissues:{"order":71.42857142857144,"status":"backlog"}

Full remote build usable example

A follow up to #79 is to get this example: https://github.com/iron-io/iron_worker_examples/tree/master/ruby_ng/image_processing

working like this:

• iron_worker upload https://github.com/iron-io/iron_worker_examples/blob/master/ruby_ng/image_processing/image_processor.worker
• iron_worker queue image_processor -p "{"image_url": ......., all the aws configs, etc from the config_example.yml .... }"

Extra points (optional):

• Pass in which image operations should be performed and the values required to perform them, for eg:

  [ "generate_thumb": {"width": 200, "height": 200, "format": "jpg"}, "operation2":....]

lib API change

Library API was changed by commit romand@b0d4801

The way code objects are created have been changed: instead of

  code = Code::Ruby.new(:workerfile => 'common.worker') do
    name 'asdf'
    exec 'worker.rb'
  end

or

  code = Code::Creator.create do
    runtime 'ruby'
    ...
  end

now we write

  code = Code.new(:runtime => 'ruby') do
    ...
  end

or

  code = Code.new do
    runtime 'ruby'
    ...
  end

or

  code = Code.new
  code.runtime 'ruby'
  ...

or

  code = Code.new(:workerfile => 'cool.worker') # with runtime set in workerfile

Before the change, block passed to Code::Creator.create was executed twice:

• at first, to detect runtime and determine corresponding subclass (Code::Ruby or Code::Binary or ...)
• second, within that subclass constructor (Code::Ruby.new or Code::Binary.new or ...)

Workerfiles content was executed twice, too.

In order to eliminate such unexpected and buggy behavior, code of subclasses was moved to modules with corresponding names, and those modules are mixed in code object when 'runtime' method is called.

As the result,

• the only 1 code class remains (Code) instead of hierarchy (Code::Base, Code::Ruby, ...)
• Code.new is the single code objects creation facility, with many good ways to specify runtime
• bug with duplicate block and workerfile evaluation is fixed
• library API is changed (have been cut)

The one way to protect customers from suddenly broken code, we can

  class Code
    class Ruby
      def initialize(*args, &block)
        IronCore::Logger.warn 'Code::Ruby.new is deprecated, call Code.new(:runtime => "ruby") instead'
        Code.new(...)
      end
    end
  end

Another way is just to document this change (in changelog, readme ...) and let customers make (quick!) fix in their code.

@iced @treeder please discuss

Show warnings when files are included but can't be found

Typo causing problems with named params?

https://github.com/iron-io/iron_worker_ruby_ng/blob/master/lib/iron_worker_ng/code/base.rb#L48

Seems like worke should be worker, which might explain why I was having some trouble passing :worker => 'test_linux' :name="GoTest".

Much better error output when things dont' go right

for example, when I forget to add code.merge_worker 'some_worker' I get:

[/Users/chad/workspace/iron/test/ng] # ruby upload.rb
/usr/local/lib/ruby/gems/1.9.1/gems/rest-client-1.6.7/lib/restclient/abstract_response.rb:48:in return!': 400 Bad Request (RestClient::BadRequest) from /usr/local/lib/ruby/gems/1.9.1/gems/rest-client-1.6.7/lib/restclient/request.rb:230:inprocess_result'
from /usr/local/lib/ruby/gems/1.9.1/gems/rest-client-1.6.7/lib/restclient/request.rb:178:in block in transmit' from /usr/local/lib/ruby/1.9.1/net/http.rb:627:instart'

Would be MUCH better to have output that tried to understand what the problem is and said "You have not included a worker file with your code package"

or something liek that

Is there a way to get the log?

If so, can you add to examples to wait then get log?

Beautify CLI output

AFTER

iron_worker upload hello

------> Detecting Configuration
Found configuration in working directory
Working with project 'Test Project' (project_id: '4e25e1d25c0dd27801000283')
------> Discovering .worker file
Found .worker file 'hello.worker'
[Ruby | PHP | .NET | Go | Binary] worker detected with path = 'hello_worker.rb'
Class name has been set to 'MyTestWorker'
------> Creating worker package
Worker upload started
------> Worker deployed with version r15
https://hud.iron.io/tq/projects/4e25e1d25c0dd27801000283/code/4fc7e44451ef0d2807087604

BEFORE

iron_worker upload hello

Working with project 'Test Project' ('4e25e1d25c0dd27801000283')
Using workerfile 'hello.worker'
Merging ruby exec with path='hello_worker.rb' and class=''
Worker 'hello' upload started
Worker 'hello' uploaded

Update rubygems info to point to this repo and docs, etc

https://rubygems.org/gems/iron_worker_ng

Schedule update

Is there a quick and elegant way to reschedule (update) workers without having to unschedule the old one and schedule the new one?

Use same contructor format as other gems

eg:

 iron_mq = IronMQ::Client.new('token' => @config["iron"]["token"], 'project_id' => @config["iron"]["project_id"])

Thoughts:

• When uploading/queueing, has to display in the console what's going on.
• After uploading package, return to CLI: link to project (or code package to download), revision number of package
• create a way to get log from CLI (or results of last run? or all runs last 24 hours?) this would be bad ass.

(more to come)

support merge_gem(:git => ...)

already wanted by customers

support merge_gem(:path => ...)

very handy, too

create separate project for tests run

Can you change order or token and project_id in constructor?

Token should go first because project_id could become optional for certain things like getting a list of projects.

CLI Tool: iron_worker MyWorker webhook

Returns the full URL to the webhook

this would be wicked cool. good/bad idea?

Document CLI tool and .worker files

stop supporting .workerfile and Workerfile, use just .worker

by numerous requests