twineworks/ruby-for-pentaho-kettle

Ruby scripting for pentaho-kettle

Ruby scripting for pentaho-kettle

An plugin for Pentaho kettle (PDI) allowing to include ruby scripts as transformation steps.

The elegance of the ruby language is paired with ultimate row processing flexibility. The step allows scripts to read, write, aggregate, consume and redirect rows to accomplish the most complex of data processing tasks in one place.

The implementation is based on JRuby. Thanks to JRuby's great Java interop, the scripting step also enables easy Java scripting in kettle.

Supported versions of pentaho-kettle

The plugin works with Kettle versions 5.x - 8.x.

How to get it?

Grab the latest release from the releases page. You can also get the plugin zip as a maven dependency from maven central.

How to install?

Decompress the release zip to <kettle-dir>/plugins and restart Spoon. The "Ruby Script" step will appear in the "Scripting" section of a transformation.

How do I write ruby scripts in kettle?

The ruby scripting step comes with a lot of samples. You can access them by opening a ruby step dialog and exploring the samples section on the left.

Features at a glance

• rows are represented as hashes, indexed by field name
• automatic conversion between all kettle data types and ruby types
• steps with no input can be used as row generators
• can redirect rows to an error handling stream
• can read from specific info steps
• can write to specific target steps
• can call your favorite Java libraries
• can use ruby gems
• kettle's step $step and transformation $trans objects are available in ruby scope for advanced scripting

Where do I report bugs and issues?

Just open issues on github.

What about ruby gems?

You can use gems, see the Ruby gems article on the project wiki. As long as JRuby likes the gem, which usually means that the gem has no unsupported native bindings, you may use gems as with any other ruby program.

How do I build the project?

mvn clean package

It creates the plugin zip in target/ruby-for-pentaho-kettle-{version}-plugin.zip.

How do I run the test suite?

Create a package, then run the tests. The packaging process unzips the plugin into target/ruby-for-pentaho-kettle, which enables kettle to find the plugin when running integration tests.

mvn clean package
mvn -DskipTests=false test

How can I contribute?

If you'd like to contribute please fork the project, add the feature or bugfix and send a pull request. If your change majorly alters the way the plugin works, we should discuss it via an open issue first.

License

The ruby-for-pentaho-kettle plugin uses the LGPL 2.1 license.

Support

Open source does not mean you're on your own. The ruby-for-pentaho-kettle plugin is developed by Twineworks GmbH. Twineworks offers commercial support and consulting services. Contact us if you'd like us to help with a project.

ruby-for-pentaho-kettle's Issues

Ruby scripting error "org.typeexit.kettle.plugin.steps.ruby.RubyStepMeta", after clicking the ruby script icon

Hi Slawo-ch,

I had downloaded & installed the latest version of "Ruby-Scripting-for-Kettle-master" from your github. But still I am facing the same error "org.typeexit.kettle.plugin.steps.ruby.RubyStepMeta"
org.pentaho.di.core.exception.KettlePluginException:
Class not found
org.typeexit.kettle.plugin.steps.ruby.RubyStepMeta
at org.pentaho.di.core.plugins.PluginRegistry.loadClass(PluginRegistry.java:471)
at org.pentaho.di.core.plugins.PluginRegistry.loadClass(PluginRegistry.java:336)
at org.pentaho.di.ui.spoon.Spoon.newStep(Spoon.java:6598)
at org.pentaho.di.ui.spoon.trans.TransGraph$5.drop(TransGraph.java:613)
at org.eclipse.swt.dnd.DNDListener.handleEvent(Unknown Source)
at org.eclipse.swt.widgets.EventTable.sendEvent(Unknown Source)
at org.eclipse.swt.widgets.Widget.sendEvent(Unknown Source)
at org.eclipse.swt.widgets.Widget.sendEvent(Unknown Source)
at org.eclipse.swt.widgets.Widget.sendEvent(Unknown Source)
at org.eclipse.swt.widgets.Widget.notifyListeners(Unknown Source)
at org.eclipse.swt.dnd.DropTarget.Drop(Unknown Source)
at org.eclipse.swt.dnd.DropTarget.Drop_64(Unknown Source)
at org.eclipse.swt.dnd.DropTarget$3.method6(Unknown Source)
at org.eclipse.swt.internal.ole.win32.COMObject.callback6(Unknown Source)
at org.eclipse.swt.internal.ole.win32.COM.DoDragDrop(Native Method)
at org.eclipse.swt.dnd.DragSource.drag(Unknown Source)
at org.eclipse.swt.dnd.DragSource.access$000(Unknown Source)
at org.eclipse.swt.dnd.DragSource$1.handleEvent(Unknown Source)
at org.eclipse.swt.widgets.EventTable.sendEvent(Unknown Source)
at org.eclipse.swt.widgets.Widget.sendEvent(Unknown Source)
at org.eclipse.swt.widgets.Display.runDeferredEvents(Unknown Source)
at org.eclipse.swt.widgets.Display.readAndDispatch(Unknown Source)
at org.pentaho.di.ui.spoon.Spoon.readAndDispatch(Spoon.java:1310)
at org.pentaho.di.ui.spoon.Spoon.waitForDispose(Spoon.java:7931)
at org.pentaho.di.ui.spoon.Spoon.start(Spoon.java:9202)
at org.pentaho.di.ui.spoon.Spoon.main(Spoon.java:648)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at org.pentaho.commons.launcher.Launcher.main(Launcher.java:92)
Caused by: java.lang.ClassNotFoundException: org.typeexit.kettle.plugin.steps.ruby.RubyStepMeta
at java.net.URLClassLoader$1.run(Unknown Source)
at java.net.URLClassLoader$1.run(Unknown Source)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(Unknown Source)
at java.lang.ClassLoader.loadClass(Unknown Source)
at java.lang.ClassLoader.loadClass(Unknown Source)
at org.pentaho.di.core.plugins.KettleURLClassLoader.loadClassFromParent(KettleURLClassLoader.java:87)
at org.pentaho.di.core.plugins.KettleURLClassLoader.loadClass(KettleURLClassLoader.java:106)
at java.lang.ClassLoader.loadClass(Unknown Source)
at org.pentaho.di.core.plugins.PluginRegistry.loadClass(PluginRegistry.java:466)
... 30 more

My Setup details:-
Pentaho version - 5.2(pdi-ce-5.2.0.0-209.zip) on Windows-8.1 OS
JAVA JDK 1.7 Please let me know how to proceed further for fixing the issue.

Let me know how to proceed further.

Thanks,
Sunil gupta M

Kettle 4.2.0-M1: error when editing script: java.lang.NoSuchMethodError: org.pentaho.di.ui.core.widget.StyledTextComp.<init

Hello, I've used this awesome Kettle plugin for long, but on the last Kettle available version, it seems it doesn't work anymore. Well the transformation run if you already have some, but when editing the Ruby script, you'll likely get that stack trace:

Unable to open dialog for this step
java.lang.NoSuchMethodError: org.pentaho.di.ui.core.widget.StyledTextComp.(Lorg/eclipse/swt/widgets/Composite;ILjava/lang/String;)V
at org.typeexit.kettle.plugin.steps.ruby.RubyStepDialog.addScriptTab(RubyStepDialog.java:1782)
at org.typeexit.kettle.plugin.steps.ruby.RubyStepDialog.getData(RubyStepDialog.java:1162)
at org.typeexit.kettle.plugin.steps.ruby.RubyStepDialog.open(RubyStepDialog.java:491)
at org.pentaho.di.ui.spoon.delegates.SpoonStepsDelegate.editStep(SpoonStepsDelegate.java:126)
at org.pentaho.di.ui.spoon.Spoon.editStep(Spoon.java:7610)
at org.pentaho.di.ui.spoon.trans.TransGraph.editStep(TransGraph.java:2749)
at org.pentaho.di.ui.spoon.trans.TransGraph.mouseDoubleClick(TransGraph.java:694)
at org.eclipse.swt.widgets.TypedListener.handleEvent(Unknown Source)
at org.eclipse.swt.widgets.EventTable.sendEvent(Unknown Source)
at org.eclipse.swt.widgets.Widget.sendEvent(Unknown Source)
at org.eclipse.swt.widgets.Display.runDeferredEvents(Unknown Source)
at org.eclipse.swt.widgets.Display.readAndDispatch(Unknown Source)
at org.pentaho.di.ui.spoon.Spoon.readAndDispatch(Spoon.java:1168)
at org.pentaho.di.ui.spoon.Spoon.start(Spoon.java:6822)
at org.pentaho.di.ui.spoon.Spoon.main(Spoon.java:552)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.pentaho.commons.launcher.Launcher.main(Launcher.java:134)

Date conversion will fail if date is null

Hello,

My test case is reading a CSV file (so with lazy conversion turned off) where I have columns as date types. It happens that some row don't have a date so value is null.
Unfortunately, currently, the Ruby step won't let you a chance to do anything as is will breaks on the null date conversion before entering the script.
In that case, we get the following stack trace:

INFO 10-03 11:51:55,205 - CSV file extraction - Header row skipped in file '/home/rvalyi/DEV/openerp/customer/Bambino/fileexamples/AB export.csv'
DEBUG 10-03 11:51:55,209 - CSV file extraction - Signaling 'output done' to 1 output rowsets.
INFO 10-03 11:51:55,209 - CSV file extraction - Finished processing (I=92, O=0, R=0, W=91, U=0, E=0)
INFO 10-03 11:51:55,225 - test - Transformation has allocated 2 threads and 1 rowsets.
INFO 10-03 11:51:55,225 - Ruby Scripting - Starting to run...
ERROR 10-03 11:51:55,233 - Ruby Scripting - Unexpected error :
ERROR 10-03 11:51:55,233 - Ruby Scripting - java.lang.NullPointerException
org.typeexit.kettle.plugin.steps.ruby.execmodels.SimpleExecutionModel.createRubyInputRow(SimpleExecutionModel.java:245)
org.typeexit.kettle.plugin.steps.ruby.execmodels.SimpleExecutionModel.onProcessRow(SimpleExecutionModel.java:398)
org.typeexit.kettle.plugin.steps.ruby.RubyStep.processRow(RubyStep.java:32)
org.pentaho.di.trans.step.RunThread.run(RunThread.java:40)
java.lang.Thread.run(Thread.java:732)

I fixed this with the following patch:

--- a/src/org/typeexit/kettle/plugin/steps/ruby/execmodels/SimpleExecutionModel.java
+++ b/src/org/typeexit/kettle/plugin/steps/ruby/execmodels/SimpleExecutionModel.java
@@ -242,7 +242,11 @@ public class SimpleExecutionModel implements ExecutionModel {
break;

                    case ValueMeta.TYPE_DATE:

•                           rubyRow.put(field, data.runtime.newTime(((Date) r[i]).getTime()));
  

•                           if (r[i] != null) {
  

•                                   rubyRow.put(field, data.runtime.newTime(((Date) r[i]).getTime()));
  

•                            } else {
  

•                                   rubyRow.put(field, null);
  

•                           }
                          break;
  
                  }
  

If you could rebuild the jar and make the new zip available that would help us in distributing the plugin.

unable to read data from CSV file input inside Ruby transformation

Hello,

first of all, well done for the work; it's amazing!

If you build an example similar to your "basic field access" example, but instead you use data from a simple "CSV input step" before entering the Ruby step, then try to print or reuse the content of the fields in the Ruby transfo: you can strange Java classes instead not Ruby types. We tried that with different CSV on two PC with String or Integer and always had the issue.

May be the CSV input is generating some specific Java subtypes that Redbrige is unable to cast to Ruby?
With the Javascript modified step or with "TerminatOOOR" such things were working.

Hope this helps.

Cannot open the dialog of Rubi step when re-load the trans

The problem is when i imported a ruby step,edited the trans,everything work fine at this time. but when i restart Kettel, load the trans with ruby step, i can not open the ruby step(double click on the Ruby step), it shows following error:
"Argument cannot be null"
and it function normally when i excuting the trans.

Kette version: 4.01 stable
System: win server 2003
others: ruby runtime 1.92

cheers
chaojie

Error using Kettle Parameters or variables in Ruby Scripting

I am trying to use a Kettle Job Parameter or variable within the Ruby Scripting plugin to use as part of a connection string to Redis. This is done within the "Set as start script" and successfully connects and the "row script" runs successfully for the first row which needs to be processed. however the step then fails with the error below. In trying to resolve the issue I have found that the problem occurs even if I just write the kettle parameter to a variable, which then never gets used. When I comment out the line writing the parameter to the variable the step processes the "row script" successfully for all source rows.

Source Code (in "Set as start script") - Fails with error below

require 'rubygems'
require 'redis'

conx = $step.get_variable("RedisTarget")
p conx

@@r = Redis.new(:host => conx, :port => 6379)

@@r.flushdb

Source Code (in "Set as start script") with variable not being used - Fails with error below

require 'rubygems'
require 'redis'

conx = $step.get_variable("RedisTarget")
p conx

@@r = Redis.new(:host => "127.0.0.1, :port => 6379)

@@r.flushdb

Source Code (in "Set as start script") with parameter code removed - Runs successfully

require 'rubygems'
require 'redis'

conx = $step.get_variable("RedisTarget")

p conx

@@r = Redis.new(:host => "127.0.0.1, :port => 6379)

@@r.flushdb

============Error Message Begin =================================
INFO 23-03 12:19:38,977 - Spoon - Starting job...
INFO 23-03 12:19:38,978 - Process Organisation - Start of job execution
INFO 23-03 12:19:38,984 - Process Organisation - Starting entry [Set variables]
INFO 23-03 12:19:38,986 - Process Organisation - Starting entry [Get User Information]
INFO 23-03 12:19:38,988 - Get User Information - Loading transformation from XML file [file:///home/martinp/Work/Projects/MiXDB/Redis/MiXFleet/DataMigration/Get User Information.ktr]
INFO 23-03 12:19:39,099 - Get User Information - Dispatching started for transformation [Get User Information]
INFO 23-03 12:19:39,102 - Get User Information - This transformation can be replayed with replay date: 2011/03/23 12:19:39
INFO 23-03 12:19:42,126 - get orgid - Finished reading query, closing connection.
INFO 23-03 12:19:42,133 - get orgid - Finished processing (I=1, O=0, R=0, W=1, U=0, E=0)
INFO 23-03 12:19:42,283 - Get User Info - Finished reading query, closing connection.
INFO 23-03 12:19:42,287 - Get User Info - Finished processing (I=70, O=0, R=1, W=70, U=0, E=0)
"127.0.0.1"
java.lang.ArrayIndexOutOfBoundsException: 5
at org.jruby.runtime.scope.ManyVarsDynamicScope.setValueDepthZero(ManyVarsDynamicScope.java:158)
at org.jruby.runtime.scope.ManyVarsDynamicScope.setValue(ManyVarsDynamicScope.java:151)
at org.jruby.embed.variable.VariableInterceptor.inject(VariableInterceptor.java:128)
at org.jruby.embed.internal.BiVariableMap.inject(BiVariableMap.java:381)
at org.jruby.embed.internal.EmbedEvalUnitImpl.run(EmbedEvalUnitImpl.java:111)
at org.typeexit.kettle.plugin.steps.ruby.execmodels.SimpleExecutionModel.onProcessRow(SimpleExecutionModel.java:404)
at org.typeexit.kettle.plugin.steps.ruby.RubyStep.processRow(RubyStep.java:32)
at org.pentaho.di.trans.step.RunThread.run(RunThread.java:40)
at java.lang.Thread.run(Thread.java:636)
ERROR 23-03 12:19:42,460 - Ruby Scripting - Unexpected error :
ERROR 23-03 12:19:42,460 - Ruby Scripting - org.jruby.embed.EvalFailedException: java.lang.ArrayIndexOutOfBoundsException: 5
org.jruby.embed.internal.EmbedEvalUnitImpl.run(EmbedEvalUnitImpl.java:141)
org.typeexit.kettle.plugin.steps.ruby.execmodels.SimpleExecutionModel.onProcessRow(SimpleExecutionModel.java:404)
org.typeexit.kettle.plugin.steps.ruby.RubyStep.processRow(RubyStep.java:32)
org.pentaho.di.trans.step.RunThread.run(RunThread.java:40)
java.lang.Thread.run(Thread.java:636)

INFO 23-03 12:19:42,461 - Ruby Scripting - Finished processing (I=0, O=0, R=2, W=1, U=0, E=1)
INFO 23-03 12:19:42,461 - Get User Information - Get User Information
ERROR 23-03 12:19:42,461 - Get User Information - Errors detected!
INFO 23-03 12:19:42,465 - Process Organisation - Finished job entry Get User Information
INFO 23-03 12:19:42,466 - Process Organisation - Finished job entry Set variables
INFO 23-03 12:19:42,470 - Get User Information - Get User Information
INFO 23-03 12:19:42,477 - Process Organisation - Job execution finished
INFO 23-03 12:19:42,478 - Spoon - Job has ended.
================== Error Message End ==================================

Martin

Sintax highlighting doesn't support accents

Hi,

first thanks a lot for this awesome step!

The problem I'm facing is that the syntax highlighting stops working properly if I use any of the spanish characters in the comments that are not ASCII, like n tilde (ñ), accented vocals (á é í ó ú). Probably this will happen for other languages.

The code runs properly but syntax highlighting is broken.

It would be nice if you could expand the characters for the comments syntax highlighting to include those characters.

Thank you very much!

question: global variable possible?

Hello Slawo,

I have a question, in the Ruby script we are using to connect to OpenERP, there is a slow initial operation: the initialization of he Ruby OpenERP client (OOOR); it usually takes around 5 seconds.
Using the "init script" tab, we are able to initialize it only once per transformation fortunately.

However, it would be cool if we could even be able to let it survive a transformation execution itself. We now tend to execute those transformations on a Carte server so the JVM keeps running.
I tested using Ruby global variables but then don't persist from one execution to another.

Do you know if there could be some special scope we could use where we could place variable we want to survive an execution?

Thank you very much.

Ruby scripting error, after clicking the ruby script icon

Hi, I am getting this error, after clicking the ruby icon i get this message. can you please point me to right direction, what is the problem.
java.lang.NullPointerException
at org.jruby.embed.internal.SingleThreadLocalContextProvider.getRuntime(SingleThreadLocalContextProvider.java:61)
at org.typeexit.kettle.plugin.steps.ruby.RubyStepSyntaxHighlighter.(RubyStepSyntaxHighlighter.java:115)
at org.typeexit.kettle.plugin.steps.ruby.RubyStepDialog.open(RubyStepDialog.java:233)
at org.pentaho.di.ui.spoon.delegates.SpoonStepsDelegate.editStep(SpoonStepsDelegate.java:136)
at org.pentaho.di.ui.spoon.Spoon.editStep(Spoon.java:7835)
at org.pentaho.di.ui.spoon.trans.TransGraph.editStep(TransGraph.java:2749)
at org.pentaho.di.ui.spoon.trans.TransGraph.mouseDoubleClick(TransGraph.java:705)
at org.eclipse.swt.widgets.TypedListener.handleEvent(TypedListener.java:195)
at org.eclipse.swt.widgets.EventTable.sendEvent(EventTable.java:84)
at org.eclipse.swt.widgets.Display.sendEvent(Display.java:4128)
at org.eclipse.swt.widgets.Widget.sendEvent(Widget.java:1457)
at org.eclipse.swt.widgets.Widget.sendEvent(Widget.java:1480)
at org.eclipse.swt.widgets.Widget.sendEvent(Widget.java:1465)
at org.eclipse.swt.widgets.Widget.notifyListeners(Widget.java:1270)
at org.eclipse.swt.widgets.Display.runDeferredEvents(Display.java:3974)
at org.eclipse.swt.widgets.Display.readAndDispatch(Display.java:3613)
at org.pentaho.di.ui.spoon.Spoon.readAndDispatch(Spoon.java:1221)
at org.pentaho.di.ui.spoon.Spoon.waitForDispose(Spoon.java:7044)
at org.pentaho.di.ui.spoon.Spoon.start(Spoon.java:8304)
at org.pentaho.di.ui.spoon.Spoon.main(Spoon.java:580)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.pentaho.commons.launcher.Launcher.main(Launcher.java:134)

Thanks

Error in Transformation having Ruby Step

I have build and deployed the Ruby step in pdis.
Next I have a transformation which uses openssl and bouncy_castle gems.
Also verified that all the gems are present in the Ruby/gems/gems folder.Executed the below command > java -Djruby.gem.home=gems -jar lib/jruby-complete.jar -S gem list
Output:
*** LOCAL GEMS ***

activesupport (4.2.1, 4.0.2)
akami (1.3.0, 1.2.0)
atomic (1.1.99 java, 1.1.14 java)
bouncy-castle-java (1.5.0147)
bson (3.0.2 java, 1.9.2 java)
builder (3.2.2)
bundler (1.6.3)
bundler-unload (1.0.2)
chronic (0.10.2)
executable-hooks (1.3.2)
gem-wrappers (1.2.4)
gyoku (1.1.0, 1.0.0)
hoe (3.13.1, 3.7.2)
httpclient (2.6.0.1, 2.3.4.1)
httpi (2.1.0, 2.0.2)
i18n (0.7.0, 0.6.9)
jruby-launcher (1.1.0 java)
jruby-openssl (0.9.7 java, 0.9.5 java, 0.9.4)
json (1.8.1 java, 1.5.0 java)
json-jruby (1.5.0 java)
mime-types (2.0)
mini_portile (0.5.2)
minitest (5.6.1, 4.7.5)
mongo (1.3.1)
multi_json (1.8.2)
nokogiri (1.6.0 java)
nori (2.3.0, 2.1.0)
rack (1.5.2)
rake (10.4.2, 10.1.0, 10.0.3)
rest-client (1.6.7)
rubygems-bundler (1.4.4)
rubyntlm (0.3.4)
rvm (1.11.3.8)
savon (2.2.0, 0.9.2)
sources (0.0.1)
thread_safe (0.3.5 java, 0.1.3 java)
tzinfo (1.2.2, 0.3.38)
uuidtools (2.1.5, 2.1.4)
wasabi (3.1.0)

_Pentaho Transformation Run Error_****

Error 👍 (root) at jar:file:/opt/pdis-5_2/plugins/steps/Ruby/lib/jruby-complete.jar!/META-INF/jruby.home/lib/ruby/shared/rubygems/custom_require.rb:1
require at jar:file:/opt/pdis-5_2/plugins/steps/Ruby/lib/jruby-complete.jar!/META-INF/jruby.home/lib/ruby/shared/rubygems/custom_require.rb:60
(root) at /data/pdis_repo/lib/api_helper.rb:1
org.pentaho.di.core.exception.KettleException:
org.jruby.embed.EvalFailedException: org.jruby.embed.EvalFailedException: (LoadError) load error: openssl -- java.lang.NoClassDefFoundError: Could not initialize class org.jruby.ext.openssl.ASN1
org.jruby.embed.EvalFailedException: (LoadError) load error: openssl -- java.lang.NoClassDefFoundError: Could not initialize class org.jruby.ext.openssl.ASN1

at org.typeexit.kettle.plugin.steps.ruby.execmodels.SimpleExecutionModel.onProcessRow(SimpleExecutionModel.java:529)

Can't using Ruby Gems

Hello everybody,
Maybe this doubt seems to be very simple, but I'm caught in there a lot of time.
First i follow the steps from the samples using de JRuby to install a gem in the Ruby Extension directory (inside and outsid of directory "../data-integration\plugins\ruby-for-pentaho-kettle\gems").
In this case i receive the Error message: " org.jruby.embed.EvalFailedException: (LoadError) no such file to load -- chronic"
I tryin change RUBY_GEM_HOME and GEM_HOME like the Sample, but in this case the Ruby Extension crash and not work.
I need to point out that i use the Extension in many projects in Production Environment.

If anybody have a little video, images or a clear explanation of how you configure the environment to use Gems I would be very grateful.
Thank u everybody.

YAML

Hi,

Kettle showed me this:

org/jruby/RubyKernel.java:1038:in require': no such file to load -- yaml (LoadError) from file:/opt/Pentaho/Kettle/launcher/../libext/jruby-ooor.jar!/META-INF/jruby.home/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:31:inrequire'
from /opt/Pentaho/Kettle/plugins/steps/Ruby/gems/gems/activesupport-3.0.4/lib/active_support/ordered_hash.rb:1:in (root)' from org/jruby/RubyKernel.java:1038:inrequire'
from file:/opt/Pentaho/Kettle/launcher/../libext/jruby-ooor.jar!/META-INF/jruby.home/lib/ruby/site_ruby/1.8/rubygems/custom_require.rb:31:in require' from /opt/Pentaho/Kettle/plugins/steps/Ruby/gems/gems/activesupport-3.0.4/lib/active_support/ordered_hash.rb:8:in(root)'
from /opt/Pentaho/Kettle/plugins/steps/Ruby/gems/gems/activemodel-3.0.4/lib/active_model/errors.rb:10:in (class ActiveResource)' from /opt/Pentaho/Kettle/plugins/steps/Ruby/gems/gems/activeresource-3.0.4/lib/active_resource/validations.rb:4:in(root)'
from /opt/Pentaho/Kettle/plugins/steps/Ruby/gems/gems/activeresource-3.0.4/lib/active_resource/validations.rb:1408:in (class Base)' from /opt/Pentaho/Kettle/plugins/steps/Ruby/gems/gems/activeresource-3.0.4/lib/active_resource/base.rb:1406:in(class ActiveResource)'
from /opt/Pentaho/Kettle/plugins/steps/Ruby/gems/gems/activeresource-3.0.4/lib/active_resource/base.rb:21:in (root)' from /opt/Pentaho/Kettle/plugins/steps/Ruby/gems/gems/activeresource-3.0.4/lib/active_resource/base.rb:24:in(root)'
from org/jruby/RubyKernel.java:1038:in require' from /opt/Pentaho/Kettle/plugins/steps/Ruby/gems/gems/ooor-1.4.2/lib/app/models/open_object_resource.rb:31:inrequire'
from /opt/Pentaho/Kettle/plugins/steps/Ruby/gems/gems/ooor-1.4.2/lib/ooor.rb:19:in (root)' from org/jruby/RubyKernel.java:1038:inrequire'
from /opt/Pentaho/Kettle/plugins/steps/Ruby/gems/gems/ooor-1.4.2/lib/ooor.rb:36:in require' from Ruby Script:3:in(root)'
ERROR ,361 - Ruby Scripting - Error desconocido :
ERROR ,361 - Ruby Scripting - org.jruby.embed.EvalFailedException: (LoadError) no such file to load -- yaml
org.jruby.embed.internal.EmbedEvalUnitImpl.run(EmbedEvalUnitImpl.java:127)
org.jruby.embed.ScriptingContainer.runUnit(ScriptingContainer.java:1229)
org.jruby.embed.ScriptingContainer.runScriptlet(ScriptingContainer.java:1245)
org.typeexit.kettle.plugin.steps.ruby.execmodels.SimpleExecutionModel.onProcessRow(SimpleExecutionModel.java:377)
org.typeexit.kettle.plugin.steps.ruby.RubyStep.processRow(RubyStep.java:32)
org.pentaho.di.trans.step.RunThread.run(RunThread.java:40)
java.lang.Thread.run(Thread.java:636)

What can I solve this issue? I tried to install yaml gems but any result.

Thanks you, great plugin

Configuring the output fields on the UI

Hi,

My use case is that I'm basically just changing the values of the all the fields in my input stream based on custom logic. Looks like I need to manually enter all the fields for my output stream on the UI. Just so that I can mark the "Update existing field?" as "Y"?

Is there any way that I can press some button to bring all my input fields to the output fields and then I just modify/remove selectively?

Thanks for this great plugin!

Cheers,

Junjun

Class not found

I have just downloaded the Ruby plugin for Kettle, the file Ruby-Scripting-for-Kettle-master.zip. In this file is a folder called Ruby-Scripting-for-Kettle-master. I have copied this folder to my folder ..\data-integration\plugins\steps and than started Spoon.

As promised a new icon appears in the scripting folder, but when I drag that to the canvas, I get an error Class not found
org.typeexit.kettle.plugin.steps.ruby.RubyStepMeta
exactly like issue nr 2.

I am running Windows 7, jdk 1.6.0_38. I have tried this in PDI 4.2.1 and PDI 4.4.0, in both versions I get this error.

Can you please help me out?

Thanks.

-dzo

How can I use the Ruby Plugin with the Java API?

Hi All,

The Ruby plugin is terrific, I'd like to use it with some transformations that I launch via the API. Is there example code anywhere of how to accomplish this? FWIW, I am using some gems as well,

Sporadic NoClassDefFoundError using Ruby Step

Similar to the issue reported in #26 we are, on occassion, getting java.lang.NoClassDefFoundError when using a Ruby Step. Otherwise it's been working reliably and fantastically. Any advice? I'll place a couple of log snippets in the comments below. We are using release 1.3.4 of the this plugin and I've verified it's using the included JRuby complete JAR (9.1.13.0). Thanks for any ideas/avenues.

Error creating Ruby Scripting Step

When dragging a new "Ruby Scripting" step into the transformation I get the following error

Error creating step
I was unable to create a new step

org.pentaho.di.core.exception.KettlePluginException:
Class not found
org.typeexit.kettle.plugin.steps.ruby.RubyStepMeta

at org.pentaho.di.core.plugins.PluginRegistry.loadClass(PluginRegistry.java:364)
at org.pentaho.di.core.plugins.PluginRegistry.loadClass(PluginRegistry.java:240)
at org.pentaho.di.ui.spoon.Spoon.newStep(Spoon.java:5500)
at org.pentaho.di.ui.spoon.trans.TransGraph$5.drop(TransGraph.java:576)
at org.eclipse.swt.dnd.DNDListener.handleEvent(Unknown Source)
at org.eclipse.swt.widgets.EventTable.sendEvent(Unknown Source)
at org.eclipse.swt.widgets.Widget.sendEvent(Unknown Source)
at org.eclipse.swt.widgets.Widget.sendEvent(Unknown Source)
at org.eclipse.swt.widgets.Widget.sendEvent(Unknown Source)
at org.eclipse.swt.widgets.Widget.notifyListeners(Unknown Source)
at org.eclipse.swt.dnd.DropTarget.Drop(Unknown Source)
at org.eclipse.swt.dnd.DropTarget.Drop_64(Unknown Source)
at org.eclipse.swt.dnd.DropTarget$3.method6(Unknown Source)
at org.eclipse.swt.internal.ole.win32.COMObject.callback6(Unknown Source)
at org.eclipse.swt.internal.ole.win32.COM.DoDragDrop(Native Method)
at org.eclipse.swt.dnd.DragSource.drag(Unknown Source)
at org.eclipse.swt.dnd.DragSource.access$0(Unknown Source)
at org.eclipse.swt.dnd.DragSource$1.handleEvent(Unknown Source)
at org.eclipse.swt.widgets.EventTable.sendEvent(Unknown Source)
at org.eclipse.swt.widgets.Widget.sendEvent(Unknown Source)
at org.eclipse.swt.widgets.Display.runDeferredEvents(Unknown Source)
at org.eclipse.swt.widgets.Display.readAndDispatch(Unknown Source)
at org.pentaho.di.ui.spoon.Spoon.readAndDispatch(Spoon.java:1161)
at org.pentaho.di.ui.spoon.Spoon.start(Spoon.java:6725)
at org.pentaho.di.ui.spoon.Spoon.main(Spoon.java:545)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at org.pentaho.commons.launcher.Launcher.main(Launcher.java:134)

Caused by: java.lang.ClassNotFoundException: org.typeexit.kettle.plugin.steps.ruby.RubyStepMeta
at java.net.URLClassLoader$1.run(Unknown Source)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(Unknown Source)
at java.lang.ClassLoader.loadClass(Unknown Source)
at java.lang.ClassLoader.loadClass(Unknown Source)
at org.pentaho.di.core.plugins.KettleURLClassLoader.loadClass(KettleURLClassLoader.java:80)
at java.lang.ClassLoader.loadClass(Unknown Source)
at org.pentaho.di.core.plugins.PluginRegistry.loadClass(PluginRegistry.java:357)
... 29 more

This is happening on both Windows and Linux

Cannot open step dialog after restart of kettle

This is a duplicate of #11, but I'm not sure how notifications work for closed tickets.
After an exit and restart of Kettle, all existing Ruby steps are not accessible to edit. A dialog pops up with the message "Argument cannot be null". Here is my comment re-stated (with some elaboration) from #11.

I'm using kettle 4.2 (latest stable) with the RubyPlugin_1.2_Kettle_4.2.zip package and seeing this exact error.

I would consider this a blocking bug since you only have one Kettle session to edit the step.

Using Kettle 32 bit on Mac OSX and setting RUBY_GEM_HOME in kettle.properties.

I downgraded to RubyPlugin_1.1_Kettle_4.2.zip and I still see this error.

Here's the stack trace

java.lang.IllegalArgumentException: Argument cannot be null
    at org.eclipse.swt.SWT.error(Unknown Source)
    at org.eclipse.swt.SWT.error(Unknown Source)
    at org.eclipse.swt.SWT.error(Unknown Source)
    at org.eclipse.swt.widgets.Widget.error(Unknown Source)
    at org.eclipse.swt.widgets.Text.setText(Unknown Source)
    at org.pentaho.di.ui.core.widget.TextVar.setText(TextVar.java:171)
    at org.typeexit.kettle.plugin.steps.ruby.RubyStepDialog.addRuntimeTab(RubyStepDialog.java:953)
    at org.typeexit.kettle.plugin.steps.ruby.RubyStepDialog.open(RubyStepDialog.java:413)
    at org.pentaho.di.ui.spoon.delegates.SpoonStepsDelegate.editStep(SpoonStepsDelegate.java:126)
    at org.pentaho.di.ui.spoon.Spoon.editStep(Spoon.java:7728)
    at org.pentaho.di.ui.spoon.trans.TransGraph.editStep(TransGraph.java:2744)
    at org.pentaho.di.ui.spoon.trans.TransGraph.mouseDoubleClick(TransGraph.java:693)
    at org.eclipse.swt.widgets.TypedListener.handleEvent(Unknown Source)
    at org.eclipse.swt.widgets.EventTable.sendEvent(Unknown Source)
    at org.eclipse.swt.widgets.Widget.sendEvent(Unknown Source)
    at org.eclipse.swt.widgets.Widget.sendEvent(Unknown Source)
    at org.eclipse.swt.widgets.Widget.sendEvent(Unknown Source)
    at org.eclipse.swt.widgets.Widget.notifyListeners(Unknown Source)
    at org.eclipse.swt.widgets.Display.runDeferredEvents(Unknown Source)
    at org.eclipse.swt.widgets.Control.sendTrackEvents(Unknown Source)
    at org.eclipse.swt.widgets.Control.actionProc(Unknown Source)
    at org.eclipse.swt.widgets.Display.actionProc(Unknown Source)
    at org.eclipse.swt.internal.carbon.OS.CallNextEventHandler(Native Method)
    at org.eclipse.swt.widgets.Widget.kEventControlTrack(Unknown Source)
    at org.eclipse.swt.widgets.Control.kEventControlTrack(Unknown Source)
    at org.eclipse.swt.widgets.Widget.controlProc(Unknown Source)
    at org.eclipse.swt.widgets.Display.controlProc(Unknown Source)
    at org.eclipse.swt.internal.carbon.OS.SendEventToEventTarget(Native Method)
    at org.eclipse.swt.widgets.Display.readAndDispatch(Unknown Source)
    at org.pentaho.di.ui.spoon.Spoon.readAndDispatch(Spoon.java:1169)
    at org.pentaho.di.ui.spoon.Spoon.start(Spoon.java:6940)
    at org.pentaho.di.ui.spoon.Spoon.main(Spoon.java:553)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.pentaho.commons.launcher.Launcher.main(Launcher.java:134)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at apple.launcher.LaunchRunner.run(LaunchRunner.java:116)
    at apple.launcher.LaunchRunner.callMain(LaunchRunner.java:51)
    at apple.launcher.JavaApplicationLauncher.launch(JavaApplicationLauncher.java:52)

"invalid multibyte char" when handling UTF-8 data in Ruby script

I have been really enjoying using the Ruby scripting step in Pentaho since finding it recently. However, I deal with a lot of Unicode data and I am running into issues trying to deal with UTF8 strings within the Ruby script step.

Whenever I try to manipulate or reference UTF8 text in the script I encounter a problem, which varies depending on the "Ruby Compatibility" configuration.

For example, simply trying to assign text to a field

$row["last"] = "文字化け"

results in either malformed garbage text (using 1.8 Compatibility) or an "invalid multibyte char (US-ASCII)" SyntaxError exception (using 1.9 Compatibility).

2012/11/07 15:09:54 - Ruby Scripting.0 - ERROR (version 4.3.0-stable, build 16786 from 2012-04-24 14.11.32 by buildguy) : Caused by: org.jruby.exceptions.RaiseException: (SyntaxError) Ruby Script:14: invalid multibyte char (US-ASCII)
2012/11/07 15:09:54 - Ruby Scripting.0 - ERROR (version 4.3.0-stable, build 16786 from 2012-04-24 14.11.32 by buildguy) : Error initializing step [Ruby Scripting]

I have tried a long list of the usual Ruby fixes to force UTF8 handling, including:

#!/bin/env ruby -wKu
# encoding: utf-8
$KCODE = "u"
Encoding.default_internal = "utf-8"

But these do not seem to be effective. When investigating the internal and external encoding states, Ruby seems to always be operating within "US-ASCII" regardless.

My apologies if I'm just missing something obvious with my configuration. I can't be sure whether or not this is a high-level problem within the Ruby scripting step itself or something deeper in the jRuby implementation or somewhere else, so I thought I would try asking here. Please let me know if I need to provide more detailed error logs or anything else.

I would greatly appreciate any help on this matter. Thank you for your time!

BUILD FAILED

when i try to run "ant install" i'm getting the following error

/var/www/html/Ruby/build.xml:17: /home/arasu/Downloads/dataintegration/

(LoadError) no such file to load -- rubygems

Hi!
I'm trying samples from dist/steps/Ruby/samples/25 - Using Ruby Gems, and receiving such errors, Maybe some thing needs to be installed?

stopping transformation doesn't work, at least under certain circumstances

If you have a Ruby script that do a long or endless loop before creating any output, then clicking on the 'stop' transformation button of Kettle is not going to kill the script.
Try with an endless loop printing something in the log for instance.
Hope this helps.

Love this in 4.x - any chance for an update in 5. ?

Hey - this is a terrific addition to the 4.x world. I'm a little cautious to jump back in as we've migrated to 5.x... any chance you can update/test with 5?

I'll download and see what happens - just curious if you know if it works.

Memory usage with Java API

Hi All,

I have the Plugin working with the Java API - but the Ruby steps don't seem to be releasing memory for long running processes (I bind to a RabbitMQ queue and process). If I do the same thing with Java - the memory gets collected.

I think it may have something to do with logging from looking at a heap dump, but I'm not sure. Anything come to mind that I might be doing wrong?

Thanks again for a great addition to Pentaho!

Updating fields

Hey Slawo,
On a few occasions I have run across the error

java.lang.IllegalStateException: Field max_event_duration_in_secondscannot be updated. I cannot find it!
    at org.typeexit.kettle.plugin.steps.ruby.RubyStepMeta.getFields(RubyStepMeta.java:94)
    at org.typeexit.kettle.plugin.steps.ruby.execmodels.SimpleExecutionModel.initMainRowStream(SimpleExecutionModel.java:223)
    at org.typeexit.kettle.plugin.steps.ruby.execmodels.SimpleExecutionModel.onProcessRow(SimpleExecutionModel.java:459)
    at org.typeexit.kettle.plugin.steps.ruby.RubyStep.processRow(RubyStep.java:32)
    at org.pentaho.di.trans.step.RunThread.run(RunThread.java:40)
    at java.lang.Thread.run(Thread.java:679)

In my step, I have the field max_event_duration_in_seconds appearing in the input meta and set as an output variable with the "Update existing field" flag set "Y". It's type is set to Integer, which is the same as what the input meta displays.

Are there any non-obvious conditions that would raise this error? This error only occurs sometimes - which leads me to believe I have an edge case/anomaly in my data - but this error occurs before any $step.log_basic commands I try to issue to identify those anomalies at a row-level.

Global Variable Doesn't Allow Repository Reference

Is it possible to make a reference to a repository job?