rdcli/rdc.etl

Extract Transform Load toolkit (python).

rdc.etl's Issues

DBLoad - Make record count to update at once dynamic

Use time since last update, duration of last update and queue length to decide to go for smaller updates if pertinent.

Create project from PasteScript template does not work when rdc.etl is installed from egg

Missing files in egg.

Traceback (most recent call last):
File "/Users/rd/Sites/MyPrj/.py-env/bin/paster", line 9, in
load_entry_point('PasteScript==1.7.5', 'console_scripts', 'paster')()
File "/Users/rd/Sites/MyPrj/.py-env/lib/python2.7/site-packages/paste/script/command.py", line 104, in run
invoke(command, command_name, options, args[1:])
File "/Users/rd/Sites/MyPrj/.py-env/lib/python2.7/site-packages/paste/script/command.py", line 143, in invoke
exit_code = runner.run(args)
File "/Users/rd/Sites/MyPrj/.py-env/lib/python2.7/site-packages/paste/script/command.py", line 238, in run
result = self.command()
File "/Users/rd/Sites/MyPrj/.py-env/lib/python2.7/site-packages/paste/script/create_distro.py", line 137, in command
template, output_dir, vars)
File "/Users/rd/Sites/MyPrj/.py-env/lib/python2.7/site-packages/paste/script/create_distro.py", line 196, in create_template
template.run(self, output_dir, vars)
File "/Users/rd/Sites/MyPrj/.py-env/lib/python2.7/site-packages/paste/script/templates.py", line 58, in run
self.write_files(command, output_dir, vars)
File "/Users/rd/Sites/MyPrj/.py-env/lib/python2.7/site-packages/paste/script/templates.py", line 127, in write_files
template_renderer=self.template_renderer)
File "/Users/rd/Sites/MyPrj/.py-env/lib/python2.7/site-packages/paste/script/copydir.py", line 71, in copy_dir
names = os.listdir(source)
OSError: [Errno 2] No such file or directory: '/Users/rd/Sites/MyPrj/.py-env/lib/python2.7/site-packages/rdc/etl/extra/tools/templates/project'

DatabaseJoin discriminant looks ignored when one part is None, inserted value is not null but ""

I have (pid, path) as discriminant, and path is empty / missing due to some buggy transform tuning. Each time the transform runs it adds a new line, with path = ""

DatabaseLoad's fetch_columns attribute does not work like other attributes in transforms

You can't override this attribute in subclass because of the way the constructor is treating this attribute, which is bad and inconsistent.

    # XXX should take self.fetch_columns into account if provided
    self.fetch_columns = {}
    if isinstance(fetch_columns, (list, tuple, )):
        self.add_fetch_column(*fetch_columns)
    elif isinstance(fetch_columns, dict):
        self.add_fetch_column(**fetch_columns)