Configuration of on_error Copy option

[snowflakedb]

Configuration of on_error Copy option

[smtakeda]

do you mean on_error='%s' where %s can be either one of them?

CONTINUE | SKIP_FILE | SKIP_FILE_num | SKIP_FILE_num% | ABORT_STATEMENT

?
I assume this change is used for Talend Connector.

[olhava]

Yes, @smtakeda, this is correct.

[olhava]

@smtakeda, I added pull requests for 2 possible solutions:
#36
#37

[smtakeda]

@olhava what behavior do you expect when the error occurs? For example, COPY with ABORT_TRANSACTION will raise an SQL error if an invalid data are included in the source. Do you want the API to raise the exception as is?

[smtakeda]

Adding a test case. This is the expected error. if this works, I'll commit the change and release next week.

    StreamLoader underTest = ...
    try
    {
      underTest.finish();
      fail("should raise an exception");
    }
    catch (Loader.ConnectionError ex)
    {
      assertThat(ex.getCause() instanceof SQLException, equalTo(Boolean.TRUE));
      SQLException sqlEx = (SQLException) ex.getCause();
      assertThat(sqlEx.getErrorCode(), equalTo(100038));
    }

[olhava]

@smtakeda Yes, in case error occurs with on_error = ABORT_TRANSACTION, it's okay to throw Loader.ConnectionError.

One more concern in case of onErrorNumPercents, I did not find the documentation about the type of percentage value. I've set the type as Integer in my pull request, maybe it should be changed to Float if it is allowed.

[smtakeda]

@olhava sounds good.
I'm making the parameter to a single string type and validate them when specified. It may not be the best, but I don't want to clutter the parameters only for ON_ERROR option:

The validation will be done with this regex:

private final static Pattern validPattern = Pattern.compile(
    "(?:ABORT_STATEMENT|CONTINUE|SKIP_FILE(?:_\\d+%?)?)"

[smtakeda]

Btw, I'm going to remove copyEmptyFieldAsEmpty. It turned out it is no longer useful as I fixed the default behavior where null and empty data are treated as is instead of converting to null at all times:
f8b07f6

[howryu]

Fixed in f3469af

[smtakeda]

Actually the fix is already out.