jhc-systems/debezium-connector-ibmi

Debezium Connector for IBM i (AS/400)

debezium-connector-ibmi's Issues

journal loops when sequence reset

Package for Kafka Connect

Hey, is there a way to package the Connector to work with Confluent Kafka Connect?

How to?

How would one build this locally and create a local debezium docker instance?

Processing journal entries stops with error "CPF7074 message RCVRNG for specified SEARCH not valid"

Processing journal entries stops with repeated error:

Call failed position JournalPosition [offset=xxx, receiver=yyy, receiverLibrary=zzz, processed=false] with error code CPF7074 message RCVRNG for specified SEARCH not valid. &N Cause . . . . . : The values specified for the RCVRNG parameter do not agree with the value for the SEARCH parameter or the command or API specifications. If the SEARCH(*ASCEND) parameter is specified for the Retrieve Journal Entry (RTVJRNE) command, the ". If the SEARCH(*DESCEND) parameter is specified for the Retrieve Journal Entry (RTVJRNE) command, the journal receivers must be specified from the newest to oldest. &P If the Display Journal (DSPJRN) command, Receive Journal Entry (RCVJRNE) command, or Retrieve Journal Entries (QjoRetrieveJournalEntries) API was used, the journal receivers must be specified from the oldest to the newest. &N Recovery . . . : The Work with Journal Attributes (WRKJRNA) command can be used to display the receiver directory for the correct arrangement of the journal receivers. Change the values of the RCVRNG parameter or the value of the SEARCH parameter. Then try the request again.

force ccsid on tables

A legacy app has written data into tables using the wrong ccsid
Please add support to force the ccsid both for jdbc and for journal chagnes

for remote calls pass the ccsid into the constructor:

AS400Text uk = new AS400Text(200, 285);

for jdbc (the default translate binary=false config is correct - don't override it)
retrieve bytes as binary then convert locally using the ccsid

try (ResultSet rs = ps.getResultSet()) {
	while (rs.next()) {
		byte[] b = rs.getBytes(1);
		new String(b, Charset.forName("CP285"));

re-snapshotting may miss entries as it refetches the current offset

The initial snapshot happens by:
1/ fetching the current journal offset
2/ snapshot all tables
3/ start processing journal entries from the offset fetched

When using the signalling features such as the table or kafka topic the journal processing is stopped and snapshotting happens in exactly the same way as above with processing resuming from the offset found just before this snapshot started - ignoring the current journal processing offset

Processing should simply resume at the offset the journal processing had got to

Support BOOLEAN

In V7R5 and later V7R4, IBM released support for BOOLEAN data types. When using this connector with a table that contains a BOOLEAN datatype, a hard error is thrown.

fetching receiver list problematic

It looks from the logs like we request the receiver list immediately after we detect a new receiver but seems to result in a list that doesn't either contain the new receiver or the receiver we are currently on

position JournalPosition [offset=405050821, receiver=xxx01341, receiverLibrary=#JRNLIB, processed=false] not found in active receivers list from xxx01325 to ... xxx01340

receiver name not detected

• It was my bad, I journaled my files with a journal that was not in the *LIB
  => I Just updated my file and now it detects everythings 😮

--> for this constraint, do you think that you'll be able to update it ? I'm not a big java developper but I scrolled through the code and saw this on this page

I think that maybe it's where the problem is. I use a query to get journal info https://gist.github.com/andymalo43/16231be5805866b523abe3271409a0e5

I hope this can help you :)

have a great day

And again : Thank you very much for your work :)

Originally posted by @andymalo43 in #40 (comment)

new CCSID decodes backwards from ascii to ebdic

The following exception on adding the ccsid:

org.apache.kafka.connect.errors.ConnectException: An exception occurred in the change event producer. This connector will be stopped.
at io.debezium.pipeline.ErrorHandler.setProducerThrowable(ErrorHandler.java:53) ~[debezium-core-2.1.2.Final.jar:2.1.2.Final]
at io.debezium.pipeline.ChangeEventSourceCoordinator.lambda$start$0(ChangeEventSourceCoordinator.java:116) ~[debezium-core-2.1.2.Final.jar:2.1.2.Final]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) [?:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]
at java.lang.Thread.run(Thread.java:833) [?:?]
Caused by: java.lang.RuntimeException: Failed to initialize snapshot context.
at io.debezium.connector.db2as400.As400SnapshotChangeEventSource.execute(As400SnapshotChangeEventSource.java:65) ~[classes/:?]
at io.debezium.connector.db2as400.As400SnapshotChangeEventSource.execute(As400SnapshotChangeEventSource.java:30) ~[classes/:?]
at io.debezium.pipeline.ChangeEventSourceCoordinator.doSnapshot(ChangeEventSourceCoordinator.java:155) ~[debezium-core-2.1.2.Final.jar:2.1.2.Final]
at io.debezium.pipeline.ChangeEventSourceCoordinator.executeChangeEventSources(ChangeEventSourceCoordinator.java:137) ~[debezium-core-2.1.2.Final.jar:2.1.2.Final]
at io.debezium.pipeline.ChangeEventSourceCoordinator.lambda$start$0(ChangeEventSourceCoordinator.java:109) ~[debezium-core-2.1.2.Final.jar:2.1.2.Final]
... 5 more
Caused by: java.lang.IllegalArgumentException: The column "ëáà|<" is referenced as PRIMARY KEY, but a matching column is not defined in table "DATABASE.SCHEMA.TABLE"!
at io.debezium.relational.TableEditorImpl.lambda$updatePrimaryKeys$0(TableEditorImpl.java:107) ~[debezium-core-2.1.2.Final.jar:2.1.2.Final]
at java.util.ArrayList.removeIf(ArrayList.java:1672) ~[?:?]
at java.util.ArrayList.removeIf(ArrayList.java:1660) ~[?:?]
at io.debezium.relational.TableEditorImpl.updatePrimaryKeys(TableEditorImpl.java:103) ~[debezium-core-2.1.2.Final.jar:2.1.2.Final]
at io.debezium.relational.TableEditorImpl.create(TableEditorImpl.java:307) ~[debezium-core-2.1.2.Final.jar:2.1.2.Final]
at io.debezium.relational.Tables.lambda$overwriteTable$2(Tables.java:194) ~[debezium-core-2.1.2.Final.jar:2.1.2.Final]
at io.debezium.util.FunctionalReadWriteLock.write(FunctionalReadWriteLock.java:84) ~[debezium-core-2.1.2.Final.jar:2.1.2.Final]
at io.debezium.relational.Tables.overwriteTable(Tables.java:187) ~[debezium-core-2.1.2.Final.jar:2.1.2.Final]
at io.debezium.jdbc.JdbcConnection.readSchema(JdbcConnection.java:1203) ~[debezium-core-2.1.2.Final.jar:2.1.2.Final]
at io.debezium.connector.db2as400.As400SnapshotChangeEventSource.readTableStructure(As400SnapshotChangeEventSource.java:140) ~[classes/:?]
at io.debezium.connector.db2as400.As400SnapshotChangeEventSource.execute(As400SnapshotChangeEventSource.java:62) ~[classes/:?]
at io.debezium.connector.db2as400.As400SnapshotChangeEventSource.execute(As400SnapshotChangeEventSource.java:30) ~[classes/:?]
at io.debezium.pipeline.ChangeEventSourceCoordinator.doSnapshot(ChangeEventSourceCoordinator.java:155) ~[debezium-core-2.1.2.Final.jar:2.1.2.Final]
at io.debezium.pipeline.ChangeEventSourceCoordinator.executeChangeEventSources(ChangeEventSourceCoordinator.java:137) ~[debezium-core-2.1.2.Final.jar:2.1.2.Final]
at io.debezium.pipeline.ChangeEventSourceCoordinator.lambda$start$0(ChangeEventSourceCoordinator.java:109) ~[debezium-core-2.1.2.Final.jar:2.1.2.Final]
... 5 more

Primary key in question was SEDOL:

ëáà|< = 0x53, 0x45, 0x44, 0x4F, 0x4C - EBDIC CP285
SEDOL = 0x53, 0x45, 0x44, 0x4F, 0x4C - ASCII

Kafka Message is junk data

I'm running the docker compose here and getting absolute junk in the kafka message. Am I doing something wrong here?