maxarndt/podyn

DynamoDB to PostgreSQL & Citus continuous replication tool

podyn's Issues

replicateData() throws a NullPointerException

The following statement throws a NullPointerException.

podyn/src/main/java/com/citusdata/migration/DynamoDBReplicator.java

Lines 223 to 225 in 666927c

	for(Future<Long> futureResult : futureResults) {
	futureResult.get();
	}

To fix this problem I need to debug the following function:

podyn/src/main/java/com/citusdata/migration/DynamoDBTableReplicator.java

Lines 236 to 285 in 666927c

	public long replicateData(int maxScanRate) {
	RateLimiter rateLimiter = RateLimiter.create(maxScanRate);
	Map<String,AttributeValue> lastEvaluatedScanKey = null;
	long numRowsReplicated = 0;
	while(true) {
	ScanResult scanResult = scanWithRetries(lastEvaluatedScanKey);
	if (addColumnsEnabled) {
	for(Map<String,AttributeValue> dynamoItem : scanResult.getItems()) {
	addNewColumns(dynamoItem);
	}
	}
	TableRowBatch tableRowBatch = new TableRowBatch();
	for(Map<String,AttributeValue> dynamoItem : scanResult.getItems()) {
	TableRow tableRow = rowFromDynamoRecord(dynamoItem);
	tableRowBatch.addRow(tableRow);
	}
	LOG.info(String.format("Replicated %d rows to table %s", tableRowBatch.size(), tableSchema.tableName));
	numRowsReplicated += tableRowBatch.size();
	/* load the batch using COPY */
	emitter.copyFromReader(tableSchema, tableRowBatch.asCopyReader());
	lastEvaluatedScanKey = scanResult.getLastEvaluatedKey();
	if(lastEvaluatedScanKey == null) {
	break;
	}
	// Account for the rest of the throughput we consumed,
	// now that we know how much that scan request cost
	double consumedCapacity = scanResult.getConsumedCapacity() != null ? scanResult.getConsumedCapacity().getCapacityUnits() : 0;
	int permitsToConsume = (int)(consumedCapacity - 1.0);
	if (permitsToConsume <= 0) {
	permitsToConsume = 1;
	}
	// Let the rate limiter wait until our desired throughput "recharges"
	rateLimiter.acquire(permitsToConsume);
	}
	return numRowsReplicated;
	}

Nested objects are mistakenly be escaped using jsonb conversion mode

Podyn generates tables with the following schema in the data (jsonb) column:

{
    "name": "Mike",
    "lastName": "Miller",
    "address": "{\"city\":\"Berlin\",\"code\":10115,\"street\":\"Some street 83\",\"country\":\"Germany\"}"
}

The value of the attribute address is escaped which leads to problems if you want for example to use an ORM to cast this JSON to a (nested) POCO.

The output of podyn should look like:

{
    "name": "Mike",
    "lastName": "Miller",
    "address": {"city":"Berlin","code":10115,"street":"Some street 83","country":"Germany"}
}

Numerics are treated as strings

If there is a numeric type in your dynamoDB, podyn treats them as strings and this ends up in the following in your PostgreSQL using jsonb mode:

{
    "someNumeric": "0.234",
    "name": "Mike",
    "lastName": "Miller"
}