Replicator syncing data very slowly

[flyingrabbit-lab]

Replicator syncing data very slowly

[randombishop]

+1
I am trying to replicate to Postgres too but on a much smaller hardware, it's been running for 5 days already and only got halfway so far. My guess is there are lots of index in the database that slow down the inserts.

[flyingrabbit-lab]

@randombishop How much data do you have already in PG? For us replicator synced more than 450 GB, which seems odd to me as our hub uses only 130 GB.

[randombishop]

Hubble is fully synced and uses 140GB.
PG is 126GB so far at this level of progress:
replicator-1 | [13:26:12.812] INFO (7): Backfilled events for 379628 of 560026 FIDs. Estimated time remaining: 2 days, 17 hours, 50 minutes, 6 seconds

[sanjayprabhu]

Hmm, 800 is a very high degree of concurrency, possible it's causing db contention. Could you try lowering it? Also how does db health look like? Are you seeing any deadlocks or timeouts?

[flyingrabbit-lab]

Cloud sql allowed 1K connections, cpu usage was under 60% (with 96 vCPU).
Deadlocks only appeared for a 3 minute period for the past week since I am trying to replicate the data.
Lowering concurrency only increases backfill time.

I have seen a few of these Info messages after starting the replicator, but no errors.
Job 73816705 took too long and has stalled.

Does the Repolicator reuse DB connections via a connection pool? It seems to me that it keeps opening and closing connections.

[flyingrabbit-lab]

@randombishop Have you managed to sync the data into PG?

[randombishop]

@flyingrabbit-lab Not yet, progress seems to be 80%:
Backfilled events for 479559 of 595301 FIDs. Estimated time remaining: 1 week, 2 days, 7 hours, 19 minutes, 14 seconds