Comments (8)
+1
I am trying to replicate to Postgres too but on a much smaller hardware, it's been running for 5 days already and only got halfway so far. My guess is there are lots of index in the database that slow down the inserts.
from hub-monorepo.
@randombishop How much data do you have already in PG? For us replicator synced more than 450 GB, which seems odd to me as our hub uses only 130 GB.
from hub-monorepo.
Hubble is fully synced and uses 140GB.
PG is 126GB so far at this level of progress:
replicator-1 | [13:26:12.812] INFO (7): Backfilled events for 379628 of 560026 FIDs. Estimated time remaining: 2 days, 17 hours, 50 minutes, 6 seconds
from hub-monorepo.
Hmm, 800 is a very high degree of concurrency, possible it's causing db contention. Could you try lowering it? Also how does db health look like? Are you seeing any deadlocks or timeouts?
from hub-monorepo.
Cloud sql allowed 1K connections, cpu usage was under 60% (with 96 vCPU).
Deadlocks only appeared for a 3 minute period for the past week since I am trying to replicate the data.
Lowering concurrency only increases backfill time.
I have seen a few of these Info messages after starting the replicator, but no errors.
Job 73816705 took too long and has stalled.
Does the Repolicator reuse DB connections via a connection pool? It seems to me that it keeps opening and closing connections.
from hub-monorepo.
@randombishop Have you managed to sync the data into PG?
from hub-monorepo.
@flyingrabbit-lab Not yet, progress seems to be 80%:
Backfilled events for 479559 of 595301 FIDs. Estimated time remaining: 1 week, 2 days, 7 hours, 19 minutes, 14 seconds
from hub-monorepo.
Related Issues (20)
- [HELP]: ContractFunctionExecutionError: The contract function "register" reverted with the following signature: 0xf90230a9 HOT 1
- (200 $USDC) castsByParent with `reverse` parameter fails on popular channels HOT 5
- Crash on start hubble v1.12.1, Failed to get snapshot 404 HOT 1
- bug: Warpcast followers count is defferent with Hub follow count ? HOT 2
- Missing Deps Documentation
- Network Docs Correction
- sync farcaster data error HOT 3
- Replicator migrate fails on existing database HOT 1
- bug(hubble): Fetch fname transfers when merge fails due to "username not registered" HOT 6
- bug(hubble): Hubble restarts periodically due to "neon::types::Deferred` was dropped without being settled" HOT 1
- Can I get information about the farcaster nodes that are currently running distributed? HOT 1
- Problem with degen points status HOT 1
- feat(shuttle): Add reverse reconciliation to shuttle
- bug(hubble): grpc stream closes unexpectedly
- bug(hubble) Missing userdata HOT 6
- No payment method accepted for Android users in Brazil. HOT 1
- Bug: Hub/Shuttle is streaming delete events but when queried hubs for latest state it says that its in merge state HOT 5
- Replicator fails to start with "Cannot find module 'bs58'" HOT 1
- bug(core) Faker library is bundled instead of being set as dependency HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from hub-monorepo.