After upgrade to 2.3.3 the conflux may stop sync at random height

[lcgogo]

After upgrade to 2.3.3 the conflux may stop sync at random height

[lcgogo]

@peilun-conflux Please help!

[735946969]

After upgrade to 2.3.3 the conflux may stop sync at random height. After stop sync the log repeats like below

2023-12-19T05:15:44.700454144+00:00 INFO  IO Worker #0         cfxcore::sta - Statistics: StatisticsInner { sync_graph: SyncGraphStatistics { inserted_block_count: 628, inserted_header_count: 177974 }, consensus_graph: ConsensusGraphStatistics { inserted_block_count: 177676, activated_block_count: 177675, processed_block_count: 177677 } }
2023-12-19T05:15:49.703267896+00:00 INFO  IO Worker #0         cfxcore::sta - Statistics: StatisticsInner { sync_graph: SyncGraphStatistics { inserted_block_count: 628, inserted_header_count: 177974 }, consensus_graph: ConsensusGraphStatistics { inserted_block_count: 177676, activated_block_count: 177675, processed_block_count: 177677 } }
2023-12-19T05:15:54.706360245+00:00 INFO  IO Worker #1         cfxcore::sta - Statistics: StatisticsInner { sync_graph: SyncGraphStatistics { inserted_block_count: 628, inserted_header_count: 177974 }, consensus_graph: ConsensusGraphStatistics { inserted_block_count: 177676, activated_block_count: 177675, processed_block_count: 177677 } }
2023-12-19T05:15:59.708748398+00:00 INFO  IO Worker #1         cfxcore::sta - Statistics: StatisticsInner { sync_graph: SyncGraphStatistics { inserted_block_count: 628, inserted_header_count: 177974 }, consensus_graph: ConsensusGraphStatistics { inserted_block_count: 177676, activated_block_count: 177675, processed_block_count: 177677 } }
2023-12-19T05:16:04.711781033+00:00 INFO  IO Worker #2         cfxcore::sta - Statistics: StatisticsInner { sync_graph: SyncGraphStatistics { inserted_block_count: 628, inserted_header_count: 177974 }, consensus_graph: ConsensusGraphStatistics { inserted_block_count: 177676, activated_block_count: 177675, processed_block_count: 177677 } }
2023-12-19T05:16:09.714621051+00:00 INFO  IO Worker #2         cfxcore::sta - Statistics: StatisticsInner { sync_graph: SyncGraphStatistics { inserted_block_count: 628, inserted_header_count: 177974 }, consensus_graph: ConsensusGraphStatistics { inserted_block_count: 177676, activated_block_count: 177675, processed_block_count: 177677 } }

After I use snapshot my inserted_block_count always 0

[lcgogo]

I find what's happened: The cfx data disk increases very quickly at these days. And the cfx will auto remove some files when disk usage reaches 100%. After remove the files, the cfx stops sync.
Because not reach the disk alert keeping time (15m or 5m I set), I am not figure it out by alert.

[cfl132]

i think there are some promble here，our node's vote was all automatically unlocked，so the total vote become zero，after this happen，i try increase one vote，but after 5 hours，it was in outqueue（should be seven days）