Not accepted mined blocks are not hidden nor marked

[team-exor]

Not accepted mined blocks are not hidden nor marked

[joeuhren]

Hi Johannes, unfortunately you have discovered one of the last big problems with the explorer that is yet to be resolved. In fact, I have not encountered an open source block explorer that properly addresses this issue, so there are probably hundreds if not thousands of coins out there suffering from the same problem.

For now, the only workaround is to do a full resync (or restore a backup that is known to be free of orphaned blocks) and make frequent mongo backups so that you can restore a good backup anytime you notice this problem.

I already have some ideas about how to resolve this issue, but it will require a major overhaul to the sync code and I don't have time to dig into that by myself just yet. Since this is an important issue that should be resolved in the best way possible, I would love to open up a dialogue with other developers and see if there are any other ideas about how best to fix this.

For anyone reading this who would like an explanation about exactly what is going on, the problem occurs when 2 or more nodes find the next block at the same time and the chain is temporarily split as a result. Half the network was given "Block A" as the next block and the other half see "Block B" as the correct block. This is typically resolved in the very next block, although it can sometimes carry on multiple blocks before a "winner" is decided. The winner will always be the the side that can discover a block faster than the other side which becomes the largest chain and the blocks from the other side become "orphaned blocks".

At the current moment, the explorer does not have a way to detect these types of chain splits and will therefore attribute both (or all) blocks as valid, which results in too many coins for certain addresses that in reality did not actually receive those coins since they belong to an orphaned blockchain which the network has rejected.

[PutinCoinPUT]

Hi Joe,

if it is too complicated to adapt the sync algorithm, what about if we just adapt the explorer database request and filter out each transaction, which got "-1" confirmations so that they won't be displayed nor added to the address - balance. Would this be (for the meantime) faster solution of this issue?

[joeuhren]

@PutinCoinPUT I think that should be 50% of the "fix". The problem is that at the moment the blocks arrive, they do not have -1 confirmations, so they are added to the explorer balances right away. I believe the sync code needs to be smart enough to see that 2 or more blocks with the same block height but different hashes have arrived, and mark that block height as special in the stats table for example. Then it can return back to the previously marked block height for the next sync to see if the blockchain has resolved the split yet (is one of the blocks showing -1 confirmations yet?) and if yes, then remove the transactions for the orphaned block(s) from the database. Perhaps it should even refrain from adding to the address balances at all if it notices the split, but it also needs to have another method which can detect and fix older orphaned blocks for those of us who are currently running an explorer and don't want to do a complete resync. I also plan to add a new menu option to go along with this "feature" which can list the orphaned blocks.

It's not impossible, but will be a bit tricky and time-consuming to get it right and test.

[PutinCoinPUT]

@joeuhren For the time being, what, if we just make a "cleaning" script, which automatically runs, let's say, once every 10 minutes or so, which cleans out all already identified "-1" orphaned blocks from the eIquidus database. This is no "beautiful" solution, I know, but, for the moment, it may prevent the explorer database from running into a highly inaccurate state again.

[joeuhren]

@PutinCoinPUT I'm personally not interested in spending time on a temporary fix. Feel free to come up with your own script and share it with us here if you are successful. I will add a proper fix sometime in the future as time permits and update the thread when it is ready.

[joeuhren]

Hi Johannes, and anyone else following this thread. Today I have posted a complete fix to the orphaned block problem: 09fa919

You do not need to reindex your explorer for this update to fix your existing orphaned blocks. This fix adds new functionality to the core block sync process that will traverse through the known blocks in search of orphans and remove them from the local database (limited info about the orphan data is copied to a new orphans collection). This includes fixes to the balance, sent and received values for individual addresses. It also includes a new optional page that can be displayed in the explorer menu that lists all known orphaned blocks that have been "fixed".

Keep in mind that after updating your explorer code, the next block sync will take an abnormally long time because it will be fixing orphans at the end of the normal sync process. Depending on how many orphans were previously synced and how large your blockchain is, this process can take many hours to complete. After that one long sync has completed, all future syncs will be fast again and will also automatically resolve new orphaned blocks as they arrive.

I would strongly recommend making a full backup of your explorer database using npm run create-backup before updating the explorer code just in case there are problems with running this new code on certain blockchains.

Assuming you have not made any custom changes to the explorer source code, you can quickly update your explorer using the npm run update-explorer cmd. If you have made custom changes to the source, you will need to cherry-pick and merge the updated code manually.

Also, if anyone is worried about having their explorer not pulling in new blocks for a few hours while the orphans are being fixed, you can always run the first sync on another computer and restore the data to the explorer server once completed by following these steps:

1. Make a full backup of your explorer database with npm run create-backup
2. Copy or transfer the backup file to another computer (backups are stored in the explorer/backups directory by default)
3. Set up an instance of eiquidus on the other computer with your coin wallet and make sure it is all working and connected to the database (NOTE: be sure to use more-or-less the same settings.json as the live server since some settings can change how the data is saved)
4. Restore the live backup file to the 2nd server using npm restore-backup <path and filename>
5. Run a block sync with npm run sync-blocks on the 2nd server and wait for it to finish
6. Once finished syncing, it is recommended to review the data in a browser to make sure the numbers look good
7. Assuming all looks good, make a full backup of the 2nd server explorer database with npm run create-backup
8. Copy or transfer the backup file back to the live server and restore it with npm restore-backup <path and filename>

@PutinCoinPUT once you are able to update your explorer and try this code out, your feedback would be greatly appreciated to know if all went well for you so we can close this issue.

[PutinCoinPUT]

Hi Joe, thanks a lot for your valued work! I will checkout the new code and install it on weekend, when I am back in my "programmers cave" ;-). I'm really happy you solved this task for all of us! I will post a feedback here on how it worked out.

Best wishes & thanks,
Johannes

[PutinCoinPUT]

Just to let everybody know: I just installed the new eIquidus explorer with integrated orphaned blocks detection and it works really fine. Thank you Joe.