Combine efforts with 0xB10C/known-mining-pools

[0xB10C]

Combine efforts with 0xB10C/known-mining-pools

[0xB10C]

Here's a file you can test with: https://github.com/0xB10C/known-mining-pools/blob/d966997c35d068eb8f4cccc32acd65f07f3c72f0/generated/pool-list.json

Feel free to comment in bitcoin-data#75 too if you run into any issues.

I'm planning to set up a CI job that generates and pushes the file automatically, which removes the burden of having to run the generate scripts every time someone touches the data.

[0xB10C]

There's now a new file to test with https://github.com/0xB10C/known-mining-pools/blob/master/generated/pool-list.json

bitcoin-data#77 is still a todo.

[nymkappa]

Closing. Once we're done refactoring the mining pool process on our end, I will fetch updates manually from https://github.com/bitcoin-data/mining-pools.

Thanks for your help on this!

[0xB10C]

As discussed on Matrix, I've merged bitcoin-data#69 which included some of the pool information from this repo. We should be synced now.

I've also came across e.g. a few addresses tagged as AntPool which are clearly not coinbase output addresses. I had previously cleaned these up in my repo. For example: 1D9jw3QHNankXxtcGVihsDK7Z7THN6j7Pg.

mining-pools/pools.json

Lines 1213 to 1216 in e73a425

	"1D9jw3QHNankXxtcGVihsDK7Z7THN6j7Pg": {
	"name": "AntPool",
	"link": "https://www.antpool.com/"
	},

As a next step, I'd be interested in hearing which format works best for mempool-space (the same old pool.json with slugs? or a new format?).

[nymkappa]

Thanks @0xB10C.

Regarding the non-coinbase addresses, I don't know why they are here, they were already in the original repo before we forked it. If the file should be soly focused on mining pool (which I think should), then that would make sense to only keep coinbase output addresses. If in the future, if we want to tag large holders on mempool.space, this should not just include miners but also exchanges and other known entities, which would be done in a different file. So your cleanup is welcome.

Regarding the format, the one you proposed is great. We want it to be easy to parse (here a simple JSON.parse will do) and easy to map to a database (rows/columns). We need to add the slug (or id, as a more general purpose field name) field which represents an unique identifier for a mining pool.

Regarding the export tool, which is a bit out of scope for this issue
For mempool version <= 2.4.0, I believe we should not provide any update to the mining pool definitions. The reason is that even if you provide an export tool to convert from the new format to the old format, my parser will not be able to spot all changes, and as a result, user's database will tag some blocks improperly.
The only workaround is to ask user to re-index everything, which of course can only be done manually since they are not upgrading.

Todo on mempool repo:

I will rewrite the mining pool parser from scratch to support this new format once validated, which should not be very complicated with that new syntax.

[0xB10C]

We need to add the slug (or id, as a more general purpose field name) field which represents an unique identifier for a mining pool.

Adding IDs in bitcoin-data#74.

I've also changed the format a bit to only allow a single URL as link instead of a list of URLs as links in bitcoin-data#71.

I will rewrite the mining pool parser from scratch to support this new format once validated, which should not be very complicated with that new syntax.

Let me know I you'd like a generated JSON-file with the separate per-pool-JSON-files as a list. I could easily generate something like this automatically once the pool data changes. Should make it easier to download it for mempool-space than having to load all JSON files separatly.

[
  {
    "id": "f2pool",
    "name": "F2Pool",
    "addresses": [
      "1KFHE7w8BhaENAswwryaoccDb6qcT6DbYY",
      "bc1q7wedv4zdu5smt3shljpu5mgns48jn299mukymc",
      "bc1qf274x7penhcd8hsv3jcmwa5xxzjl2a6pa9pxwm"
    ],
    "tags": [
      "七彩神仙鱼",
      "🐟"
    ],
    "link": "https://www.f2pool.com"
  },
  {
    "id": "slush",
    "name": "SlushPool",
    "addresses": [
      "1AqTMY7kmHZxBuLUR5wJjPFUvqGs23sesr",
      "1CK6KHY6MHgYvmRQ4PAafKYDrg1ejbH1cE"
    ],
    "tags": [
      "/slush/"
    ],
    "link": "https://slushpool.com"
  },
...
]

[nymkappa]

Let me know I you'd like a generated JSON-file with the separate per-pool-JSON-files as a list

Yes we definitely need one single file with a list of JSON objects because we detect file changes using the sha sum of the file using github API https://github.com/mempool/mempool/blob/343a48818b9057b7fd0b428fef1e46205acca8c8/backend/src/tasks/pools-updater.ts#L102-L118

[0xB10C]

Renamed Slush to Braiins in bitcoin-data#78