Remote denylists and watching system (proposal)

[hsanjuan]

Remote denylists and watching system (proposal)

[hsanjuan]

HTTP polling has been introduced at #22

[lidel]

We want to leverage this and switch ipfs.io and dweb.link to use RAINBOW_DENYLISTS=https://badbits.dwebops.pub/badbits.deny.

Did some initial triage today:

• #38 with proposal to leverage HTTP caching
  • not a blocker, but will improve cache hits
• https://github.com/protocol/badbits.dwebops.pub/issues/32733 to make upstream badbits published at https://badbits.dwebops.pub/badbits.deny append-only
  • @hsanjuan would appreciate any info you remember on this, why the upstream list is sorted? is it for takedown automation PRs to never cause merge conflict? I'd like to fix upstream but need to understand constraints first.
  • Lmk, if it is a rabbit hole. I suspect http caching support together with detection of non-append-only lists makes more sense as it is more generic.

[hsanjuan]

Hey, nopfs watches denylists and reads any new lines appended to them. Adding updates in append-only fashion allows to do this without having to re-read the whole file.

I don't think #38 is a must. If the list upstream is append only, you:

• Download it
• Read from len(size_of_download file) for every update using the Range header. If no updates happened you will be reading a 0-range and it's essentially a no-op method (equivalent to checking for a If-Modified-Since tag), otherwise you obtain only the part of the content that you append to the local copy, and nopfs processes accordingly.

I don't know if you saw, but the badbits list is published in append-only format here: https://denyli.st/badbits.deny.txt and that is what I used for my defunct gateway.

I have a github action that reads https://badbits.dwebops.pub/badbits.deny and finds any new lines and turns appends them to https://denyli.st/badbits.deny... so far so good, it's been going for months.

So you can use RAINBOW_DENYLISTS=https://denyli.st/badbits.deny.txt already. In the meantime I would update badbits to be append only and not have to rely on a 3rd party.