fjahr/asmap-data

Demo repository for how a similar repository could be used in Bitcoin Core

asmap-data's Issues

Collaborative Launch 1706536800 (2024-01-29 14:00 GMT, Kartograf v0.4.4)

Next collaborative launch is planned for 1706536800. Please make sure to upgrade to Kartograf v0.4.4 which was released today (https://github.com/fjahr/kartograf/releases/tag/0.4.4).

./run map -w=1706536800 -irr -rv

#7 wasn't a success but it was expected that not every run would have sufficient matches. The robustness improvement for the download issue should be an improvement though and the hashes of downloaded files are now printed in the output right away, which should make debugging mismatches a bit easier. I hope this one can the first clean run that concisely shows what the process could look like when being run routinely.

Thanks for your participation in advance! Upon completion please post your hash in the comments. When we have majority for one hash, we will open a PR, similar to #6.

Collaborative Launch 1706191200 (2024-01-25 14:00 GMT, Kartograf v0.4.2)

Next collaborative launch is planned for 1706191200. Please make sure to upgrade to Kartograf v0.4.2 which was released today (https://github.com/fjahr/kartograf/releases/tag/0.4.2).

./run map -w=1706191200 -irr -rv

#4 has been a success but a lot of issues were fixed along the way making it hard to parse and grasp the actual process. I hope this one can the first clean run that concisely shows what the process could look like when being run routinely.

Thanks for your participation in advance! Upon completion please post your hash in the comments. When we have majority for one hash, we will open a PR, similar to #6.

Collaborative Launch 1724248800 (Wednesday, August 21, 2024 2:00 PM GMT)

Next collaborative launch is planned for 1724248800. Please make sure to use Kartograf v0.4.6 and this time please also try to use the latest rpki-client version. The run will work with an older version, too, but I would like to test if our success rate can be improved if everyone is one the latest version. This could hint at "silent conflicts" produced by different versions of rpki-client. See here for more info on this: fjahr/kartograf#16.

./run map -w=1724248800 -irr -rv

EDIT: nix users please use kartograf 0.4.7. This ensures you will be on the latest rpki-client version.

Demo using Collaborative Launch feature at 1702994400 (Kartograf 0.4.1)

In Kartograf 0.3.1 a new feature was added that allows users to start the mapping process at the exact same time at a previously agreed timestamp. This has shown some promise to result in the same or at least very similar results for the final mapping file across several participants. If multiple participants get the same result independently, trust in the ASMap file is further minimized as the input data generation is effectively federated. From the point of the creation of the input data, the process of arriving at the final ASMap file is reproducible with open-source tools and thus completely trustless.

As a demo of how this could work in the future, I propose the following timestamp to start a mapping process collaboratively: 1702994400. This is Tuesday, December 19, 2023 2:00:00 PM GMT, the same time of day at which the Bitcoin Core IRC meeting happens on Thursdays.

What you need to do to participate:

1. If you haven't used it already, get the latest version of Kartograf, install instruction are in the README. To ensure everything is set up correctly, maybe run ./run map once before setting the collaborative launch. If you have used Kartograf in the past you will probably still need to upgrade to the latest version, 0.3.1 at least.
2. When everything is set up you just need to run ./run map -w=1702994400 -irr -rv. The program runs and waits until the timestamp is hit, you just need to ensure that your computer is running until then and let it finish the process.
3. When the process is finished you will see a hash of the final result printed to the screen, please post it here as a comment.

A few notes, especially if you have tried Kartograf previously:

• Runtime: Kartograf now uses multithreading, which means runtime can now be dramatically shorter than previously but it depends on the machine a lot and during certain periods you will see a lot of CPU usage. For reference, on a 2-year-old Macbook Pro with basic specs I expect to see ~3h for everything.
• Network: Kartograf has been restructured to download everything it needs as fast as possible in the beginning since this also helps the chances of getting matching final results among participants. You should see a big spike in network activity in the beginning for a few minutes and then the tools runs basically fully offline. Since this is quite a bit of data and finishing fast means higher chance of success, I wouldn't recommend tethering over your mobile maybe. On the other hand, we also want to stress test this feature so if you only have a slow connection available, please still participate if you can!
• Data: This is a full run collecting RPKI, IRR and Collectors data (from Routeviews). In previous demos didn't do this because of the long runtime.

Let me know if there are any questions. If you plan to participate, happy to see you confirm beforehand here as well.

Collaborative Launch 1710770400 (Monday, March 18, 2024 2:00:00 PM GMT)

Next collaborative launch is planned for 1710770400. Please make sure to use Kartograf v0.4.4.

./run map -w=1710770400 -irr -rv

Sporadic connection issues with `ftp.arin.net`

In preparation for today's collaborative run (#7), I ran kartograf twice yesterday. The first run failed because I got disconnected from the FTP server while downloading arin.db.gz; the second run succeeded.

To add some redundancy, I scheduled the collaborative on two machines (one at home, another at Contabo). Unfortunately, both runs failed. Here's the output:

[virtu@gravity:~/kartograf]$ ./run map -w=1706191200 -irr -rv

--- Start Kartograf ---

Kartograf version: 0.4.2
Using rpki-client version 8.6.
Coordinated launch mode: Waiting until 1706191200 (2024-01-25 15:00:00 CET) to launch mapping process.
Countdown: 0 second(s)
Starting...
The epoch for this run is: 1706191200 (2024-01-25 14:00:00 UTC, local: 2024-01-25 15:00:00 CET)

--- Fetching RPKI ---

Downloaded TAL for AFRINIC to /home/virtu/kartograf/data/1706191200/rpki/tals/afrinic.tal
Downloaded TAL for APNIC to /home/virtu/kartograf/data/1706191200/rpki/tals/apnic.tal
Downloaded TAL for ARIN to /home/virtu/kartograf/data/1706191200/rpki/tals/arin.tal
Downloaded TAL for LACNIC to /home/virtu/kartograf/data/1706191200/rpki/tals/lacnic.tal
Downloaded TAL for RIPE to /home/virtu/kartograf/data/1706191200/rpki/tals/ripe.tal
Downloading RPKI Data
...finished in 0:07:36.020574

--- Fetching IRR ---

Downloading afrinic.db.gz
Downloading apnic.db.route.gz
Downloading apnic.db.route6.gz
Downloading arin.db.gz
Traceback (most recent call last):
  File "/home/virtu/kartograf/./run", line 93, in <module>
    Kartograf.map(args)
  File "/home/virtu/kartograf/kartograf/kartograf.py", line 63, in map
    fetch_irr(context)
  File "/home/virtu/kartograf/kartograf/timed.py", line 10, in wrapper
    result = func(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^
  File "/home/virtu/kartograf/kartograf/irr/fetch.py", line 28, in fetch_irr
    ftp = FTP(host)
          ^^^^^^^^^
  File "/nix/store/qp5zys77biz7imbk6yy85q5pdv7qk84j-python3-3.11.6/lib/python3.11/ftplib.py", line 121, in __init__
    self.connect(host)
  File "/nix/store/qp5zys77biz7imbk6yy85q5pdv7qk84j-python3-3.11.6/lib/python3.11/ftplib.py", line 162, in connect
    self.welcome = self.getresp()
                   ^^^^^^^^^^^^^^
  File "/nix/store/qp5zys77biz7imbk6yy85q5pdv7qk84j-python3-3.11.6/lib/python3.11/ftplib.py", line 244, in getresp
    resp = self.getmultiline()
           ^^^^^^^^^^^^^^^^^^^
  File "/nix/store/qp5zys77biz7imbk6yy85q5pdv7qk84j-python3-3.11.6/lib/python3.11/ftplib.py", line 230, in getmultiline
    line = self.getline()
           ^^^^^^^^^^^^^^
  File "/nix/store/qp5zys77biz7imbk6yy85q5pdv7qk84j-python3-3.11.6/lib/python3.11/ftplib.py", line 218, in getline
    raise EOFError
EOFError

Looks like EOFError signals the server closed the connection.

Interrupt could be nicer

I did a ctrl+c during the "Validating RPKI" phase. The output looks a bit noisy and it also didn't stop, at least not before I gave up waiting after two minutes and ctrl + c again, which responded quickly.

$ ./run map -w 1702648800

--- Start Kartograf ---

Using rpki-client version 8.5.
Coordinated launch mode: Waiting until 1702648800 (2023-12-15 15:00:00 CET) to launch mapping process.
The epoch for this run is: 1702648800 (2023-12-15 14:00:00 UTC, local: 2023-12-15 15:00:00 CET)

--- Fetching RPKI ---

Downloading RPKI Data
...finished in 0:01:16.535906

--- Validating RPKI ---

Validating RPKI ROAs
^CTraceback (most recent call last):
  File "/home/sjors/dev/kartograf/./run", line 92, in <module>
    Kartograf.map(args)
  File "/home/sjors/dev/kartograf/kartograf/kartograf.py", line 68, in map
    validate_rpki_db(context)
  File "/home/sjors/dev/kartograf/kartograf/timed.py", line 10, in wrapper
    result = func(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^
  File "/home/sjors/dev/kartograf/kartograf/rpki/fetch.py", line 38, in validate_rpki_db
    with ThreadPoolExecutor() as executor:
  File "/home/sjors/.pyenv/versions/3.11.7/lib/python3.11/concurrent/futures/_base.py", line 647, in __exit__
    self.shutdown(wait=True)
  File "/home/sjors/.pyenv/versions/3.11.7/lib/python3.11/concurrent/futures/thread.py", line 235, in shutdown
    t.join()
  File "/home/sjors/.pyenv/versions/3.11.7/lib/python3.11/threading.py", line 1119, in join
    self._wait_for_tstate_lock()
  File "/home/sjors/.pyenv/versions/3.11.7/lib/python3.11/threading.py", line 1139, in _wait_for_tstate_lock
    if lock.acquire(block, timeout):
       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
KeyboardInterrupt

Collaborative Launch 1710424800 (Thursday, March 14, 2024 2:00:00 PM GMT)

Next collaborative launch is planned for 1710424800. Please make sure to use Kartograf v0.4.4.

./run map -w=1710424800 -irr -rv