rrybarczyk/asmap-rs

A tool to assist the asmap project read and parse RIS raw data from the RIPE NCC.

asmap-rs's Issues

Overwriting downloaded files

Does not overwrite immediately after downloading each file, but only after all of them are successfully downloaded.

Doesn't require input to be gzipped

Some files come not zipped.
Ideally we don't ask the user to zip them, but just handle it as is.

Create Error to handle AS_SET

If AS_PATH is an unordered AS_SET, pass custom error up from AsPathParser to FindBottleneck where the associated IP/mask and RIB Entry can be logged.

Actually look into corrupted file

I looked at the code for a bit #26
It seems like the issue was caused by a corruption in a file.

I'm not sure in which state the buffer appears after we advance past the problematic field.
But I think it doesn't necessary happen at the end of the file, and can happen in the middle.

There are 2 potentially bad outcomes:

1. we misinterpret the rest of the file, and then make wrong decisions when finding bottleneck (very bad)
2. we ignore rest of the file (bad but alright I guess)

I'm not sure it's actually easy to recover after the problem. @i7i can you take a look at bytes?

If it's not possible to reliably recover to a good state after passing one bad record, I think we should skip an entire file so that (1) doesn't happen.

panics in debug build on integer underflow

[gmaxwell@argonaut asmap-rs]$ time ./target/debug/asmap-rs find-bottleneck --dir dump/
Reading in and parsing dump/rrc00-latest-bview.gz
Skipping unsupported MRT record from TABLE_DUMP_V2.
Reading in and parsing dump/rrc01-latest-bview.gz
Skipping unsupported MRT record from TABLE_DUMP_V2.
Reading in and parsing dump/rrc02-latest-bview.gz
Skipping unsupported MRT record from TABLE_DUMP_V2.
thread 'main' panicked at 'attempt to subtract with overflow', src/as_path_parser.rs:96:43
note: run with RUST_BACKTRACE=1 environment variable to display a backtrace

real 157m53.960s
user 157m53.079s
sys 0m0.490s

Error on `find-bottleneck` if data dir dne

Bottleneck doesn't try to create output dir

There is a bit of an inconsistency between download and find-bottleneck: download will create the output dir if it doesn't exist, so the command in the readme will succeed even im the dump dir does not exist. On the other hand the find-bottleneck example will fail if the bottleneck dir doesn't exist and it will do so at the very end.

Ideally both commands would work consistently, as a quick fix we could also change the readme or do a check at the beginning of find-bottleneck to make sure that it fails early at least.

Rename struct Address -> RoutingPrefix

I think the name "Address" doesn't make sense.

ip + mask is should be called "routing prefix"

Skip file instead of quitting if invalid gz header

Add progress bar

• file disk reads
• file parsing
• bottleneck finding

Make warnings more readable

Right now when we print a warning with a list of records for an anomaly (for example multi-AS ip), it's very hard to understand whats going on. I suggest printing just 2 lists which enough to show the anomaly

More tests

• Test invalid gzip header
• Test accuracy of logic

Optimize

Currently asmap-rs takes 15 Gb of RAM, need to get that down to 4Gb.

Rationale from @naumenkogs: "Ideally, any user should have the right to be paranoid and do everything on their own, including constructing their own asmap locally. Obviously, requiring constructing asmaps on the phone is not reasonable, but maybe comparing to compiling Bitcoin Core is reasonable. This takes 2-4Gb of RAM."

For profiling and narrowing down the inefficient code:

• Use Rust's benchmark feature for tests
• Use cargo flamegraph (requires bin to work)