job/irrexplorer

IRR explorer

IRR Explorer

Note: IRR Explorer v2

The NLNOG Foundation supported by RIPE NCC commissioned a complete rewrite of IRR Explorer: https://github.com/dashcare/irrexplorer

This v1 repository has been archived.

Explore IRR & BGP data in near real time

Background

IRR Explorer was written to make it easier to debug data in the IRR system. An example is to verify whether you would be impacted by deployment if filtering strategies such as "IRR Lockdown".

video: https://ripe70.ripe.net/archives/video/21/ pdf: https://ripe70.ripe.net/presentations/52-RIPE70_jobsnijders_irrlockdown.pdf

Setup

Copy irrexplorer_config.yml.dist to irrexplorer_config.yml and edit it to your liking.

Note that mirroring of many IRR databases is not possible without getting access granted.

( cd db ; ./bootstrap )

This will download IRR database snapshots and setup the database. It will take some time.

Continously updating BGP and IRR sources:

./worker

Finally start the web interface

./irexwww

There is query tool as well:

./query 1.2.3.0/24

It is rather basic and need some work though.

irrexplorer's Issues

Different type of warning for RFC1918 space

Looking at http://irrexplorer.nlnog.net/search/10.0.0.0

we should probably offer as advice "WTF IS THIS SHIT" instead of the current suggestion to properly register the RFC1918 space. :-)

Also maybe separate warnings for when a route-object exists versus only seeing it in BGP source.

Current rfc1918 (or we can call it 'bogon') space: 10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16, fe80::/7, 2001:db8::/32, 169.254.0.0/16 and some other blocks.

uppercase everything everywhere

To present people with a consistent experience across the board and to adhere to decade old tradition, it might be better if everything is upper case (autnum/as-set/source/members/you name it!). This would eliminate the need for a third-party client to normalise on their side because irrexplorer already normalises.

Investigate the use of database for IRRexplorer

Using py-radix requires a lot of manual work, and restricts the data model significantly.

E.g., it makes it impossible to have start+end time for certain prefix announce and do efficient queries to see what has happened for a certain prefix within a give timeframe. Such functionality can be extremely useful during dissemination of route leaks, spam origin, traffic hijacking etc.

It seems that PostgreSQL already supports the basic data type:
http://www.postgresql.org/docs/current/interactive/functions-net.html

And since 9.4 can build indexes that support efficient queries of overlaps (super/subnet):
http://michael.otacoo.com/postgresql-2/postgres-9-4-feature-highlight-gist-inet-datatype/

Further it might be worth partitioning the data on disk per database.

ARIN (non-nonauth) IRR Seems to be Missing

eg http://irrexplorer.nlnog.net/search/397444 doesn't show 2620:6e:a000::/48 as listed in ARIN IRR (which it is, just not ARIN-nonauth).

objects with colons cause issues

Do a search for something like: 23456.

There are many AS-SETS that include 23456, such as AS134262:AS-AIRTEL-IN.

If you search on this as-set in IRR Explorer it will return an error.

Multiple origin support

When multiple route objects exist for a route in the RIPE database only one origin AS is listed.

Example: 89.255.254.0/24

fix 'refresh' script

I want the script to work like: ./refresh ripe but it seems it will delete the entire routes table rather then just ripe.

mbrs-by-ref not honored

> whois -h whois.radb.net AS517:AS-REFTEST | egrep "mbrs-by-ref|members"
mbrs-by-ref:    AS286-MNT
members:        as517

> whois -h whois.radb.net -i member-of AS517:AS-REFTEST | egrep "member-of|aut-num|mnt-by"
aut-num:        AS2858
member-of:      AS517:AS-REFTEST
mnt-by:         RIPE-NCC-END-MNT
mnt-by:         AS286-MNT

irrexplorer returns (but no surprise, it doesn't parse mbrs-by-ref / member-of at all):

Macro Expansion
as_macro          depth  path              source  members 
AS517:AS-REFTEST  1      AS517:AS-REFTEST  ripe    as517
Showing 1 to 1 of 1 entries

Cosmetic bug?

http://irrexplorer.nlnog.net/prefix/192.176.144.0/24

Home Page - Missing "Source code available on Github."

Minor issue. Should the "Source code available on Github." also be listed on the home page? Only appears after you do a search.

Deep AS Query

Currently as number query only returns prefixes directly associated with the AS number (bgp homed/registered prefix).

This looks nice, but doesn't tell the full story. The query should return all covered/covering prefixes related to the initial set of prefixes. This will show any hijacked routes / invalid IRR prefixes entries and so forth.

This is already implemented in report.query_as_deep.
However the query is somewhat heavy for large AS (2914 takes 36 seconds on my laptop).
Furthermore 6to4 addresses pollute the results rather drastically and should probably be filtered. If done at query level, this might help performance, but I don't think it will do a lot.

/me puts on thinking hat.

Service starts before all trees are ready

The event driven stuff spawns before all databases are ready (in particular the ripe one).

504 gateway timeouts

I'm not sure if that would be the right place with issues that may be more related to the running instance http://irrexplorer.nlnog.net than the actual code.
If not than sorry about that - feel free to close this issue.
I have not done any analysis what is the actual root cause of IRR explorer not responding to http request (and NGINX acting as the reverse proxy responding with http 504 instead) so I don't know if this is more infrastructure issue or a problem with a tool. Anyway taking into consideration that requests for other prefixes work well and 125.21.91.0/24 consequently fails there might an issue with the irrexplorer itself which hangs on processing the request.

root@GB3TO01ICENOC02:~# curl -vv http://irrexplorer.nlnog.net/json/prefix/125.21.91.0/24?_=1566288430606
*   Trying 185.27.174.141...
* TCP_NODELAY set
* Connected to irrexplorer.nlnog.net (185.27.174.141) port 80 (#0)
> GET /json/prefix/125.21.91.0/24?_=1566288430606 HTTP/1.1
> Host: irrexplorer.nlnog.net
> User-Agent: curl/7.58.0
> Accept: */*
>
< HTTP/1.1 504 Gateway Time-out
< Server: nginx/1.14.0 (Ubuntu)
< Date: Tue, 20 Aug 2019 09:00:22 GMT
< Content-Type: text/html
< Content-Length: 192
< Connection: keep-alive
<
<html>
<head><title>504 Gateway Time-out</title></head>
<body bgcolor="white">
<center><h1>504 Gateway Time-out</h1></center>
<hr><center>nginx/1.14.0 (Ubuntu)</center>
</body>
</html>
* Connection #0 to host irrexplorer.nlnog.net left intact
root@GB3TO01ICENOC02:~#

IPv6 prefixes probably shouldn't be adviced as legacy

Quite often there will be v6 prefixes in IRR, but not in BGP.

While there can be many reasons for this, I'm guessing the most common is that the prefix has been assigned, but the network (or its as-free customer) hasn't deployed IPv6 yet. The advice should reflect this.

Sorting of prefix column in prefix.html is dumb

Currently it will sort like this:

1.x.x.x
10.x.x.x
2.x.x.x
etc

Would be good if the default sort method for that column is something that does a better job at numeric sorting :-)

Prefix link should be in prefix, not as

The link to prefix info is nice, but should be on the prefix, not AS.

The as should link to either some information about the network (peeringdb?) or maybe list all prefixes for that AS.

Google's ASN 15169 crashes irrexplorer

Do not click: hxxp://irrexplorer.nlnog.net/search/15169

Seems google has a lot of IRR objects, nearly 7000:

Vurt:~ job$ bgpq3 as15169 | wc -l
    6715

Either irexwww or browsers in general have trouble with the size of it.

Safari 11.1 doesn't display results, CORS error with cdnjs

Safari 11.1 errors on the js loading, CORS issues with the HSTS preload lists/HTTPS redirect it looks like.
cdnjs/cdnjs#4328 (comment) looks like the same issue with a suggested workaround.

Add AFRINIC to default sources list

AS number/macro parser guesses wrong sometimes

IRRexplorer thinks this is an as number, not a set:
AS9498:AS-BHARTI-IN

dbname seems to be missing in irrexplorer_config.yml.dist

I get this when starting IRRExplorer locally:

htj@pyrite:~/src/irrexplorer$ python irrexplorer.py
Traceback (most recent call last):
File "irrexplorer.py", line 205, in
worker = NRTMWorker(feedconfig, lookup_queues[name], result_queues[name])
File "irrexplorer.py", line 125, in init
self.dbname = feedconfig['dbname']
KeyError: 'dbname'

I've added some manual dbname field to in my irrexplorer_config.yml, which seems to fix this. Currently just using altdb, apnic, etc. Not sure if that is right, or if it matter. If it is right, it could just the entry name instead of a seperate field.

display most specific route at top of list

When you search for an IP or prefix which is covered by other routes in either BGP or IRR, it might make sense to either highlight the most specific entry or place the most specific entry at the top

example: http://irrexplorer.nlnog.net/prefix/216.107.151.110

Less Specific Route ou Route6 IRR Objects resulting RED advice

Route and Route6 IRR Object doesn't have GE LE as ROAs.
So, its no possible to define the max netmask allowed with an entry.

The /24 and /48 are the De Facto stantard in max prefix lenght allowed in DFZ.

In pratical terms of filtering, for an Small ISP with a single /22 and a /32, two simple entries with orign of his on ASN are enough to ensure the correct filtering on his upstreams and peers.

But, on IRRExplorer, if an ASN creates those two IRR Route and Route6 that I mentioned, and by some reason he needs to disagregate( 😭 ), the disagragated prefixes contained on the less specific Route/Route6 object will be presented as a RED-WRONG-THING...
"Prefix in DFZ, but no route-object with correct origin anywhere"

What I'm seeing to happen is that same ASNs are also diagregating their IRR Objects just to get the "every thing is GOOD-And-Green" on IRRExplorer.

The result of that is larger preffix lists on ITPs, em harder work to reduce that work.

My wish/suggestion is that a prefix found on DFZ that in contained on a more specific IRR entry(until /24 to v4 and /48 to v6) could get also a Green status...
Not exactly "Looks good: BGP origin consistent with AS in route-objects".

Maybe a lighter green with "Looks good: BGP origin consistent with AS in a less specific route-objects".

Confusing rpki results

Since i created ROA's for some of our resources the results are a bit confusing:

We announce a /29 and in the roa we state it can have a max prefix of /48. Looks like a lot of the possible subnets of the /29 are also shown. Idem for ipv4

It take a lot of time to parse al this data

Expand AS-[0-9]+ style macro's

Bug in IrrExplorer, tool doesn't work for AS-57344 (At least not the expansion part.)
In utils.py ... guess we should catch AS- or if int(data[2:]) is negative, treat it as ASmacro.

Show more as object info

AS-macros and as numbers are somewhat anonymous currently.

Consider adding as-name and desr to site. Currently this is not stored in the database, so we have to add some stuff for that.

Say which IRR has a record

ex:
http://irrexplorer.nlnog.net/search/2001:4860::/32

Would be nice to know which IRR has the record in question

Add "search exact" method

Sometimes you just don't need all the other entries in IRR or BGP that are more specific

proposed name: /exact_prefix/<path:prefix>

and then just only display that exact prefix + output from subprocess whois -h rr.ntt.net $prefix

Bug or feature?

http://irrexplorer.nlnog.net/prefix/8.8.8.0/24

bogus error on foreign object w/ different origin

Lookup: http://irrexplorer.nlnog.net/prefix/80.94.64.0

Gives "Proper RIPE DB object, but foreign objects with different origin also exist", although both radb & ripe report "5580".

go fix pls thx bye.

Handle AS23456 properly

For instance:

http://irrexplorer.nlnog.net/prefix/80.210.80.0/20

I am not exactly sure what the solution here is. Is it safe to simply ignore anything with AS23456?

BGP table should refresh or exabgp

Currently the BGP table does not refresh itself every 300 seconds, but the thread becomes non-responsive after trying a refresh: there are bugs here https://github.com/job/irrexplorer/blob/master/irrexplorer/bgp.py#L141

Option 1: fix refresh capability and use the lg01.infra.ring.nlnog.net table

Option 2: get rid of the table fetch & parser, launch an instance of exabgp in a subprocess, and parse the updates from exabgp STDOUT (JSON blobs) in real time to maintain the radix tree

Confusing advise for RIPE-based prefix with 1 ROA (AS0), no IRR and no BGP

Looking up a RIPE-based prefix with 1 ROA (AS0), no IRR and no BGP:

Route objects in foreign registries exist, but no BGP origin. Consider moving IRR object to RIPE DB or deleting them.

As discussed with @job, this is confusing, because there is no route object at all. This should instead lead to $whatever advise we think is good.

Flash/jinja form error on request

• Running on http://0.0.0.0:5000/
  sleeping 300 seconds before reconnecting to rr.ntt.net (LEVEL3)
  127.0.0.1 - - [15/May/2015 12:51:16] "GET / HTTP/1.1" 500 -
  Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/flask/app.py", line 1836, in call
  return self.wsgi_app(environ, start_response)
  File "/usr/lib/python2.7/dist-packages/flask/app.py", line 1820, in wsgi_app
  response = self.make_response(self.handle_exception(e))
  File "/usr/lib/python2.7/dist-packages/flask/app.py", line 1403, in handle_exception
  reraise(exc_type, exc_value, tb)
  File "/usr/lib/python2.7/dist-packages/flask/app.py", line 1817, in wsgi_app
  response = self.full_dispatch_request()
  File "/usr/lib/python2.7/dist-packages/flask/app.py", line 1477, in full_dispatch_request
  rv = self.handle_user_exception(e)
  File "/usr/lib/python2.7/dist-packages/flask/app.py", line 1381, in handle_user_exception
  reraise(exc_type, exc_value, tb)
  File "/usr/lib/python2.7/dist-packages/flask/app.py", line 1475, in full_dispatch_request
  rv = self.dispatch_request()
  File "/usr/lib/python2.7/dist-packages/flask/app.py", line 1461, in dispatch_request
  return self.view_functionsrule.endpoint
  File "/home/htj/src/irrexplorer/irrexplorer.py", line 478, in index_old

  File "/usr/lib/python2.7/dist-packages/flask/templating.py", line 128, in render_template
  context, ctx.app)
  File "/usr/lib/python2.7/dist-packages/flask/templating.py", line 110, in _render
  rv = template.render(context)
  File "/usr/lib/python2.7/dist-packages/jinja2/environment.py", line 969, in render
  return self.environment.handle_exception(exc_info, True)
  File "/usr/lib/python2.7/dist-packages/jinja2/environment.py", line 742, in handle_exception
  reraise(exc_type, exc_value, tb)
  File "/home/htj/src/irrexplorer/templates/index.html", line 4, in top-level template code
  {% import "bootstrap/utils.html" as util %}
  File "/home/htj/src/irrexplorer/flask_bootstrap/templates/bootstrap/base.html", line 1, in top-level template code
  {% block doc -%}
  File "/home/htj/src/irrexplorer/flask_bootstrap/templates/bootstrap/base.html", line 4, in block "doc"
  {%- block html %}
  File "/home/htj/src/irrexplorer/flask_bootstrap/templates/bootstrap/base.html", line 20, in block "html"
  {% block body -%}
  File "/home/htj/src/irrexplorer/flask_bootstrap/templates/bootstrap/base.html", line 23, in block "body"
  {% block content -%}
  File "/home/htj/src/irrexplorer/templates/index.html", line 44, in block "content"
  {{ wtf.quick_form(form, form_type='inline', button_map={'submit_button': 'primary'}) }}
  File "/home/htj/src/irrexplorer/flask_bootstrap/templates/bootstrap/wtf.html", line 154, in template
  {{ form.hidden_tag() }}
  UndefinedError: 'main.InputForm object' has no attribute 'hidden_tag'

Do I need something special here?

RPKI results not showing up

Hi!

Recently the RPKI results stopped showing up. Does anyone know if there is an issue?

Regards,
Sergio.

display latest irr/bgp sync at the bottom of the page

So people have a sense of 'freshness' (maybe also an indicator when the next refresh will be? or even a real time display of what worker.py is doing if you go to /console?)

Better error message when database is unavailable

If the database is down, one just gets a red box. Not a proper error.

Confusing results

This briefly confused us. We searched for 144.202.0.0/17, and got these results:

It looks like it went up a level to the /16, then back down to the next /17?

(attached a screenshot because we're already working on cleaning up the incorrect /16)

Empty as macro

http://irrexplorer.nlnog.net/search/AS-COLOCLUE

Has member in ripe irr.

loss of functionality

all queries return "The server encountered an internal error and was unable to complete your request. Either the server is overloaded or there is an error in the application."

Setup instructions are out of date

To recreate:

• Get a fresh clone of the repo.
• mkvirtualenv irrexplorer
• pip install -r requirements.txt
• ./query 1.2.3.0/24

10:54:14 [dotwaffle@engelbert:~/Code/irrexplorer] [irrexplorer] master ± ./query 1.2.3.0/24
Traceback (most recent call last):
  File "./query", line 47, in <module>
    main()
  File "./query", line 22, in main
    result = report.prefix_report(db, so.value)
AttributeError: 'module' object has no attribute 'prefix_report'

Additionally, irr.yml no longer exists in the repo.

Report includes prefixes that are out of scope

IRRExplorer seem to include prefixes that are not in scope of the requested prefix.

For instance:
http://irrexplorer.nlnog.net/prefix/193.10.68.0/24

query for supernet of IRR/BGP prefixes

Please consider supporting a query for a supernet (summary) of multiple IRR/BGP prefixes.
example: 141.36.0.0/16 -> currently "Could not find any matching prefix in IRR or BGP tables for 141.36.0.0/16"

I would like to see the combined results for all longer prefixes, in this case:
141.36.180.0/23
141.36.186.0/23

Remove clean-up suggestion on RFC7607 ROAs

One thing to consider is that when I publish an AS0 origin ROA (in my case for an IXP - http://irrexplorer.nlnog.net/search/206.80.238.0) is that IRRExplorer reports it as a possibly legacy object that could be cleaned up.

In the small edge case of not being visible in BGP, and having an ROA of AS0, perhaps we'd want to use different language for the result and color it green?

replication bug

Error during IRR update:  ERROR: serials (32143748 - 32198870) don't exist!
ERROR: the above error probably means that the gap between your seed snapshot and the current SERIAL on the stream source is too large.
rr.ntt.net 43 SAVVIS
Streaming from rr.ntt.net:43/savvis from serial 397001
START Version: 3 savvis 397001-397186
Changes: add_route: 140  del_route: 35
Error during IRR update:  invalid cidr value: "216.52.0.1/21"
LINE 1: SELECT create_route ('216.52.0.1/21', 30282, 'savvis');
                             ^
DETAIL:  Value has bits set to right of mask.

rr.ntt.net 43 TC
Error during IRR update:  current transaction is aborted, commands ignored until end of transaction block

appconfig.cfg is missing from distribution

Traceback (most recent call last):
File "irrexplorer.py", line 507, in
create_app().run(host="0.0.0.0", debug=True, use_reloader=False)
File "irrexplorer.py", line 449, in create_app
app.config.from_pyfile('appconfig.cfg')
File "/usr/lib/python2.7/dist-packages/flask/config.py", line 128, in from_pyfile
with open(filename) as config_file:
IOError: [Errno 2] Unable to load configuration file (No such file or directory): '/home/htj/src/irrexplorer/appconfig.cfg'

:-(

Any chance this could be added?

I've also managed to get myself blacklisted from RADB and APNIC. I hope it it only temporary :-).

Strip whitespace from user input in search box

We should strip leading or trailing whitespace from the search box. This can occur when you copypasta and do it a bit wrong.

Seems to work ok for AS numbers - e.g.

http://irrexplorer.nlnog.net/search/%208330
works

but:
http://irrexplorer.nlnog.net/search/%205.57.80.0/20
doesn't

IRR Explorer has weak newline handling

commit afe956b seems to fix NRTM, but breaks ./bootstrap. In the downloaded .db.gz files there are 'random' newlines all over the place, maybe at the head of the file, maybe at the end, who knows! :-)

Maybe some rewrite to start parsing the previous lines the moment a \n was found is better then the current 'lookahead' approach?

AS number input

Any chance of being able to input an AS number aside from a prefix? I understand that it is still early in development.

Cool stuff otherwise. I can see we have some legacy stuff to cleanup.

Any chance/etc for some speedups?

Only display columns which have any data

Currently we list all IRR databases in the prefix.html UI, regardless of actual data in that IRR for any of the prefixes requested in the view. A more condense UI is easier to read. Seems lots of people forget to scroll to the right.