Quality assurance / v3 consistency checks

[mxsasha]

Quality assurance / v3 consistency checks

[mxsasha]

A test has been run using 320.000 queries from rr.ntt.net against a local version of IRRDv2.3 (as run by NTT) and IRRDv4. 35.000 queries produced errors in both instances as they were invalid, 20.000 NRTM queries were skipped, along with all asdot queries.

To be fixed:

• !oMAINT-AS7091: inet-rtr 198.32.175.80` appears to be returned by the psql query but not in the whois result.
• !oMAINT-AS2527: as-sets AS-SO-NET(6) from NTTCOM are missing in whois response, JPIRR versions present. Both are returned by the psql query, likely related to previous issue. All are included in the whois output when filtering with -T as-set.
• A number of routes are missing from the v4 psql DB: 66.241.152.0/24/ARIN for !gAS36221, , 193.187.153.0/24/ARIN for !gas395800, 23.223.195.0/24 for -i origin AS20940.
• The tests for !i queries should be re-run after search order is consistent, as that significantly affects results.
• Queries for certain mntners produce a partial object, notably tenue-mnt and -i mnt-by foo
• Responses with -K flag are not filtered for uniqueness, e.g. -K -r -i origin AS127 leads to duplicate results due to duplicates present in different sources.
• -K responses should not include member-of
• Research performance of !gAS7552 and !6as38547.
• Leaf/dangling set members should only be included in non-recursive !i queries

Notes about v2.3:

• Returns exact matches even when asked for a less specific match, and sometimes does not include the actual less specific match
• Accepts combined ,Lo parameters for !r which are invalid and processed incorrectly.
• Some !i responses may have random extra newlines?
• !iAS-BAKKER returns a mntner name from mbrs-by-ref, which is invalid.
  Origin searches for AS40224, AS28001, AS20324, AS14522, AS9541 and others return double responses, i.e. identical primary key and source, from sources ARIN-WHOIS/RPKI.
• -x queries are accepted without spaces, which is invalid
• 186.211.120.0.24/LEVEL3 is missing for !gAS262747 because it's origin is in asdot

Other than this, I did not notice significant differences in query responses.

[mxsasha]

New run with fixes: 329982 queries ran, 216 with different results, 35724 produced errors on both instances, 74280 invalid queries were skipped, 19877 NRTM queries were skipped.

General conclusions:

• A few bugs have surfaced in IRRDv4, noted below.
• IRRDv2 set resolving (with !i queries) is very buggy, particularly in terms of correct source prioritisation, and when using mbrs-by-ref/member-of. This means that it is not possible to fully track down all query response differences, particularly because these queries are very complicated to track down. However, all researched queries fit IRRDv2 bugs.
• IRRDv2 accepts certain invalid queries and makes a best effort to parse them. IRRDv4 will not.
• IRRDv2 can store return duplicate objects (same RPSL primary key and source), even though their descriptions are different. IRRDv4 enforces PK+source uniqueness.

Possible bugs in IRRDv4:

• !iRS-KROOT-V6,1and !iAS12654:RS-RIS,1 return IPv6 prefixes in IRRDv4 -> moved to #134

Details on differences in responses that are considered bugs in IRRDv2, or intentional behaviour changes in IRRDv4:

• !gAS262747 is missing 186.211.120.0/24 because its origin is ASDOT
• !gas49112 is missing 176.52.224.0/20 because of #52
• -T route6 -i origin AS14613 is missing 2001:470:33::1/48, as is 2001:420:3400::/34 for AS109. 2620:10a:e0::/40 for AS11250, due to #62
• !i,AS-4761, !i,AS-55805, !i,AS-DIRECT_PEERS, !i,AS-DRUKNET-TRANSIT, !i,AS-SET-GTI, !i,AS-TIG-GROUP, !i,AS-VECTANT, !i,AS9583:AS-SIFY, !man,as2914, !r1.1.1.1,Lo, -KSI, -i origin 10204, -i origin 132359, -i origin 134069, -i origin 137263, -i origin 29457 are missing a response in v4 because it is an invalid query format. IRRDv2 makes an attempt to interpret the query regardless, with varying degrees of success.
• !iAS-AKAMAI, !iAS-FORTIS, !iAS-INTERCLOUD, !iAS-IPTEL, !iAS-MEEBO, !iAS-MITSOL, !iAS-ORION, (and possible ,1 variants) are missing a response in v4, because the source prioritisation places RADB over RIPE, and there is an empty RADB as-set with that name. IRRDv2 appears not to consistently follow its own source prioritisation.
• !iAS-9299ARCSTAR,1, AS-ITERANET AS-INFOCOM and others have different (but not empty results) also because IRRDv2 does not correctly follow its own source prioritisation. For example, for AS9299ARCSTAR, AS36997 in RIPE-NONAUTH has member-of AS-INFOCOM which is indirectly part of AS9299ARCSTAR. However, v2 appears to select AS36997 from AFRINIC, which has no member-of, so AS36997 only appears in an IRRDv4 answer. This is expected to explain many of the differences in !i set resolving queries. (This is not adequately explained by source selection stickiness to the original source of AS-9299ARCSTAR.)
• Queries in the format of !iRS-COMM_GFDAB,1, !iRS-COMM_GFDAE,1, etc. have an empty response in IRRDv2, but many prefixes in IRRDv4. The route-sets, which occur in both NTTCOM and INTERNAL have mbrs-by-ref:ANY set. The route objects in NTTCOM do not have a member-of set, but the ones in INTERNAL do. Even though the preferred route-set is from NTTCOM, IRRDv4 will search for any object in the included sources with member-of set to the route-set, because prioritisation would not make a meaningful difference in the result. IRRDv2 does appear to be sticky in which sources it uses to look for route(6) objects, which is inconsistent with documentation.
• !iAS-ALPHACRON, !iAS-ANEXIA, !iAS-AS19905, !iAS-BAKKER, !iAS-BYTESOURCE, !iAS-CHKALOVSK, !iAS-COMM-IT, !iAS-EDIS, !iAS-FFGT, !iAS-HKCOLO, !iAS-I3B, !iAS-I3B-TRANSIT-V4, !iAS-I3B-TRANSIT-V6, !iAS-IGLOO22225, !iAS-IP-IT, !iAS-IWAN, !iAS-MEINSYSTEM, !iAS-MNCDN, !iAS-NETSTYLE, !iAS-NOE-LFWV, !iAS-PHORUS, !iAS-PRUNK, !iAS-RENOME, !iAS-ROUTELABEL, !iAS-RWILLIAMS, !iAS-SASEUROPE, !iAS-SELFNET, !iAS-SERVICENOW, !iAS-SET-ITC, !iAS-SET-SCOPESKY, !iAS-SET-STW, !iAS-SOFIA-CONNECT, !iAS-SPINSATNET, !iAS-TDC, !iAS-TEAMVIEWER, !iAS-TELETRONIC, !iAS-TOMMYBOWDITCH, !iAS-VIRTELA-COMM, !iAS12486:AS-YEMEN-NET, !iAS30873:AS-YEMENNET-CUSTOMERS, !iAS3320:AS-AUTH-PILOT-BLACKHOLE-FROM-AS13194 give different responses in v4, because v2 incorrectly includes the value(s) of the mbrs-by-ref attribute.
• !iAS-ATSTELECOM, !iAS-E_MOUSE have a different answer due to incorrect case sensitivity in filtering the original query input from the AS set output in v2
• !iAS-TWIG,1 have different answers due to v2 including an extra comma before the start of the output
• !oMAINT-AS2527, !oMAINT-AS7091, -T mntner maint-nttcom-bb, -T mntner maint-nttcom-ra, -T mntner maint-nttcom-sla, -i mnt-by MAINT-AS10217 -i mnt-by as790-mnt, -i mnt-by foo, have different answers, because of slightly different masking of auth lines
• !r1.1.1.0/24,l, !r193/21,l have more answers in v2, because v2 does not actually filter for less specifics, but incorrectly filters for exact match or less specific
• -F AS2914, -K 193/21 -K as1234, -k 193.0.0.1, have different answers, because IRRDv4 includes the less specific objects that match too (in AS2914's case an as-block in the AFRINIC db)
• route6 queries in the format -T route6 -i origin $AS produce different results for as14522, as17732, as19114, as19754, as23678, as23838, as24482, as25697, as25780, as27901, as28001, as30479, as32875, as33517, as36915, as3816, as38719, as45177, as45224, as55569, as55803, as56264, as58717, as58779, as64098, as9541. This also happens for -i origin AS$ queries for as14522, as19194, as20324, as28001, as29761, as40224, as45925, as46261, as9541. The issue is that IRRDv2 returns "duplicate" results from sources ARIN-WHOIS and RPKI, i.e. objects with the same source and same primary key. The descriptions are different, but those are not part of a route(6) PK. IRRDv4 enforces PK/source uniqueness and therefore does not return duplicates.
• Queries for object templates have different output due to slightly different formatting of the template.