False negatives

[arlolra]

False negatives

[Ryman]

Phillip will send his code in a few days, so I just did some simple grepping on the consensus so far. I'll have to look deeper into issues where the rules aren't matching behaviour, see what the descriptor says about them.

For the last group, will need to re-run to get the relay fingerprint and ip to figure out which relay caused the issue. (Could also grep tordnsel history, but that could be coincidental?)

Unique Check2 errors

##in consensus ** should have accepted
199.48.147.40 - accept on 443 (Amunet6)
83.46.34.138 - reject but NOT rejecting 443 (r Unnamed UsTgd3p5Lbl1sXL3NiUkn9Ie0NQDsvTkaMoWq4eBIjUlvYpqt+tDQA)
85.197.43.75 - reject but NOT rejecting 443 (armselig)
120.56.175.53 - reject but NOT rejecting 443 (kapler)
120.56.160.58 - reject but NOT rejecting 443 (hosg)

##in consensus ** should have rejected
188.242.194.157 - reject 1-65535 (torDewus)
66.180.193.219 - reject 1-65535 (tordienet)
93.31.155.175 - reject 1-65535

Unique Check1

46.163.115.187 - in consensus - accept on 443 (surfanon03)

Bad for both

##in consensus ** should have rejected
78.142.175.70 - reject 1-65535 (SCggHJTyWbtzDA9YZi) (bandwidth is also 16)

##not in consensus - likely different public-ip
91.207.183.244 
31.185.27.1
76.10.128.88
72.52.91.19
178.140.198.156
208.89.209.124

[arlolra]

Thanks @Ryman

These two,

188.242.194.157 - reject 1-65535 (torDewus)
66.180.193.219 - reject 1-65535 (tordienet)

actually have a more complex policy that includes,

reject 38.229.70.0/24:*

What you're seeing there is just the summary.

Since IsTor uses,

var DefaultTarget = AddressPort{"38.229.70.31", 443}

we have an answer there.

[arlolra]

This one,

83.46.34.138 - reject but NOT rejecting 443 (r Unnamed UsTgd3p5Lbl1sXL3NiUkn9Ie0NQDsvTkaMoWq4eBIjUlvYpqt+tDQA)

is 52C4E0777A792DB975B172F73625249FD21ED0D4 which TorDNSEL thought,

ExitNode 52C4E0777A792DB975B172F73625249FD21ED0D4
Published 2013-10-24 15:43:10
LastStatus 2013-10-24 21:02:57
ExitAddress 88.22.146.119 2013-10-24 16:03:24

at the time (10-11). However, it's no longer in the output at noon.

[arlolra]

85.197.43.75 - reject but NOT rejecting 443 (armselig)
120.56.175.53 - reject but NOT rejecting 443 (kapler)
120.56.160.58 - reject but NOT rejecting 443 (hosg)

These three it probably should have gotten right and, indeed, if I just take a consensus and exit list from that time as input, it does. They all seem to be changing IPs every few hours over the month, so perhaps a caching issue somewhere along the line.

[arlolra]

Again, for 199.48.147.40 - accept on 443 (Amunet6) TorDNSEL thinks,

ExitNode 0E07902A1004F72BFDFF9497B9FA07604619F111
Published 2013-10-25 04:06:47
LastStatus 2013-10-25 05:03:03
ExitAddress 199.48.147.36 2013-10-25 05:37:28

Amunet6 is part of a set, Amunet 1-8, which yield,

199.48.147.35
199.48.147.36
199.48.147.37
199.48.147.39
199.48.147.39
199.48.147.41
199.48.147.41
199.48.147.42

instead of https://atlas.torproject.org/#search/amunet

[arlolra]

This one's kind of interesting, 93.31.155.175 - reject 1-65535
TorDNSEL gets it right and it's in the consensus, so you'd think it'd be there.

Edit: scratch that getting tired and grepping wrong. Don't have an answer for this one.

[Ryman]

Haven't had a chance to dive into the descriptors yet, but for:

reject 38.229.70.0/24:*
What you're seeing there is just the summary.

Since IsTor uses,

var DefaultTarget = AddressPort{"38.229.70.31", 443}
we have an answer there.

I guess we should update this, is there a neat way we can keep this up to date? (Currently I'm seeing check & check2 @ 38.229.72.22)

And for Amunet6(etc), are you thinking we'll need to allow lookups via TorDNSEL ips and the ip provided in descriptors for isTor? (Or perhaps trying to handle families, which is basically a trust system?)

These three it probably should have gotten right and, indeed, if I just take a consensus and exit list from that time as input, it does. They all seem to be changing IPs every few hours over the month, so perhaps a caching issue somewhere along the line.

Might be worth updating exits.Update to group by ip rather than fingerprint? Should allow us to retain older mappings from ip -> descriptor rather than saying one fingerprint = one ip?

[arlolra]

@Ryman In case you missed it, check2 is now live on https://check.torproject.org

Did you happen to get the scripts to rerun those tests?

Thanks for all your efforts.

[Ryman]

Caught a few trac update mails, good job! 👍

Did you happen to get the scripts to rerun those tests?

No, haven't received an updated, I just sent a reminder now. If I don't get an update/code by EOW, I'll look to hack a quick equivalent up with Stem over the weekend.

Basic functionality is just grab a consensus, build circuits exiting from each exit flagged relay (or all relays?) and hit check.tpo for success/fail right? Nothing additional required?

[arlolra]

Basic functionality is just grab a consensus, build circuits exiting from each exit flagged relay (or all relays?)

If there's no exit flag, could you exit? :)

and hit check.tpo for success/fail right? Nothing additional required?

Yup.

[Ryman]

Scripts here https://github.com/NullHypothesis/exitmap

Plan to review tonight and run tomorrow. Will update!

[Ryman]

Just a heads up that this might take a while longer to meet our needs, but it looks like we've still got a number of failures. <I don't think it's worthwhile to investigate the IP's below until we get a fingerprint, but one of us should look into the fingerprinted ones when able.>

Here's some failures for 2013/11/22 16:08:00 GMT:

31.185.27.1 
63.147.126.67 *
66.180.193.219 *
72.52.91.19
77.249.18.242 *
78.142.175.70 *
81.7.13.111 *
85.214.57.119 
91.76.128.65 *
95.211.139.146 *
188.242.194.157 
198.180.167.4 *
208.89.209.124 *

Another run at 16:46 - I've marked the IPs above with stars if they're in both and below if it's unique to below.

63.147.126.67
66.180.193.219
77.249.18.242
78.142.175.70
81.7.13.111
91.76.128.65
91.207.183.244 *
93.31.155.175 *
95.211.139.146
198.180.167.4
208.89.209.124

It's tricky to get an associated fingerprint for each circuit with the current code structure but I'll look to do that over a rewrite first.

Due to the rerun's and that when I check single fingerprints e.g. 9F9D031FA907CB9AE0746B73CD735D3EC3378843, it still fails and yet it's missing from the second set of results, so it looks like we can expect more failures than those listed above since it seems to be bombing during execution at some point. (I'm thinking it's tor timing out circuits before they get tested and not retrying cleanly)

5 of these are Atlas with exit policies that should have worked for the current DefaultTarget (although check is running on a different IP 38.229.70.31 vs 38.229.72.22) All of these have been up for multiple hours (one for over 66days) so I'm not sure why it wouldn't have been accepted by check.

https://atlas.torproject.org/#details/9F9D031FA907CB9AE0746B73CD735D3EC3378843
https://atlas.torproject.org/#details/03FF94D9E5001DD2290BC3B19FA7F59CE1E30279
https://atlas.torproject.org/#details/059FC4D27DF934739215E48DADF71DF4DD56DDAE
https://atlas.torproject.org/#details/9B0AF2F5C6A4AD23E883CDC040BAB2CF63972849
https://atlas.torproject.org/#details/65C35C03571307D7546D6978605A6B11B473F6EE

[Ryman]

2013/11/22 22:34 GMT

BA998001D8F8E82EEFBDFA209EF57C0D6ED43F03 Unnamed 72.52.91.19 False
059FC4D27DF934739215E48DADF71DF4DD56DDAE vc 91.76.128.65 False
9B0AF2F5C6A4AD23E883CDC040BAB2CF63972849 tordienet 66.180.193.219 False
9F9D031FA907CB9AE0746B73CD735D3EC3378843 torDewus 188.242.194.157 False
5D263037FC175596B3A344132B0B755EB8FB1D1C anduinExit01 31.185.27.1 False
5B0B9946BB6750169A34878E768598E209ADFEDF Unnamed 63.147.126.67 False
62BEE61EB88D4A81C3BA3931D6FA999D706AC4D5 Radiolama 62.149.12.153 False
EB3D0EEECE0396984D840DD2A6ECF9C65DE8D471 9ZCC3 91.207.183.244 False
9E56EA2C1C0644FCBEF2EDCC90D29A1584AB056C torrelaymadavide 85.214.57.119 False
BFF292795AD1E09F1AB4EE744D5C6C7576B06306 rondienode1 81.7.13.111 False
03FF94D9E5001DD2290BC3B19FA7F59CE1E30279 SCggHJTyWbtzDA9YZi 78.142.175.70 False
8BA8F6255F207761B252672C86D71688D640BCC8 caprica 198.180.167.4 False
9BFA01F07560A4185CE5A118AEE3C8A6A57ED020 qesyrocp 95.211.139.146 False
ED6A0EC56B459BC03F3342900DF6A1548F75A286 snowden 208.89.209.124 False

Still haven't managed to fix the timeout problem so it's still not a full result set per run, but the above is: fingerprint, nickname, IP check.tpo showed, (false is failure)

[arlolra]

5 of these are Atlas with exit policies that should have worked for the current DefaultTarget (although check is running on a different IP 38.229.70.31 vs 38.229.72.22) All of these have been up for multiple hours (one for over 66days) so I'm not sure why it wouldn't have been accepted by check.

https://atlas.torproject.org/#details/9F9D031FA907CB9AE0746B73CD735D3EC3378843
https://atlas.torproject.org/#details/03FF94D9E5001DD2290BC3B19FA7F59CE1E30279
https://atlas.torproject.org/#details/059FC4D27DF934739215E48DADF71DF4DD56DDAE
https://atlas.torproject.org/#details/9B0AF2F5C6A4AD23E883CDC040BAB2CF63972849
https://atlas.torproject.org/#details/65C35C03571307D7546D6978605A6B11B473F6EE

Ok, I figured out what's going on here. All of these have an exit policy summary that says, reject 1-65535. If you look at the exitips.py script, you'll notice it ignores relays in the consensus that don't allow exiting is_exiting_allowed() == False based on the summary, which I guess is a pretty poor assumption to make. https://github.com/arlolra/check/blob/master/scripts/exitips.py#L54

[arlolra]

Ok, deployed that fix and the 5 you mentioned now return true,

9F9D031FA907CB9AE0746B73CD735D3EC3378843 torDewus 188.242.194.157
03FF94D9E5001DD2290BC3B19FA7F59CE1E30279 SCggHJTyWbtzDA9YZi 78.142.175.70
059FC4D27DF934739215E48DADF71DF4DD56DDAE vc 91.76.128.65
9B0AF2F5C6A4AD23E883CDC040BAB2CF63972849 tordienet 66.180.193.219
65C35C03571307D7546D6978605A6B11B473F6EE MyNickIsNobody 93.31.155.175

plus two more,

D5F2C65F4131A1468D5B67A8838A9B7ED8C049E2 che 213.65.180.228
C4D41BFB33993F9D64699726A445283CE307E30C ididnteditheconfig 166.70.15.14

[arlolra]

Ok, that last commit should fix,

EB3D0EEECE0396984D840DD2A6ECF9C65DE8D471 9ZCC3 91.207.183.244
ED6A0EC56B459BC03F3342900DF6A1548F75A286 snowden 208.89.209.124
BA998001D8F8E82EEFBDFA209EF57C0D6ED43F03 Unnamed 72.52.91.19
62BEE61EB88D4A81C3BA3931D6FA999D706AC4D5 Radiolama 62.149.12.153
9BFA01F07560A4185CE5A118AEE3C8A6A57ED020 qesyrocp 95.211.139.146
9E56EA2C1C0644FCBEF2EDCC90D29A1584AB056C torrelaymadavide 85.214.57.119

leaving only these three, which I'm not sure about.

5D263037FC175596B3A344132B0B755EB8FB1D1C anduinExit01 31.185.27.1
8BA8F6255F207761B252672C86D71688D640BCC8 caprica 198.180.167.4
BFF292795AD1E09F1AB4EE744D5C6C7576B06306 rondienode1 81.7.13.111

I think it's time for another run. See how things are progressing.

[arlolra]

8BA8F6255F207761B252672C86D71688D640BCC8 caprica 198.180.167.4
https://atlas.torproject.org/#details/8BA8F6255F207761B252672C86D71688D640BCC8

This one TorDNSEL will never find because it isn't listening on 443 or 80 so it can build circuits to itself.

[Ryman]

25/11/2013 22:26

Down to 5 (but run seemed to fail for last 80-90 routers)

BFF292795AD1E09F1AB4EE744D5C6C7576B06306 rondienode1 81.7.13.111 False
5D263037FC175596B3A344132B0B755EB8FB1D1C anduinExit01 31.185.27.1 False
E1B6FF05FE098E9652D3A8C28FC7E1E48DD3B9CE cityConnect 87.64.82.40 False
DD1AB742EFF955C5248FD880003ADF80D86D4B17 kote 77.51.237.90 False
3D4B517843B5E6B71E40BED6E8C3D26DEFFACB3B MeinServer5 84.189.118.221 False

[Ryman]

25/11/2013 23:26

Down to the remaining 2 that you'd flagged before, so does seem the other 3 were timing issues :)

5D263037FC175596B3A344132B0B755EB8FB1D1C anduinExit01 31.185.27.1 False
BFF292795AD1E09F1AB4EE744D5C6C7576B06306 rondienode1 81.7.13.111 False

Promising, but still not a 'full' run!

[Ryman]

26/11/2013 13:50

Same as before

5D263037FC175596B3A344132B0B755EB8FB1D1C anduinExit01 31.185.27.1 False
BFF292795AD1E09F1AB4EE744D5C6C7576B06306 rondienode1 81.7.13.111 False

Published address is 198.180.167.5, I'm guessing caprica exits from a number of addresses as it's been correct sometimes and wrong at other times.

8BA8F6255F207761B252672C86D71688D640BCC8 caprica 198.180.167.4 False

New router - ~1hr10mins uptime

5C7547A2803D26477DE3FAAFBEF1A50D629C7C78 default 62.182.194.21 False

[Ryman]

26/11/2013 17:30

Favourites

5D263037FC175596B3A344132B0B755EB8FB1D1C anduinExit01 31.185.27.1 False
BFF292795AD1E09F1AB4EE744D5C6C7576B06306 rondienode1 81.7.13.111 False

Wrong IPs

8BA8F6255F207761B252672C86D71688D640BCC8 caprica 198.180.167.4 False
951ACA179D6E447D552D5398269282F6AB87BA2B Unnamed 50.7.13.28 False

~1-2hours uptime

222E8CB6B8B763AF09929F6D8D623BCE77A11074 CrossRoads 5.254.139.56 False
52E24E092E1E3CC0DFAAC3B0ECFE1086633C7C07 Musicmail 120.59.164.189 False
4E8FBAAB78C038F66B6DD844FB94F95DCEA54B70 vasyalee 109.122.36.187 False
6C9AAFEA27FDF5206E6D59F67AA3DCC29C60072C Parker 120.56.173.15 False
106EA56537CCB405C4301A9F6ACCD19864B211AB Ubu 86.70.225.149 False
E0BABE4195E33C1FC339C0C23C29F958E0B26DEC Unnamed 188.242.18.139 False
7CABD88CFED3023082D45CD87B96D8030624A038 armselig 89.0.239.100 False
67001EA817A6D80960A793DB971BC82EB64516AC Anon 217.10.97.25 False

Just over 2 hours

AC4E847242C31D290CF998F5E361777E74B42B0B MoyaVidalia 90.154.196.44 False

3 hours?

BDC81E5976A3B4BECED6F27DF7C955811050597A tomassey 125.255.2.30 False

[arlolra]

TorDNSEL is listening on, TestDestinationAddress 38.229.72.22:8080,8443,110,5190,6667,6697,9030

Testing our favourites from above:

With a torrc corresponding to anduinExit01,

ExitNodes 5D263037FC175596B3A344132B0B755EB8FB1D1C
StrictNodes 1

Contrary to the stated exit policy, I can only exit on port 443.

For rondienode1,

ExitNodes BFF292795AD1E09F1AB4EE744D5C6C7576B06306
StrictNodes 1

I can exit on all the above ports except the last one, 9030, which makes sense because it isn't included in the exit policy. So, still not sure what's going on here.

[arlolra]

Published address is 198.180.167.5, I'm guessing caprica exits from a number of addresses as it's been correct sometimes and wrong at other times.

No, as I said above, caprica's policy is accept 80 443 and since TorDNSEL isn't listening there, it'll never find it. I'm guessing the reason why you only see it from time to time is because of timeout issues you were seeing.

[arlolra]

As for the timing issue between when TorDNSEL runs and when Check updates, I'm not sure what to do. TorDNSEL is still updating its exit list 22 minutes after the hour (seems to be consistently finished by then). Does it makes more sense to update at 23 to pick up these stragglers?

[Ryman]

As for the timing issue between when TorDNSEL runs and when Check updates, I'm not sure what to do. TorDNSEL is still updating its exit list 22 minutes after the hour (seems to be consistently finished by then). Does it makes more sense to update at 23 to pick up these stragglers?

It may be more robust to have a more frequent (5min?) cron job that checks to see if the exitlist file has changed or not and takes action if so?

Re: caprica , yeah, I forgot it was in the above comments.

Contrary to the stated exit policy, I can only exit on port 443.

I wonder how prevalent this is across exit relays. If we can't get TorDNSEL to work with 443 anytime soon, I wonder if we should consider a minimal hardcoded list or reach out the relay operators? (I'm not a fan of either option really)

[arlolra]

It may be more robust to have a more frequent (5min?) cron job that checks to see if the exitlist file has changed or not and takes action if so?

TorDNSEL runs the exit tests when it gets a signal from tor that the consensus was updated, which only happens once, at the hour. I think the right thing to do is try and figure out what's taking it so long or maybe switch to Philipp's scanner, which he seems to be improving.

I wonder how prevalent this is across exit relays. If we can't get TorDNSEL to work with 443 anytime soon, I wonder if we should consider a minimal hardcoded list or reach out the relay operators? (I'm not a fan of either option really)

I emailed the contact for anduinExit01 because of the conflict in stated exit policy but no response so far. Roger seemed to indicate that relay operators like the attention. I'm not a fan of hardcoding a list but it could be a stopgap until we get TorDNSEL running on 443 (which may or may not be a while).