potential additions

[blackjack8]

potential additions

[BlackJack8]

This is the Output from jawz101:
https://github.com/BlackJack8/iOSAdblockList/files/2544590/output.zip

I dont know which version he used

[BlackJack8]

Hey there!
It really depends on the adblocker - Most automatically resolve www.
Just like in the browser, the www. often just gets removed when opening the page.

I actually looked up if all domains still exist on my list, and they do somehow. If they are still being used, well, idk.
I‘m going to look it up with the tool you listed above.

Thanks for your help!

[BlackJack8]

If there are some issues with the www. at the beginning, i can of course remove that! I tried it with only one adblocker

[jawz101]

What ad blocker are you using?

Here's that list of potentially down sites.
short version
down-hosts.txt

detailed version
down_or_potentially_down.txt

or split
404.txt
409.txt
410.txt
400.txt
403.txt

how to read
https://en.wikipedia.org/wiki/List_of_HTTP_status_codes#4xx_Client_errors

From the wikipedia page, 400, 404 and 410 have a very strong indication of not existing while the others are possible.

If you have a whois or nslookup command on your computer you could try testing one of the others on the list versus one of the ones on these lists to see the difference.
nslookup ads.ezboard.com non existent example versus nslookup google.com one that is found.

NSlookup goes out to your DNS server & checks to see if it can resolve a domain name to IP address(es).

Then, another utility is called whois. I usually go out to the web and find a site that has a web version of this tool. What a whois does is tell you who owns the domain. (also note some people buy one with anonymizer services to hide their details. ICANN is a pretty official site. another is maybe GoDaddy's since they sell domain names. There are tons of sites that offer this utility as well as command-line utilities or bulk checking gui tools that can look them up too.

web examples:
https://whois.icann.org/en/lookup?name=google.com
https://www.godaddy.com/whois/results.aspx?domain=google.com

Sometimes I use these tools to see who might really own a domain just to see what sort of business they are.

[BlackJack8]

I‘m using Adblock by Futuremind

„Here's that list of potentially down sites.
short version
down-hosts.txt

detailed version
down_or_potentially_down.txt

or split
404.txt
409.txt
410.txt
400.txt
403.txt“

Quick question: Are these all from my list? It seems weird because some domains from there are not even in my list.

Thank you for the help!

[jawz101]

err.. I don't think some are. I was combining your list with someone else's that said he was actively monitoring his iPhone traffic so, yeah, there are some on the lists that are extras which you can just ignore. Unfortunately I'm back @ work so I can't re-run it with just your entries.

[jawz101]

cleanlist.txt

I kept working with it while I'm here and, ignoring the extra domains, this would be your list cleaned of dead hosts. I sort by primary domain so I can see who owns clumps of domains.
ex: so like s0.2mdn.net and 2mdn.net are clumped together instead of spread across the list. If you've ever used Privacy Badger on Chrome or Firefox, they did this to their add-on after I started reorganizing their lists for readability :P

Now I'm going through and looking at some of the companies I haven't seen before. Trying to find Android games with airpush, aark, etc. I can't believe iPhone would allow ads in the notification panel.

Anyways, a few comments: personally I'm hesitant to block the ones starting with ns.example.com the reason is usually these are name servers. If the main domain - in this case example.com - was indeed an ad company, then sure. Otherwise, these might have legit purposes. Also, I personally don't want to block facebook chat but that's just a matter of opinion on what I want to consider ad/tracking. And the ones starting with ocsp.... some would consider a newer security check. see here. They're usually built into browsers to check to see if SSL certificates are revoked as you browse the web. It's up to you. I mean, it's a matter of opinion. It would make your phone less chatty but it might be used to make sure your really visiting the site you want to visit.

[BlackJack8]

I actually removed some domains since my last update, i would be pleased if you can do the same with the updated one when i get home in about 2-3 hours.

[jawz101]

Will do. Yeah, that PyFunceble program is a little tricky to get set up. The dev is trying to work through an easier install method at my request but- yeah- it's just a little tricky at least for me to get python figured out. Oh.. that cleanlist file's first entry has some extra dots. I use a couple of tools that process ends of lines and the first line it always doesn't clean up

[BlackJack8]

ok, updated it.

[jawz101]

Thanks. I'll work on it in about an hour.

I was just thinking- how do you collect these domains? Does your ad blocker log all DNS requests or something and you're picking out the unnecessary ones? Is it traffic you see while just running various apps and games or also while you're browsing on your phone? Is it just from use on the phone?

The reason I ask is I'm also considering just merging yours straight into mine since there's so much overlap between Android and iOS mobile ad companies so I might as well.

I'm also curious where in the world you live (like North America, Europe, Asia, etc.) I'm in North America and the reason I ask is I'm curious if some of the different ad networks we're seeing might be due to the different sorts of regional markets we are in. For instance, I'd rarely download a Chinese shopping app or something. Just curious.

[BlackJack8]

Yes, my adblock logs all invoked domains.
If you want to merge it, feel free to. You helped me a lot!
I live in germany :)

[jawz101]

here's the output
output.zip

[BlackJack8]

Thanks!
But some domains in the „invalid“ file are still being used, like adclear.teufel.de, 1049.b.akamai.net, d3anogn3pbtk4v.cloudfront.net etc.
Why didn‘t it recognize that?

[jawz101]

Do you have an example of what I'd do to see any of those working successfully? If that company decides to stop using it or something that would be a reason one gets caught as currently active. Other than that, I'll have to ask the author of that tool what the reason might be if you see use of one still.

[BlackJack8]

It is just minor, like around 5 domains.
All of them appear on the internet, but i sadly don‘t know on which exactly...
I just recently saw 1049.b.akamai.net, cloudfront.net so it‘s reasonably unlikely they are discontinued.
Didn‘t see adclear.teufel.de in the last time though...

[BlackJack8]

Ok so just by accessing the optout site from teufel, adclear.teufel.de appears.

[jawz101]

Just by going here? https://www.teufel.de/datenschutz.html

I don't see that analytics.teufel.de in my dns logs nor in uBlock Origin when i visit that URL.

As for the akamai and cloudfront ones, perhaps they are addresses that are dynamically created. Say, when you and only you visit that web page, it assigns you a unique identifier in the form of an actual subdomain. Those content delivery networks, such as cloudfront, akamai, cloudflare, leaseweb, etc. do tricky things to work for many businesses who use their product. One subdomain could represent a unique company or even uniquely identify you uniquely.

https://en.m.wikipedia.org/wiki/Content_delivery_network

Example: this full domain name looks like unique identifier for one particular user

b07aa28a-b286-4659-ff2e-79a1d5dfd31f.api.pushspring.com

That's just too formulaic to be anything but generated when you yourself connected to that server. I try to guess if that's the case and don't include those types of entries on my list because no one else would ever hit them.

https://en.m.wikipedia.org/wiki/Universally_unique_identifier

[BlackJack8]

https://www.teufel.de/datenschutz-opt-out.html
It is adclear.teufel.de, not analytics.teufel.de

[jawz101]

ah... well, I don't know what to say for that. Perhaps the PyFunceble author might be reached to see if something was wrong. I'm going to close this for now though

[funilrys]

Hello there!

May I get your INVALID and INACTIVE files ? You may have discovered a critical issue.

Also are you using the stable or dev version ?

About adclear.teufel.de I'll look at it as it is really strange ....

Have a nice day/night.

Cheers,
Nissar

[jawz101]

@funilrys I was using the dev version - whatever the latest one was at that time I ran it.

@BlackJack8 just curious what made you include it on your list? I just tried doing an nslookup from work and it didn't resolve here either. I went back to the web page too and checked it with uBlock Origin to see what it's doing. It looks like that webpage tried to load 5 images from that subdomain but they all return as broken links even when I have that domain unblocked.

In uBlock Origin's logging feature it can show you a preview of what each image is that a page loads so you can create rules to block them. But with the 5 images from that subdomain the little preview window shows a broken link to an image which makes me wonder if they have old code on that page that is referencing some pics they used to have on an old subdomain. Like maybe it's still trying to get the images even though they aren't really there?

[BlackJack8]

Wow, very weird what is happening there.
I‘ve just seen it in my dns proxy log, so i included it.
That means this domain is obsolete, right?

Thank you for the help!

[funilrys]

Hi @BlackJack8, I can confirm it! I did run my tool over 5 countries in 3 different continents and I get the same result. Same with ping, nslookup and dig.

I also checked 1049.b.akamai.net, d3anogn3pbtk4v.cloudfront.net and your INACTIVE list, I also get the same.

May I invite you to read this page (for the dev version - which will be soon the stable version) and/or this page (for the current stable) ?

It's just some simple flowcharts which may help you understand what we do behind PyFunceble.

If you have any questions or need help for something else, feel free to ping me!

Have a nice day/night.

Cheers,
Nissar

[BlackJack8]

Ok, thank you two for the help! I definetly learned something new :)