umkus/ip-index

A fast offline IP lookup library. Detects VPN/hosting.

IP Index

A self-sufficient web-service providing info about an IP, particularly its ASN, country of origin and whether it's a hosting or not.

Project contains:

• Data-center ASNs CSV list
• Dockerized webservice (nginx and nodejs)
• NPM library

Updates daily. False positives are possible, use at own risk.

Usage

For a quick and easy start and evaluation you can start the docker container directly:

docker run --pull always -d -it -p 80:4000 --rm ghcr.io/umkus/ipindex-node:latest

Alternatively for a more advanced usage with pre-configured nginx throttling and caching run this:

# git clone [email protected]:Umkus/ip-index.git
# cd ip-index
docker compose -f docker-compose.yml up -d

Now open this url in your browser: http://localhost/?ip=8.8.8.8

You will see the following data structure:

[
  {
    "start": "134744064",
    "end": "134744319",
    "subnet": "8.8.8.0/24",
    "asn": 15169,
    "hosting": true,
    "country": "US",
    "handle": "GOOGLE",
    "description": "Google",
    "subnetsNum": 956
  }
]

Why this exists

Most existing solutions to detect VPNs/Proxies provide HTTP APIs or binary databases on a subscription model. Downsides of the existing projects might be at least one of the following:

• Not cost-effective
• Not portable
• Not fast enough

This solution is:

• Free
• Portable (Docker image)
• Fast and efficient (caching, throttling reverse proxy)

Methods of validation

One of the target ASN IPs is checked against one or more of the known IP scoring services:

• https://www.ipqualityscore.com/free-ip-lookup-proxy-vpn-test/lookup/1.1.1.1
• https://www.ip2location.com/demo/1.1.1.1
• https://scamalytics.com/ip/1.1.1.1

Acknowledgments

• This product includes IP2Location LITE data available from http://www.ip2location.com.
• This product includes GeoLite2 data created by MaxMind, available from https://www.maxmind.com.

ip-index's Issues

Start - end | What is it?

 "start": 134217728,
    "end": 142606335,

Datacenter list accuracy

Your datacenter list (https://github.com/Umkus/ip-index/blob/master/dist/datacenters.netset) is incredible large. I'm wondering how accurate it is.

I programmatically counted the number of IP-addresses that is cover. It's: 1.273.450.192
While the ipcat dataset (https://github.com/client9/ipcat) only contains 95.959.476.
The number of IP-addresses in your dataset is 30% of the complete IPv4 address space. So the datacenter dataset is soo large that it is hard to belief that it can be that large. It would mean that at least 30% of all theoretically available IPv4 addresses are currently owned by datacenters.

[Advice] Best practices for running this in production

Very nice looking tool you've put together here. We are looking at using this for an analytics service.

How would you recommend this tool be implemented in production, e.g. with AWS?

Apple Private Relay third party ASNs are in asns_dcs.csv

A word of caution when using asns_dcs.csv to check if an IP's ASN seen from an Apple device is in the list and so it should be blocked or handled specifically as datacenter originated traffic.

The Apple users who enabled the Icloud+ Private Relay feature will have, from your listening web server POV,
an IP in some Cloudflare, Akamai, Fastfly ranges.
E.g. you can see that often the ASN of an Iphone with Private Relay enabled is 13335, i.e. Cloudflare,
which is (13335), as it should be, here in asns_dcs.csv.

So, the correct way to handle that is, in case of Apple UA, to always check for the "asOrganization" (or equivalent, I'm talking here Cloudflare workers cf object jargon, with ip2location the key/name is, if IPV4, "Provider") of this specific IP:
if the remote client is an Apple Private Relay enabled device/software you will get, as asOrganization/Provider/whatever, "iCloud Private Relay", and this means that the hit is from a real user, not from a datacenter / hosting server.

I still dont know what would happen (and if this is possible) if an Apple user has both the Private Relay feature enabled and a VPN or similar set in browser/device, but I think and hope that this would be a corner case.

Feature Request: Ability to query IPV6 addresses

Currently, IPv6 queries are not supported, could they be? :)

No longer frequently updates

Hey, I noticed that this repo no longer has daily release tags ever since some changes where you removed the db zip from the repo. Is this intended?

Why does this not contain anything from Nord or ExpressVPN

I checked your netset file for multiple IPs from Nord VPN, NONE of them are in the list.

Try it for yourself https://nordvpn.com/de/servers/tools/

ASN Datacenter

Hello, reporting another ASN.

397540

Most requests are returning []

Hi, since 15 October most requests are returning []. Seems like data are missing?

License file is missing

Hey, I discovered this great project coming from client9/ipcat#154

Went through the list of datasets that that are provided and - from what I gathered - they can be used as a replacement from the unmaintained https://github.com/client9/ipcat

One thing seems missing though, I couldn't find any License file. It would let developers know how they may or may not use this project.

client9/ipcat uses the GPL v3 License to "prevent commercial data providers from scooping up this data without compensation or attribution".

Here's a great website on the subject: https://choosealicense.com/

IP 166.198.252.12 returns []

166.198.252.12 --> []

Data centers

Hello, there are two data centers missing from the data center list.

42473, ANEXIA
64249, Charles River Operation

Cheers

proxy 45.61.188.135 returns []

This VPN IP: 45.61.188.135 returns []

AS398083 missing in asn-ip

I know this is a different project, but I can't open an issue over there, and I assume as a result of this issue, 5.133.124.0/24 will cause issues with this project.

This isn't by far the only AS that is affected, the following numbers and possibly more are missing: 940, 1636, 3577, 4879, 6485, 6565, 7363, 10425, 10475, 10931, 10981, 11011, 11044, 11091, 11223, 11316, 11490, 11610, 11636, 11825, 11945, 11967, 11969, 11974, 12176, 12240, 13224, 13317, 13339, 13342, 13492, 13520, 13717, 13980, 14046, 14076, 14350, 14428, 14439, 14461, 14545, 14552, 14594, 14605, 14651, 14889, 15037, 15115, 15178, 15182, 15200, 15220, 15329, 15804, 16615, 16666, 16769, 16804, 16809, 17341, 18604, 18691, 18775, 18781, 18796, 18798, 18805, 18821, 18976, 19023, 19081, 19140, 19142, 19176, 19330, 19359, 19507, 19521, 19533, 19658, 19700, 19708, 19926, 20089, 20118, 20190, 20200, 20227, 20421, 20443, 20447, 21584, 21666, 21695, 21714, 21861, 21986, 22063, 22160, 22226, 22271, 22272, 22484, 22538, 22539, 22694, 22733, 22736, 22743, 22774, 22830, 23021, 23176, 23190, 23194, 23288, 23357, 23411, 23448, 23507, 25568, 25600, 25641, 25647, 25800, 25831, 25922, 25978, 26015, 26016, 26018, 26051, 26137, 26140, 26286, 26295, 26341, 26345, 26520, 26643, 26663, 26682, 26701, 26712, 26759, 26799, 26861, 26922, 26986, 27174, 27180, 27247, 27289, 27291, 27335, 27384, 27426, 27470, 28102, 29763, 29829, 29855, 29987, 30004, 30006, 30037, 30104, 30155, 30265, 30442, 30550, 30654, 30699, 31781, 31813, 32021, 32042, 32103, 32121, 32187, 32203, 32282, 32346, 32358, 32467, 32497, 32534, 32581, 32596, 32772, 32842, 33017, 33023, 33042, 33051, 33123, 33137, 33232, 33245, 33394, 33417, 33484, 33625, 33628, 33684, 33732, 35948, 36002, 36267, 36283, 36357, 36465, 36774, 36893, 36900, 36911, 36928, 36979, 37034, 37155, 37169, 37320, 37338, 37442, 37475, 37476, 37544, 37555, 37591, 37703, 39991, 40186, 40255, 40279, 40340, 40503, 40600, 40687, 40745, 40751, 40779, 40909, 40920, 46086, 46123, 46161, 46226, 46228, 46233, 46355, 46501, 46559, 46614, 46673, 46766, 46772, 46815, 46876, 46916, 47044, 47069, 47086, 52761, 53298, 53367, 53419, 53552, 53657, 53715, 53751, 53771, 53775, 53854, 53929, 53947, 54148, 54209, 54218, 54231, 54262, 54278, 54309, 54338, 54424, 54427, 54517, 54518, 54590, 54752, 54753, 54776, 54787, 55289, 62711, 62828, 63092, 63190, 63198, 63305, 63387, 63390, 63401, 64203, 64257, 65656, 65999, 227171, 261050, 264245, 264379, 264493, 265039, 268652, 269062, 269948, 327869, 327928, 327933, 328007, 328226, 328369, 393266, 393305, 393314, 393320, 393322, 393377, 393446, 393491, 393493, 393588, 393866, 394076, 394156, 394183, 394293, 394376, 394402, 394519, 394607, 394721, 394787, 394788, 394856, 394879, 394936, 394939, 395270, 395281, 395350, 395438, 395514, 395858, 395898, 396015, 396330, 396993, 397096, 397124, 397186, 397306, 398083, 398118, 1440665

nvm

Edit: Nvm, found what I was looking for. Please delete this.

False positive

This one seems to be a false positive:

394330

It belongs to LTD Broadband

https://www.ip2location.com/as394330

https://ltdbroadband.com/

They don't seem to offer any vpn or datacenter services and are clearly a fiber/wireless ISP

Also, is there any info how the list is made, you just do a check on one of the 3 and if it shows up you mark it? Maybe something like a confidence score can limit false positives?

about VPN info

How can I get the information if one IP belongs to a VPN

The repository is damn huge

The repository size is extremely large (20Gib+ ?):

git clone [email protected]:Umkus/ip-index.git
Klone nach 'ip-index' ...
remote: Enumerating objects: 7552, done.
remote: Counting objects: 100% (150/150), done.
remote: Compressing objects: 100% (85/85), done.
Empfange Objekte:  39% (2971/7552), 4.29 GiB | 10.94 MiB/s

It also requires a lot of time for the initial clone process to complete.
I suppose running git gc is in order.

Expose finer grained black list matches

Just a feature request to expose these fine grained matches, might be a bit more work due to how the netset files are built?

is_tor
is_anonymous
is_vpn

Potential bug - Missing IP ranges in datacenters

I couldn't reopen the previous issue so here goes:
#19 (comment)

The following IP range belonging to 42473 is not in the datacenter table, despite the ASN being listed in matches as a datacenter.

131 | 2204368896 | 2204369919 | 42473

This might be a bug, but I have not checked if this is a problem for other ranges.

Here's a test IP: 2204369629.

Originally posted by @ptibom in #19 (comment)

Fetching new versions daily

Hi, thanks for a great repo! Curious to hear how you'd recommend fetching daily versions when using the Docker image in production?

It'd be simple to spin down the Docker, grab the latest image and spin that up. However, we'd of course like to do the daily update of the IP list w/o downtime. Have you considered adding that functionality to the Docker image?

AS29447

Hi, it seems that AS29447 is incorrectly reported.

Question about the swap from Sqlite

Hello, I'm using an old version of your repo with the sqlite file.

I'm just curious why you made the decision to swap from sqlite?

(Trying to decide if I should keep it or go with your new system.)

Thank you

2 likely DCH ASNs not in asns_dcs.csv

8660
12650