jsdelivr / globalping Goto Github PK
View Code? Open in Web Editor NEWA global network of probes to run network tests like ping, traceroute and DNS resolve
Home Page: https://www.jsdelivr.com/globalping
A global network of probes to run network tests like ping, traceroute and DNS resolve
Home Page: https://www.jsdelivr.com/globalping
Check this video https://dl.dropboxusercontent.com/s/ebuatjli404csd7/chrome_OXy6yz7yq5.mp4
note the time parameter
manual is: 5,6,1,13...
api is: 0,0,10,0,10...
After testing it turns running the same dig command inside the docker container produces those weird results. Need to understand why dig inside Docker is stupid
How about we expose each probe's unique ID and make the test quarable by the ID?
Example use-case:
I want to run a test from Amazon probe located in Canada. There is no way to set 2 filters at the moment, like country:canada + asn:amazon to run a test only on that specific probe.
But if we had IDs then the user could use /probes to filter as much as he wants and then simply use the IDs he needs
Lets also integrate anti-abuse logic. To stop people using probes to connect to malicious IPs and domains.
Any request to a banned domain should be stopped by the API before even getting to any probe.
Lists I propose using:
Its important since people could be running probes in their homes so we need to do our best to protect them. Lists can be auto-updated during API deployment
Originally posted by @jimaek in https://github.com/jsdelivr/globalping/issues/1#issuecomment-1044580719
How about we also block probes from connecting that have proxy_description=vpn and tor-*
plus maybe proxy_type=anonymous,aol,blackberry,corporate,
?
https://developer.fastly.com/reference/vcl/variables/geolocation/client-geo-proxy-description/
https://developer.fastly.com/reference/vcl/variables/geolocation/client-geo-proxy-type/
hosting example https://globalping-geoip.global.ssl.fastly.net/142.132.251.61
A probe hosted behind a VPN would only create problems. If the user is in China with the VPN server in Germany it would get registered in our system as a German probe. Then if someone tried to use the probe he would get 100ms latency to ping Germany from Germany.
Hm, just like the title says - we need US states as locations.
API will be used in browsers directly so we must enable CORS.
see https://www.npmjs.com/package/@koa/cors
We need an endpoint that will list all connected and available probes to build a map on a website. Probably it will be a list of probe.location
objects since we don't have other probes information that should be shared publically.
Geographical coordinates are required there to place a marker on a map. MaxMind provides location information for IP addresses so it must be added to the probes as well.
GET /v1/probes
[
{
"continent": "EU",
"region": "eastern-europe",
"country": "UA",
"city": 706483,
"asn": 123456,
"latitude": 49.98081,
"longitude": 36.25272
},
...
]
One of the key use-cases of the dns command is benchmarking the performance of resolvers and dns servers.
So every dns command should have a "querytime" parameter that says how long the query actually took.
dig already returns that value by default.
Unrelated but we should also include the "server" parameter of dig as well. Its important to understand what resolver was used for the query.
;; Query time: 11 msec
;; SERVER: 8.8.8.8#53(8.8.8.8)
We already have this info as part of https://globalping-geoip.global.ssl.fastly.net/
"as": {
"name": "internetia sp.z o.o.",
So why not use it. I suggest we start collecting this data from all probes and exposing in our API.
And make it part of POST for measurements too to all users to do limit their tests to certain providers.
Example use-case: I want to understand my performance from AWS to my CDN. Or to choose which Hetzner location I should be using. I could then run a latency test from "network: hetzner".
This means it should also be part of #51
docker logs a
Checking for the latest version
Current version 0.2.2
Latest version 0.2.2
[2022-04-02 14:10:51] [INFO] [16] [general] Start probe version 0.2.2 in a production mode
[2022-04-02 14:10:52] [ERROR] [16] [general] connection to API failed failed to collect probe metadata
Not sure why it fails since both of our APIs have geo IP data.
curl ipinfo.io
{
"ip": "139.99.172.23",
"hostname": "vps-197e15b9.vps.ovh.ca",
"city": "Sydney",
"region": "New South Wales",
"country": "AU",
"loc": "-33.8678,151.2073",
"org": "AS16276 OVH SAS",
"postal": "1001",
"timezone": "Australia/Sydney",
"readme": "https://ipinfo.io/missingauth"
https://globalping-geoip.global.ssl.fastly.net/139.99.172.23
This PR is a nice start to filter out the basic stuff #92
But its still simple to get the probes to target LAN endpoints. Simply create a DNS record test.domain.com A 192.168.1.1
and ask the probe to ping test.domain.com, it will resolve to a local IP and return results of the LAN test. This will be especially problematic with HTTP.
We need a way to prevent that completely. The obvious solution is to add a check on the probe side and if the resolved IP of any measurement type is LAN then kill the test and return an error.
But there might be a better way to do this
Yes, I wanted to use throng
for clustering but it means that every single probe will create multiple separate ws clients.
Originally posted by @zarianec in #37 (comment)
[0.2.3] mcallen (undefined), US, NA, 20473
https://globalping-geoip.global.ssl.fastly.net/216.238.82.66
https://ipinfo.io/216.238.82.66
Our logic is that if the 2 Fastly DBs are not unanimous then we fall back to ipinfo. So in that case that probe should have Texas as state. But at the moment its missing.
I think when we fallback to ipinfo we try to use MaxMind's State value? Which obviously is missing since it thinks its Mexico
The user facing parameters. On the backend we will need to set our own stuff.
ping
traceroute
dns
Global parameters:
The CLI and other UI will have a virtual smart field "location". E.g. location=Eastern Europe|Athens|as3454|Germany
and it will return a ready json to be used to provide the IDs to the static fields above. This requires a separate location resolver endpoint.
Exponential backoff associated with failing tests. Need to make sure that probes with poor results are not ruining the tests. The goal is to prevent dial-up users from breaking the service.
Probably need to also log probes in "backing-off" state to make sure we dont lose half our network due to a bug in this system.
Ping has the same issue as #57 but at the end it doesnt even return the output, e.g.
{
"id": "xCZhqmMDXldUel7f",
"type": "ping",
"status": "finished",
"createdAt": 1648571655024,
"updatedAt": 1648571685028,
"results": [
{
"probe": {
"continent": "EU",
"region": "western-europe",
"country": "NL",
"state": null,
"city": "amsterdam",
"asn": 14061,
"longitude": 4.8897,
"latitude": 52.374
},
"result": {
"rawOutput": "\n--- 141.164.34.61 ping statistics ---\n15 packets transmitted, 14 received, 6.66667% packet loss, time 2808ms\nrtt min/avg/max/mdev = 275.943/276.093/276.595/0.165 ms, pipe 2\n"
}
}
]
}
Resolve location endpoint for magic like "Ping from 10 locations in Eastern Europe".
Its needed for easier development of web and cli tools.
It should also support partial names like "frankfurt" when in reality the probes are at "frankfurt am main" or "amazon" when full is "amazon.com inc"
Example inputs:
The output should be something easily usable for the measurement endpoint.
Message
Unexpected token
in JSON at position 95
Backtrace (last 10 lines)
SyntaxError: Unexpected token
in JSON at position 95
at JSON.parse (<anonymous>)
node_modules/co-body/lib/json.js:57 parse
node_modules/co-body/lib/json.js:41 AsyncFunction.module.exports
at runMicrotasks (<anonymous>)
at processTicksAndRejections (node:internal/process/task_queues:96:5)
node_modules/koa-bodyparser/index.js:100 async
node_modules/koa-bodyparser/index.js:85 async
node_modules/@koa/cors/index.js:59 async
at async file:///app/dist/lib/http/middleware/ratelimit.js:26:5
View on AppSignal: https://appsignal.com/jsdelivr/sites/6230aa4683eb670b2c24f6e9/exceptions/incidents/10?timestamp=2022-04-05T17:57:27Z
Currently API previews dont work
2022-03-29T14:49:12.656703+00:00 app[web.1]: WARNING: NODE_ENV value of 'production' did not match any deployment config file names.
2022-03-29T14:49:12.656851+00:00 app[web.1]: WARNING: See https://github.com/lorenwest/node-config/wiki/Strict-Mode
2022-03-29T14:49:13.085654+00:00 app[web.1]: [2022-03-29 14:49:13] [ERROR] [22] [global] failed to start cluster connect ECONNREFUSED 127.0.0.1:6379
REDIS_TLS_URL
, REDIS_URL
vars it sets automatically.Current measurement request payload:
{
"locations": [
{
"type": "country",
"value": "FR",
"limit": 3 // per-location limit
},
{
"type": "country",
"value": "PL",
"limit": 4 // per-location limit
}
],
"measurement": {
"type": "ping",
"target": "hello.com",
"packets": 3
},
"limit": 10 // global limit
}
Global limit and per-location limit are mutually exclusive - it means that if the global limit is present in the request, the per-location limit is not allowed and vice-versa.
Because we are not limited by a single location type per request - the resulting probes set must contain unique records only.
const probes = [
{ id: 1, country: "PL", contient: "EU" },
{ id: 2, country: "UA", contient: "EU" },
{ id: 3, country: "PL", contient: "EU" },
{ id: 4, country: "NL", contient: "EU" },
];
const request = [
{ type: "country": value: "PL", limit: 1 },
{ type: "continent": value: "EU", limit: 4 },
]
const result = [
// because of request[0]
{ id: 3, country: "PL", contient: "EU" },
// because of request[1] - even though 4 probes from the EU was requested - only 3 was found
{ id: 2, country: "UA", contient: "EU" },
{ id: 4, country: "NL", contient: "EU" },
{ id: 1, country: "PL", contient: "EU" },
]
This is the easiest one. We must find the exact amount of probes for each location. If we don't have enough probes for a specific location, only these we have will be used and the final probes count may be smaller than expected.
Example:
Probes: UA:10; PL:4; NL:3
Request: UA:5; PL: 7; NL:3
Result: UA:5; PL:4; NL:3
The final amount of probes shouldn't exit the global limit. It may be smaller if we don't have enough probes in total though.
At least 1 probe from each requested location must be present in the results (if we have probes there at all ofc.).
If some locations don't have enough probes the result must be filled with probes from locations that have them.
Example:
Probes: UA:6; PL:2; NL:1
Request: UA; PL; NL - Limit: 6
Result: UA:3; PL:2; NL:1
For World Wide (WW) requests, we must apply "smart" logic where we break the world into groups and the final probes set must contain a predefined percentage of probes from each group.
percentage | location |
---|---|
5 | Africa |
15 | Asia |
30 | Europe |
10 | Oceania |
30 | North America |
10 | South America |
Example:
For a WW request with a global limit of 100, the result will contain 30 probes from Europe, 15 from Asia, etc.
POST /v1/measurements
must accept no more than 100 requests per minute
Make sure to write the exit the limit event to logs and report to appSignal so we can see it.
When MaxMind fails to resolve a parameter attempt to use a fallback service. Need to decide what to use, an option is Fastly for free https://globalping-geoip.global.ssl.fastly.net/
They use https://www.digitalelement.com/ which sounds pretty serious and reliable
Dont even try to resolve them with maxmind, just check and return an error immediatly https://en.wikipedia.org/wiki/Reserved_IP_addresses
When I run ping tests without setting a location e.g.
{"measurement":{"type":"ping","target":"google.com"},"locations":[],"limit":1}
the response always comes from 1 specific probe. In this case its always Johannesburg
{"id":"WPWKOLw8unO7Xljg","type":"ping","status":"in-progress","createdAt":1649698775713,"updatedAt":1649698776228,"results":[{"probe":{"continent":"AF","region":"southern-africa","country":"ZA","state":null,"city":"johannesburg","asn":199524,"longitude":28.0436,"latitude":-26.2023,"network":"g-core labs s.a."},"result":{"rawOutput":"PING google.com (172.217.169.78) 56(84) bytes of data.\n64 bytes from 172.217.169.78: icmp_seq=1 ttl=114 time=158 ms\n"}}]}
We had a logic with weights and priorities and the probe should be selected at random based on those weights
\"measurement\" does not match any of the allowed types"
MVP done, rest is in progress
We need a new http type of measurement. Probably use https://github.com/sindresorhus/got. It supports everything except HTTP3 but its coming soon. The URL must be manually built from the parameters.
http
host:
header. Default is $target (optional)Set internal parameters on our side:
globalping probe (https://github.com/jsdelivr/globalping)
request: 10000
Output:
Must support the same global parameters, especially the resolver. But make sure we still check the hostname or IP part of the URL against our blacklist.
Global parameters:
What should we use in prod?
@zarianec @patrykcieszkowski please vote/comment
We must unify the error response format for the API.
I suggest something similar to Stripe error formats:
{
"error": {
"message": "Something bad happened",
"type": "api_error",
}
}
For validation errors, we also need to provide a list of failed fields with a description:
{
"error": {
"message": "Validation Failed",
"type": "invalid_request_error",
"params": {
"field_name": "Message"
}
}
}
It seems we try to ping IPs like 800.800.80.80
and we just get a measurement stuck in in-progress
.
Same for hostnames like dfdf
Validate IPs and hostnames to make sure they are valid
Plan so far: Use .md to document the API in this repo. Once ready add the link to the API too #23
Add a probe version information here.
This requires modification to the probe as well that can be tracked here
We have a probe in Spain, Madrid but the API has mixed the data, it says Madrid USA:
"location": {
"continent": "NA",
"region": "northern-america",
"country": "US",
"city": "madrid",
"asn": 20473,
"latitude": 40.4165,
"longitude": -3.7026,
"network": "choopa llc"
These 2 DBs have wrong data and different cities https://globalping-geoip.global.ssl.fastly.net/208.85.17.142
By our GeoIP logic that means we should fallback to data by ipinfo https://ipinfo.io/208.85.17.142
But it looks like we falled back only for city and kept the rest from a different DB. We can't mix this kind of info, so the fallback should be for all location values.
Even with 3 IP data sources I get lots of wrong locations with datacenter IPs. And now while I control all of them I can either remove the wrong probes completely or make a request to maxmind and ipinfo to update their data.
But soon we will have 0 control over them and if we detect an Australian IP as American it will end up heavily influencing the results with people not understanding why their USA to USA test is so slow.
The problem is that even if a DB gets an IP's geo correctly we can't know it without a human review. So the current logic of 2vs1 is still the most optimal one.
So I am thinking how we could complement the existing system. Some ideas:
To me none of the above ideas are great. So lets keep this issue open until we can come up with something better
We need a simple UI demo for easier testing of the API. It can be as simple as an HTML file with a bunch of inputs for everything that our API supports which will also show the results in real-time by requesting the measurement ID every 500ms.
Output the raw output with original formatting as main output with all the additional metadata included below
e.g. If I ping something random the hostname won't get resolved and I will get this error
# ping google12344.com
ping: google12344.com: Name or service not known
But our API will keep running for 30 seconds until it times out and returns
{
"id": "HVXCBTN9KIj8HIWN",
"type": "ping",
"status": "finished",
"createdAt": 1648656181924,
"updatedAt": 1648656211933,
"results": [
{
"probe": {
"continent": "EU",
"region": "eastern-europe",
"country": "PL",
"state": null,
"city": "krakow",
"asn": 43939,
"longitude": 19.9968,
"latitude": 49.9979
},
"result": {}
}
]
}
I think it will be a major problem for our users. e.g. a user using our web tools or even CLI will get blocked for 30 seconds while he waits for a problematic test to finish.
We need a way to understand when tests fail and return the error. I dont think it should be too hard because the binary command will fail and exit, I dont understand why we have to wait for something. If the binary exited, then catch it and return the output of the error.
This applies to all commands.
Alongside well-known geographical location types like country, continent, city, we must also support custom regions for better and simpler routing.
These regions should be defined as a group of countries. Request validation must be tweaked to allow only predefined regions.
Probes routing (#9 ) should support it as well.
Proposed regions are:
name |
---|
Caribbean |
Southern Asia |
Middle Africa |
Northern Europe |
Southern Europe |
Western Asia |
South America |
Polynesia |
Australia and New Zealand (?) |
Western Europe |
Eastern Africa |
Western Africa |
Eastern Europe |
Central America |
Northern America |
South-Eastern Asia |
Southern Africa |
Eastern Asia |
Northern Africa |
Melanesia |
Micronesia |
Central Asia |
Limit to 1 connection per IP address from probes. Don't allow a user to start 200 containers on his PC and DDOS us.
In some cases, it may be useful to combine "world" type location with other location types:
I think your example could be useful when you want to cover the world + extra probes from a specific location. To avoid creating a huge POST with every single country. Limits and everything else would work as any other location
"locations": [
{
"type": "world"
},
{
"type": "country",
"value": "NL"
}
]
This will require:
Right now we have a simple anti-abuse system that simply rate-limits the amount of POST requests we accept per IP. Thats fine but its only an anti-abuse system.
We also need an extensive system to allow us to limit the amount of actual tests a user can run. e.g. I want to set 200 tests limit per hour per IP address with no authentication.
In theory that IP can now run either 1 measurement with limit of probes set to 200, and if we actual have 200 probes we run the test and return the results. But all future measurement requests will be declined until the 1 hour limit resets.
Or it can run 200 POST requests asking for 1 probe in each, same thing from the limit's point of view.
Need to make sure here that we correctly count them. So if a user has a limit of 200 and asks for 200 probes in Poland but we only have 50 probes in Poland that means that after the tests finish he will still have 150 tests remaining for that hour.
Later we will expand this for custom limits per API key and plan it's subscribed to, but at the moment its not necessary.
Needs #153
Add support for DNS command on the API side.
For the probe implementations - see jsdelivr/globalping-probe#6
A small raspberry pi should not be getting the same amount of tests as an 8 core intel server.
Maybe run a local benchmark during startup or something? Because an 8 core Atom is not the same as an 8 core Xeon either
@patrykcieszkowski We didn't come up with anything concrete yet. Artem planned to do some manual benchmarks and see if he can come up with usable data to then integrate into the code.
Maybe you have an idea how to do this?
Run a few tests like a traceroute to a far away location like 141.164.34.61
The raw output will flash random lines. Like first line 8 of the traceroute, then line 14, then nothing, then 2 random lines and then the full output. It must emulate the way it appears in the command line, one line after the other
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.