Giter Site home page Giter Site logo

jsdelivr / globalping Goto Github PK

View Code? Open in Web Editor NEW
211.0 10.0 27.0 3.03 MB

A global network of probes to run network tests like ping, traceroute and DNS resolve

Home Page: https://www.jsdelivr.com/globalping

TypeScript 93.12% JavaScript 6.58% Shell 0.06% HTML 0.12% Dockerfile 0.12%
probes network-analysis network performance ripe-atlas globalping devops sysops api benchmark

globalping's Introduction

Globalping Header

Gain insights into your network routing from any location in the world!
Troubleshoot anycast issues, monitor CDN and DNS performance, perform uptime monitoring,
and build your own network tools for personal or public use.


Invitation to contribute!

Everyone is welcome to contribute – here are some ways you can get involved:

  • Suggest ideas for new features.
  • Submit pull requests and fixes for existing issues.
  • Help us with quality control and testing across systems. Report any problems, bugs, or bad user experience you find.
  • Contribute to the documentation.
  • Write tutorials and articles and help spread the word about Globalping on your blog or other platforms.

Refer to CONTRIBUTING.md for more information.

The Globalping Platform

Globalping allows anyone to run networking commands such as ping, traceroute, dig, and mtr on probes distributed around the globe. Our goal is to provide a free, user-friendly API for everyone to build interesting networking tools and services.

Many users will likely prefer alternative ways to use the platform rather than working directly with the API. Therefore, to make Globalping accessible to all kinds of users, we're constantly expanding and improving our tools, all of which leverage the full potential of the Globalping API:

Learn more about Globalping on www.jsdelivr.com/globalping

Our major sponsors

We thank our sponsors who contribute to the development of Globalping and help us expand our probe network!

Gcore xTom Edis Global

Support the Globalping community and platform

We welcome any individual or company interested in supporting Globalping's growth and our mission to make the internet a faster place for everyone. Here are some ways you can help:

  • Run multiple probes: Globalping relies on a globally distributed network of probes for running network measurement requests. We're happy to list anyone who can host at least six probes as a donor on GitHub and our website.
  • Become a GitHub sponsor: By becoming a GitHub sponsor of jsDelivr, you are supporting both the jsDelivr CDN and the Globalping platform. Sponsors contributing $10 or more per month can request a hardware probe to install in their home or office.
  • Become a hardware probe provider: Sponsor a batch of co-branded hardware probes, including your own stickers and swag, to ship to your users or hand out at conferences.

Quick start – Run your first tests

Whether you're new to network testing or are a seasoned pro, getting started with Globalping is straightforward. Let's check out how you can run your first tests using our various tools and integrations:

globalping latency test from google cloud

Our website offers the fastest way to get started. Run tests instantly from anywhere and experiment with the different options for different test types. For each test, you can view the probes used on a map and get a detailed view of the individual test results below.

Upgrade your network debugging capabilities by installing the Globalping CLI tool. Get access to our global network of probes without leaving your terminal!

Install the CLI on Linux, macOS, or Windows using the command for your package manager:

#Ubuntu
curl -s https://packagecloud.io/install/repositories/jsdelivr/globalping/script.deb.sh | sudo bash
apt install globalping

#RHEL
curl -s https://packagecloud.io/install/repositories/jsdelivr/globalping/script.rpm.sh | sudo bash
dnf install globalping

#MacOS
brew tap jsdelivr/globalping
brew install globalping

#Windows
winget install globalping
OR
choco install globalping

And then run your tests:

$ globalping traceroute google.com from Western Europe --limit 2
> EU, DE, Frankfurt, ASN:210546
traceroute to google.com (142.250.185.78), 20 hops max, 60 byte packets
 1  10.0.0.1 (10.0.0.1)  0.747 ms  0.714 ms
 2  10.2.0.97 (10.2.0.97)  5.482 ms  5.511 ms
 3  10.10.0.1 (10.10.0.1)  613.232 ms  613.268 ms
 4  ae6-ffm21.core2.ffm3.de (45.138.175.101)  9.150 ms  9.188 ms
 5  ae1-core2.core3.ffm3.de (45.138.175.105)  6.027 ms  6.030 ms
 6  142.250.171.196 (142.250.171.196)  5.562 ms  5.749 ms
 7  209.85.244.249 (209.85.244.249)  48.691 ms  48.812 ms
 8  142.250.209.243 (142.250.209.243)  5.742 ms  5.743 ms
 9  fra16s48-in-f14.1e100.net (142.250.185.78)  5.716 ms  5.717 ms

> EU, NL, Zwolle, ASN:50673
...

Learn more about the Globalping CLI in its repository

Globalping REST API

If you want to build something custom or simply learn more about all the available options and data we provide, check out the Globalping REST API.

Creating a new measurement test is as simple as:

POST https://api.globalping.io/v1/measurements
{
    "limit": 10,
    "locations": [],
    "target": "jsdelivr.com",
    "type": "ping",
    "measurementOptions": {
        "packets": 5
    }
}

Read the full API documentation and explore our dev demo

Slack App

Improve collaboration in your workspace by installing our Slack app, enabling everyone to interact with Globalping without leaving Slack. The app is especially handy for NOC, OPS, and Support teams collaborating on debugging networking issues and discussing results.

Add to Slack

To initiate a command with the Globalping Slack app, use the slash command /globalping. Apart from that, you can use the same test command structure as with our other tools and integrations. For example, to get started, type /globalping help.

Example commands to try:


/globalping ping 8.8.8.8 from Germany

/globalping traceroute jsdelivr.com from South America --limit 2

Important

To make the app post test results in Slack Threads, mention it using @globalping. For example,  @globalping ping google.com

Learn more about the Slack app on our blog.

GitHub Bot

Automatically post network test results into any public GitHub issue with our GitHub bot. Mention it with @globalping, followed by the command you want to run. Otherwise, you can follow the same test command structure as with our other tools and integrations.

Example commands to try:

@globalping ping 8.8.8.8 from Germany
@globalping traceroute jsdelivr.com from South America --limit 2

Learn more about the GitHub bot on our blog.

Globalping command structure

We make sure that all integrations and tools provide a consistent experience, so writing tests looks almost identical whether you're using the CLI tool, the Slack app, or another official integration.

Follow this structure when writing measurement commands:

globalping [command] [target] from [location] [flags]

Examples:

  • globalping ping google.com from aws
  • globalping ping google.com from Berlin, South America --limit 2

Let's look at the components:

Available test types

Globalping supports the following:

  • ping
  • traceroute
  • mtr
  • dns (similar to dig)
  • http (similar to curl GET and HEAD)
Target

The target represents the destination for your test. This can be a domain name or an IP address for most test types.

Location

The location field can process different locations, including continents, regions, countries, cities, US states, and ASNs (prefixed by "AS," e.g., from AS80085). You can also specify measurement IDs from previous tests to reuse the same probes.

Tip

Check out our best pracises and tips to learn how to define locations effectively.

Best practices and tips

Learn to use Globalping in the most optimal way!

Test with "magic" 🧙

All integrations connect to the same API and use our "magic" field as location input. This is the best way to interact with Globalping, as it maintains a consistent and straightforward user experience, allowing you to reuse logic and parameters across all tools.

So, for example, when you run a test from "Germany" using the CLI, Slack app, or any other official integration, our API's magic parser processes the location.

Note

Developers who want stricter and more predictable control over the selected probes and user input can use the individual location parameters when making an API call, which expect you to provide each city, country, and tag in a standardized way.

The magic field supports a wide range of parameters and location combinations, including countries, continents, cities, US states, regions (Western Europe), ASNs, ISP names, eyeball or data center tags, and cloud region names (us-east-2).

Reselect probes ♻️

You can also provide the "magic" field with a measurement ID to have the API select the same probes used in a previous measurement.

For example:

  • from WZIAtMx4LLhzit02

Important

This is a best-effort action, and if some of the probes are no longer online, they will be missing from the new results. Additionally, as measurements expire (lifetime depends on user type), you should not hard-code measurement IDs, as new tests will break after a measurement expires.

You can obtain the measurement ID through the "share" functionality. For example, use --share in the CLI, find the "Share URL" section at the bottom of the web results or receive it by calling the API directly.

Some practical use cases:

  • Increase the reliability of network endpoint comparisons, such as CDN, DNS, and edge compute benchmarking, as well as provider comparisons.
  • Support troubleshooting by preselecting the probes in problematic networks and running different tests, with different parameters if needed but with the same probes, until your issue is solved.
  • Emulate a continuously running ping or mtr by reusing probes and stitching together the output of different measurements into a single output.

Understand the "world" location 🌍

The world location is special and uses a pseudo-random algorithm to select probes while maintaining certain proportions per continent.

Important

If you provide no location, the system defaults to using "world".

For example, when requesting tests from world --limit 100, the system aims to return probes proportionally from Africa (5 probes), Asia (15 probes), Europe (30 probes), Oceania (10 probes), North America (30 probes), and South America (10 probes).

Basic location targeting 📍

Our aim is to enable you to provide whatever location feels right – some examples:

  • from usa or from united states - A random probe in the USA
  • from new york -  A random probe in NYC
  • from aws - A random probe hosted with Amazon
  • from WZIAtMx4LLhzit02 - Selects the same probes that were used for the specified measurement

We'd only get one probe in these examples because we didn't specify a limit. But, if you select a large region, for instance, you expect to get multiple results. Use the --limit flag to define the number of tests to run, which is set to one (1) by default.

For example:

  • from north america selects one probe in either USA or Canada.
  • from north america --limit 2 selects one probe in USA and one in Canada.

Other location examples:

  • europe, africa, asia
  • east asia, north europe, eastern europe
  • greece, canada, mexico, japan
  • california, texas
  • frankfurt, dallas, sydney
  • as396982, as16509
  • comcast, google, ovh, orange, t-mobile
  • eyeball, datacenter

Note

Providing no location defaults to world, selecting a probe from a random location.

Stack location filters 🍔

You can combine multiple locations using the plus symbol +, and use them as filters to define your desired probe locations more accurately.

Examples:

  • from comcast+california returns a probe on the Comcast network in California.
  • from google+europe returns a Google Cloud probe in Europe.
  • from google+germany returns a Google Cloud probe in Germany (useful if you forget the cloud region name!).

You can combine as many parameters as you like:

  • from cogent+europe+datacenter returns a probe hosted at Cogent in Europe, tagged as a data center probe. Or you could use the eyeball tag to target probes that aren't part of a data center.

Define multiple locations 🌐

To run tests at multiple locations in a single request, list them with a comma, for example:

  • from Berlin,South America,Idaho --limit 4.

Naturally, you can combine filters and multiple locations at the same time:

  • from amazon+france,ovh+france --limit 2 returns a probe hosted on AWS in France and a probe hosted on OVH in France.

You can also define your own "world" location, for example:

  • from germany, greece, uk, usa, canada, japan, china, south africa --limit 20

returns probes from these countries in roughly equal proportions to meet the limit of 20.

Important

The limit parameter is global and applies to the location as a whole, not allowing the API to return more than the limit. If you want to set custom limits per location, use our API directly instead of the "magic" field.

Leverage system tags (eyeball networks and more) 🏷️

Many probes are going to be tagged by our system. At the moment, this includes:

  • Google Cloud and AWS cloud region names. For example, aws-eu-west-1 and gcp-us-south1. These tags follow the respective provider's naming scheme and are prefixed with their name.
  • Eyeball and data center network tags. eyeball and datacenter. eyeball probes are hosted with ISPs that provide internet access to regular people and small businesses. As the name suggests, datacenter tags are intended for probes hosted in a data center.

Things to keep in mind

Probes share no UUIDs

Our probes don't expose unique IDs that you can use to target them explicitly. Instead, we recommend fine-tuning the probe selection by using and combining filters or specifying IDs from previous measurements, as described in the best practice section above.

This ensures that popular locations, like from Amsterdam or from AWS, are automatically load-balanced across multiple probes within the same location, preventing the overload of specific probes.

Best-effort results

When requesting a specific number of probes, there is no guarantee that the API will respond with the exact amount.

Probe availability rules

Globalping exposes and lets you interact with probes that are currently online.

Here's when a probe is considered online and available through our API:

  • It's up-to-date. A probe running an older version will be forced to auto-update before it becomes available.
  • It passes quality control tests. Every probe must pass a test, ensuring it's connected to a reasonably stable network without packet loss. This test is done regularly; if a probe fails, it gets taken offline.
  • It's not behind a VPN. We block probes that we detect are hosted behind VPNs or other proxying techniques, as they would report incorrect latency and routing data, making them unusable.

This whole process is completely automated and managed by our platform.

Join the network – Run a probe

Globalping relies on the community to expand its probe network. While we maintain our own probes in key locations, we welcome any support from both corporate partners and individuals. Join our network and help make the internet faster for everyone by running a probe (or several).

Setup instructions

You can run a Globalping probe on any internet-accessible device that can run a Docker container. For example, you can use a rented VPS, a dedicated server with spare capacity, or even your locally hosted Raspberry PI.

Use this command to create and run the Globalping probe container:

docker run -d --log-driver local --network host --restart=always --name globalping-probe ghcr.io/jsdelivr/globalping-probe

And it works on x86 and ARM architectures. Podman instructions

Notes on probe security and customization

  • Probes don't open any ports or accept any incoming connections. They can only establish a connection with our API.
  • We include regularly updated lists and databases of domains and IPs associated with malware or potentially dangerous content and completely ban them at the API  level.
  • Tests scale to the amount of available CPU cores. Our code is very lightweight and shouldn't use too many of your resources. Therefore, in most cases, we recommend running our probe as is. However, if you're worried, you can use --cpuset-cpus="0-2" with your Docker command to limit the number of available cores.
  • We rate-limited all users on the API level to prevent the abuse of the network.
  • No local network tests are allowed, only public endpoints.

Learn more in the Globalping Probe respository.

Limits | WIP

Our platform has multiple limits to prevent abusive behaviour while motivating people to contribute to the sustainability of our platform. Here's an overview:

Global limits

These limits apply per IP address for all Globalping users:

  • 100 POST requests per minute per IP. No GET limits are implemented to support "real-time" use cases.
  • A single measurement is limited to 200 probes per location and 500 total probes.

Unauthenticated users

Anyone can connect to and use our API without requiring any credentials. For users without authentication, we limit the number of tests an IP address can run:

  • 100 tests per hour

Note

A test is defined as a successful measurement the platform runs and returns to the user. For example, a limit of 10 tests means that users can either run 10 measurements with the probe limit set to 1 per measurement or a single measurement with the probe limit set to 10.

Registered jsDelivr users – Free

All registered jsDelivr users get an API key for authentication, granting them higher limits:

  • 200 tests per hour

GitHub Sponsors

As a GitHub Sponsor of jsDelivr, your contributions help us continue the development of all projects under the jsDelivr Organization.

As a thanks, we upgrade your account to receive higher limits.

Custom limits

Feel free to reach out if you need a custom limit for your API key.

We're happy to provide higher limits for researchers, non-profits, and other open-source projects.

Support and feedback

If you are stuck or want to give us your feedback, please open a new issue.

Development

Please refer to CONTRIBUTING.md for more information.

globalping's People

Contributors

alexey-yarmosh avatar ayuhito avatar davideredics avatar deepsourcebot avatar dependabot[bot] avatar hackedserver avatar inventarsarah avatar jimaek avatar kewlamogh avatar martinkolarik avatar patrykcieszkowski avatar sukkaw avatar taras-pinchuk-uinno avatar zarianec avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

globalping's Issues

Need to understand failed tests

e.g. If I ping something random the hostname won't get resolved and I will get this error

# ping google12344.com
ping: google12344.com: Name or service not known

But our API will keep running for 30 seconds until it times out and returns

{
  "id": "HVXCBTN9KIj8HIWN",
  "type": "ping",
  "status": "finished",
  "createdAt": 1648656181924,
  "updatedAt": 1648656211933,
  "results": [
    {
      "probe": {
        "continent": "EU",
        "region": "eastern-europe",
        "country": "PL",
        "state": null,
        "city": "krakow",
        "asn": 43939,
        "longitude": 19.9968,
        "latitude": 49.9979
      },
      "result": {}
    }
  ]
}

I think it will be a major problem for our users. e.g. a user using our web tools or even CLI will get blocked for 30 seconds while he waits for a problematic test to finish.

We need a way to understand when tests fail and return the error. I dont think it should be too hard because the binary command will fail and exit, I dont understand why we have to wait for something. If the binary exited, then catch it and return the output of the error.

This applies to all commands.

Prevent all LAN access

This PR is a nice start to filter out the basic stuff #92

But its still simple to get the probes to target LAN endpoints. Simply create a DNS record test.domain.com A 192.168.1.1 and ask the probe to ping test.domain.com, it will resolve to a local IP and return results of the LAN test. This will be especially problematic with HTTP.

We need a way to prevent that completely. The obvious solution is to add a check on the probe side and if the resolved IP of any measurement type is LAN then kill the test and return an error.

But there might be a better way to do this

http command

We need a new http type of measurement. Probably use https://github.com/sindresorhus/got. It supports everything except HTTP3 but its coming soon. The URL must be manually built from the parameters.

http

  • method {HEAD} - The type of request to make. GET or HEAD. Default is HEAD (optional)
  • path {/path/somewhere.jpg} - the path part that we will add to "target" to build the URL. Must support query strings, hashbangs and everything else. Default none (optional)
  • protocol {https} - the protocol to use and to be added to the URL. Technically if user selects HTTPS we just make it part of the URL. But if he selects HTTP2 or 3 then we use HTTPS in the URL and set extra parameters to support http/2 and http/3. Supported HTTP,HTTPS,HTTP2,HTTP3. Default HTTPS. (optional)
  • port {543} - Port to connect to, default is 443 if HTTPS/http2/http3 and 80 if HTTP. (Optional.)
  • host {string.com} - the value for the host: header. Default is $target (optional)
  • headers {array} - key:value array that allows the user to set custom headers except from "host" which is above. Default non (optional)

Set internal parameters on our side:

  • user-agent should be globalping probe (https://github.com/jsdelivr/globalping)
  • 10 second total timeout request: 10000
  • disable all caches
  • followRedirect: false
  • set dns resolver if present in global option
  • dnsLookupIpVersion:4
  • http2 is set based on user but note this: "If you prefer to use the newest version, set both request to http2wrapper.auto and http2 to true."
  • rejectUnauthorized:false to accept expired certs
  • Don't actually download anything to disk. It should all be in memory and sent to the API.
  • Limit download size of content to 10kb!!! If the content being fetched is longer then we need to stop it immediately and send back to the API the first 10kb. This should be enough to fetch simple content like JSON endpoints or raw HTML pages.
  • Make sure its multi-threaded same as the rest jsdelivr/globalping-probe#10

Output:

  • raw output. If HEAD then the raw headers. If GET then body without headers.
  • parsed output of headers in all cases
  • performance meta-data: dns lookup, tcp connection, tls handshake, time to first byte, download, total time
  • tls certificate meta-data like issuer, domain, issue and expiration date, valid bool...

Must support the same global parameters, especially the resolver. But make sure we still check the hostname or IP part of the URL against our blacklist.

Global parameters:

  • target - hostname or IP address (required)
  • resolver {1.1.1.1} - empty means local resolve (optional)
  • limit {5} - empty means 1 per selected location (optional)
  • continent - (optional)
  • country - (optional)
  • region - (optional)
  • state - (optional)
  • city - (optional)
  • asn - (optional)"

Exponential backoff for probes

Exponential backoff associated with failing tests. Need to make sure that probes with poor results are not ruining the tests. The goal is to prevent dial-up users from breaking the service.
Probably need to also log probes in "backing-off" state to make sure we dont lose half our network due to a bug in this system.

Limit of measurements

Right now we have a simple anti-abuse system that simply rate-limits the amount of POST requests we accept per IP. Thats fine but its only an anti-abuse system.

We also need an extensive system to allow us to limit the amount of actual tests a user can run. e.g. I want to set 200 tests limit per hour per IP address with no authentication.
In theory that IP can now run either 1 measurement with limit of probes set to 200, and if we actual have 200 probes we run the test and return the results. But all future measurement requests will be declined until the 1 hour limit resets.

Or it can run 200 POST requests asking for 1 probe in each, same thing from the limit's point of view.

Need to make sure here that we correctly count them. So if a user has a limit of 200 and asks for 200 probes in Poland but we only have 50 probes in Poland that means that after the tests finish he will still have 150 tests remaining for that hour.

Later we will expand this for custom limits per API key and plan it's subscribed to, but at the moment its not necessary.

Needs #153

Add probes listing endpoint

We need an endpoint that will list all connected and available probes to build a map on a website. Probably it will be a list of probe.location objects since we don't have other probes information that should be shared publically.

Geographical coordinates are required there to place a marker on a map. MaxMind provides location information for IP addresses so it must be added to the probes as well.

GET /v1/probes

[
  {
    "continent": "EU",
    "region": "eastern-europe",
    "country": "UA",
    "city": 706483,
    "asn": 123456,
    "latitude": 49.98081,
    "longitude": 36.25272
  },
  ...
]

API Documentation

Plan so far: Use .md to document the API in this repo. Once ready add the link to the API too #23

Add metric to appsignal

  1. connected probes count
  2. total measurements per minute
  3. measurements per type per minute
  4. total results count in Redis (?)
  5. measurement time per type (not sure that app signal supports that)

API errors response format

We must unify the error response format for the API.

I suggest something similar to Stripe error formats:

{
  "error": {
    "message": "Something bad happened",
    "type": "api_error",
  }
}

For validation errors, we also need to provide a list of failed fields with a description:

{
  "error": {
    "message": "Validation Failed",
    "type": "invalid_request_error",
    "params": {
      "field_name": "Message"
    }
  }
}

IP Geo v2

Even with 3 IP data sources I get lots of wrong locations with datacenter IPs. And now while I control all of them I can either remove the wrong probes completely or make a request to maxmind and ipinfo to update their data.
But soon we will have 0 control over them and if we detect an Australian IP as American it will end up heavily influencing the results with people not understanding why their USA to USA test is so slow.

The problem is that even if a DB gets an IP's geo correctly we can't know it without a human review. So the current logic of 2vs1 is still the most optimal one.

So I am thinking how we could complement the existing system. Some ideas:

  1. What Patryk said. Run some latency test. e.g. if our DB logic says the IP is in Dallas then have the probe ping an IP address we 100% know is located in Dallas (e.g. aws endpoints). Then if the latency is higher than 5-10ms that means the DB was wrong. But this has lots of potential pitfalls. What if we dont have a static endpoint anywhere close to the probe? How do we decide the exact number of ms as the threshold? And do we do after the test fails? Sounds too complicated and unreliable.
  2. Manual override rules. It's more of a hack than a solution. But basically a config file in Github where we could write IPs or IP ranges and the exact location as we see fit. This would guarantee accuracy but its not scalable. Also we could make the corrections only after someone reports a mistake, so if nobody reports anything we won't be able to fix anything. Also it doesn't work long term. A correct fix now could be wrong in 6 months when the IPs get moved to a different datacenter.
  3. User data. We could allow the users to pass env vars that correct their reported IP GEO but then it becomes a question of trust. It makes us open to abuse and malicious/troll activity.

To me none of the above ideas are great. So lets keep this issue open until we can come up with something better

Validate IPs and hostnames

It seems we try to ping IPs like 800.800.80.80 and we just get a measurement stuck in in-progress.
Same for hostnames like dfdf

Validate IPs and hostnames to make sure they are valid

Probe IDs

How about we expose each probe's unique ID and make the test quarable by the ID?
Example use-case:
I want to run a test from Amazon probe located in Canada. There is no way to set 2 filters at the moment, like country:canada + asn:amazon to run a test only on that specific probe.
But if we had IDs then the user could use /probes to filter as much as he wants and then simply use the IDs he needs

Launch TODO

  • Need a CLI tool.
  • jsdelivr.com/globalping landing page. Description, probes map and stats, how to contribute, how to use, small demo
  • Logo
  • Banner for Github readme and social
  • Full Documentation on Github
  • Grafana with stats
  • Remove debug info from the API

SyntaxError happened in POST [unknown route]

Message

  Unexpected token 
 in JSON at position 95

Backtrace (last 10 lines)

SyntaxError: Unexpected token 
 in JSON at position 95
    at JSON.parse (<anonymous>)
node_modules/co-body/lib/json.js:57 parse
node_modules/co-body/lib/json.js:41 AsyncFunction.module.exports
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
node_modules/koa-bodyparser/index.js:100 async
node_modules/koa-bodyparser/index.js:85 async
node_modules/@koa/cors/index.js:59 async
    at async file:///app/dist/lib/http/middleware/ratelimit.js:26:5

View on AppSignal: https://appsignal.com/jsdelivr/sites/6230aa4683eb670b2c24f6e9/exceptions/incidents/10?timestamp=2022-04-05T17:57:27Z

Probe fails to connect

 docker logs a
Checking for the latest version
Current version 0.2.2
Latest version 0.2.2
[2022-04-02 14:10:51] [INFO] [16] [general] Start probe version 0.2.2 in a production mode
[2022-04-02 14:10:52] [ERROR] [16] [general] connection to API failed failed to collect probe metadata

Not sure why it fails since both of our APIs have geo IP data.

 curl ipinfo.io
{
  "ip": "139.99.172.23",
  "hostname": "vps-197e15b9.vps.ovh.ca",
  "city": "Sydney",
  "region": "New South Wales",
  "country": "AU",
  "loc": "-33.8678,151.2073",
  "org": "AS16276 OVH SAS",
  "postal": "1001",
  "timezone": "Australia/Sydney",
  "readme": "https://ipinfo.io/missingauth"

https://globalping-geoip.global.ssl.fastly.net/139.99.172.23

Consider banning VPNs

How about we also block probes from connecting that have proxy_description=vpn and tor-* plus maybe proxy_type=anonymous,aol,blackberry,corporate,?
https://developer.fastly.com/reference/vcl/variables/geolocation/client-geo-proxy-description/
https://developer.fastly.com/reference/vcl/variables/geolocation/client-geo-proxy-type/

hosting example https://globalping-geoip.global.ssl.fastly.net/142.132.251.61

A probe hosted behind a VPN would only create problems. If the user is in China with the VPN server in Germany it would get registered in our system as a German probe. Then if someone tried to use the probe he would get 100ms latency to ping Germany from Germany.

Support ISP as metadata

We already have this info as part of https://globalping-geoip.global.ssl.fastly.net/

"as": {
"name": "internetia sp.z o.o.",

So why not use it. I suggest we start collecting this data from all probes and exposing in our API.
And make it part of POST for measurements too to all users to do limit their tests to certain providers.

Example use-case: I want to understand my performance from AWS to my CDN. Or to choose which Hetzner location I should be using. I could then run a latency test from "network: hetzner".

This means it should also be part of #51

UI demo

We need a simple UI demo for easier testing of the API. It can be as simple as an HTML file with a bunch of inputs for everything that our API supports which will also show the results in real-time by requesting the measurement ID every 500ms.

Output the raw output with original formatting as main output with all the additional metadata included below

Block malware IPs and domains

Lets also integrate anti-abuse logic. To stop people using probes to connect to malicious IPs and domains.
Any request to a banned domain should be stopped by the API before even getting to any probe.
Lists I propose using:

Its important since people could be running probes in their homes so we need to do our best to protect them. Lists can be auto-updated during API deployment

Originally posted by @jimaek in https://github.com/jsdelivr/globalping/issues/1#issuecomment-1044580719

Real-time results are in random order

Run a few tests like a traceroute to a far away location like 141.164.34.61

The raw output will flash random lines. Like first line 8 of the traceroute, then line 14, then nothing, then 2 random lines and then the full output. It must emulate the way it appears in the command line, one line after the other

Consider combining "world" location with other location types

In some cases, it may be useful to combine "world" type location with other location types:

I think your example could be useful when you want to cover the world + extra probes from a specific location. To avoid creating a huge POST with every single country. Limits and everything else would work as any other location

"locations": [
    {
        "type": "world"
    },
    {
        "type": "country",
        "value": "NL"
    }
]

This will require:

  1. adding a new "virtual" location "world"
  2. add a validation rule to filter multiple "world" locations in the request
  3. think about how to combine different probe selection strategies

DNS needs performance metadata

One of the key use-cases of the dns command is benchmarking the performance of resolvers and dns servers.
So every dns command should have a "querytime" parameter that says how long the query actually took.
dig already returns that value by default.

Unrelated but we should also include the "server" parameter of dig as well. Its important to understand what resolver was used for the query.

;; Query time: 11 msec
;; SERVER: 8.8.8.8#53(8.8.8.8)

Probe geo mixed city and country

We have a probe in Spain, Madrid but the API has mixed the data, it says Madrid USA:

"location": {
"continent": "NA",
"region": "northern-america",
"country": "US",
"city": "madrid",
"asn": 20473,
"latitude": 40.4165,
"longitude": -3.7026,
"network": "choopa llc"

These 2 DBs have wrong data and different cities https://globalping-geoip.global.ssl.fastly.net/208.85.17.142
By our GeoIP logic that means we should fallback to data by ipinfo https://ipinfo.io/208.85.17.142

But it looks like we falled back only for city and kept the rest from a different DB. We can't mix this kind of info, so the fallback should be for all location values.

Commands and parameters to support

The user facing parameters. On the backend we will need to set our own stuff.

ping

  • packets {6} - The number of packets to send. max 16. Default is 3 (optional)
  • quick {true} - boolean, a special fast test that returns a single value in ms per location, instead of full ping output. Default false (optional)

traceroute

  • protocol {TCP} - Traceroute protocol. ICMP|TCP|UDP. Default is UDP (optional)
  • port {90} - Port to use if TCP is selected above. Default 80 (optional)

dns

  • type {CNAME} - DNS query type to use. Default A. Support ['A' or 'AAAA' or 'ANY' or 'CNAME' or 'DNSKEY' or 'DS' or 'MX' or 'NS' or 'NSEC' or 'PTR' or 'RRSIG' or 'SOA' or 'TXT' or 'SRV'] (optional)
  • protocol {TCP} - Traceroute protocol. TCP|UDP. Default is UDP (optional)
  • port {90} - Supports both UDP and TCP. Default 53 (optional)

Global parameters:

  • target - hostname or IPv4 (required)
  • resolver {1.1.1.1} - empty means local resolve (optional)
  • limit {5} - empty means 1 per selected location (optional)
  • continent - (optional)
  • country - (optional)
  • region - (optional)
  • state - (optional)
  • city - (optional)
  • asn - (optional)

The CLI and other UI will have a virtual smart field "location". E.g. location=Eastern Europe|Athens|as3454|Germany and it will return a ready json to be used to provide the IDs to the static fields above. This requires a separate location resolver endpoint.

Limit probes per ip

Limit to 1 connection per IP address from probes. Don't allow a user to start 200 containers on his PC and DDOS us.

Custom regions as locations for probes

Alongside well-known geographical location types like country, continent, city, we must also support custom regions for better and simpler routing.

These regions should be defined as a group of countries. Request validation must be tweaked to allow only predefined regions.
Probes routing (#9 ) should support it as well.

Proposed regions are:

name
Caribbean
Southern Asia
Middle Africa
Northern Europe
Southern Europe
Western Asia
South America
Polynesia
Australia and New Zealand (?)
Western Europe
Eastern Africa
Western Africa
Eastern Europe
Central America
Northern America
South-Eastern Asia
Southern Africa
Eastern Asia
Northern Africa
Melanesia
Micronesia
Central Asia

Get preview URLs working

Currently API previews dont work

2022-03-29T14:49:12.656703+00:00 app[web.1]: WARNING: NODE_ENV value of 'production' did not match any deployment config file names.
2022-03-29T14:49:12.656851+00:00 app[web.1]: WARNING: See https://github.com/lorenwest/node-config/wiki/Strict-Mode
2022-03-29T14:49:13.085654+00:00 app[web.1]: [2022-03-29 14:49:13] [ERROR] [22] [global] failed to start cluster connect ECONNREFUSED 127.0.0.1:6379
  1. Do we need them? @zarianec @patrykcieszkowski
  2. If we do then lets fix them. First we need to enable the free redis instances on each preview app and then use the REDIS_TLS_URL, REDIS_URL vars it sets automatically.

Benchmark probes / auto-scale tests per probe

A small raspberry pi should not be getting the same amount of tests as an 8 core intel server.
Maybe run a local benchmark during startup or something? Because an 8 core Atom is not the same as an 8 core Xeon either

@patrykcieszkowski We didn't come up with anything concrete yet. Artem planned to do some manual benchmarks and see if he can come up with usable data to then integrate into the code.
Maybe you have an idea how to do this?

MVP

  • API + websocket server
  • probe agent
  • Resolve location endpoint for magic like "Ping from 10 locations in Eastern Europe"
  • Real-time results
  • Endpoint to build a map of active probes
  • Auto-updates within a docker container. Assume the user will never pull a new Docker container
  • Auto-scaling of tests per probe. A small raspberry pi should not be getting the same amount of tests as an 8 core intel server. Maybe run a local benchmark during startup or something? Because an 8 core Atom is not the same as an 8 core Xeon either
  • Exponential backoff associated with failing tests. Need to make sure that probes with poor results are not ruining the tests. The goal is to prevent dial-up users from breaking the service
  • Limit to 1 connection per IP address from probes. Don't allow a user to start 200 containers on his PC and DDOS us.

ping has no raw output

Ping has the same issue as #57 but at the end it doesnt even return the output, e.g.

{
  "id": "xCZhqmMDXldUel7f",
  "type": "ping",
  "status": "finished",
  "createdAt": 1648571655024,
  "updatedAt": 1648571685028,
  "results": [
    {
      "probe": {
        "continent": "EU",
        "region": "western-europe",
        "country": "NL",
        "state": null,
        "city": "amsterdam",
        "asn": 14061,
        "longitude": 4.8897,
        "latitude": 52.374
      },
      "result": {
        "rawOutput": "\n--- 141.164.34.61 ping statistics ---\n15 packets transmitted, 14 received, 6.66667% packet loss, time 2808ms\nrtt min/avg/max/mdev = 275.943/276.093/276.595/0.165 ms, pipe 2\n"
      }
    }
  ]
}

GitHub bot

  • MVP Done. Only countries work
  • Needs the resolve endpoint to make it understand all locations

Resolve location endpoint

Resolve location endpoint for magic like "Ping from 10 locations in Eastern Europe".
Its needed for easier development of web and cli tools.

It should also support partial names like "frankfurt" when in reality the probes are at "frankfurt am main" or "amazon" when full is "amazon.com inc"

Example inputs:

  • Africa
  • AS80085
  • Germany
  • Athens
  • Frankfurt
  • Western Europe
  • Asia
  • USA
  • PL
  • EU
  • Colorado
  • New York
  • NYC
  • California
  • Hetzner (#70 )
  • Amazon (#70)

The output should be something easily usable for the measurement endpoint.

No location tests are stuck

When I run ping tests without setting a location e.g.

{"measurement":{"type":"ping","target":"google.com"},"locations":[],"limit":1}

the response always comes from 1 specific probe. In this case its always Johannesburg

{"id":"WPWKOLw8unO7Xljg","type":"ping","status":"in-progress","createdAt":1649698775713,"updatedAt":1649698776228,"results":[{"probe":{"continent":"AF","region":"southern-africa","country":"ZA","state":null,"city":"johannesburg","asn":199524,"longitude":28.0436,"latitude":-26.2023,"network":"g-core labs s.a."},"result":{"rawOutput":"PING google.com (172.217.169.78) 56(84) bytes of data.\n64 bytes from 172.217.169.78: icmp_seq=1 ttl=114 time=158 ms\n"}}]}

We had a logic with weights and priorities and the probe should be selected at random based on those weights

Probes selection rules

Current measurement request payload:

{
    "locations": [
        {
            "type": "country",
            "value": "FR",
            "limit": 3 // per-location limit
        },
        {
            "type": "country",
            "value": "PL",
            "limit": 4 // per-location limit
        }
    ],
    "measurement": {
        "type": "ping",
        "target": "hello.com",
        "packets": 3
    },
    "limit": 10 // global limit
}

Global limit and per-location limit are mutually exclusive - it means that if the global limit is present in the request, the per-location limit is not allowed and vice-versa.

Because we are not limited by a single location type per request - the resulting probes set must contain unique records only.

const probes = [
	{ id: 1, country: "PL", contient: "EU" },
	{ id: 2, country: "UA", contient: "EU" },
	{ id: 3, country: "PL", contient: "EU" },
	{ id: 4, country: "NL", contient: "EU" },
];

const request = [
	{ type: "country": value: "PL", limit: 1 },
	{ type: "continent": value: "EU", limit: 4 },
]

const result = [
	// because of request[0]
	{ id: 3, country: "PL", contient: "EU" },
	

	// because of request[1] - even though 4 probes from the EU was requested - only 3 was found
	{ id: 2, country: "UA", contient: "EU" }, 
	{ id: 4, country: "NL", contient: "EU" },
	{ id: 1, country: "PL", contient: "EU" },
]

Case 1. With locations and limit per location

This is the easiest one. We must find the exact amount of probes for each location. If we don't have enough probes for a specific location, only these we have will be used and the final probes count may be smaller than expected.

Example:
Probes: UA:10; PL:4; NL:3
Request: UA:5; PL: 7; NL:3
Result: UA:5; PL:4; NL:3

Case 2. With locations and global limit

The final amount of probes shouldn't exit the global limit. It may be smaller if we don't have enough probes in total though.
At least 1 probe from each requested location must be present in the results (if we have probes there at all ofc.).
If some locations don't have enough probes the result must be filled with probes from locations that have them.

Example:
Probes: UA:6; PL:2; NL:1
Request: UA; PL; NL - Limit: 6
Result: UA:3; PL:2; NL:1

Case 3. Without locations and global a limit (e.g. World Wide)

For World Wide (WW) requests, we must apply "smart" logic where we break the world into groups and the final probes set must contain a predefined percentage of probes from each group.

percentage location
5 Africa
15 Asia
30 Europe
10 Oceania
30 North America
10 South America

Example:
For a WW request with a global limit of 100, the result will contain 30 probes from Europe, 15 from Asia, etc.

Add rate limits on API

POST /v1/measurements must accept no more than 100 requests per minute

Make sure to write the exit the limit event to logs and report to appSignal so we can see it.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.