kiwix/mirrors-qa

Q/A tools for Kiwix Download Mirrors

mirrors-qa

Q/A tools for Kiwix Download Mirrors

mirrors-qa's Issues

Worker network issue

From the very installation of the worker, I noticed that the mullvad-check request (executed on the wireguard container) at the beginning of the manager failed.
I thought it could be a race condition in the initialization and left it for later.
Having left the worker all week end I realize it failed to process tests and logs are full of

[2024-08-05 07:38:49,088: INFO] Sleeping for 300.0s
[2024-08-05 07:44:01,252: ERROR] docker api error for exec_command (attempt 1): Unable to execute command ['curl', '--interface', 'wg0', '-s', 'https://am.i.mullvad.net/json'] in mirrors-qa-wireguard, result: b'', status: 6
[2024-08-05 07:44:16,682: ERROR] docker api error for exec_command (attempt 2): Unable to execute command ['curl', '--interface', 'wg0', '-s', 'https://am.i.mullvad.net/json'] in mirrors-qa-wireguard, result: b'', status: 6
[2024-08-05 07:44:37,776: ERROR] docker api error for exec_command (attempt 3): Unable to execute command ['curl', '--interface', 'wg0', '-s', 'https://am.i.mullvad.net/json'] in mirrors-qa-wireguard, result: b'', status: 6
[2024-08-05 07:45:06,493: ERROR] error while processing tasks Unable to execute command ['curl', '--interface', 'wg0', '-s', 'https://am.i.mullvad.net/json'] in mirrors-qa-wireguard, result: b'', status: 6
[2024-08-05 07:45:06,493: INFO] Sleeping for 300.0s
[2024-08-05 07:50:18,091: ERROR] docker api error for exec_command (attempt 1): Unable to execute command ['curl', '--interface', 'wg0', '-s', 'https://am.i.mullvad.net/json'] in mirrors-qa-wireguard, result: b'', status: 6
[2024-08-05 07:50:35,415: ERROR] docker api error for exec_command (attempt 2): Unable to execute command ['curl', '--interface', 'wg0', '-s', 'https://am.i.mullvad.net/json'] in mirrors-qa-wireguard, result: b'', status: 6
[2024-08-05 07:50:57,949: ERROR] docker api error for exec_command (attempt 3): Unable to execute command ['curl', '--interface', 'wg0', '-s', 'https://am.i.mullvad.net/json'] in mirrors-qa-wireguard, result: b'', status: 6
[2024-08-05 07:51:23,729: ERROR] error while processing tasks Unable to execute command ['curl', '--interface', 'wg0', '-s', 'https://am.i.mullvad.net/json'] in mirrors-qa-wireguard, result: b'', status: 6

That exit-code 6 error in curl is a resolve error. Executing without -s is explicit: curl: (6) Could not resolve host: am.i.mullvad.net

It's not a DNS-only issue as the following dont work either:

root@d0dc5d0d7006:/# curl --interface wg0 --resolve 'am.i.mullvad.net:443:45.83.223.233' https://am.i.mullvad.net/json
curl: (28) Failed to connect to am.i.mullvad.net port 443 after 129775 ms: Could not connect to server
root@d0dc5d0d7006:/# curl --interface wg0 --connect-to 'am.i.mullvad.net:443:45.83.223.233:443' https://am.i.mullvad.net/json
curl: (28) Failed to connect to 45.83.223.233 port 443 after 130806 ms: Could not connect to server

I can see that at the moment, it's linked to one of the US server but we didn't receive any data

root@d0dc5d0d7006:/# wg show
interface: wg0
  public key: ew/S+MT5IX/wKx4ITIWoLCqhrC66mS6jK2cdnMP60Ho=
  private key: (hidden)
  listening port: 40760
  fwmark: 0xca6c

peer: NODBnzBgLO5Itv+HOyQer7jpVn1UdCKfv9uwAAvvMkA=
  endpoint: 185.156.46.130:51820
  allowed ips: 0.0.0.0/0, ::/0
  transfer: 0 B received, 4.80 MiB sent

That endpoint IP is publicly ping-able but it does not work from the container.

Basic repo architecture

Basic architecture

We need to bootstrap the repository with the basic architecture for the main components:

mirrors-qa/
├── .github
│   ├── workflows
│   │   ├── backend-QA.yaml
│   │   └── ...
├── backend
│   ├── Dockerfile
│   ├── pyproject.toml
│   └── ...
├── dev
│   └── docker-compose.yaml
├── frontend
│   └── README
└── worker
    ├── Dockerfile
    ├── pyproject.toml
    └── ...

backend

Backend is a Python project and should thus be based on our bootstrap.

It's a FastAPI project backed by a PostgreSQL DB. DB schemas is to be set in code. This will use SQLalchemy as we use on zimfarm dispatcher and metrics.

Stub API would just expose an helloworld for now

worker

Backend is a Python project and should thus be based on our bootstrap.
It should be packaged as one or more docker images (with a compose in this case).

As a foreign component, we want it to be a simple, dumb, task executor. It is composed of a main loop awaiting test requests.

frontend

At this stage, we've agreed on not developping a GUI from scratch.
How we'll consume the data is not yet decided but we're leaning towards deploying a metabase instance which will be connected to the DB.
Frontend would thus mostly consist in carefuly crafted exports/dashboards in metabase (metabase stores all this inside a dedicated DB: H2/PostgreSQL/MySQL)

dev

A docker-compose should be provided for developers to easily setup the whole project

Missing countries upload from manager

It's not a bug per say but not having the feature is preventing any worker use.

With the current implementation, there is no Country in the DB. It can be created manually for that's not what we want/decided. We want countries to reflect worker capabilities.
ATM this information is never sent from worker to Backend so unless the Backend admin manually adds countries, there will never be countries in the DB and thus never be Tests…

Proposed solution:

• GET /workers/{worker}/countries endpoint (just to know what's in there, as its easy to build)
• PUT /workers/{worker}/countries: list of ISO country codes. Creates any Country not in DB. Replaces worker-country association with new content
• manager start: retrieves and converts list of countries from config files ; then after Auth, it PUTs to that endpoint.

Database schema

Here's a list of the information I think we'd want in the DB.

Given we may want to use Metabase to consume data, we must have all information in-DB as metabase will connect directly and we wont be able to leverage computed stuff in the backend.
This must be taken into consideration when designing the DB. Might have an impact on DB size but that's not an issue

Mirrors

• ID
• Base URL
• Enabled
• Region
• ➡️ Country
• ASN
• Score
• Latitude
• Longitude
• Country Only
• Region Only
• AS Only
• Other Countries

ASN and below params are duplicates from the MB DB that would only be helpful once we intend to configure MB remotely. We may skip all this for now. Also, those information are not available on mirrors.html

Countries

• Code
• Name

Workers

• ID
• ➡️ Countries
• Auth Info
• Last Seen On

Tests

• ID
• ➡️ Mirror
• ➡️ Worker*
• ➡️ Country
• RequestedOn
• StartedOn
• Status: Missed | Succeeded | Errored
• Error*
• IP
• ASN
• ISP
• Location (City)
• Latency
• Download Size
• Download duration
• Download speed

@elfkuzco, use this as direction and not specification. You are invited to suggest otherwise. Also at this stage, we want a basis to build upon. No doubt this will evolve as we implement.

Questions for Dashboard Development

In order to build a dashboard that accurately gives insights on the results of the speedtest, it is important to ask the right questions before building the chart to answer the question.

Currently, here are some of the questions I have come up with:

• What is the status distribution for tests per mirror?
• What is the maximum speed from each test country across all the mirrors?
• What is the minimum speed from each test country across all the mirrors?
• What is the average latency of requests from a test country across all the mirrors?
• Which ISP offers the maximum/minimum speeds?

NOTE: These questions allow for optional filtering based on worker ID.

Below is a sample anwering the first two questions.

CLI command feedback

CLI commands, even with --verbose (eg mirrors-qa-backend --verbose update-worker --countries fr,us,de hetzner1) has no feedback. That's unfortunate as the user has no clue whether something happened or not and can only trust the return code.

Worker manager

Worker is configured with the Backend API URL and credentials. For now, as we envision having only a single worker, we can have the worker's configuration (List of countries supported) done in the DB directly.

Worker communicates with the Backend via the Backend API.
Polling Backend API periodically is very reliable and transparent. In addition, we may have a zmq based push system that allows for instant operations on backend scheduling.

The worker is composed of a main manager that is mostly this poll loop awaiting test requests. Once a request arrive, the worker informs the backend that it is processing it.

Then it starts the actual test process:

• informs the network container to reconfigure for requested country
• ensures that this succeeded
• starts the network-blind test container (which reports results to filesystem) and awaits its completion.

Once the tester has completed, results are read and submitted to the API (and deleted from disk?).
Worker is back on its loop.

Communication between the manager container and the tester one be done docker exec.

Alternative is to use filesystem: tester to store its output into a well known file onto a shared volume/mount. We used both methods on zimfarm, reliably.

Basic repo architecture/2

• Working FastAPI
• Stub DB access via backend
• Stub tests using DB
• Stub worker script doing nothing
• Working docker images
• Published docker images
• Working dev docker compose

Decide on a VPN provider

Criteria:

• offers wireguard (modern protocol we'll instrument)
• good connectivity (reputation)
• has a large number of server locations
• server locations includes areas of interest to us:
  • where we think we have a large user base
  • where we lack mirrors
  • ➡️ mostly South-East Asia & Africa

I've already identified Mullvad for which we paid for 6months for development:

• 684 servers in 44 countries full list
• Includes Hong Kong, Indonesia, Singapore, South Africa and Thailand
• we can retrieve the servers via an API
• it's whats best in terms of privacy
• cost is fixed, stable and affordable: $5/mo.

Not scheduling

ATM, the single worker has nothing to do:

[2024-08-06 12:07:38,656: INFO] Found 45 country codes from configuration files.
[2024-08-06 12:07:38,846: INFO] Updated the list of countries for worker to 45 countries.
[2024-08-06 12:07:39,014: INFO] Fetched 0 test(s) from Backend API
[2024-08-06 12:07:39,015: INFO] Sleeping for 300.0s

and the scheduler creates nothing

[2024-08-06 12:11:00,960: INFO] No idle workers found.                                                                                                                                                                       
[2024-08-06 12:11:00,960: INFO] Sleeping for 10800.0 seconds.

The worker sleep duration is set for 5mn (300s) so well under the 1h interval so it should not be an issue.
Looking at the last_seen_on value (used for idle matching), it says August 6, 2024, 11:17 AM which is when the worker last uploaded results… Are we only updating last_seen_on on results upload?

Automated Download Speed Testing for download.kiwix.org

This ticket tracks the Kiwix GSoC 20232024 Project Automated Download Speed Testing for download.kiwix.org until code contributions and/or specific tickets requires creating its own dedicated repository.

Candidates, contributors, this ticket is the preferred location to discuss this project. Ask your questions here and mentors (@rgaudin, @kelson42) shall respond.

Mandatory reads:

• https://github.com/kiwix/overview/blob/main/CONTRIBUTING.md
• https://github.com/kiwix/overview/blob/main/CODE_OF_CONDUCT.md

---

Automated Download Speed Testing for download.kiwix.org

Objective: create a solution that automatically tests the download speed of download.kiwix.org from various locations on Earth. This will help Kiwix to ensure fast and reliable downloads for all users and improve the overall user experience, as 80% of our users are located in the Global South. By automating the testing process, Kiwix can save time and resources (we’re a non-profit, after all).

Technologies: Python, VPN, datacenters

Description:

Kiwix provides offline access to Wikipedia and other educational content to millions of users worldwide. Ensuring fast and reliable downloads from download.kiwix.org is critical to delivering a good user experience. However, it is challenging to accurately test download speeds from different locations on earth due to network latency and other factors.

The objective of this project is to create an automated solution to test the download speed of download.kiwix.org from various locations on earth using VPN or datacenters. The solution will use python to run the tests and record the results. The tests will be run at regular intervals and the results will be stored in a database. This will allow Kiwix to monitor download speeds over time and identify any issues that may be affecting performance.

Key Deliverables:

• A python script that tests download speed from various locations on earth.
• A database to store the test results.
• A web-based interface to view the results.
• Documentation for how to set up and use the solution.

Skills required:

• Strong knowledge of python programming language
• Familiarity with network testing and performance analysis.
• Experience with VPNs and datacenters.
• Knowledge of databases and web development is a plus.

Difficulty: Easy, but probably a longer (350 hours) project

Automatically retrieve metabase data

Now that we have the data we want in metabase, we need to prepare for its automated use.
Ultimate goal is to reconfigure mirrorbrain using this data but this will require external glue code that's not a concern at the moment. Also we still dont know how we'll configure (see kiwix/container-images#263)

Here we want to have a sample script that can take params (filters) and queries the Metabase questions we built and retrieves its information in a parsable form.

Scheduling

Scheduling is entirely done in the backend.

The backend knows about the Active workers (those that showed themselves within some interval –1h?) and thus the testable countries.

Schedule code is ran periodically:

• If there's no idle worker ➡️ quit (await next period)
• get list of countries for idle workers
• create test entries for all possible countries in DB
• notify worker that there are entries
• worker either receives notification or polls API
• worker retrieves detail about one test ; informs API that it is processing it
• worker runs the test and submits the data to the API
• worker is back on its loop

Schedule clean-up is also ran periodically and consists in expiring Test requests for which data never arrived.

Tester worker

Tester worker is the independent speedtest image/container that does the actual speedtest.

It's input is an URL to test and returns test results as JSON

{
  "started_on": "ISO formatted datetime",
  "status": "success|error",
  "error": "Error reason",
  "ip": "Current Public IP",
  "asn": "Current Public IP AS",
  "isp": "Current Public IP ISP",
  "country": "Current Public IP Country",
  "location": "Current Public IP City",
  "latency": "Average Ping result to netloc of URL",
  "size": "Size in bytes of downloaded URL",
  "duration": "Duration in seconds (micro?) of the download",
  "speed": "Speed in bytes per second of the download"
}

I'm not sure about IP-related data. We don't want to store just the IP as associated data might change over time and we want the DB to have all meaningful data. Still we could:

• query an online service (simplest but prone to failures or denial)
• embedded a free IP database (need to check if we can get it all in free sets) but it's more work and needs periodic updates
• submit just the IP and have the backend do what's on the previous line

Create test files

All the mirrors must serve the same file for use with the speed test. Location and name needs to accommodate all servers.
Although it should not require any mirror intervention, an email must be sent to admins informing them about this new file and its purpose.

File should be random binary data and weight 50MB.
We may also host a 1MB file for simulation/devel for use where speed accuracy is not important

Document mirrorbrain behavior

In order to limit the scope of this project, and to understand what are the available variables we can act upon, we need to have a very clear understanding of how the load balancer works: what information is used and how in order to compute a request response.

I suppose it's all in mirrorbrain's documentation.

I am sharing here the configuration (removed the admin's details) as some would matter.

id	identifier	baseurl	baseurl_ftp	baseurl_rsync	enabled	status_baseurl	region	country	asn	prefix	ipv6_only	score	scan_fpm	last_scan	comment	operator_name	operator_url	public_notes	lat	lng	country_only	region_only	as_only	prefix_only		other_countries	file_maxsize
5	mirror.kiwix	https://mirror.download.kiwix.org/	ftp://mirror.download.kiwix.org/	rsync://rsyncd-service/self.download.kiwix.org/	TRUE	TRUE	eu	fr	0		FALSE	500	473	2023-01-27 15:44:17.793 +0000	Added - Wed Mar 8 15:11:24 2017	Kiwix	https://www.kiwix.org			48.860	2.350	FALSE	FALSE	FALSE	FALSE		0
3	mirror2	http://mirror2.kiwix.org	ftp://mirror2.kiwix.org		FALSE	TRUE			0		FALSE	3	0		Added - Mon Apr 30 15:05:32 2012				46.000	2.000	FALSE	FALSE	FALSE	FALSE		0
6	kiwix	http://mirror.kiwix.org	ftp://mirror.kiwix.org		FALSE	TRUE			0		FALSE	1	512		Added - Thu Jul 28 12:04:35 2011				46.000	2.000	FALSE	FALSE	FALSE	FALSE		0
7	mirror3	http://mirror3.kiwix.org	ftp://mirror3.kiwix.org		FALSE	FALSE	eu	fr	0		FALSE	10	845		Added - Thu Jan 9 20:16:22 2014				0.000	0.000	FALSE	FALSE	FALSE	FALSE		0
8	netcologne.de	http://mirror.netcologne.de/kiwix/	ftp://mirror.netcologne.de/kiwix/		FALSE	TRUE	eu	de	0		FALSE	5000	300		Added - Fri Jul 31 11:22:47 2015 \n\n*** scanned and enabled at Fri Jul 31 11:27:04 2015.				0.000	0.000	FALSE	FALSE	FALSE		FALSE		0
4	mirror.tn	https://mirror.ati.tn/wiki/			FALSE	TRUE	af	tn	0		FALSE	5000	0	2019-05-25 17:59:54.125 +0000	"Added - Sat Sep 19 20:25:51 2015
*** reenabled by mirrorprobe at Sat May 25 17:47:28 2019."				0.000	0.000	FALSE	FALSE	FALSE	FALSE		0
9	fau.de	https://ftp.fau.de/kiwix/	ftp://ftp.fau.de/kiwix/		TRUE	TRUE	eu	de	0		FALSE	5000	636	2023-01-27 15:40:24.841 +0000	Added - Thu Mar 12 17:53:10 2015	Friedrich-Alexander-Universität	https://www.fau.de/		0.000	0.000	FALSE	FALSE	FALSE	FALSE			0
18	mirror-sites-ca.mblibrary.info	https://mirror-sites-ca.mblibrary.info/mirror-sites/download.kiwix.org/		rsync://mirror-sites-ca.mblibrary.info/download.kiwix.org/	TRUE	TRUE	na	ca	0		FALSE	100	0		Added - Wed Aug 16 09:43:38 2023	MB Group		https://www.mbgroup.global/		0.000	0.000	FALSE	FALSE	FALSE	FALSE		0
17	mirror-sites-fr.mblibrary.info	https://mirror-sites-fr.mblibrary.info/mirror-sites/download.kiwix.org/		rsync://mirror-sites-fr.mblibrary.info/download.kiwix.org/	TRUE	TRUE	eu	fr	0		FALSE	100	0		Added - Wed Aug 16 09:36:48 2023	MB Group		https://www.mbgroup.global/		0.000	0.000	FALSE	FALSE	FALSE	FALSE		0
10	isoc.org.il	https://mirror.isoc.org.il/pub/kiwix/			TRUE	TRUE	EU	IL	0		FALSE	100	0	2023-01-27 15:40:28.352 +0000	Added - Sun Jul 17 14:22:46 2011	Israel Internet Association	https://isoc.org.il		31.500	34.750	TRUE	FALSE	FALSE	FALSE		0
12	nluug.nl	https://ftp.nluug.nl/pub/kiwix/	ftp://ftp.nluug.nl/pub/kiwix/	rsync://ftp.nluug.nl/kiwix/	TRUE	TRUE	eu	nl	0		FALSE	3000	494	2023-01-27 15:40:27.225 +0000	Added - Tue Feb 28 14:18:32 2017	NLUUG	https://nluug.nl		52.094	5.119	FALSE	FALSE		FALSE	FALSE		0
2	mirror.accum.se	https://saimei.ftp.acc.umu.se/mirror/kiwix.org/	ftp://mirror.accum.se/mirror/kiwix.org/	rsync://mirror.accum.se/mirror/kiwix.org/	TRUE	TRUE	eu	se	0		FALSE	3000	83	2023-01-27 15:40:24.117 +0000	Added - Wed Jun 7 11:34:05 2017	Academic Computer	Club in Umeå	https://www.accum.se		59.329	18.069	FALSE	FALSE	FALSE	FALSE		0
16	laotzu-mirror.accum.se	https://laotzu.ftp.acc.umu.se/mirror/kiwix.org/	ftp://mirror.accum.se/mirror/kiwix.org/	rsync://mirror.accum.se/mirror/kiwix.org/	TRUE	TRUE	eu	se	0		FALSE	3000	0		Added - Tue Mar 21 11:01:30 2023	Academic Computer Club in Umeå		https://www.accum.se		59.329	18.069	FALSE	FALSE	FALSE	FALSE		0
14	your.org	https://ftpmirror.your.org/pub/kiwix/	ftp://ftpmirror.your.org/pub/kiwix/	rsync://ftpmirror.your.org/pub/kiwix/	TRUE	TRUE	na	us	0		FALSE	5000	0	2023-01-27 15:40:31.488 +0000	"Added - Wed Oct 9 10:52:56 2013
*** scanned and enabled at Sat Nov 9 16:45:16 2013."	Your.org	https://your.org		0.000	0.000	FALSE	FALSE	FALSE	FALSE		0
1	dotsrc.org	https://mirrors.dotsrc.org/kiwix/	ftp://mirrors.dotsrc.org/kiwix/	rsync://mirrors.dotsrc.org/kiwix/	TRUE	TRUE	eu	dk	0		FALSE	5000	0	2023-01-27 15:40:21.219 +0000	Added - Wed Mar 1 11:22:34 2017	dotsrc	https://dotsrc.org		55.676	12.568		FALSE	FALSE	FALSE	FALSE		0
15	md.mirrors.hacktegic.com	https://md.mirrors.hacktegic.com/kiwix-md/		rsync://md.mirrors.hacktegic.com/kiwix-md/	TRUE	TRUE	eu	md	0		FALSE	3000	0		Added - Mon Feb 13 09:41:46 2023	md.mirrors.hacktegic.com	https://md.mirrors.hacktegic.com/		46.940		28.916	FALSE	FALSE	FALSE	FALSE		0
11	wikimedia	https://dumps.wikimedia.org/kiwix/			TRUE	TRUE	na	us	0		FALSE	3000	0	2023-01-27 15:40:26.346 +0000	Added - Sun May 15 09:58:02 2011	Wikimedia Foundation	https://wikimedia.org		37.770	-122.393	FALSE	FALSE	FALSE	FALSE		0
19	mirror-sites-in.mblibrary.info	https://mirror-sites-in.mblibrary.info/mirror-sites/download.kiwix.org/		rsync://mirror-sites-in.mblibrary.info/download.kiwix.org/	TRUE	TRUE	as	in	0		FALSE	100	0		Added - Wed Aug 16 09:49:41 2023	MB Group		https://www.mbgroup.global/		0.000	0.000	FALSE	FALSE	FALSE	FALSE		0
13	mirrorservice.org	https://www.mirrorservice.org/sites/download.kiwix.org/	ftp://ftp.mirrorservice.org/sites/download.kiwix.org/	rsync://ftp.mirrorservice.org/download.kiwix.org/	TRUE	TRUE	eu	gb	0		FALSE	5000	0	2023-01-27 15:40:22.308 +0000	"Added - Sat Nov 9	16:34:19 2013
*** scanned and enabled at Sat Nov 9 16:41:26 2013.
*** scanned and enabled at Sat Nov 9 16:42:27 2013."	University of Kent	https://www.kent.ac.uk/computing		0.000	0.000	FALSE	FALSE	FALSE	FALSE		0

mirrobrain.csv

@kelson42, from our recent call, I suppose the *_only params might not be properly set.

Backend API

Tentative list of endpoints we'd need at first. We mostly want CRUD on tests and authentication. API is restful JSON of course.

Endpoint	Usage
POST /auth/authenticate	Authenticate users (workers)
GET /tests	List of tests from DB.
POST /tests	Not needed, tests created by scheduler
GET /tests/:id	Test entry details
PATCH /tests/:id	for worker to update the test with status and/or results

GET /tests can be public. It must support pagination and filtering (worker, country, etc)

create-worker should work off a public key

I'm not sure how we ended up requesting an RSA private key for create-worker but that's a mistake.
The backend doesn't need to and has no interest in knowing the worker's private key. We only need (and actually only use) the public key.

Update mirrors DB

We need to initialize our DB (#6) with mirrors information and we'll also need to update them periodically.
For this, we should rely on the online list of mirrors that we have.

https://download.kiwix.org/mirrors.html

It's in HTML format but all information can be extracted easily.

Let's use the URL's hostname as ID for now and also assume that any DB mirror not found in the page should be marked disabled

Deploy MVP

k8s

• Database
• Database Backup
• Backend/API
• Metabase
• Metabase DB (if not using the embedded H2)
• Metabase Database Backup

On dedicated, new machine

• Worker

---

We'll need to provision a dedicated machine for the worker.

Assumptions:

• we have no specific CPU/RAM/Disk requirement because the machine will mostly be downloading 50MB to RAM then switch to doing the same from another mirror.
• we must not pay ingress fees as the machine will be constantly downloading.

Machine	EUR/m	Bandwidth
Dedibox Start-2-SSD	10	250Mbps
SCW PLAY2-PICO	14	100Mbps
SCW DEV1-M	18	300Mbps