Q/A tools for Kiwix Download Mirrors
kiwix / mirrors-qa Goto Github PK
View Code? Open in Web Editor NEWQ/A tools for Kiwix Download Mirrors
License: GNU General Public License v3.0
Q/A tools for Kiwix Download Mirrors
License: GNU General Public License v3.0
From the very installation of the worker, I noticed that the mullvad-check request (executed on the wireguard container) at the beginning of the manager failed.
I thought it could be a race condition in the initialization and left it for later.
Having left the worker all week end I realize it failed to process tests and logs are full of
[2024-08-05 07:38:49,088: INFO] Sleeping for 300.0s
[2024-08-05 07:44:01,252: ERROR] docker api error for exec_command (attempt 1): Unable to execute command ['curl', '--interface', 'wg0', '-s', 'https://am.i.mullvad.net/json'] in mirrors-qa-wireguard, result: b'', status: 6
[2024-08-05 07:44:16,682: ERROR] docker api error for exec_command (attempt 2): Unable to execute command ['curl', '--interface', 'wg0', '-s', 'https://am.i.mullvad.net/json'] in mirrors-qa-wireguard, result: b'', status: 6
[2024-08-05 07:44:37,776: ERROR] docker api error for exec_command (attempt 3): Unable to execute command ['curl', '--interface', 'wg0', '-s', 'https://am.i.mullvad.net/json'] in mirrors-qa-wireguard, result: b'', status: 6
[2024-08-05 07:45:06,493: ERROR] error while processing tasks Unable to execute command ['curl', '--interface', 'wg0', '-s', 'https://am.i.mullvad.net/json'] in mirrors-qa-wireguard, result: b'', status: 6
[2024-08-05 07:45:06,493: INFO] Sleeping for 300.0s
[2024-08-05 07:50:18,091: ERROR] docker api error for exec_command (attempt 1): Unable to execute command ['curl', '--interface', 'wg0', '-s', 'https://am.i.mullvad.net/json'] in mirrors-qa-wireguard, result: b'', status: 6
[2024-08-05 07:50:35,415: ERROR] docker api error for exec_command (attempt 2): Unable to execute command ['curl', '--interface', 'wg0', '-s', 'https://am.i.mullvad.net/json'] in mirrors-qa-wireguard, result: b'', status: 6
[2024-08-05 07:50:57,949: ERROR] docker api error for exec_command (attempt 3): Unable to execute command ['curl', '--interface', 'wg0', '-s', 'https://am.i.mullvad.net/json'] in mirrors-qa-wireguard, result: b'', status: 6
[2024-08-05 07:51:23,729: ERROR] error while processing tasks Unable to execute command ['curl', '--interface', 'wg0', '-s', 'https://am.i.mullvad.net/json'] in mirrors-qa-wireguard, result: b'', status: 6
That exit-code 6
error in curl is a resolve error. Executing without -s
is explicit: curl: (6) Could not resolve host: am.i.mullvad.net
It's not a DNS-only issue as the following dont work either:
root@d0dc5d0d7006:/# curl --interface wg0 --resolve 'am.i.mullvad.net:443:45.83.223.233' https://am.i.mullvad.net/json
curl: (28) Failed to connect to am.i.mullvad.net port 443 after 129775 ms: Could not connect to server
root@d0dc5d0d7006:/# curl --interface wg0 --connect-to 'am.i.mullvad.net:443:45.83.223.233:443' https://am.i.mullvad.net/json
curl: (28) Failed to connect to 45.83.223.233 port 443 after 130806 ms: Could not connect to server
I can see that at the moment, it's linked to one of the US server but we didn't receive any data
root@d0dc5d0d7006:/# wg show
interface: wg0
public key: ew/S+MT5IX/wKx4ITIWoLCqhrC66mS6jK2cdnMP60Ho=
private key: (hidden)
listening port: 40760
fwmark: 0xca6c
peer: NODBnzBgLO5Itv+HOyQer7jpVn1UdCKfv9uwAAvvMkA=
endpoint: 185.156.46.130:51820
allowed ips: 0.0.0.0/0, ::/0
transfer: 0 B received, 4.80 MiB sent
That endpoint IP is publicly ping-able but it does not work from the container.
We need to bootstrap the repository with the basic architecture for the main components:
mirrors-qa/
├── .github
│ ├── workflows
│ │ ├── backend-QA.yaml
│ │ └── ...
├── backend
│ ├── Dockerfile
│ ├── pyproject.toml
│ └── ...
├── dev
│ └── docker-compose.yaml
├── frontend
│ └── README
└── worker
├── Dockerfile
├── pyproject.toml
└── ...
Backend is a Python project and should thus be based on our bootstrap.
It's a FastAPI project backed by a PostgreSQL DB. DB schemas is to be set in code. This will use SQLalchemy as we use on zimfarm dispatcher and metrics.
Stub API would just expose an helloworld for now
Backend is a Python project and should thus be based on our bootstrap.
It should be packaged as one or more docker images (with a compose in this case).
As a foreign component, we want it to be a simple, dumb, task executor. It is composed of a main loop awaiting test requests.
At this stage, we've agreed on not developping a GUI from scratch.
How we'll consume the data is not yet decided but we're leaning towards deploying a metabase instance which will be connected to the DB.
Frontend would thus mostly consist in carefuly crafted exports/dashboards in metabase (metabase stores all this inside a dedicated DB: H2/PostgreSQL/MySQL)
A docker-compose should be provided for developers to easily setup the whole project
It's not a bug per say but not having the feature is preventing any worker use.
With the current implementation, there is no Country in the DB. It can be created manually for that's not what we want/decided. We want countries to reflect worker capabilities.
ATM this information is never sent from worker to Backend so unless the Backend admin manually adds countries, there will never be countries in the DB and thus never be Tests…
Proposed solution:
GET /workers/{worker}/countries
endpoint (just to know what's in there, as its easy to build)PUT /workers/{worker}/countries
: list of ISO country codes. Creates any Country not in DB. Replaces worker-country association with new contentHere's a list of the information I think we'd want in the DB.
Given we may want to use Metabase to consume data, we must have all information in-DB as metabase will connect directly and we wont be able to leverage computed stuff in the backend.
This must be taken into consideration when designing the DB. Might have an impact on DB size but that's not an issue
ASN and below params are duplicates from the MB DB that would only be helpful once we intend to configure MB remotely. We may skip all this for now. Also, those information are not available on mirrors.html
@elfkuzco, use this as direction and not specification. You are invited to suggest otherwise. Also at this stage, we want a basis to build upon. No doubt this will evolve as we implement.
In order to build a dashboard that accurately gives insights on the results of the speedtest, it is important to ask the right questions before building the chart to answer the question.
Currently, here are some of the questions I have come up with:
NOTE: These questions allow for optional filtering based on worker ID.
Below is a sample anwering the first two questions.
CLI commands, even with --verbose
(eg mirrors-qa-backend --verbose update-worker --countries fr,us,de hetzner1
) has no feedback. That's unfortunate as the user has no clue whether something happened or not and can only trust the return code.
Worker is configured with the Backend API URL and credentials. For now, as we envision having only a single worker, we can have the worker's configuration (List of countries supported) done in the DB directly.
Worker communicates with the Backend via the Backend API.
Polling Backend API periodically is very reliable and transparent. In addition, we may have a zmq based push system that allows for instant operations on backend scheduling.
The worker is composed of a main manager that is mostly this poll loop awaiting test requests. Once a request arrive, the worker informs the backend that it is processing it.
Then it starts the actual test process:
Once the tester has completed, results are read and submitted to the API (and deleted from disk?).
Worker is back on its loop.
Communication between the manager container and the tester one be done docker exec
.
Alternative is to use filesystem: tester to store its output into a well known file onto a shared volume/mount. We used both methods on zimfarm, reliably.
Criteria:
I've already identified Mullvad for which we paid for 6months for development:
ATM, the single worker has nothing to do:
[2024-08-06 12:07:38,656: INFO] Found 45 country codes from configuration files.
[2024-08-06 12:07:38,846: INFO] Updated the list of countries for worker to 45 countries.
[2024-08-06 12:07:39,014: INFO] Fetched 0 test(s) from Backend API
[2024-08-06 12:07:39,015: INFO] Sleeping for 300.0s
and the scheduler creates nothing
[2024-08-06 12:11:00,960: INFO] No idle workers found.
[2024-08-06 12:11:00,960: INFO] Sleeping for 10800.0 seconds.
The worker sleep duration is set for 5mn (300s) so well under the 1h interval so it should not be an issue.
Looking at the last_seen_on value (used for idle matching), it says August 6, 2024, 11:17 AM
which is when the worker last uploaded results… Are we only updating last_seen_on
on results upload?
This ticket tracks the Kiwix GSoC 20232024 Project Automated Download Speed Testing for download.kiwix.org until code contributions and/or specific tickets requires creating its own dedicated repository.
Candidates, contributors, this ticket is the preferred location to discuss this project. Ask your questions here and mentors (@rgaudin, @kelson42) shall respond.
Mandatory reads:
Objective: create a solution that automatically tests the download speed of download.kiwix.org from various locations on Earth. This will help Kiwix to ensure fast and reliable downloads for all users and improve the overall user experience, as 80% of our users are located in the Global South. By automating the testing process, Kiwix can save time and resources (we’re a non-profit, after all).
Technologies: Python, VPN, datacenters
Description:
Kiwix provides offline access to Wikipedia and other educational content to millions of users worldwide. Ensuring fast and reliable downloads from download.kiwix.org is critical to delivering a good user experience. However, it is challenging to accurately test download speeds from different locations on earth due to network latency and other factors.
The objective of this project is to create an automated solution to test the download speed of download.kiwix.org from various locations on earth using VPN or datacenters. The solution will use python to run the tests and record the results. The tests will be run at regular intervals and the results will be stored in a database. This will allow Kiwix to monitor download speeds over time and identify any issues that may be affecting performance.
Key Deliverables:
Skills required:
Difficulty: Easy, but probably a longer (350 hours) project
Now that we have the data we want in metabase, we need to prepare for its automated use.
Ultimate goal is to reconfigure mirrorbrain using this data but this will require external glue code that's not a concern at the moment. Also we still dont know how we'll configure (see kiwix/container-images#263)
Here we want to have a sample script that can take params (filters) and queries the Metabase questions we built and retrieves its information in a parsable form.
Scheduling is entirely done in the backend.
The backend knows about the Active workers (those that showed themselves within some interval –1h?) and thus the testable countries.
Schedule code is ran periodically:
Schedule clean-up is also ran periodically and consists in expiring Test requests for which data never arrived.
Tester worker is the independent speedtest image/container that does the actual speedtest.
It's input is an URL to test and returns test results as JSON
{
"started_on": "ISO formatted datetime",
"status": "success|error",
"error": "Error reason",
"ip": "Current Public IP",
"asn": "Current Public IP AS",
"isp": "Current Public IP ISP",
"country": "Current Public IP Country",
"location": "Current Public IP City",
"latency": "Average Ping result to netloc of URL",
"size": "Size in bytes of downloaded URL",
"duration": "Duration in seconds (micro?) of the download",
"speed": "Speed in bytes per second of the download"
}
I'm not sure about IP-related data. We don't want to store just the IP as associated data might change over time and we want the DB to have all meaningful data. Still we could:
All the mirrors must serve the same file for use with the speed test. Location and name needs to accommodate all servers.
Although it should not require any mirror intervention, an email must be sent to admins informing them about this new file and its purpose.
File should be random binary data and weight 50MB
.
We may also host a 1MB
file for simulation/devel for use where speed accuracy is not important
In order to limit the scope of this project, and to understand what are the available variables we can act upon, we need to have a very clear understanding of how the load balancer works: what information is used and how in order to compute a request response.
I suppose it's all in mirrorbrain's documentation.
I am sharing here the configuration (removed the admin's details) as some would matter.
id | identifier | baseurl | baseurl_ftp | baseurl_rsync | enabled | status_baseurl | region | country | asn | prefix | ipv6_only | score | scan_fpm | last_scan | comment | operator_name | operator_url | public_notes | lat | lng | country_only | region_only | as_only | prefix_only | other_countries | file_maxsize | |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
5 | mirror.kiwix | https://mirror.download.kiwix.org/ | ftp://mirror.download.kiwix.org/ | rsync://rsyncd-service/self.download.kiwix.org/ | TRUE | TRUE | eu | fr | 0 | FALSE | 500 | 473 | 2023-01-27 15:44:17.793 +0000 | Added - Wed Mar 8 15:11:24 2017 | Kiwix | https://www.kiwix.org | 48.860 | 2.350 | FALSE | FALSE | FALSE | FALSE | 0 | ||||
3 | mirror2 | http://mirror2.kiwix.org | ftp://mirror2.kiwix.org | FALSE | TRUE | 0 | FALSE | 3 | 0 | Added - Mon Apr 30 15:05:32 2012 | 46.000 | 2.000 | FALSE | FALSE | FALSE | FALSE | 0 | ||||||||||
6 | kiwix | http://mirror.kiwix.org | ftp://mirror.kiwix.org | FALSE | TRUE | 0 | FALSE | 1 | 512 | Added - Thu Jul 28 12:04:35 2011 | 46.000 | 2.000 | FALSE | FALSE | FALSE | FALSE | 0 | ||||||||||
7 | mirror3 | http://mirror3.kiwix.org | ftp://mirror3.kiwix.org | FALSE | FALSE | eu | fr | 0 | FALSE | 10 | 845 | Added - Thu Jan 9 20:16:22 2014 | 0.000 | 0.000 | FALSE | FALSE | FALSE | FALSE | 0 | ||||||||
8 | netcologne.de | http://mirror.netcologne.de/kiwix/ | ftp://mirror.netcologne.de/kiwix/ | FALSE | TRUE | eu | de | 0 | FALSE | 5000 | 300 | Added - Fri Jul 31 11:22:47 2015 \n\n*** scanned and enabled at Fri Jul 31 11:27:04 2015. | 0.000 | 0.000 | FALSE | FALSE | FALSE | FALSE | 0 | ||||||||
4 | mirror.tn | https://mirror.ati.tn/wiki/ | FALSE | TRUE | af | tn | 0 | FALSE | 5000 | 0 | 2019-05-25 17:59:54.125 +0000 | "Added - Sat Sep 19 20:25:51 2015 | |||||||||||||||
*** reenabled by mirrorprobe at Sat May 25 17:47:28 2019." | 0.000 | 0.000 | FALSE | FALSE | FALSE | FALSE | 0 | ||||||||||||||||||||
9 | fau.de | https://ftp.fau.de/kiwix/ | ftp://ftp.fau.de/kiwix/ | TRUE | TRUE | eu | de | 0 | FALSE | 5000 | 636 | 2023-01-27 15:40:24.841 +0000 | Added - Thu Mar 12 17:53:10 2015 | Friedrich-Alexander-Universität | https://www.fau.de/ | 0.000 | 0.000 | FALSE | FALSE | FALSE | FALSE | 0 | |||||
18 | mirror-sites-ca.mblibrary.info | https://mirror-sites-ca.mblibrary.info/mirror-sites/download.kiwix.org/ | rsync://mirror-sites-ca.mblibrary.info/download.kiwix.org/ | TRUE | TRUE | na | ca | 0 | FALSE | 100 | 0 | Added - Wed Aug 16 09:43:38 2023 | MB Group | https://www.mbgroup.global/ | 0.000 | 0.000 | FALSE | FALSE | FALSE | FALSE | 0 | ||||||
17 | mirror-sites-fr.mblibrary.info | https://mirror-sites-fr.mblibrary.info/mirror-sites/download.kiwix.org/ | rsync://mirror-sites-fr.mblibrary.info/download.kiwix.org/ | TRUE | TRUE | eu | fr | 0 | FALSE | 100 | 0 | Added - Wed Aug 16 09:36:48 2023 | MB Group | https://www.mbgroup.global/ | 0.000 | 0.000 | FALSE | FALSE | FALSE | FALSE | 0 | ||||||
10 | isoc.org.il | https://mirror.isoc.org.il/pub/kiwix/ | TRUE | TRUE | EU | IL | 0 | FALSE | 100 | 0 | 2023-01-27 15:40:28.352 +0000 | Added - Sun Jul 17 14:22:46 2011 | Israel Internet Association | https://isoc.org.il | 31.500 | 34.750 | TRUE | FALSE | FALSE | FALSE | 0 | ||||||
12 | nluug.nl | https://ftp.nluug.nl/pub/kiwix/ | ftp://ftp.nluug.nl/pub/kiwix/ | rsync://ftp.nluug.nl/kiwix/ | TRUE | TRUE | eu | nl | 0 | FALSE | 3000 | 494 | 2023-01-27 15:40:27.225 +0000 | Added - Tue Feb 28 14:18:32 2017 | NLUUG | https://nluug.nl | 52.094 | 5.119 | FALSE | FALSE | FALSE | FALSE | 0 | ||||
2 | mirror.accum.se | https://saimei.ftp.acc.umu.se/mirror/kiwix.org/ | ftp://mirror.accum.se/mirror/kiwix.org/ | rsync://mirror.accum.se/mirror/kiwix.org/ | TRUE | TRUE | eu | se | 0 | FALSE | 3000 | 83 | 2023-01-27 15:40:24.117 +0000 | Added - Wed Jun 7 11:34:05 2017 | Academic Computer | Club in Umeå | https://www.accum.se | 59.329 | 18.069 | FALSE | FALSE | FALSE | FALSE | 0 | |||
16 | laotzu-mirror.accum.se | https://laotzu.ftp.acc.umu.se/mirror/kiwix.org/ | ftp://mirror.accum.se/mirror/kiwix.org/ | rsync://mirror.accum.se/mirror/kiwix.org/ | TRUE | TRUE | eu | se | 0 | FALSE | 3000 | 0 | Added - Tue Mar 21 11:01:30 2023 | Academic Computer Club in Umeå | https://www.accum.se | 59.329 | 18.069 | FALSE | FALSE | FALSE | FALSE | 0 | |||||
14 | your.org | https://ftpmirror.your.org/pub/kiwix/ | ftp://ftpmirror.your.org/pub/kiwix/ | rsync://ftpmirror.your.org/pub/kiwix/ | TRUE | TRUE | na | us | 0 | FALSE | 5000 | 0 | 2023-01-27 15:40:31.488 +0000 | "Added - Wed Oct 9 10:52:56 2013 | |||||||||||||
*** scanned and enabled at Sat Nov 9 16:45:16 2013." | Your.org | https://your.org | 0.000 | 0.000 | FALSE | FALSE | FALSE | FALSE | 0 | ||||||||||||||||||
1 | dotsrc.org | https://mirrors.dotsrc.org/kiwix/ | ftp://mirrors.dotsrc.org/kiwix/ | rsync://mirrors.dotsrc.org/kiwix/ | TRUE | TRUE | eu | dk | 0 | FALSE | 5000 | 0 | 2023-01-27 15:40:21.219 +0000 | Added - Wed Mar 1 11:22:34 2017 | dotsrc | https://dotsrc.org | 55.676 | 12.568 | FALSE | FALSE | FALSE | FALSE | 0 | ||||
15 | md.mirrors.hacktegic.com | https://md.mirrors.hacktegic.com/kiwix-md/ | rsync://md.mirrors.hacktegic.com/kiwix-md/ | TRUE | TRUE | eu | md | 0 | FALSE | 3000 | 0 | Added - Mon Feb 13 09:41:46 2023 | md.mirrors.hacktegic.com | https://md.mirrors.hacktegic.com/ | 46.940 | 28.916 | FALSE | FALSE | FALSE | FALSE | 0 | ||||||
11 | wikimedia | https://dumps.wikimedia.org/kiwix/ | TRUE | TRUE | na | us | 0 | FALSE | 3000 | 0 | 2023-01-27 15:40:26.346 +0000 | Added - Sun May 15 09:58:02 2011 | Wikimedia Foundation | https://wikimedia.org | 37.770 | -122.393 | FALSE | FALSE | FALSE | FALSE | 0 | ||||||
19 | mirror-sites-in.mblibrary.info | https://mirror-sites-in.mblibrary.info/mirror-sites/download.kiwix.org/ | rsync://mirror-sites-in.mblibrary.info/download.kiwix.org/ | TRUE | TRUE | as | in | 0 | FALSE | 100 | 0 | Added - Wed Aug 16 09:49:41 2023 | MB Group | https://www.mbgroup.global/ | 0.000 | 0.000 | FALSE | FALSE | FALSE | FALSE | 0 | ||||||
13 | mirrorservice.org | https://www.mirrorservice.org/sites/download.kiwix.org/ | ftp://ftp.mirrorservice.org/sites/download.kiwix.org/ | rsync://ftp.mirrorservice.org/download.kiwix.org/ | TRUE | TRUE | eu | gb | 0 | FALSE | 5000 | 0 | 2023-01-27 15:40:22.308 +0000 | "Added - Sat Nov 9 | 16:34:19 2013 | ||||||||||||
*** scanned and enabled at Sat Nov 9 16:41:26 2013. | |||||||||||||||||||||||||||
*** scanned and enabled at Sat Nov 9 16:42:27 2013." | University of Kent | https://www.kent.ac.uk/computing | 0.000 | 0.000 | FALSE | FALSE | FALSE | FALSE | 0 |
@kelson42, from our recent call, I suppose the *_only
params might not be properly set.
Tentative list of endpoints we'd need at first. We mostly want CRUD on tests and authentication. API is restful JSON of course.
Endpoint | Usage |
---|---|
POST /auth/authenticate |
Authenticate users (workers) |
GET /tests |
List of tests from DB. |
/tests |
Not needed, tests created by scheduler |
GET /tests/:id |
Test entry details |
PATCH /tests/:id |
for worker to update the test with status and/or results |
GET /tests
can be public. It must support pagination and filtering (worker, country, etc)
I'm not sure how we ended up requesting an RSA private key for create-worker
but that's a mistake.
The backend doesn't need to and has no interest in knowing the worker's private key. We only need (and actually only use) the public key.
We need to initialize our DB (#6) with mirrors information and we'll also need to update them periodically.
For this, we should rely on the online list of mirrors that we have.
https://download.kiwix.org/mirrors.html
It's in HTML format but all information can be extracted easily.
Let's use the URL's hostname as ID for now and also assume that any DB mirror not found in the page should be marked disabled
We'll need to provision a dedicated machine for the worker.
Assumptions:
50MB
to RAM then switch to doing the same from another mirror.Machine | EUR/m | Bandwidth |
---|---|---|
Dedibox Start-2-SSD |
10 | 250Mbps |
SCW PLAY2-PICO |
14 | 100Mbps |
SCW DEV1-M |
18 | 300Mbps |
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.