Comments (63)
Here's the latest data from dir.friendica.social with the above query:
20180430-site-health.zip
from dir.
I'll admit I didn't expect to have so much fun when I agreed to maintain the Friendica Directory.
from dir.
I'm not entirely clear on this point yet. But I think once we understand the probability of how the ping (X) value affects the response time (Y), we should know how to "factor" it.
The log function -- and here I understand this a little better -- is merely to normalise the curve of your factored values. If you were to divided Y/X and your curve consequently looks like this, you cannot really do the zoning properly.
So if you use log(Y/X), you could already do some zoning, even Y/X is just a random factoring. The above graph then should look less skewed.
from dir.
The MrPetovan/dir:master branch has been updated. You can use it to test the ping feature. Additionally, you can now use a CLI console tool to trigger a probe on a specific domain or site-health-id manually.
from dir.
I love everything about this.
from dir.
Just for clarification: When we pull and push as defined in sync-targets, we only transfer information about profiles and not any health related server information? Correct?
Yes, health is computed locally.
from dir.
I'm analysing the data that dir.hubup.pro has generated over the last few days (since we are collecting avg_ping
). I currently have only 176 nodes, about 100 less than the last dir.friendica.social dataset.
Preliminary, it looks promising. I think there are similarities in the quartiles, despite that actually the values for discounted_request_time
are very different. I'll wait for a few more days to get more data and will than show a detailed box chart to compare both directories.
I'm currently looking into ways how to calculate the quantiles and extreme values. I found something that looks like a potential direction for how to do quantiles in php (see below).
Once we know Q1 -3 you can also calculate the lower "extreme value" (i.e. Q3 + 1.5 x IQR).
function Median($Array) {
return Quartile_50($Array);
}
function Quartile_25($Array) {
return Quartile($Array, 0.25);
}
function Quartile_50($Array) {
return Quartile($Array, 0.5);
}
function Quartile_75($Array) {
return Quartile($Array, 0.75);
}
https://blog.poettner.de/2011/06/09/simple-statistics-with-php/
from dir.
Here the code with fixed values.
//Speed scoring.
if (intval($time) > 0) {
//Pentaly / bonus points.
if ($time > 515) {
$current -= 10; //Bad speed.
} elseif ($time > 251) {
$current -= 5; //Still not good.
} elseif ($time > 181) {
$current += 0; //This is normal.
} elseif ($time > 71) {
$current += 5; //Good speed.
} else {
$current += 10; //Excellent speed.
}
and here with dynamic points based on all discounted_request_time
values sorted.
//Speed scoring.
if (intval($time) > 0) {
//Pentaly / bonus points.
if ($time > $lower_iqr) {
$current -= 10; //Bad speed.
} elseif ($time > $Quartile_75) {
$current -= 5; //Still not good.
} elseif ($time > $Quartile_50) {
$current += 0; //This is normal.
} elseif ($time > $Quartile_25) {
$current += 5; //Good speed.
} else {
$current += 10; //Excellent speed.
}
lower_iqr
= Quartile_75
+ 1.5 x (Quartile_75
- Quartile_25
)
from dir.
Some preliminary results. Here a comparison of the two directories, one in Western Europe and the other Southeast Asia.
The datasets were taken at different times and have different total number of nodes (181 v. 270).
Results based on this equation:
discounted_request_time
= request_time
- (avg_ping
* coefficient
)
The coefficient:
dir.friendica.social-20180430 = 3.49712908930528
dir.hubup.pro-20180512 = 0.146000348540368
from dir.
Here you are:
I did deduplicate base_url but I didn't added the nurl
column. It shouldn't skew the data too much.
from dir.
Here's my plan to change the policy:
- Ping all the known reachable servers
- Divide the average response time by the average ping latency
- Create break points based on percentiles on the resulting data
- Grade servers on a curve based on this data so that the absolute value doesn't matter as much.
My first impulse would be to do it once, take the resulting break points and hard-code them in the health scoring process, but ideally the break point should be updated regularly. The issue is that it requires to ping every know server which takes significant amount of time especially with 5 tries.
Maybe it could be a rolling thing where the ping is checked at the same time as the server health, and the break points are moved less regularly.
Another idea would be to have a fully proportional score change based on min/max values, this wouldn't require any break points.
from dir.
The following is a proposed direction by a trained econometrician, whom I consulted about how to conceptually handle the issue by determining meaningful values for "server health".
Conditional distribution
request_time (Y) is a random variable and we are interested how it relates to another variable called avg_ping (X).
The most we can know about how X affects Y is contained in the conditional distribution of Y given X. This information is summarised by the conditional probability function, defined by:
for all values of x such that:
The interpretation is most easily be seen when X and Y are discreet. Then,
where the right-hand side is read as "the probability that Y = y given that X = x."
Skewness
Once we understand the probabilities we need to test whether the data set is skewed or normal. If skewed, we use a log function to normalise it.
Zoning
With a normal distribution we can easily define different zones to the left and right of the bell cure crest. Here we can establish the cut off points for the different level of healthiness of the server.
from dir.
Here are the native math functions: http://php.net/manual/en/book.math.php
For statistics, it's a separate module: http://php.net/manual/en/book.stats.php
from dir.
Thanks for asking and for the work, I've a question: the bell curve studies only one variable (the response time in our case), how do you factor in the ping? Through the potential log function?
from dir.
Are we waiting on your friend then, or do we have to do anything by ourselves?
from dir.
I'm going to test the log function for the time being. Next Friday, I'll see him again. He will try to get someone who will do the empirical part for us on a spreadsheet. Once we know the factor formula and the benchmark values, it can be coded.
Today's discussion was very brief (after some other meeting) and were merely conceptual. I put up the concept for people to comment. But I understand this is quite advanced and also partially beyond my own expertise.
from dir.
Beyond my own as well!
from dir.
So the LOG-normal functions works as predicted. Here a before and after on the "request_time" variable:
This would be ready for zoning, if it had the ping factored in. I'll try to give it go even before the next meeting.
--
But there is a remaining issue, as we have an incomplete data set. I think, we need all known nodes because the inclusion of these 257 nodes out of more than 1000 is done rather arbitrary.
@MrPetovan do you think you could obtain data for all known nodes? All we need to ensure is that there is no duplication (e.g. http vs https).
from dir.
There’s a difference between “known node” and “active node”. I voluntarily limited the dataset to active nodes only because ping and request times don’t make sense for dead nodes.
from dir.
That's good. Sorry my confusion!
Yes, if that's all active nodes out there that's great.
Do you have a script for generating the data?
from dir.
No, it was a simple SQL query against the site-health
table, including only nodes with a positive health score.
from dir.
Sorry, this is the part I don't understand.
In my site-health
table I have 25 entries that are of positive value. The renaming 400 servers have a negative score. This includes for example my own server, your server, libranet.de.
So my question how do you make that leap from a negative health score to being inactive?
from dir.
Because the score penalty for connection timeout is very high, so anything with the lowest health score (-100) is unlikely to be active.
from dir.
But could this not be part of the latency problem we're trying to solve?
Could you have a look at my table taken from this side of the world? There are some active nodes with -100.
--
It's not very important to have the complete dataset now, because I hope that we can do the health levels dynamically in the future based on percentage zones with relevant cut off points on each site (see here: #43 (comment)). But it may help us to define the zones properly up front. It would rule out any distortion.
We can do this any time once we have the correct probability formula and agreed on the different zones in a spreadsheet for checking.
In the future, each directory might once in awhile calculate automatically the cut off values for health levels based on our predefined zones. This presumably will be done on the basis of all known nodes in the DB.
from dir.
The important field for available node is dt_last_seen
. It's the last time the directory got a successful response. Compare this value with dt_last_probed
which marks the last time the directory tried to probe the server. Any time there's a difference between those two date, a score penalty is applied to the server.
Also remember that high response time will also penalize servers, this is why you have mass-trespass.uk
at -20 despite a successful last probe. And since you're far from most servers in this list, your scores will be overall lower than what I would get with a directory hosted in France.
This is why you probably should take into account nodes where dt_last_seen
and dt_last_probed
have a value and whose values aren't more than X [time unit] apart.
from dir.
Yes, that makes sense. Failed probes, like outdated code, will reduce the overall score of course.
When I said dynamically calculated levels, I purely meant value of the response_time moderate by ping_avg.
My own node running on a piece of junkware is a good example; it has a very high response time but a relatively low ping (from my directory) and high high (from yours). The probability distribution formula will actually reduce my node's response_time value because of the low ping, but it probably will be only a tiny bit. My node will still very likely get a low score in terms of the high RT/ping value. That's the part we are interested in; to see how each percentage zone pans out along the curve at the crest and the base.
Are you able to see why I'm interested in the whole dataset of all known nodes? Do you think I could easily generate this myself for my directory by running a simple query?
from dir.
Of course I see the interest, but I'm weary of inactive nodes skewing the data set in a specific direction.
The other table to consider is site-probe
, you'll find the result of successful probes. You can use average values to get a single set of ping/request time per server, and you can join site-health
so that even current inactive nodes historical data will be included, increasing the sample size without skewing the data set.
The query to extract the data would go along those lines (I can't test until 8 PM EST):
SELECT `base_url`, AVG(`request_time`), AVG(`avg_ping`)
FROM `site-probe`
JOIN `site-health` ON `site-health`.`id` = `site-probe`.`site_health_id`
WHERE `request_time` IS NOT NULL
AND `avg_ping` IS NOT NULL
GROUP BY `site-health`.`id`
from dir.
Thanks! No rush at all.
The conditional probability equation will not produce a value for any node that has a zero ping.
The very long request_times that just end up flat-lining at the base will turn into a steep drop through the normal-log function. See the before and after graph (see here: #43 (comment))
We will do a purely qualitative judgment with the zoning as to how narrow each zone will. This will ensure that very slow machines can never have a medium or high rating.
from dir.
How can there be a ping of zero? Or do you mean a missing value?
from dir.
Less than a millisecond ping. Could happen.
from dir.
Thanks MrPetovan for the data!!
@tobiasd highlights an important point. I think for the equation to give a probability, the ping value must be greater than zero. A superfast 0.001ms ping would still work.
In practice, this means admins who block pings will not have a health value. There are currently nodes that return a fast request_time but have a "0" ping, presumably because it failed.
Rather than giving these nodes a bad health score, they should be given a special status as "unknown health" or something like this. All the nodes that I saw fitting this category, were not open for registration. So this would be the trade off: if you run an open registration node, you need to be pingable. Otherwise you don't get a score.
from dir.
One could use the average ping time for those nodes that block ping. I mean a fix value we find during the evaluation of the health determination round.
from dir.
Yes, that's a possibility, but it's open for manipulation. So if you have a slow request_time and a faster than average ping, you just block your ping and instantly get a better score.
It's a kind of shared risk issue. There are many legitimate reasons why people block ping, but if everyone were to block ping than the directories would not work any more.
Of course we would still provide individual outputs for nodes without ping, similar to screenshot below. But there would not be a final overall score; instead of the heart being green or whatever colour, the heart would just not be shown or something like this, while clearly indicating the node is active. Mostly it wouldn't matter, because only nodes open for registration are listed in ../servers.
from dir.
If someone is gaming the health test that way, we can just introduce a penalty by a horrible factor.
The fixed time could also be at the lower end of the bell-curve; average plus half of the full width half max value or so. Then the fixed time is more likely no desired value for gamers.
from dir.
Yes, that would work. But than it would make the node's health look worse than it really is.
Do you think, it's important to have an overall score in such cases?
from dir.
I don't think there will be gaming of the value, hence my suggestion with the average value to be neutral in that metric. I don't really have an opinion about "no health" value in the listing and if that would be better or worse for the node in terms of "advertising" the node to new users.
from dir.
That's a good point! "Unknown health" can sound worse than "below average health" for some people. We probably need better explanations that would help people to interpret the finer points, either way.
If we give those nodes an overall score, I think a penality would be good mainly for encouraging people to allow pings were this is possible. Mainly a penalty for not sharing the collective risk of contributing to the average and standard deviation ping values.
How exactly that penalty would look like (so that's not an invitation to fiddle the system), we need to see with the data and the calculated probabilities. Something like you said, average or above average ping value, or even a simple 0 = 1.
from dir.
Hypolite I have a question about the datatbase, in particular about the ping value.
Looking at the site-probe
table, I cannot see any ping value. What am I missing here?
I am able to query the db for the request_time and able to join it with health-site
, by omitting avg_ping`.
from dir.
Either:
- You need to update your database according to the
dfrndir.sql
file since there's no automatic update.
Or: - I need to update the
dfrndir.sql
file as the ping could be adir.friendica.social
-only feature at the moment. 😅
(Pretty sure it's the second one)
from dir.
ping could be a dir.friendica.social-only feature
Ahh... it's not in the source yet, you're saying. Oh well. 😀
from dir.
What's the easiest way? I'll update my db and add a table? If you give me a hint, I might be able to do it.
But does the current code actually collect pings or only your unpublished version?
from dir.
Only my unpublished version as well. The easiest way is for me to commit my code.
from dir.
Sorry for the untimeliness :/
from dir.
I am outlining here what seems like a workable solution. @Ken-Ko , one of the team members who is the main driver behind this, has just joined us on github and will follows the implementation thorough and advise us further if necessary.
The initially proposed idea of treating this as a conditional probability problem has been rejected, as it will lead to unnecessary levels of complexity. Instead the more plausible way is to tackle this with a linear regression, i. e. ordinary least squares (OLS).
This will give a "discounted_request_time" based on the request time in relation to the ping value. We will only need basic mathematical operations (addition, subtraction, division, multiplication) and exponentiation. This also will avoid the problem with a zero/ failed ping as 0 will get no discount. Here the basic equation:
discounted_request_time
= request_time
- (avg_ping
* Coefficient)
The Coefficient based on the current dataset (dir.friendica.social-20180430-site-health) is 3.49712908930528
This results in the following hard-coded equation:
discounted_request_time
= request_time
- (avg_ping
* 3.49712908930528)
In the next post, I'll outline how to calculate the Coefficient; so we can do this dynamically, as new nodes come online, or disappear, or hardware changes.
For the moment, I would like you to look at the actual data and inspect Column H (Discounted Y_a). There you see all nodes sorted by "speed" discounted_request_time
. The faster the node, the lower the value. Some fast/ superfast nodes have negative values. We are not really interested in the actual value itself, but in which (of the six) zones each node will fall accordingly.
There is a very nice chart in the excel file that shows the distribution and will give us an idea about the different zones. Unfortunately, the chart is only visible in a specific propitiatory version (which I think is MS Excel 2016). I hope to show this chart here once we get an exported graphic file. If you happen to have that particular version, you can already see the preview. Otherwise just look at the ods file for the data.
The zoning will be done dynamically too in the future. I'm working on this at the moment, but we can also hard-code the values for the time being. This means with the above equation we can have this up and running for testing in no time.
dir-friendica-social-20180430-site-health-OLS.zip
from dir.
Wow, thank you and your team for your work!
from dir.
Scoring (Q1-3)
The data gives us the following three quartiles:
First Quartile: 75
Second Quartile: 118
Third Quartile: 251
In a boxplot the whole data looks like this:
Here in more detail with the outliers removed, generated by BoxPlotR:
http://shiny.chemgrid.org/boxplotr/
So the quick hard-coded fix will look like this:
//Speed scoring.
if (intval($time) > 0) {
//Pentaly / bonus points.
if ($time > 520) {
$current -= 10; //Bad speed.
} elseif ($time > 251) {
$current -= 5; //Still not good.
} elseif ($time > 181) {
$current += 0; //This is normal.
} elseif ($time > 71) {
$current += 5; //Good speed.
} else {
$current += 10; //Excellent speed.
}
from dir.
AVG(avg_ping
)
I'm looking at the average ping values as they come in at dir.hubup.pro.
I think, there seems to be an error in how the average of avg_ping
is calculated. E.g. https://libranet.de in Germany gets 0.0894 or https://social.isurf.ca in California 0.9265 from Thailand.
Is it that a certain number of 0 values are ending up in the average?
from dir.
I don't know, I'm just taking the raw result from the ping command. Have you tried running a manual ping command against the same domains to see if there's a difference?
from dir.
Looking at the avg_ping
column in site-probe
everything is correct! The ping for https://libranet.de is 236 and for https://social.isurf.ca 227, identical to the values taken manually.
The problem occurs when running a query the database to generate an output. I can actually see some error messages in addition to the distorted output.
So the ping seems to work without any issues. I'll check what's going on with my db or phpmyadmin setup.
Just for clarification: When we pull and push as defined in sync-targets
, we only transfer information about profiles and not any health related server information? Correct?
from dir.
I managed to traced the error that occurred when querying the database. It was a result of having modified the table index when I adding the two new columns. It's all fixed now and works as expected. 🙂
from dir.
This looks good, but I'm not sure what it would be for. Determining the cutoff points?
from dir.
Here I'm just showing again the proposed zoning based the data for dir.friendica.social 20180430
A = Excellent
B = Good
C = Normal
D = Still not good
E = Bad
from dir.
This looks good, but I'm not sure what it would be for. Determining the cutoff points?
Yes. See above.
Quartile_25
= Q1
Quartile_50
= Q2
Quartile_75
= Q3
from dir.
@MrPetovan the explanation for the coefficient, I gave, is incorrect. #43 (comment)
I'll try to give the correct version shortly. Hope you have not already coded this.
from dir.
Duplication of nodes
I have noticed there are some duplications in the base_url
table.
Something like:
http://meld.de/
https://meld.de/
But we are quite sure there is only one node running there, despite the difference in protocols.
Even more concerning are duplications of entries with identical protocols. So for instance in Hypolite's dataset there are ten (10!) entries for https://libranet.de and about 15 for https://friendica.ladies.community each with different request_time
values.
What's going on there and how to fix this?
from dir.
The behavior is even stranger than you expect. These are the only 15 redundant base_urls in the dir.friendica.social database:
The first issue is that there isn't a UNIQUE key on the base URL. The second issue is that there's no reduction to a normalized URL (without https
) which would allow to rule out HTTP/HTTPS duplicates.
from dir.
Ohh.. what did we let ourselves in for here... 😲
I think this issue has effected the stats somehow. The coefficients are too different. Could you run this query again with:
WHERE `request_time` IS NOT NULL
AND `avg_ping` IS NOT NULL
We would like to run some further tests. Thanks.
from dir.
Of course, go ahead!
from dir.
[OK, I deleted some of my redundant posts above]
Ko tested the two new datasets for us and we found some interesting developments. I'm summarising a three page long report here and will give the practical implication.
For "dir.frienica.social" the removal of duplicated nodes seemed to make the relationship between request_time
and avg_ping
even stronger. This is good.
However, for "dir.hubup.pro" the data showed there was no relationship between request_time
and avg_ping
. The above (see #43 (comment)) rather different coefficients were already some indication of this. These two graphs might give you further some idea of the problem.
After removing all servers with zero avg_ping
value (and some outliers), we now have established that in the Thai dataset, there is also a significant relationship between request_time
and avg_ping
. Which is good, because it allows us to use the OLS equations as planned.
discounted_request_time
= request_time
- (avg_ping
* coefficient
)
Here the coefficients (plus p-values about the likelihood of no relationship)
"dir.frienica.social = 3.316080104 (p = 0.0)
"dir.hubup.pro" = 4.965583808 (p = 0.001)
Practical implication
The calculation of the coefficient and the Q1, Q2, Q3, and IQR values (see here #43 (comment)) must exclude all nodes with zero avg_ping
. These nodes (providing they have a request_time
that is not zero) will of course still get a health score, but will not contribute to determining the coefficient and speed zones.
from dir.
But then each directory server has to automatically recalculate the coefficients from time to time--right?
from dir.
Correct, and also its speed score zones.
Here an example for zones: #43 (comment)
from dir.
OK, here the coefficient. Please excuse this non-standard notation. I hope this makes sense.
Coefficient
=SUM of all x*y / SUM of all x^2
x = avg_ping
- (AVERAGE of all avg_ping
WHERE avg_ping
is NOT zero)
y = request_time
- (AVERAGE of all request_time
WHERE avg_ping
is NOT zero)
from dir.
Moved to friendica/friendica-directory#4
from dir.
Related Issues (20)
- Dynamic version scoring HOT 5
- Locations Sidebar Results HOT 2
- Support the "account-type" HOT 12
- Remove all Javascript HOT 1
- Add search on tags HOT 2
- health page not working HOT 3
- wrong download link
- Make the maintenance.html page configurable HOT 1
- bug location HOT 2
- Feature list of recommended public servers broken HOT 3
- Short statistic on the directory servers HOT 4
- Decentralized Network Clarification HOT 4
- Profiles with censored=1 should not be updated / unblocked by cron_sync.php HOT 16
- Missing isset() calls causing noisy logs HOT 1
- Update Version Check, including partical automation HOT 2
- Feedback loop between multiple servers with push enabled HOT 21
- Missing UNIQUE key for base_url and reduction to normalized URL HOT 4
- Filter for explicit content nodes in Server list HOT 1
- Next version of the directory HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from dir.