All frameworks based on Node.js must be tested on the same docker image.

[KostyaTretyak]

All frameworks based on Node.js must be tested on the same docker image.

[joanhey]

I am creating benchmarks since 2008, and since that time I have seen almost everything.
Some testing with very different size responses for the same body, other testing and say that is faster, but in reality they are testing a 502 from the sever, not from the framework, etc etc etc

I'm the first that want a fair benchmark, and that all use the best tools for that, without make tricks.

But I say you that if one docker image is slower, the best thing is to create an issue to the creators of the dockerfile, not in the benchmark.
The people that see the benchmarks don't know which docker image is using.

Use the benchmark as a tool to see problems in your framework or toolset. And inform about problems to create a faster and healthy ecosystem to your framework or/and language.

Please don't use the benchmark only like a competition of results numbers.
Thanks.

[joanhey]

Sorry, but Distmod still never run with the Citrine Servers. The PR is still not merged.
If you made your test in local, possibly will be very different numbers that with the Citrine Servers.

And you are welcome to change any framework benchmark, to have better results.

[KostyaTretyak]

@joanhey, it is not entirely clear what you wanted to say in this comment. Granted, my local tests won't be the same as on Citrine Servers, but what does that change? Do you think the tests will change disproportionately?

As for the fact that I can change any test to improve it... The change I proposed is probably not to everyone's liking, so changing the rules should be done by the owner of these tests in the first place.

[joanhey]

I can assure you that the results can change a lot in local vs Citrine servers.

Also the results in my pc, can be very different than in your pc.
The benchmark need to be always run with the same servers.
Any small config change in the server, will change a lot of results.

[KostyaTretyak]

Absolutely agree. Just what does it change in the context of my issue? Do you doubt that frameworks using different versions of docker images will show different results?

[joanhey]

No, but i'll be not a big change.
And if it's exist a big difference, perhaps is better to create an issue in the creators of the docker image.

I know form PHP, that using the official PHP docker images, a lot of times is a lot slower than use a normal Ubuntu image and download the required packages to run PHP.
But a benchmark is for that, to show discrepancies. The problem is not in the benchmark it's in the docker file.
And the correct thing is create an issue to the docker file, not in the benchmark.

[KostyaTretyak]

Tests should have the same initial conditions for all frameworks as much as possible. As you can guess, it is not a big problem for me if my framework shows better results (among Node.js frameworks). But I want a fair competition. My proposed changes add precision to the results of this competition.

[nbrady-techempower]

We've discussed this a lot in the past and it's not something we plan on requiring at this time. There are reasons why framework maintainers may want their framework running on a specific version of a language/engine/etc and we leave it up to those maintainers to decide.