Comments (12)
Should we remove frameworks like gnet? It only implements plaintext and isn't actually doing any parsing / routing - it just scans to the \r\n\r\n and sends a canned response which doesn't meet the test requirements.
from frameworkbenchmarks.
I like it a lot, but it exist a problem long ago.
At the moment the framework is removed, all the history of the framework will disappear from the Rounds.
Like I said before the Rounds need to be immutable.
For example, in PHP we need to change the name because was php5, after the change plain PHP don't appear in the old Rounds.
We have the numbers, and the work done, but don't show in the Rounds.
from frameworkbenchmarks.
one thing i have been thinking is not quite fair is to combine results from different framework mutations together into the composite score. surely composite score should reflect a single configuration and that configuration's performance across all benches?
for example, if we look at ntex, which was top of the last official round, the different flavours get wildly different scores across the different benchmarks. is it fair to pick the best mutation in each category and combine those for composite? is it even possible to run a single service on ntex which would score highly across all benches? it doesn't seem so, but this is surely what the composite score should be measuring?
maybe a better system would be to sum up the scores across all benchmarks for a particular mutation and then, for each framework, choose the mutation that got the best composite score?
maybe this has been raised before. sorry for bringing it up again if so.
I agree with you on the composite score issue. Besides incompatible features it's a common practice in the bench that frameworks implement low level json
and/or plaintext
to boost their composite score which is questionable to say at least.
Speaking of ntex
from what I see the current bench has to choose one async runtime which means it's tokio
or async-std
flavor scores can't be achieved at the same time. That said it's possible to modify the code to combine multiple runtimes and get the best of them which would be a big refactor but it can be done.
from frameworkbenchmarks.
Rules like these show how popular the project is and I agree with both.
On top of it I suggest composite score being calculated per mutation which would offer a quick view of per mutation detail.
from frameworkbenchmarks.
@nbrady-techempower on the number of mutations I proposed #8055 some time ago but then left it given the community feedbacks. It might be worth a while re-check it
from frameworkbenchmarks.
One framework to remove: Baratine. The domain baratine.io is not registered to the project anymore (careful, clickbait!), and the github project has last changes 7 years ago (https://github.com/baratine/baratine)
from frameworkbenchmarks.
In reality Baratine is marked as Stripped.
Why not bypass all the stripped frameworks from the runs ??
from frameworkbenchmarks.
In reality Baratine is marked as Stripped.
Why not bypass all the stripped frameworks from the runs ??
I disagree. In xitca-web Stripped bench is used to avoid polluting the default leaderboard while keep perf tracking of low level system software like OS and lang(and/or program) runtime at the same time. In fact Stripped is a fairly arbitrary category because there are even more unrealistic bench marked as Realistic. Unless there is a unified standard to determine what bench must be Stripped or not it's unfair to bypass them.
from frameworkbenchmarks.
@fakeshadow Ok.
I'm happy that is useful this information.
And about the what need to be Stripped, I think that it's a work of all the devs here, help to clarify the requirements and also to identify the frameworks than bypass these requirements.
from frameworkbenchmarks.
@fakeshadow Ok. I'm happy that is useful this information.
And about the what need to be Stripped, I think that it's a work of all the devs here, help to clarify the requirements and also to identify the frameworks than bypass these requirements.
Unfortunately the meaning of "Realistic" is subjective and from the existing bench code it's clear we have very divided opinions among bench maintainers. Therefore I doubt a common ground can be reached easily.
Actually I'm fine with the current configuration where the category is up to the maintainers to decide. When people look into the code and figure it out they would know which framework and it's community share the same opinion.
In other word as long as stripped bench can run in non official bench I personally find it's fine. As for broken(or outdated) bench I believe we can use βbrokenβ tag to stop them from hogging resources in runs.
from frameworkbenchmarks.
one thing i have been thinking is not quite fair is to combine results from different framework mutations together into the composite score. surely composite score should reflect a single configuration and that configuration's performance across all benches?
for example, if we look at ntex, which was top of the last official round, the different flavours get wildly different scores across the different benchmarks. is it fair to pick the best mutation in each category and combine those for composite? is it even possible to run a single service on ntex which would score highly across all benches? it doesn't seem so, but this is surely what the composite score should be measuring?
maybe a better system would be to sum up the scores across all benchmarks for a particular mutation and then, for each framework, choose the mutation that got the best composite score?
maybe this has been raised before. sorry for bringing it up again if so.
from frameworkbenchmarks.
@MarkReedZ , your project also has bugs:
#9055
from frameworkbenchmarks.
Related Issues (20)
- Round 22 results site shows "woo" test with Racket in the language column whereas it is written in Common Lisp
- New execution mode "profiling" HOT 6
- Enhancement request: disable pg_stat_statements when running anything but validation
- PHP 8.3 update [info]
- Inconsistent composite score best score computation HOT 2
- Holiday Break HOT 8
- Where to find the exact code that was used for Round22? HOT 1
- Expired SSL `tfb-status.techempower.com` HOT 1
- Actix failing build HOT 9
- New Server Set up HOT 57
- Most of the best-performing frameworks don't survive temporary db connectivity loss HOT 25
- [IMPORTANT] To further test fairness. Please consider this for next benchmark tests. HOT 1
- gRPC Framework benchmark HOT 1
- Please upgrade hardware for extreme testings HOT 1
- Increase concurrency for future rounds to better showcase performance on large servers HOT 2
- Randomize Framework order in runs HOT 11
- Add GraalVM native build for Quarkus HOT 3
- Add container name in tfb for local metrics HOT 1
- Create benchmarks using seastar HOT 1
- [Python] mrhttp: a project with many problems
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
π Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. πππ
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google β€οΈ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from frameworkbenchmarks.