techempower/frameworkbenchmarks

Source for the TechEmpower Framework Benchmarks project

Welcome to TechEmpower Framework Benchmarks (TFB)

If you're new to the project, welcome! Please feel free to ask questions here. We encourage new frameworks and contributors to ask questions. We're here to help!

This project provides representative performance measures across a wide field of web application frameworks. With much help from the community, coverage is quite broad and we are happy to broaden it further with contributions. The project presently includes frameworks on many languages including Go, Python, Java, Ruby, PHP, C#, F#,Clojure, Groovy, Dart, JavaScript, Erlang, Haskell, Scala, Perl, Lua, C, and others. The current tests exercise plaintext responses, JSON serialization, database reads and writes via the object-relational mapper (ORM), collections, sorting, server-side templates, and XSS counter-measures. Future tests will exercise other components and greater computation.

Read more and see the results of our tests on cloud and physical hardware. For descriptions of the test types that we run, see the test requirements section.

If you find yourself in a directory or file that you're not sure what the purpose is, checkout our file structure in our documentation, which will briefly explain the use of relevant directories and files.

Quick Start Guide

To get started developing you'll need to install docker or see our Quick Start Guide using vagrant

1. Clone TFB.

    $ git clone https://github.com/TechEmpower/FrameworkBenchmarks.git
   

2. Change directories

    $ cd FrameworkBenchmarks
   

3. Run a test.

    $ ./tfb --mode verify --test gemini
   

Explanation of the ./tfb script

The run script is pretty wordy, but each and every flag is required. If you are using windows, either adapt the docker command at the end of the ./tfb shell script (replacing ${SCRIPT_ROOT} with /c/path/to/FrameworkBenchmarks), or use vagrant.

The command looks like this: docker run -it --rm --network tfb -v /var/run/docker.sock:/var/run/docker.sock -v [FWROOT]:/FrameworkBenchmarks techempower/tfb [ARGS]

• -it tells docker to run this in 'interactive' mode and simulate a TTY, so that ctrl+c is propagated.
• --rm tells docker to remove the container as soon as the toolset finishes running, meaning there aren't hundreds of stopped containers lying around.
• --network=tfb tells the container to join the 'tfb' Docker virtual network
• The first -v specifies which Docker socket path to mount as a volume in the running container. This allows docker commands run inside this container to use the host container's docker to create/run/stop/remove containers.
• The second -v mounts the FrameworkBenchmarks source directory as a volume to share with the container so that rebuilding the toolset image is unnecessary and any changes you make on the host system are available in the running toolset container.
• techempower/tfb is the name of toolset container to run

A note on Windows

• Docker expects Linux-style paths. If you cloned on your C:\ drive, then [ABS PATH TO THIS DIR] would be /c/FrameworkBenchmarks.
• Docker for Windows understands /var/run/docker.sock even though that is not a valid path on Windows, but only when using Linux containers (it doesn't work with Windows containers and LCOW). Docker Toolbox may not understand /var/run/docker.sock, even when using Linux containers - use at your own risk.

Quick Start Guide (Vagrant)

Get started developing quickly by utilizing vagrant with TFB. Git, Virtualbox and vagrant are required.

1. Clone TFB.

    $ git clone https://github.com/TechEmpower/FrameworkBenchmarks.git
   

2. Change directories

    $ cd FrameworkBenchmarks/deployment/vagrant
   

3. Build the vagrant virtual machine

    $ vagrant up
   

4. Run a test

    $ vagrant ssh
    $ tfb --mode verify --test gemini
   

Add a New Test

Either on your computer, or once you open an SSH connection to your vagrant box, start the new test initialization wizard.

    vagrant@TFB-all:~/FrameworkBenchmarks$ ./tfb --new

This will walk you through the entire process of creating a new test to include in the suite.

Resources

Official Documentation

Our official documentation can be found in the wiki. If you find any errors or areas for improvement within the docs, feel free to open an issue in this repo.

Live Results

Results of continuous benchmarking runs are available in real time here.

Data Visualization

If you have a results.json file that you would like to visualize, you can do that here. You can also attach a runid parameter to that url where runid is a run listed on tfb-status like so: https://www.techempower.com/benchmarks/#section=test&runid=fd07b64e-47ce-411e-8b9b-b13368e988c6. If you want to visualize them or compare different results files on bash, here is an unofficial plaintext results parser

Contributing

The community has consistently helped in making these tests better, and we welcome any and all changes. Reviewing our contribution practices and guidelines will help to keep us all on the same page. The contribution guide can be found in the TFB documentation.

Join in the conversation in the Discussions tab, on Twitter, or chat with us on Freenode at #techempower-fwbm.

frameworkbenchmarks's Issues

Add ASP.NET?

Node.JS mysql async.parallel redundant (or needs to be changed)

As mysql protocol does not allow for parallel queries on one connection to have real parallel queries on need to acquire multiple db connections from pool. If test tries to emulate 'N queries, and each depends on previous result' than async.parallel here is unnecessary, if it's actually 'execute N independent queries' then it might be better to get a connection from pool for each sub-query

Play benchmarks should use the newest RC

see here: playframework/playframework#899

Change php-fpm worker config

From meritt at http://news.ycombinator.com/item?id=5500014

"I'd set minimum idle to something like 16 or 32. php-fpm will not create more than 32 workers/sec.

What happens now is 256 workers running and 256 simultaneous requests occur. So php-fpm sees 256 workers busy, 0 idle. The minimum idle is 256, so it attempts to start 256 additional processes."

Incomplete test cases - compairing apples and orranges

Hi,
I've worked through some of the tests and noticed that certain framework tests don't implement all tests.
I see a issue with that in the sense that it removes parts of the programs logic and simplifies the case of handling.

Taking the netty test (which did best) as an example I don't see any logic regarding the path in the source which seems a bit like cheating to me.

I'd suggest not to rank frameworks with an incomplete implementation or at least force them to contain the same logic in regards to routing as other tests do.

Cheers,
Heinz

Change config to apc.stat=0 in php.ini

Changing apc.stat to 0 will prevent apc from checking the mtime of files, improving performance. I have not made this change since you will also need to change your setup to call apc_clear_cache() or restart the server when you deploy changes, or stale versions of files will be used

About PHP

One thing you have to understand about PHP is nobody except the shared webhosts use the technologies you're using.

First of all, any serious PHP app should be using APC by default. It's the nature of how PHP works by default vs other language deployments. Second, use php-fpm + nginx web server (or at least worker mpm in Apache 2.4 with php-fpm or fastcgi).

C++: Add Casablanca

https://casablanca.codeplex.com/

PHP is not a Framework or a Platform

This statement is good, but doesn't cover PHP:

We're using the word "framework" loosely to refer to platforms, micro-frameworks, and full-stack frameworks.

PHP is none of those things, it's a language just like Ruby or Python, and it is not a framework for the same reason Ruby, Python, JavaScript or Java are not frameworks.

PHP is not a platform either, and only becomes one platform when you utilise Apache or other web software such as Nginx. As you are testing with apache list "php" as "apache-php", just like you have listed ruby as "rack-ruby".

Beyond that, PHP 5.4 is the current stable, so use that. Its much faster than PHP 5.3. It would be like using Ruby 1.8.x, which is considerably slower than 1.9.x.

Also, nobody in their right minds is going to put their site live without APC or Zend Optimizer, so enable one of those instead.

Finally, React in there for a NodeJS-style setup. The author of this is very adamant it's not a framework, but since this project is already fragrantly disregarding widely accepted definitions of these terms it fits right in with the other systems you've been benchmarking.

compairing 'peak' performance

I see a issue with comparing 'peak' performance especially over a set of different tests (aka different concurrency levels).

This seems a bit odd to me, I'd have expected to look at the lowest number - do a worst case "you'll get at least X" comparison instead of "when you're lucky you might even get X" best case. Especially with concurrency which is likely not to end up in the frameworks sweet spot.

Alternatives would be a average, weighted average or multiple benchmarks at different concurrency levels.

New Test: Spray.io for Scala

http://spray.io/

Benchmark with PyPy

This makes both gunicorn and the json module significantly faster with no other changes, which should also affect overall Python performance significantly, particularly for long-running processes less affected by JIT warmup.

Connection pool should be configurable for all FWs

Some defaults, for some frameworks, perform better on some machines. tricky.

Or when that is too much hassle, it should be configured to the same size on all FWs regardless of the default. Then we at least know the db test outcomes are "at a pool size of n".

Ruby: Add Reel, Webmachine

Hi there! I'm the creator of an actor system + I/O framework + web server for Ruby: Celluloid, Celluloid::IO, and Reel respectively.

It seems like you guys have created a pretty interesting framework for performance benchmarking. My only complaint is that I don't think your existing tests of Ruby projects are covering the state-of-the-art and therefore make Ruby look rather bad.

I've recently done some major bottleneck elimination in Celluloid::IO and am getting rather promising numbers on JRuby, especially when InvokeDynamic is enabled. It'd be great to get my web server Reel included in your tests. Reel is also supported by the Webmachine framework, which would give a better indicator of the speed when utilized with a useful web framework as opposed to just a bare bones web server. Your benchmarks seem to run the entire gauntlet in this regard so I think including both makes sense.

I'm definitely willing to help out here however possible, and just wanted to open a tracking ticket to let you know ;)

New Test: PHP on Kohana

http://kohanaframework.org/

Concurrency levels

Okay I hope I don't start to get annoying ;)

The concurrency levels used for the benchmark seem ridiculous low, if someone has between 8 and 265 concurrent requests on a web application that does virtually no work the framework should not be a concern.

Modern frameworks talk about concurrency levels in the range of tens of thousands if not millions of concurrent requests I'd think something of 10k upwards would be a more interesting and valuable information.

PS: I'll get some test hardware the next days and will try to run this tests under 10k+ conditions and provide some results if interested.

More typical server for CPython

The simple way gunicorn is used right now is more typical of local testing than production. When using CPython in a performance-conscious way, it is customary to serve using uwsgi or if using gunicorn, to have it use an optimized worker like gevent or meinheld. At any rate, these servers are designed to serve from behind something else, e.g. nginx (e.g. uwsgi using uwsgi_pass). This is common knowledge in the Python community.

All of these dependencies can be installed with 'pip install': gunicorn, gevent, uwsgi, meinheld, using CPython 2.7.

New Test: PHP on Laravel

http://laravel.com/

As suggested in issue #1.

New Test: PHP on Symfony

http://symfony.com/

As suggested in issue #1.

Ruby: Add Goliath

http://goliath.io/

Update Infrastructure Versions for PHP and MongoDB

PHP 5.3.x will be entering it's end-of-life cycle this month. The testing infrastructure should at least include a runner for PHP 5.4.x. An additional runner for PHP 5.5.x might also be a good idea to start now, given that it is currently winding it's way through beta.

MongoDB is currently at version 2.4 and should be upgraded as well.

Erlang: Add ChicagoBoss

http://www.chicagoboss.org/

This is in progress.

sudo: no tty present and no askpass program specified

I'm getting an error when trying to run a test with my own machine as both a client and server:

~/FrameworkBenchmarks :master ⚡ ./run-tests.py -u me --max-threads 4 --test flask-python 

=====================================================
  Preparing up Server and Client ...
=====================================================

net.core.somaxconn = 1024
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_tw_recycle = 1
kernel.shmmax = 134217728
kernel.shmall = 2097152
me@localhost's password: 
Welcome to Ubuntu 12.10 (GNU/Linux 3.5.0-26-generic x86_64)

 * Documentation:  https://help.ubuntu.com/

sudo: no tty present and no askpass program specified
sudo: no tty present and no askpass program specified
sudo: no tty present and no askpass program specified
sudo: no tty present and no askpass program specified
sudo: no tty present and no askpass program specified
sudo: no tty present and no askpass program specified

=====================================================
  Parsing Results ...
=====================================================

Time to complete: 6 seconds
Results are saved in results/ec2/20130405200905

How do I prevent this?

More sorting on results

We received a number of requests to allow sorting by latency/stddev/etc on the blog posts.

Erlang: Add Zotonic

http://zotonic.com/

Include latency and maximum concurrent connections benchmarks

In a century of multi-core computers request per second is still relevant, but things like latency and maximum concurrent connections become much more important. Latency can be horrible with still reasonable QPS, but with horrible latency a framework in test won't be usable and acceptable in many-many cases.

Sometimes it is beneficial to sacrifice a bit of sequential speed for much better concurrency. Today's 24+-core machines with huge amounts of RAM can handle millions of concurrent connections, but some frameworks are much better at that then the other. This will need some work on kernel parameters tuning, e.g.:

http://blog.whatsapp.com/index.php/2012/01/1-million-is-so-2011/

New Test: PHP on Fuel

http://fuelphp.com/

PHP: Try PDO::ATTR_EMULATE_PREPARES => false

See if this helps the php-raw test.

New Test: Bottle for Python

http://bottlepy.org/

Adding tests for ServiceStack

I would be interested in adding tests for ServiceStack (.net framework). To run tests under linux, this would require mono for compilation, and either mono_fastcgi on nginx or mod_mono on apache2 to run. Is this something that could be supported by the test framework?

New Test: PHP on Yii Framework

http://www.yiiframework.com/

As suggested in issue #1

PHP: Use React

http://reactphp.org/

As suggested in issue #29.

New Test: TreeFrog Framework

http://www.treefrogframework.org/

Haskell: Add Happstack

http://www.happstack.com/page/view-page-slug/1/happstack

New Test: PHP on Zend Framework

http://framework.zend.com/

As suggested in issue #1

New Test: Wt for C++

http://www.webtoolkit.eu/wt#/

New Test: Warp for Haskell

http://www.haskell.org/haskellwiki/Web/Frameworks

NodeJS is single threaded, Java Servlet is multi-threaded: wrong NodeJS results?

In real, nodeJS apps run with cluster. Cluster helps NodeJS to utilize processor cores.
A Java servlet is typically multithreaded. That means, multiple requests to the same servlet will be executed at the same time.

More information: http://nodejs.org/api/cluster.html

I think, node will produce far better results after clustering.

Upgrade mongoDB to 2.4

As suggested in Issue #5

Benchmarking tool suggestion.

I don't know WeigHTTP enough to make any comparison but I I've been recommanded this one often and it does work quite well:

https://github.com/wg/wrk

Go - Unfair example

Go is a concurrent system programming language the strength is its ability to produce scalable and reliable software. In your example you set the MAXPROCESS config high but don´t use gorotines to make it parallel/concurrent that´s like feature cutting and that´s no scenario where the language can show his nature of working.

New Test: PHP on Slim

http://www.slimframework.com/

Python: Add CherryPy

http://www.cherrypy.org/

Don't use warbler for JRuby

The apps currently use Warbler to turn Ruby apps into WAR files on JRuby. While this is certainly a supported option, it's one of the slowest ways to deploy a JRuby application, due to the additional overhead of converting servlet headers back into byte[]-based strings and other things.

This puts JRuby at a disadvantage.

I would recommend running the tests with a purpose-built Ruby server like Puma or Torquebox. For this particular case, either would do a better job, but Puma's probably the easiest to get up and going.

New Test: Finagle for Scala

http://twitter.github.io/finagle/

PHP: Use MySQLi instead of PDO

Nazariy suggested using MySQLi in issue #1.

Alternative MySQL connectors for Django

knappador suggested the following connectors:

https://pypi.python.org/pypi/django-mysqlpool/0.1-8
http://stackoverflow.com/questions/2636536/how-to-make-django-work-with-unsupported-mysql-drivers-such-as-gevent-mysql-or-c
https://github.com/esnme/ultramysql

New Test: Compojure with clojure/java.jdbc

Instead of Korma. Add to existing Compojure "app" but as a "raw" DB test.

New Test: PHP on Silex

http://silex.sensiolabs.org/

Erlang: Add Nitrogen

http://nitrogenproject.com/