django/djangobench

Harness and benchmarks for evaluating Django's performance over time

Django

Django is a high-level Python web framework that encourages rapid development and clean, pragmatic design. Thanks for checking it out.

All documentation is in the "docs" directory and online at https://docs.djangoproject.com/en/stable/. If you're just getting started, here's how we recommend you read the docs:

• First, read docs/intro/install.txt for instructions on installing Django.
• Next, work through the tutorials in order (docs/intro/tutorial01.txt, docs/intro/tutorial02.txt, etc.).
• If you want to set up an actual deployment server, read docs/howto/deployment/index.txt for instructions.
• You'll probably want to read through the topical guides (in docs/topics) next; from there you can jump to the HOWTOs (in docs/howto) for specific problems, and check out the reference (docs/ref) for gory details.
• See docs/README for instructions on building an HTML version of the docs.

Docs are updated rigorously. If you find any problems in the docs, or think they should be clarified in any way, please take 30 seconds to fill out a ticket here: https://code.djangoproject.com/newticket

To get more help:

• Join the #django channel on irc.libera.chat. Lots of helpful people hang out there. Webchat is available.
• Join the django-users mailing list, or read the archives, at https://groups.google.com/group/django-users.
• Join the Django Discord community.
• Join the community on the Django Forum.

To contribute to Django:

• Check out https://docs.djangoproject.com/en/dev/internals/contributing/ for information about getting involved.

To run Django's test suite:

• Follow the instructions in the "Unit tests" section of docs/internals/contributing/writing-code/unit-tests.txt, published online at https://docs.djangoproject.com/en/dev/internals/contributing/writing-code/unit-tests/#running-the-unit-tests

Supporting the Development of Django

Django's development depends on your contributions.

If you depend on Django, remember to support the Django Software Foundation: https://www.djangoproject.com/fundraising/

djangobench's Issues

Add benchmarks for migration

@MarkusH already did some work in this direction.

The sample applications were generated by this script.

Tracker App

Hi @adamchainz,

So I've made a bit more progress and I've now made a "thing", which I thought it worth sharing at this stage. I've thoroughly enjoyed this little project so far, so even if not useful (although I hope it will be), the lessons learned here are valuable for me personally.

Anyway, I've set up a GitHub action to run djangobench and to publish the outputs in a datasette. I've currently got it running benchmarks at the rate of one commit per hour, to build up a bit of history.

https://djangobench-tracker.herokuapp.com/django/bench

The next step is to have a think about how to visualise this data. There are some charting options in the datasette already, but would be good to have a more dashboard approach... Something like this maybe?

https://speed.python.org/comparison/

(Sidenote, interesting that django templates are much faster in 3.7 and 3.8 than 3.6, with 3.9 being faster again. )

template_render benchmark failing on 1.5alpha

This works:
djangobench --vcs=git --control=1.3 --experiment=1.4 template_render

This Doesn't
djangobench --vcs=git --control=1.4 --experiment=master template_render

(django-dev)element:django (master)$ djangobench --vcs=git --control=1.4 --experiment=master template_render
Running benchmarks: template_render
Control: Django 1.4 (in git branch 1.4)
Experiment: Django 1.5.dev20120725205848 (in git branch master)

Running 'template_render' benchmark ...
Traceback (most recent call last):
  File "/Users/preston/Projects/Python/virtualenvs/django-dev/bin/djangobench", line 9, in <module>
    load_entry_point('djangobench==0.9', 'console_scripts', 'djangobench')()
  File "/Users/preston/Projects/code/forks/djangobench/djangobench/main.py", line 311, in main
    continue_on_errror = args.continue_on_errror
  File "/Users/preston/Projects/code/forks/djangobench/djangobench/main.py", line 62, in run_benchmarks
    experiment_data = run_benchmark(benchmark, trials, experiment_env)
  File "/Users/preston/Projects/code/forks/djangobench/djangobench/main.py", line 111, in run_benchmark
    out, _, _ = perf.CallAndCaptureOutput(command + ['-t', 1], env, track_memory=False, inherit_env=[])
  File "/Users/preston/Projects/code/forks/djangobench/djangobench/perf.py", line 1026, in CallAndCaptureOutput
    raise RuntimeError("Benchmark died: " + stderr)
RuntimeError: Benchmark died: Traceback (most recent call last):
  File "/Users/preston/Projects/code/forks/djangobench/djangobench/benchmarks/template_render/benchmark.py", line 38, in <module>
    'description': ('Render a somewhat complex, fairly typical template '
  File "/Users/preston/Projects/code/forks/djangobench/djangobench/utils.py", line 71, in run_benchmark
    benchmark_result = benchmark()
  File "/Users/preston/Projects/code/forks/djangobench/djangobench/benchmarks/template_render/benchmark.py", line 32, in benchmark
    render_to_response('permalink.html', context)
  File "/Users/preston/Projects/code/forks/django/django/shortcuts/__init__.py", line 20, in render_to_response
    return HttpResponse(loader.render_to_string(*args, **kwargs), **httpresponse_kwargs)
  File "/Users/preston/Projects/code/forks/django/django/template/loader.py", line 172, in render_to_string
    return t.render(Context(dictionary))
  File "/Users/preston/Projects/code/forks/django/django/template/base.py", line 141, in render
    return self._render(context)
  File "/Users/preston/Projects/code/forks/django/django/template/base.py", line 135, in _render
    return self.nodelist.render(context)
  File "/Users/preston/Projects/code/forks/django/django/template/base.py", line 831, in render
    bit = self.render_node(node, context)
  File "/Users/preston/Projects/code/forks/django/django/template/base.py", line 845, in render_node
    return node.render(context)
  File "/Users/preston/Projects/code/forks/django/django/template/loader_tags.py", line 123, in render
    return compiled_parent._render(context)
  File "/Users/preston/Projects/code/forks/django/django/template/base.py", line 135, in _render
    return self.nodelist.render(context)
  File "/Users/preston/Projects/code/forks/django/django/template/base.py", line 831, in render
    bit = self.render_node(node, context)
  File "/Users/preston/Projects/code/forks/django/django/template/base.py", line 845, in render_node
    return node.render(context)
  File "/Users/preston/Projects/code/forks/django/django/template/defaulttags.py", line 366, in render
    return strip_spaces_between_tags(self.nodelist.render(context).strip())
  File "/Users/preston/Projects/code/forks/django/django/template/base.py", line 831, in render
    bit = self.render_node(node, context)
  File "/Users/preston/Projects/code/forks/django/django/template/base.py", line 845, in render_node
    return node.render(context)
  File "/Users/preston/Projects/code/forks/django/django/template/defaulttags.py", line 419, in render
    raise e
django.core.urlresolvers.NoReverseMatch: Reverse for '' with arguments '()' and keyword arguments '{}' not found.

(django-dev)element:django (master)$ 

Add documentation of all the command line options

Benchmark results

I’d like to see Django performance benchmark results from the past – where can I go to see any results produced with djangobench, or with any other benchmark harness like this?

I’m interested in:

1. Django performance from release to release
2. Django performance across Python versions

pyperformance benchmarks

Hi,

I stumbled across this discussion today and after a bit of digging around I found a PR adding support for user-provided benchmark suites.

Maybe this is something to investigate?

Decide on a version support policy

There is at least once benchmark which checks for compatibility before Django 1.2... It is nice to keep this is as long as possible but with changes to migrations/syncdb, app loading and the startup process it is becoming harder.

I'm not averse to supporting say, all Django >= 1.0, but I feel perhaps 1.4 would be more reasonable at this point.

Python 3 compatibility?

Could be useful to compare revisions of Django >= 1.5 running under python 3.

The port will need to also take in account things like the simplejson dependency.

Installation issues

Contrary to install instructions, I don't seem to be able to install from pip.

Also, installing from the setup.py fails to work correctly - the 'benchmarks' directory is not copied it seems. If I do a "setup.py develop" I can get a working setup.

Fix template_render and url_reverse

These fail on master.

Add GitHub actions checks for flake8 and isort

ValueError: could not convert string to float - "short version" example

When I follow the "short version" example, I get this error:

$ djangobench --control=1.2 --experiment=master
Running all benchmarks
Control: Django 1.2 (in git branch 1.2)
Experiment: Django 1.9.dev20150415021140 (in git branch master)

Running 'default_middleware' benchmark ...
Traceback (most recent call last):
  File "/Users/audreyr/.virtualenvs/experiments/bin/djangobench", line 8, in <module>
    load_entry_point('djangobench==0.10', 'console_scripts', 'djangobench')()
  File "/Users/audreyr/code/third-party/djangobench/djangobench/main.py", line 397, in main
    experiment_python=args.experiment_python,
  File "/Users/audreyr/code/third-party/djangobench/djangobench/main.py", line 73, in run_benchmarks
    env=control_env)
  File "/Users/audreyr/code/third-party/djangobench/djangobench/main.py", line 145, in run_benchmark
    data_points = [float(line) for line in message.get_payload().splitlines()]
ValueError: could not convert string to float: d

Remove initial_data fixtures

We don't automatically load them any more anyway, and they make the benchmarks harder to understand. There are 18 to remove.

Eu

negative time execution for 'default_middleware' benchmark

$ djangobench  --control stable/1.11.x --experiment master default_middleware
Running benchmarks: default_middleware
Control: Django 1.11.6.dev20170906232200 (in git branch stable/1.11.x)
Experiment: Django 2.0.dev20170915140012 (in git branch master)

Running 'default_middleware' benchmark ...
Min: 0.000026 -> -0.000037: -0.7010x faster
Avg: 0.000233 -> 0.000210: 1.1073x faster
Not significant
Stddev: 0.00126 -> 0.00148: 1.1770x larger (N = 50)

Fix the startup benchmark

Should this now test django.setup() performance?