Giter Site home page Giter Site logo

radszy / benchmarks Goto Github PK

View Code? Open in Web Editor NEW

This project forked from kostya/benchmarks

0.0 0.0 0.0 1.38 MB

Some benchmarks of different languages

License: MIT License

C 5.58% Crystal 3.57% D 5.19% Go 2.97% JavaScript 4.18% Nim 3.20% Python 4.17% Ruby 7.88% Scala 3.76% Brainfuck 4.08% C++ 12.14% Rust 6.78% Java 6.89% Julia 2.58% Shell 0.23% C# 5.98% Perl 4.16% Makefile 12.30% Haskell 3.88% Clojure 0.47%

benchmarks's Introduction

Table of Content

Overview

The benchmarks follow the criteria:

  • They are written as the average software developer would write them, i.e.

    • The algorithms are implemented as cited in public sources;
    • The libraries are used as described in the tutorials, documentation and examples;
    • The used data structures are idiomatic.

  • The used algorithms are similar between the languages (as the reference implementations), variants are acceptable if the reference implementation exists.

  • All final binaries are releases (optimized for performance if possible) as debug performance may vary too much depending on the compiler.

My other benchmarks: jit-benchmarks, crystal-benchmarks-game

Measurements

The measured values are:

  • time spent for the benchmark execution (loading required data and code self-testing are not measured);
  • memory consumption of the benchmark process, reported as base + increase, where base is the RSS before the benchmark and increase is the peak increase of the RSS during the benchmark;
  • energy consumption of the CPU (PP0 package) during the benchmark.

All values are presented as: median±median absolute deviation.

UPDATE: 2021-07-06

Test Cases

Brainfuck

Testing brainfuck implementations using two code samples (bench.b and mandel.b). Supports two mode:

  • Verbose (default). Prints the output immediately.
  • Quiet (if QUIET environment variable is set). Accumulates the output using Fletcher-16 checksum, and prints it out after the benchmark.

Brainfuck

bench.b

Language Time, s Memory, MiB Energy, J
C++/g++ 0.921±0.033 1.48±00.00 + 0.00±00.00 16.51±00.81
Racket (Syntax Objects) 1.433±0.053 106.75±00.60 + 0.00±00.00 30.49±02.37
C/gcc 1.848±0.076 0.54±00.01 + 0.00±00.00 39.40±02.59
Rust 1.864±0.062 2.12±00.06 + 0.00±00.00 35.78±02.93
D/ldc2 1.864±0.041 2.97±00.03 + 0.00±00.00 34.74±01.31
Kotlin 1.868±0.048 40.89±00.11 + 0.48±00.10 35.11±02.66
D/gdc 1.943±0.066 6.30±00.05 + 0.00±00.00 41.12±02.40
OCaml 2.140±0.055 2.59±00.04 + 2.51±00.06 48.78±02.79
Go/gccgo 2.175±0.057 20.99±00.34 + 0.00±00.00 39.59±03.11
Go 2.201±0.075 3.45±00.01 + 0.00±00.00 48.46±01.43
Java 2.216±0.047 38.08±00.10 + 0.88±00.14 43.65±02.77
Vala/gcc 2.251±0.068 3.72±00.06 + 0.00±00.00 48.29±01.13
C/clang 2.288±0.040 0.52±00.03 + 0.00±00.00 43.78±03.77
Racket 2.300±0.156 115.73±00.07 + 2.32±00.52 52.15±04.19
Nim/gcc 2.364±0.029 1.78±00.04 + 0.00±00.00 47.10±04.60
V/gcc 2.366±0.027 0.55±00.01 + 0.00±00.00 54.95±01.10
Nim/clang 2.367±0.070 2.32±00.03 + 0.00±00.00 48.55±04.79
C#/.NET Core 2.374±0.107 34.24±00.03 + 0.01±00.00 49.14±04.65
F#/.NET Core 2.403±0.046 36.74±00.05 + 0.30±00.00 42.82±02.23
Crystal 2.491±0.116 3.30±00.05 + 0.00±00.00 51.84±03.87
Vala/clang 2.500±0.048 3.64±00.03 + 0.00±00.00 48.54±02.57
MLton 2.822±0.058 1.42±00.04 + 0.25±00.00 53.42±03.42
Chez Scheme 2.996±0.129 24.88±00.00 + 4.21±00.03 62.23±04.69
Julia 3.264±0.050 197.94±00.14 + 0.75±00.03 54.51±01.22
D/dmd 3.535±0.134 3.67±00.04 + 0.00±00.00 77.38±03.85
Scala 3.599±0.080 55.41±00.10 + 261.15±04.05 77.90±03.26
Node.js 4.208±0.175 31.23±00.07 + 1.33±00.00 82.18±10.81
C#/Mono 4.421±0.031 20.01±00.04 + 0.00±00.00 77.47±03.78
V/clang 4.676±0.107 0.50±00.00 + 0.00±00.00 102.59±03.08
Haskell (MArray) 4.885±0.166 3.72±00.09 + 1.11±00.00 95.27±06.35
Lua/luajit 6.784±0.057 2.88±00.02 + 0.00±00.00 116.90±02.65
Ruby/truffleruby (--jvm) 7.802±0.288 580.68±15.18 + 679.16±31.19 281.34±11.68
Ruby/truffleruby 8.594±0.336 414.21±00.64 + 541.02±07.15 209.20±18.29
Python/pypy 15.296±0.235 65.41±00.09 + 45.49±00.04 301.08±09.78
Haskell 15.741±0.410 3.79±00.06 + 0.82±00.00 372.84±21.06
Lua 57.706±2.164 2.93±00.03 + 0.00±00.00 1175.05±76.72
Ruby (--jit) 61.809±1.173 14.08±00.02 + 0.23±00.00 1150.99±65.44
Ruby 96.539±2.652 13.99±00.05 + 0.00±00.00 1747.94±66.65
Ruby/jruby 107.399±2.175 186.72±01.62 + 130.70±19.19 2264.93±46.81
Elixir 118.727±4.015 56.04±01.20 + 0.00±00.00 2381.04±129.88
Python 227.596±5.242 10.34±00.04 + 0.00±00.00 5115.22±223.46
Tcl (FP) 285.989±6.642 4.25±00.08 + 0.00±00.00 5522.61±280.47
Perl 354.862±6.481 6.51±00.05 + 0.00±00.00 7796.46±338.55
Tcl (OOP) 544.022±11.873 4.25±00.08 + 0.00±00.00 11120.82±814.03

mandel.b

Mandel in Brainfuck

Language Time, s Memory, MiB Energy, J
C/gcc 12.423±0.162 0.50±00.00 + 1.11±00.02 271.35±04.89
Rust 12.586±0.178 2.03±00.06 + 0.25±00.00 284.06±11.12
C++/g++ 12.910±0.304 3.18±00.05 + 0.47±00.00 289.47±07.83
D/gdc 13.800±0.242 6.72±00.05 + 0.52±00.00 281.84±12.34
D/ldc2 13.902±0.141 3.02±00.05 + 0.77±00.00 239.36±05.93
Go 16.117±0.330 3.40±00.08 + 1.27±00.01 339.69±16.91
Kotlin 17.034±0.522 40.80±00.12 + 0.88±00.32 329.00±31.56
V/gcc 17.119±0.737 1.67±00.04 + 1.03±00.00 359.04±34.09
Racket (Syntax Objects) 17.444±0.120 106.37±00.15 + 72.70±00.26 332.19±21.78
C/clang 18.167±0.686 0.54±00.01 + 1.09±00.03 381.14±34.92
C#/.NET Core 19.055±0.224 34.28±00.02 + 0.96±00.00 341.42±04.72
Vala/gcc 19.880±0.559 3.52±00.04 + 2.06±00.02 375.90±18.11
Vala/clang 22.178±0.405 3.54±00.03 + 2.06±00.03 424.87±38.99
Nim/clang 22.230±0.709 2.31±00.05 + 0.51±00.00 449.93±38.24
Nim/gcc 23.202±0.447 1.81±00.06 + 0.51±00.00 414.37±29.55
Java 23.358±0.630 38.01±00.15 + 1.42±00.13 438.99±41.36
Go/gccgo 23.445±0.761 21.30±00.07 + 1.28±00.00 443.45±33.96
Crystal 24.551±1.035 3.32±00.01 + 0.46±00.01 518.51±24.86
Scala 25.120±0.293 55.66±00.09 + 141.27±00.33 530.51±11.43
V/clang 26.637±1.047 1.66±00.03 + 1.03±00.00 538.04±45.94
F#/.NET Core 35.454±0.288 36.79±00.07 + 2.03±00.03 634.62±03.81
OCaml 37.562±1.228 3.84±00.01 + 6.69±00.39 775.01±52.89
Racket 38.757±0.747 115.69±00.28 + 2.32±00.26 732.68±17.03
Node.js 39.350±0.187 31.24±00.05 + 5.94±00.01 846.41±04.74
Chez Scheme 40.316±1.217 25.48±00.04 + 3.64±00.00 863.75±64.05
D/dmd 42.491±0.121 3.70±00.05 + 0.77±00.00 963.22±03.28
C#/Mono 44.502±0.272 20.11±00.02 + 0.88±00.00 980.24±07.72
MLton 51.981±1.646 1.41±00.04 + 4.11±00.00 1104.45±82.03
Haskell (MArray) 60.083±0.844 3.78±00.10 + 2.50±00.13 1379.17±42.02
Python/pypy 64.354±0.552 65.38±00.11 + 46.41±00.02 1229.14±09.32
Julia 78.738±1.108 198.50±00.04 + 0.78±00.05 1455.76±70.63
Ruby/truffleruby (--jvm) 134.368±3.421 584.53±06.57 + 745.92±114.67 2794.06±50.97
Ruby/truffleruby 150.668±4.223 414.64±00.37 + 585.68±11.10 2907.83±122.54
Haskell 226.291±4.319 4.04±00.10 + 25.91±00.00 4356.11±160.91
Lua/luajit 247.906±5.559 2.87±00.03 + 0.86±00.00 4766.50±239.31

Base64

Testing base64 encoding/decoding of the large blob into the newly allocated buffers.

Base64

Language Time, s Memory, MiB Energy, J
C/gcc (aklomp) 0.155±0.004 1.90±00.03 + 0.00±00.00 3.70±00.39
Rust 1.186±0.015 2.49±00.10 + 0.01±00.00 21.80±01.84
C/gcc 1.218±0.036 1.85±00.00 + 0.00±00.00 23.94±01.91
Nim/gcc 1.254±0.014 2.17±00.05 + 4.44±00.00 28.92±01.10
Crystal 1.612±0.015 3.72±00.03 + 1.83±00.04 39.30±00.13
Nim/clang 1.634±0.061 2.67±00.07 + 4.38±00.00 36.75±01.05
D/ldc2 1.967±0.032 3.41±00.03 + 3.66±00.00 35.17±00.62
V/gcc 1.977±0.035 2.12±00.05 + 0.00±00.00 34.56±02.67
Ruby 2.019±0.072 14.45±00.03 + 56.75±00.53 46.19±02.13
Ruby (--jit) 2.132±0.068 14.43±00.04 + 57.92±00.42 41.36±02.27
Vala/clang 2.142±0.078 0.00±00.00 + 0.00±00.00 46.28±02.00
D/gdc 2.175±0.121 7.06±00.04 + 3.46±00.00 44.27±06.07
Java 2.202±0.066 38.90±00.21 + 280.75±09.17 49.87±01.20
Vala/gcc 2.224±0.030 0.00±00.00 + 0.00±00.00 42.00±03.33
Scala 2.403±0.086 53.57±00.13 + 312.97±21.36 52.53±02.64
Kotlin 2.406±0.019 40.63±00.16 + 327.63±29.57 48.30±01.93
C++/g++ (libcrypto) 2.408±0.048 5.39±00.03 + 0.07±00.00 56.94±01.66
Go 2.558±0.017 4.56±00.04 + 5.61±00.26 49.88±00.83
V/clang 2.782±0.142 1.82±00.26 + 0.13±00.13 55.12±04.19
Node.js 2.792±0.046 31.72±00.05 + 36.18±00.07 58.48±03.21
Perl (MIME::Base64) 2.901±0.089 14.12±00.06 + 0.08±00.06 62.70±02.33
PHP 3.062±0.142 15.76±00.11 + 0.00±00.00 64.64±06.94
Go/gccgo 3.514±0.009 22.45±00.34 + 7.19±00.29 74.67±01.32
D/dmd 3.825±0.098 3.80±00.02 + 3.69±00.03 72.83±04.90
Python 4.112±0.145 10.09±00.02 + 0.18±00.00 81.15±07.44
Tcl 4.896±0.097 4.83±00.04 + 0.19±00.01 87.33±03.34
Python/pypy 5.049±0.052 65.29±00.02 + 45.68±00.04 97.29±08.84
F#/.NET Core 5.668±0.090 37.03±00.03 + 37.94±05.46 107.68±02.72
C#/.NET Core 5.707±0.069 34.48±00.09 + 39.49±06.99 112.02±02.38
Julia 6.078±0.079 216.92±00.09 + 52.82±10.27 120.10±06.44
Ruby/truffleruby (--jvm) 6.928±0.193 568.10±10.90 + 364.17±12.04 138.24±11.06
C#/Mono 7.404±0.387 20.56±00.04 + 18.48±00.02 174.66±04.52
Ruby/jruby 12.104±0.464 188.35±04.81 + 134.19±05.02 247.75±11.06
Perl (MIME::Base64::Perl) 16.679±0.661 15.40±00.10 + 0.22±00.06 363.19±30.65
Ruby/truffleruby 23.147±0.630 405.72±01.19 + 305.43±02.85 441.44±36.07

Json

Testing parsing and simple calculating of values from a big JSON file.

Few notes:

Json

Language Time, s Memory, MiB Energy, J
C++/g++ (DAW JSON Link NoCheck) 0.088±0.001 113.03±00.02 + 0.00±00.00 1.92±00.02
C++/g++ (simdjson On-Demand) 0.096±0.001 113.21±00.05 + 59.81±00.00 2.07±00.08
C++/g++ (DAW JSON Link) 0.124±0.002 112.80±00.20 + 0.00±00.00 2.56±00.05
Rust (Serde Custom) 0.137±0.003 111.86±00.08 + 0.00±00.00 3.15±00.18
Rust (Serde Typed) 0.151±0.004 111.87±00.07 + 11.70±00.19 2.73±00.18
C++/g++ (gason) 0.158±0.002 113.00±00.04 + 96.74±00.06 4.07±00.20
C++/g++ (simdjson DOM) 0.172±0.003 113.28±00.03 + 176.60±00.00 3.68±00.16
C++/g++ (RapidJSON) 0.227±0.006 113.03±00.03 + 128.77±00.06 5.02±00.42
D/ldc2 (Mir Amazon's Ion DOM) 0.229±0.006 112.70±00.04 + 16.26±00.00 4.97±00.44
D/ldc2 (Mir Asdf DOM) 0.243±0.004 112.67±00.07 + 61.34±00.01 5.26±00.16
C++/g++ (RapidJSON Precise) 0.290±0.009 113.07±00.02 + 128.71±00.05 6.29±00.56
C++/g++ (Boost.JSON) 0.529±0.020 113.13±00.01 + 435.76±00.06 11.13±00.67
C++/g++ (RapidJSON SAX) 0.589±0.009 112.79±00.02 + 0.00±00.00 10.30±00.40
C++/g++ (RapidJSON SAX Precise) 0.638±0.025 112.82±00.03 + 0.00±00.00 13.41±01.12
Go (jsoniter) 0.695±0.023 231.35±00.02 + 13.59±00.28 14.49±01.06
Crystal (Schema) 0.733±0.011 113.73±00.03 + 42.72±00.03 15.30±00.17
Crystal (Pull) 0.764±0.019 113.71±00.04 + 18.18±00.03 15.22±01.41
Java (DSL-JSON) 0.803±0.025 264.04±00.14 + 311.38±02.85 20.76±00.82
V/clang 0.847±0.020 111.15±00.01 + 496.21±00.06 19.21±01.32
Rust (Serde Untyped) 0.854±0.029 111.89±00.08 + 839.98±00.00 17.32±01.54
Python/pypy 0.862±0.012 285.48±00.09 + 121.42±00.01 16.56±00.28
Node.js 0.877±0.025 251.02±00.03 + 78.29±00.74 23.02±00.55
Julia (JSON3) 0.960±0.024 368.78±00.22 + 373.25±00.20 18.56±00.83
V/gcc 0.969±0.043 111.14±00.01 + 496.21±00.00 20.08±02.07
Perl (Cpanel::JSON::XS) 0.986±0.010 124.67±00.04 + 402.78±00.03 23.49±00.48
Crystal 1.030±0.023 113.72±00.02 + 396.45±00.01 22.79±00.49
C#/.NET Core (System.Text.Json) 1.060±0.043 479.22±00.09 + 138.78±00.06 22.38±02.34
Go 1.097±0.036 117.39±00.05 + 83.29±00.11 24.76±01.61
Nim/clang (Packedjson) 1.103±0.041 112.47±00.10 + 294.04±00.13 25.56±01.59
PHP 1.277±0.046 124.99±00.14 + 682.01±00.00 25.92±02.27
Nim/gcc (Packedjson) 1.380±0.057 112.05±00.11 + 293.91±00.00 28.11±02.64
Go/gccgo 1.450±0.030 142.23±06.08 + 96.03±00.17 33.10±01.45
C++/g++ (json-c) 1.532±0.030 112.93±00.19 + 1216.23±00.11 36.83±01.14
Clojure 1.609±0.044 473.39±03.17 + 576.83±05.74 46.18±01.78
Nim/clang 1.610±0.055 112.41±00.08 + 925.03±00.06 37.65±01.70
C#/.NET Core 1.730±0.058 487.98±00.04 + 294.07±00.00 37.47±04.26
CPython (UltraJSON) 1.780±0.023 121.82±00.03 + 549.14±01.04 41.90±00.80
C++/g++ (Nlohmann) 1.809±0.011 113.15±00.02 + 447.88±00.00 34.77±01.10
Nim/gcc 1.844±0.070 112.11±00.04 + 923.29±00.06 37.81±03.36
Python 1.848±0.014 120.09±00.03 + 377.21±00.00 38.12±01.17
C#/Mono 2.236±0.089 476.19±00.08 + 0.00±00.00 47.40±05.36
D/gdc 2.283±0.028 116.61±00.04 + 600.39±00.00 53.37±00.83
Haskell 2.291±0.051 4.44±00.05 + 4.68±00.03 43.10±02.05
Ruby 2.426±0.058 124.00±00.02 + 410.69±00.01 57.63±01.08
Ruby (YAJL) 2.443±0.047 123.96±00.02 + 283.34±00.00 44.94±01.47
Scala (uPickle) 2.516±0.053 284.73±00.09 + 762.67±23.36 59.39±04.12
F#/.NET Core (System.Text.Json) 2.577±0.083 485.37±00.05 + 453.88±02.45 55.03±05.61
D/ldc2 2.585±0.057 112.92±00.06 + 680.09±00.04 53.34±01.70
Ruby (--jit) 2.589±0.022 124.01±00.06 + 410.80±00.04 50.43±00.51
Rust (jq) 3.584±0.035 113.83±00.02 + 777.86±00.90 81.39±00.50
Ruby/jruby 3.876±0.084 453.42±10.48 + 1412.22±28.09 108.53±06.77
C++/g++ (Boost.PropertyTree) 4.483±0.118 112.95±00.04 + 1440.06±00.00 101.16±06.49
D/dmd 5.098±0.075 113.65±00.06 + 680.25±00.01 102.46±04.22
Vala/gcc 5.473±0.217 0.00±00.00 + 0.00±00.00 99.96±06.82
Vala/clang 5.602±0.056 0.00±00.00 + 0.00±00.00 101.12±00.60
Perl (JSON::Tiny) 12.605±0.279 125.30±00.07 + 528.63±00.10 231.12±05.85
Ruby/truffleruby (--jvm) 16.962±0.464 721.72±12.93 + 1648.53±20.81 479.86±14.66
Ruby/truffleruby 40.369±0.806 738.03±01.52 + 2269.36±70.04 834.73±65.49

Matmul

Testing allocating and multiplying matrices.

Matmul

Language Time, s Memory, MiB Energy, J
D/ldc2 (lubeck) 0.081±0.002 6.70±00.06 + 55.90±00.12 4.58±00.10
Python (NumPy) 0.107±0.003 27.62±00.10 + 57.64±00.06 6.15±00.19
Nim/gcc (Arraymancer) 0.158±0.029 5.58±00.10 + 57.28±00.12 8.16±00.82
Nim/clang (Arraymancer) 0.189±0.019 6.44±00.14 + 57.38±00.07 9.07±00.70
C++/g++ (Eigen) 0.203±0.007 3.60±00.08 + 85.24±00.01 4.59±00.22
Julia (threads: 8) 0.207±0.004 251.36±00.42 + 53.11±00.12 11.03±00.42
Java (ND4J) 0.232±0.023 145.25±01.16 + 87.50±00.13 10.77±00.94
Julia (threads: 1) 0.552±0.013 251.36±00.14 + 53.60±00.09 12.89±00.44
Julia (no BLAS) 1.220±0.018 216.72±00.33 + 51.63±00.00 30.49±00.74
D/ldc2 1.991±0.007 3.52±00.12 + 70.11±00.13 44.85±00.56
D/dmd 2.135±0.024 3.59±00.10 + 70.12±00.01 48.46±00.73
D/gdc 2.149±0.057 6.63±00.06 + 70.71±00.02 48.48±03.20
Java 3.244±0.025 38.55±00.15 + 80.66±00.55 76.29±01.27
Vala/gcc 3.271±0.021 0.00±00.00 + 0.00±00.00 79.07±01.08
C/gcc 3.357±0.055 2.01±00.04 + 68.06±00.00 77.55±01.22
Nim/gcc 3.372±0.017 2.56±00.05 + 65.48±00.00 81.10±03.55
Vala/clang 3.426±0.024 0.00±00.00 + 0.00±00.00 74.62±00.92
Rust 3.445±0.016 2.71±00.12 + 68.32±00.00 74.65±00.66
Scala 3.449±0.027 55.68±00.06 + 153.92±00.12 82.04±01.31
Nim/clang 3.469±0.050 3.07±00.02 + 66.00±00.00 74.45±00.98
Go 3.543±0.023 3.80±00.06 + 73.36±00.09 80.52±03.47
Go/gccgo 3.564±0.086 21.59±00.18 + 72.83±00.16 78.12±02.04
Swift 3.646±0.088 145.20±00.05 + 60.64±00.03 86.24±03.09
Crystal 3.728±0.104 4.15±00.12 + 59.64±00.11 79.90±06.27
V/gcc 3.757±0.127 1.87±00.05 + 69.35±00.00 80.73±07.42
Kotlin 3.800±0.048 38.80±00.22 + 80.71±00.38 88.40±02.97
Node.js 4.925±0.088 35.33±00.09 + 71.08±00.44 89.30±04.41
Python/pypy 6.541±0.073 65.85±00.13 + 69.18±00.04 114.00±03.02
C#/.NET Core 7.373±0.141 33.97±00.11 + 69.11±00.00 133.57±05.53
C#/Mono 10.775±0.202 20.06±00.06 + 69.06±00.02 240.98±06.98
V/clang 14.013±0.187 2.20±00.04 + 69.35±00.00 237.08±15.02
Ruby/truffleruby 46.906±0.541 573.59±00.56 + 684.43±03.60 1118.10±14.38
Ruby/truffleruby (--jvm) 68.521±0.596 650.54±12.48 + 693.22±91.72 1633.85±53.37
Ruby 220.767±2.987 15.22±00.05 + 68.61±00.03 4856.07±55.59
Ruby (--jit) 225.230±3.868 15.20±00.02 + 68.91±00.03 4870.24±53.87
Python 241.153±8.661 10.53±00.02 + 68.58±00.00 4955.07±173.31
Tcl 358.681±3.162 7.26±00.03 + 400.44±00.00 6919.11±392.92
Perl 405.584±6.809 9.00±00.06 + 599.62±00.04 9364.17±679.44
Ruby/jruby 500.132±7.243 263.98±04.55 + 1004.45±37.01 10848.95±221.31

Primes

Testing:

  • generating primes using the optimized sieve of Atkin;
  • prefix search for their decimal numbers using Trie data structure.

Primes

Language Time, s Memory, MiB Energy, J
Zig 0.074±0.001 1.75±00.01 + 52.99±00.76 1.58±00.07
Crystal 0.161±0.002 3.29±00.02 + 88.38±02.07 3.48±00.09
Java 0.178±0.003 36.80±00.49 + 105.34±01.04 5.49±00.27
C++/g++ 0.187±0.006 3.18±00.03 + 84.79±00.00 3.38±00.18
Rust 0.215±0.006 2.30±00.10 + 72.88±00.01 4.14±00.08
Node.js 0.285±0.002 31.04±00.08 + 175.48±00.32 7.67±00.26
Scala 0.441±0.011 55.58±00.07 + 244.52±01.62 14.76±00.36
Python/pypy 0.993±0.011 64.87±00.14 + 248.92±00.11 20.48±00.50
Ruby/truffleruby 1.306±0.039 400.03±02.35 + 442.08±02.63 32.38±02.15
Ruby/truffleruby (--jvm) 1.656±0.032 566.38±22.45 + 345.39±14.33 59.10±01.74
Ruby (--jit) 2.421±0.048 14.05±00.03 + 150.22±00.01 61.83±02.87
Ruby 2.617±0.091 13.98±00.03 + 149.89±00.00 48.52±04.63
Ruby/jruby 3.064±0.072 187.46±03.86 + 345.62±19.78 71.19±01.85
Python 5.844±0.168 10.11±00.04 + 235.83±00.77 100.59±05.67

Tests Execution

Environment

CPU: Intel(R) Core(TM) i7-10710U

Base Docker image: Debian GNU/Linux bullseye/sid

Language Version
.NET Core 5.0.204
C#/.NET Core 3.9.0-6.21160.10 (59eedc33)
C#/Mono 6.12.0.122
C/clang 11.0.1
C/gcc 10.2.1
Chez Scheme 9.5.4
Clojure "1.10.3"
Crystal 1.0.0
D/dmd v2.097.0
D/gdc 10.2.1
D/ldc2 1.26.0
Elixir 1.10.3
F#/.NET Core 11.3.2.0 for F# 5.0
Go go1.16.5
Go/gccgo 10.2.1
Haskell 9.0.1
Java 16.0.1
Julia v"1.6.1"
Kotlin 1.5.20
Lua Lua 5.4
Lua/luajit LuaJIT 2.1.0-beta3
MLton 20210117
Nim 1.4.8
Node.js v16.4.0
OCaml 4.11.1
PHP 7.4.15
Perl v5.32.1
Python 3.9.2
Python/pypy 7.3.5-final0 for Python 3.7.10
Racket "8.1"
Ruby 3.0.1p64
Ruby/jruby 9.2.19.0
Ruby/truffleruby 21.1.0
Rust 1.53.0
Scala 3.0.0
Swift swift-5.4.2-RELEASE
Tcl 8.6
V 0.2.2 0ac0ab6
Vala 0.48.17
Zig 0.8.0

Using Docker

Build the image:

$ docker build docker/ -t benchmarks

Run the image:

$ docker run -it --rm -v $(pwd):/src benchmarks <cmd>

where is:

  • versions (print installed language versions);
  • shell (start the shell);
  • brainfuck bench (build and run Brainfuck bench.b benchmarks);
  • brainfuck mandel (build and run Brainfuck mandel.b benchmarks);
  • base64 (build and run Base64 benchmarks);
  • json (build and run Json benchmarks);
  • matmul (build and run Matmul benchmarks);
  • primes (build and run Primes benchmarks);

Please note that the actual measurements provided in the project are taken semi-manually (via shell) as the full update takes days and could have occassional issues in Docker.

There is a Makefile that could be used to simplify Docker usage:

  • make build (build the image);
  • make versions (run the image with the versions command);
  • make shell (run the image with the `shell' command);
  • make toc (utility rule to update ToC in this README, requires git-markdown-toc available in PATH).

Please note that the make shell rule requires cpupower utility installed that is invoked with sudo to set cpufreq's performance governon (it runs the CPU at the maximum frequence to eliminate throttling issues).

Manual Execution

Makefiles contain recipes for building and executing tests with the proper dependencies. Please use make run (and make run2 where applicable). The measurements are taken using analyze.rb script:

$ cd <test suite>
$ ../analyze.rb make run
$ ../analyze.rb make run[<single test>]

Please note that the measurements could take hours. It uses 10 iterations by default, but it could be changed using ATTEMPTS environment variable:

$ ATTEMPTS=1 ../analyze.rb make run

Prerequisites

Please use Dockerfile as a reference regarding which packages and tools are required.

For all (optional):

  • Powercap for reading energy counters in Linux (Debian package powercap-utils).

For Python:

  • NumPy for matmul tests (Debian package python3-numpy).
  • UltraJSON for JSON tests (Debian package python3-ujson).

For C++:

  • Boost for JSON tests (Debian package libboost-dev).
  • JSON-C for JSON tests (Debian package libjson-c-dev).

For Rust:

  • libjq for jq test (Debian packages libjq-dev, libonig-dev and environment variable JQ_LIB_DIR=/usr/lib/x86_64-linux-gnu/).

For Java, Scala:

  • Coursier for downloading Maven artifacts.

For Lua:

  • LuaRocks for installing dependencies (Debian package luarocks).

For Haskell:

  • network for TCP connectivity between the tests and the test runner.
  • raw-strings-qq for raw string literals used in tests.

For Perl:

  • cpanminus for installing modules from CPAN (Debian package cpanminus).

For Vala:

  • JSON-GLib for JSON tests (Debian package libjson-glib-dev).

Contribution

Please follow the criteria specified in the overview. Besides that please ensure that the communication protocol between a test and the test runner is satisfied:

  • The test runner listens on localhost:9001;
  • All messages are sent using TCP sockets closed immediately after the message has been sent;
  • There are two messages sent from a test (it establishes the measurement boundary):
    1. The beginning message having the format name of the test/tprocess ID (the process ID is used to measure the memory consumption). Please note that the name of the test couldn't use Tab character as it's a delimiter;
    2. The end message with any content (mostly it's "stop" for consistency).
  • The test runner could be unavailable (if the test is launched as is) and the test should gracefully handle it.

Makefile guide

Binary executables

If the test is compiled into a single binary, then two sections of the Makefile require changes:

  • append a new target (the final binary location) into executables variable;
  • append the proper target rule.

Compiled artifacts

If the test is compiled, but can't be executed directly as a binary, then three sections of the Makefile require changes:

  • append a new target (the final artifact location) into artifacts variable;
  • append the proper target rule to compile the test;
  • append run[<target_artifact>] rule to run the test.

Scripting language

If the test doesn't require compilation, then two sections of the Makefile requires changes:

  • append run[<script_file>] into all_runners variable;
  • append run[<script_file>] rule to run the test.

benchmarks's People

Contributors

9il avatar akarin123 avatar beached avatar cmcaine avatar dbohdan avatar dtolnay avatar erickt avatar gavr123456789 avatar goldenreign avatar jackstouffer avatar k-bx avatar kostya avatar lqdc avatar martinnowak avatar miloyip avatar nuald avatar orthoxerox avatar philnguyen avatar pmarcelll avatar proyb6 avatar radszy avatar rap2hpoutre avatar ricvelozo avatar sfesenko avatar simonbyrne avatar snadrus avatar tchaloupka avatar tkelman avatar w-diesel avatar zapov avatar

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.