jgpc42/lein-jmh

Leiningen plugin for jmh-clojure

Adding to your project

Add [lein-jmh "0.3.0"] to your :plugins section. For example:

(defproject your-project #_...
  :plugins [[lein-jmh "0.3.0"]])

What is it?

Leiningen plugin for running jmh-clojure benchmarks.

Usage

Run lein help jmh to get started.

As mentioned in the help, by default, a jmh.edn file at the root of your project is used to configure lein-jmh. Please see the sample file for a complete guide. The task takes an optional single argument that gives the task and/or benchmark options. If omitted, all defined benchmarks will be run:

$ lein jmh

Give a map to configure the runner. For example, to run all benchmarks that match a selector from a data file in an alternate location:

$ lein jmh '{:file "benchmarks/parser.edn", :select :decode}'

Additionally, the available JMH profilers may be listed with: lein jmh :profilers.

A more involved example can be found here.

More information

This plugin is a very thin wrapper for the jmh-clojure-task library. Please see the documentation there for full usage instructions and extended examples.

Note about tiered compilation

The JVM option -XX:TieredStopAtLevel=1 is normally set automatically by Leiningen when running code in your project. This option speeds up JVM startup time but is normally problematic for benchmarking as it disables the C2 compiler.

Since lein-jmh merges the :jmh profile automatically when running benchmarks, adding the following to your project's :profiles key should be sufficient for most users:

:profiles {:jmh {:jvm-opts []}}

Alternatively, use :fork and specify different :jvm :args to override the Leiningen parent process arguments. This can be specified in your jmh.edn file, or globally via the task options map.

Running the tests

lein test

Or, lein test-all for all supported Clojure versions.

License

Copyright © 2017-2024 Justin Conklin

Distributed under the Eclipse Public License, the same as Clojure.

lein-jmh's Issues

‘EOF while reading’ when passing no arguments to jmh task

All invocations of this task using just lein jmh are currently broken,
because of the default argument "" that is passed to
leiningen.jmh/jmh. "" cannot be read with read-string.

Try lein jmh in lein-jmh’s own sample project for repro.

Variance?

Is it possible to print the (sample) variance for a benchmark? I am trying to compute a confidence interval for the ratio of the means of two experiments based on the "Rigorous Benchmarking in Reasonable Time" paper.

Better support for tools.cli

Using JMH in a managed manner with CLI tools has more friction than using the lein plugin.
Would supporting a tool API help with that?

I would not mind working on it if you're interested

Thanks

Clarify format of jmh.edn file

The file extension of the jmh.edn file would point to it being edn,
but it doesn’t seem to be edn. For example, the sample file is clearly
not valid edn, as it has an anonymous function literal and a symbol that
resolves to a static method of a Java class (Thread/sleep).
edn is a data format that isn’t
tied to a particular language (I’m sure you know this).

As far as I understand, currently jmh.edn may contain arbitrary
Clojure code but must evaluate to a map. I’m not sure if the file should
be called jmh.edn then; certain expectations come with the extension:
purely declarative, no Clojure constructs.

Me, I would welcome a pure edn benchmark file.

Possible changes: call the default file jmh.clj; actually make it an
edn file (perhaps with the proviso that unqualified symbols will be in
clojure.core); document clearly what the contents of the file may or
may not be.

Option for printing results in an easy to read, tabular format

This is a feature request for your consideration.

lein-jmh does a very fine job as a thin wrapper around jmh-clojure, but
I believe the default data-oriented presentation of the benchmark
results is not easy to use for a human reader.

JMH itself presents benchmark results in a table-like format, where
differences between benchmarks really can be seen at a glance.

Please consider providing an option for outputting results in a similar
human-readable at-a-glance format. Thank you!

Lein jmh doesn't use source-paths when loading code

Hi there,

It seems like this plugin isn't using the :source-paths value from the project.clj file.

I've run into an issue with one of my projects where I can't access any functions from my namespaces in the jmh.edn file. Whenever I add a function from my a namespace in my project, lein jmh fails with:

java.lang.RuntimeException: value did not resolve to a fn: jmh-test-more-src.more-core/more-gen-fn

The project uses a maven style layout, where I've provided source-paths. (and java-source-paths but that doesn't seem related based on my repro below)

I've got a repo with a minimal reproduction here: https://github.com/lfn3/lein-jmh-source-path-issue-repro