softprops/cappi

the sweetest sbt plugin your microbenchmarks will ever meet

cappi

The sweetest sbt plugin your microbenchmarks will ever meet.

Cappi runs Caliper benchmarks

Install

This project targets sbt version 0.13. If you should want 0.12 support drop me an issue

To install Cappi in your sbt project, add the following to your project's plugin configuration, typically in a project/plugins.sbt file.

resolvers += Resolver.url(
  "bintray-sbt-plugin-releases",
    url("http://dl.bintray.com/content/sbt/sbt-plugin-releases"))(
      Resolver.ivyStylePatterns)

addSbtPlugin("me.lessis" % "cappi" % "0.1.1")

Then mix cappiSettings into your build defintion, typically in a build.sbt file.

seq(cappiSettings:_*)

Usage

Cappi will run your benchmarks, but it needs to know what they are first. By default, Cappi defines two tasks to run your bench marks benchmark and benchmarkOnly both scoped to the cappi task.

Resolving Benchmarks

In order to run benchmarks Cappi makes the assumption that you've named your benchmark in a source file with a name ending with "Benchmark.scala". Cappi will look for this under your src/test/scala directory. It makes the assumption that you are packaging your benchmarks in folders that mirror the scala package they are under.

For example, the class foo.BarBenchmark

package foo
class BarBenchmark extends com.google.caliper.SimpleBenchmark {
 // ...
}

would be expected to be located in a file named src/test/scala/foo/BarBenchmark.scala

You can see the full list of resolved benchmark class names by running the following the the sbt REPL

show cappi::benchmarks

Alternatively, you can override the benchmarks setting in your build definition

(cappi.Keys.benchmarks in cappi.Keys.cappi) := Seq("foo.BazBenchmark")

Running Benchmarks

To run your benchmarks, simply run the following in the sbt REPL. This will run all benchmarks.

cappi::benchmark

Alternatively you can run just a target benchmark with benchmarkOnly providing a fully qualified Benchmark class name.

cappi::benchmarkOnly com.you.YourBenchmark

Caliper on your classpath

To run benchmarks, you need to first write benchmarks. To write benchmarks you need caliper on your classpath. To remove the need for you doing this yourself, Cappi will add caliper to your test classpath. This make getting started a much smoother process.

To override the calpier version in use, "0.5-rc1" by default, override the caliperVersion setting.

caliperVersion in cappi := Some("custom-version")

If you do not wish to have Cappi add this to your test classpath for some reason, say your benchmarks are not actually under your test class path. Set the caliperVersion setting to None

caliperVersion in cappi := None

props

This plugin continues on the path that this plugin started.

Doug Tangren (softprops) 2013

cappi's Issues

Enumerations support

I'd like to generate arrays according to distribution like this.

Any sugar to do it with cappi?

add support for sbt 0.12

This would be good if possible...

Casing is incorrect on show cappi::benchmarks

I have a class called HllBenchmark, but it gets listed as HLLBenchmark. If I copy/paste that, I get a confusing error.

Not really sure how the casing is getting changed.

cappi does not allow to specify parameters to Caliper

Hi,
I need to execute Caliper with some extra options (in my case, I need to use -J). A cappi::caliperOptions setting key would be nice. Is it possible to add such a feature?

Upgrade to caliper 1.0-beta-1

Thanks for a great plugin!

Exception caused by: com.google.caliper.ConfigurationException: No benchmarks defined

Having

package mypack.math

class MedianBenchmark extends com.google.caliper.SimpleBenchmark {
  def increment(reps: Int): Unit =
    for (i <- 0 to reps)
      (1 to 1000).sorted.apply(500)
}

and while running cappi::benchmark causes error:

> cappi::benchmark
[info] Running com.google.caliper.Runner mypack.math.MedianBenchmark
[error] An exception was thrown from the benchmark code.
[error] com.google.caliper.UserException$ExceptionFromUserCodeException: An exception was thrown from the benchmark code.
[error]     at com.google.caliper.ScenarioSelection.prepareSuite(ScenarioSelection.java:142)
[error]     at com.google.caliper.ScenarioSelection.select(ScenarioSelection.java:83)
[error]     at com.google.caliper.Runner.runOutOfProcess(Runner.java:373)
[error]     at com.google.caliper.Runner.run(Runner.java:97)
[error]     at com.google.caliper.Runner.main(Runner.java:423)
[error] Caused by: com.google.caliper.ConfigurationException: No benchmarks defined in mypack.math.MedianBenchmark
[error]     at com.google.caliper.SimpleBenchmark.<init>(SimpleBenchmark.java:54)
[error]     at mypack.math.MedianBenchmark.<init>(MedianBenchmark.scala:3)
[error]     at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
[error]     at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
[error]     at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
[error]     at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
[error]     at com.google.caliper.ScenarioSelection.prepareSuite(ScenarioSelection.java:134)
[error]     ... 4 more
[trace] Stack trace suppressed: run last *:cappi for the full output.
[error] (*:cappi::benchmark) Nonzero exit code returned from runner: 1
[error] Total time: 0 s, completed May 15, 2014 1:22:12 PM

I am using sbt 0.13 and scala 2.11.0.

How to fix it?

cappi does not work with more than one benchmark

Hi,
I am trying to use cappi in my project (jandom-devel/jandom). When there is more than one benchmark, the command cappi::benchmark does not work, since the com.google.caliper.Runner class does not allow more than one benchmark class.

This is the output I get:

> show cappi::benchmarks
[info] ArrayBuffer(it.unich.sci.jandom.JandomBenchmark, it.unich.sci.jandom.BoxDoubleBenchmark)
[success] Total time: 0 s, completed Sep 3, 2013 1:01:01 PM
> cappi::benchmark
[info] Compiling 1 Scala source to /home/amato/git/Jandom/core/target/scala-2.10/classes...
[info] Running com.google.caliper.Runner it.unich.sci.jandom.JandomBenchmark it.unich.sci.jandom.BoxDoubleBenchmark
[error] Error: Multiple benchmark classes specified: [it.unich.sci.jandom.JandomBenchmark, it.unich.sci.jandom.BoxDoubleBenchmark]
[info] 
[info] Usage: Runner [OPTIONS...] <benchmark>
[info] 
[info]   <benchmark>: a benchmark class or suite
[info] 
[info] OPTIONS
[info] 
[info]   -D<param>=<value>: fix a benchmark parameter to a given value.
[info]         Multiple values can be supplied by separating them with the
[info]         delimiter specified in the --delimiter argument.
[info] 
[info]         For example: "-Dfoo=bar,baz,bat"
[info] 
[info]         "benchmark" is a special parameter that can be used to specify
[info]         which benchmark methods to run. For example, if a benchmark has
[info]         the method "timeFoo", it can be run alone by using
[info]         "-Dbenchmark=Foo". "benchmark" also accepts a delimiter
[info]         separated list of methods to run.
[info] 
[info]   -J<param>=<value>: set a JVM argument to the given value.
[info]         Multiple values can be supplied by separating them with the
[info]         delimiter specified in the --delimiter argument.
[info] 
[info]         For example: "-JmemoryMax=-Xmx32M,-Xmx512M"
[info] 
[info]   --delimiter <delimiter>: character or string to use as a delimiter
[info]         for parameter and vm values.
[info]         Default: ","
[info] 
[info]   --warmupMillis <millis>: duration to warmup each benchmark
[info] 
[info]   --runMillis <millis>: duration to execute each benchmark
[info] 
[info]   --captureVmLog: record the VM's just-in-time compiler and GC logs.
[info]         This may slow down or break benchmark display tools.
[info] 
[info]   --measureMemory: measure the number of allocations done and the amount of
[info]         memory used by invocations of the benchmark.
[info]         Default: off
[info] 
[info]   --vm <vm>: executable to test benchmark on. Multiple VMs may be passed
[info]         in as a list separated by the delimiter specified in the
[info]         --delimiter argument.
[info] 
[info]   --timeUnit <unit>: unit of time to use for result. Depends on the units
[info]         defined in the benchmark's getTimeUnitNames() method, if defined.
[info]         Default Options: ns, us, ms, s
[info] 
[info]   --instanceUnit <unit>: unit to use for allocation instances result.
[info]         Depends on the units defined in the benchmark's
[info]         getInstanceUnitNames() method, if defined.
[info]         Default Options: instances, K instances, M instances, B instances
[info] 
[info]   --memoryUnit <unit>: unit to use for allocation memory size result.
[info]         Depends on the units defined in the benchmark's
[info]         getMemoryUnitNames() method, if defined.
[info]         Default Options: B, KB, MB, GB
[info] 
[info]   --saveResults <file/dir>: write results to this file or directory
[info] 
[info]   --printScore: if present, also display an aggregate score for this run,
[info]         where higher is better. This number has no particular meaning,
[info]         but can be compared to scores from other runs that use the exact
[info]         same arguments.
[info] 
[info]   --uploadResults <file/dir>: upload this file or directory of files
[info]         to the web app. This argument ends Caliper early and is thus
[info]         incompatible with all other arguments.
[info] 
[info]   --debug: run without measurement for use with debugger or profiling.
[info] 
[info]   --debug-reps: fixed number of reps to run with --debug.
[info]         Default: "1000"
[trace] Stack trace suppressed: run last JandomExtended/*:cappi for the full output.
[error] (JandomExtended/*:cappi::benchmark) Nonzero exit code returned from runner: 1
[error] Total time: 3 s, completed Sep 3, 2013 1:03:01 PM