gridgain/yardstick

GridGain Community Edition

GridGain Community Edition (GCE) is a hardened, high performance, open source in-memory computing platform. Built on Apache® Ignite™, it includes additional functionality, tuning and patches developed by GridGain to deliver optimal performance.

Getting Started

For information on how to get started with GridGain CE, please visit: Getting Started.

yardstick's Issues

Not able to trim the results

For any benchmark, while reporting, we generally focus on middle part of load, ignoring start and end as noise. For this we need trim functionality.

[Urgent] Do the number of drivers specified in DRIVER_HOSTS have effect when running manually?

Hi,

When running using manual scripts, does the number of drivers specified in DRIVER_HOSTS run clients accordingly? In other words, when I specify DRIVER_HOSTS=localhost, it should run one driver. When I specify DRIVER_HOSTS=localhost,localhost, it should run two drivers, and so on. Is that right? I feel it always runs 1 driver regardless how many values you specify!!, again when using the manual scripts.

Also, when changing any of values in benchmark.properties file, it is supposed that I don't need to re-build again? That's right? Just wanted to make sure.

Thank you in advance!

Javadoc failure

Branch qa-1586
Command: maven clean install -DskipTests

Result:

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-javadoc-plugin:2.9.1:jar (attach-javadocs) on project yardstick: MavenReportException: Error while creating archive: 
[ERROR] Exit code: 1 - /Users/isuntsov/YARDSTICK_DOCKER/yardstick/src/main/java/org/yardstickframework/runners/Worker.java:41: warning: no @return
[ERROR]     protected List<WorkResult> workOnHosts() {
[ERROR]                                ^
[ERROR] /Users/isuntsov/YARDSTICK_DOCKER/yardstick/src/main/java/org/yardstickframework/runners/FullRunner.java:142: warning: no @return
[ERROR]     public int run() {
[ERROR]                ^
[ERROR] /Users/isuntsov/YARDSTICK_DOCKER/yardstick/src/main/java/org/yardstickframework/runners/RunContext.java:131: warning: no @param for args
[ERROR]     public static RunContext getRunContext(String[] args) {
[ERROR]                              ^
[ERROR] /Users/isuntsov/YARDSTICK_DOCKER/yardstick/src/main/java/org/yardstickframework/runners/RunContext.java:131: warning: no @return
[ERROR]     public static RunContext getRunContext(String[] args) {
[ERROR]                              ^
[ERROR] /Users/isuntsov/YARDSTICK_DOCKER/yardstick/src/main/java/org/yardstickframework/runners/RunContext.java:269: error: invalid use of @return
[ERROR]      * @return Value
[ERROR]        ^
[ERROR] /Users/isuntsov/YARDSTICK_DOCKER/yardstick/src/main/java/org/yardstickframework/runners/RunContext.java:302: warning: no @return
[ERROR]     public List<String> getFullHostList() {
[ERROR]                         ^
[ERROR] /Users/isuntsov/YARDSTICK_DOCKER/yardstick/src/main/java/org/yardstickframework/runners/RunContext.java:312: warning: no @return
[ERROR]     public List<String> getFullUniqList() {
[ERROR]                         ^
[ERROR] /Users/isuntsov/YARDSTICK_DOCKER/yardstick/src/main/java/org/yardstickframework/runners/Worker.java:17: warning: no @param for runCtx
[ERROR]     public Worker(RunContext runCtx, WorkContext workCtx) {
[ERROR]            ^
[ERROR] /Users/isuntsov/YARDSTICK_DOCKER/yardstick/src/main/java/org/yardstickframework/runners/Worker.java:17: warning: no @param for workCtx
[ERROR]     public Worker(RunContext runCtx, WorkContext workCtx) {
[ERROR]            ^
[ERROR] /Users/isuntsov/YARDSTICK_DOCKER/yardstick/src/main/java/org/yardstickframework/runners/docker/DockerCheckWorker.java:14: warning: no description for @param
[ERROR]      * @param runCtx
[ERROR]        ^
[ERROR] /Users/isuntsov/YARDSTICK_DOCKER/yardstick/src/main/java/org/yardstickframework/runners/docker/DockerCheckWorker.java:15: warning: no description for @param
[ERROR]      * @param workCtx
[ERROR]        ^
[ERROR] /Users/isuntsov/YARDSTICK_DOCKER/yardstick/src/main/java/org/yardstickframework/runners/docker/DockerCleanContWorker.java:9: warning: no description for @param
[ERROR]      * @param runCtx
[ERROR]        ^
[ERROR] /Users/isuntsov/YARDSTICK_DOCKER/yardstick/src/main/java/org/yardstickframework/runners/docker/DockerCleanContWorker.java:10: warning: no description for @param
[ERROR]      * @param workCtx
[ERROR]        ^
[ERROR] /Users/isuntsov/YARDSTICK_DOCKER/yardstick/src/main/java/org/yardstickframework/runners/docker/DockerCleanImagesWorker.java:13: warning: no description for @param
[ERROR]      * @param runCtx
[ERROR]        ^
[ERROR] /Users/isuntsov/YARDSTICK_DOCKER/yardstick/src/main/java/org/yardstickframework/runners/docker/DockerCleanImagesWorker.java:14: warning: no description for @param
[ERROR]      * @param workCtx
[ERROR]        ^
[ERROR] /Users/isuntsov/YARDSTICK_DOCKER/yardstick/src/main/java/org/yardstickframework/runners/docker/DockerCleanImagesWorker.java:16: warning: no @param for imagesToClean
[ERROR]     public DockerCleanImagesWorker(RunContext runCtx,
[ERROR]            ^
[ERROR] /Users/isuntsov/YARDSTICK_DOCKER/yardstick/src/main/java/org/yardstickframework/runners/docker/DockerCollectWorker.java:13: warning: no description for @param
[ERROR]      * @param runCtx
[ERROR]        ^
[ERROR] /Users/isuntsov/YARDSTICK_DOCKER/yardstick/src/main/java/org/yardstickframework/runners/docker/DockerCollectWorker.java:14: warning: no description for @param
[ERROR]      * @param workCtx
[ERROR]        ^
[ERROR] /Users/isuntsov/YARDSTICK_DOCKER/yardstick/src/main/java/org/yardstickframework/runners/docker/DockerWorker.java:30: warning: no description for @param
[ERROR]      * @param runCtx
[ERROR]        ^
[ERROR] /Users/isuntsov/YARDSTICK_DOCKER/yardstick/src/main/java/org/yardstickframework/runners/docker/DockerWorker.java:31: warning: no description for @param
[ERROR]      * @param workCtx
[ERROR]        ^
[ERROR] 
[ERROR] Command line was: /Library/Java/JavaVirtualMachines/jdk1.8.0_152.jdk/Contents/Home/bin/javadoc @options @packages
[ERROR] 
[ERROR] Refer to the generated Javadoc files in '/Users/isuntsov/YARDSTICK_DOCKER/yardstick/target/apidocs' dir.
[ERROR] 
[ERROR] -> [Help 1]
[ERROR] 

How to generate graph for server resources

We get charts generated for driver resources (memory, cpu, network and io). How the same can be generated for servers?

how to increase memory bandwidth in running yardstick benchmarks?

How can I increase memory bandwidth in running yardstick benchmarks?
I ran PutGetBenchmark.
For increasing memory bandwidth,
I increased the value range and number of threads in the .properties file, and increased the Key-Value value.

On the Hazelcast platform, memory bandwidth is up to 20-30 GB/s.
However, on apache ignite the memory bandwidth is up to around 8GB/s.
What should I configure?

REMOTE_USER is not used in some commands

If Unix users are different on the different hosts, REMOTE_USER option should be enough to specify a user on the remote host for benchmark-run-all.sh. However, a lot of commands will still fail as they don't actually use REMOTE_USER.

Warmup time should be separate than test time

It seems that the warm-up time (-w) is part of the test duration (-d), which is confusing. Ideally, warm-up is initial time where system does lots of one time lazy initialization stuff. For bench marking we focus on core operation and thus ignores initial stuff done in warm-up time.

Test duration is time of interest for which benchmark reports are generated, which should be separate than the warm-up time.

.jfr server nodes are damaged when the test is stopped

Good afternoon. About sh stopping the server node from yardstick. Now there are used commands
pkill -9 -f "Dyardstick. server"
pkill -9 -f "benchmark-server-restarter-start.sh"

the -9 key forcibly terminates processes. A problem has been identified that if the jfr recording option is active at startup, then with this method of shutting down the server node, the last jfr file does not correctly complete its recording (it is not possible to open it later), as a result, all jfrs recorded during the test are not stitched into a single jfr file.
Make simple edits for yourself. I just deleted the -9 key from the server node stop command.

pkill-f "Dyardstick. server"
pkill-f "benchmark-server-restarter-start.sh"

I understand that there were probably reasons to use pkill -9-f, so I would like to know if it is possible to leave it like this. If so, how can this be added to the repository so that further builds are built with the corrected sh script?

Thank you very much in advance.

Results folders are not generated on Windows when running the manual scripts

Hi,

I tried to run the manual scripts (the server and driver) on the *_windows *_and everything went fine. It generates the logs folders and there are no exceptions in the logs. However, at the end of running, the results (output) folders are not generated. Any idea?

P.S: In *_linux *_they are generated properly!

Thank you in advance.

Need to add restart functionality

Need to have instruction to have opportunity to restart server/servers according to configuration.

It is impossible to use Lambda expressions inside of benchmarks

Hi,

I had a strange problem which took me quite some time to figure out: one of my custom benchmarks was not picked up by Yardstick's BenchmarkLoader - without reporting any error.

Digging into it revealed that the org.reflections dependency used inside of Yardstick is not able to handle Java 8 classes with Lambda expressions. Please update this dependency! I've updated/added the following dependencies and it worked out:

<dependency>
    <groupId>org.reflections</groupId>
    <artifactId>reflections</artifactId>
    <version>0.9.11</version>
</dependency>

<!-- Now required for javax.annotation.Nullable availablility. -->
<dependency> 
    <groupId>com.google.code.findbugs</groupId>
    <artifactId>jsr305</artifactId>
    <version>3.0.2</version>
</dependency>

See also: https://github.com/gmuecke/reflections/issues/184

This should be a pretty easy fix, improving the Java 8 compatibility significantly.

FYI: Adding the above org.reflections dependency to your benchmark project's pom will replace the outdated reflections library during the mvn install (which worked for me as a workaround).

The multiple drivers need to sync up on before starting benchmark

I am using 4 drivers (client instances) in a benchmark, however they are not in sync, resulting into incorrect benchmark results. They starts ony afer other but do not sync up, thus first driver who comes up starts benchamrk operations firsts and so finishes first.

See the time difference between starting of each driver.

$ grep "Starting main test (warmup finished)" logs_drivers/*
logs_drivers/131936_0_pnq-gst-dev06.log:<13:20:34><yardstick> Starting main test (warmup finished).
logs_drivers/131949_1_pnq-gst-dev06.log:<13:20:45><yardstick> Starting main test (warmup finished).
logs_drivers/132002_2_pnq-gst-dev06.log:<13:20:58><yardstick> Starting main test (warmup finished).
logs_drivers/132015_3_pnq-gst-dev06.log:<13:21:12><yardstick> Starting main test (warmup finished).

Same with test finish:

$ grep "Finishing main test" logs_drivers/*
logs_drivers/131936_0_pnq-gst-dev06.log:Finishing main test [ts=1436255734062, date=Tue Jul 07 13:25:34 IST 2015]
logs_drivers/131949_1_pnq-gst-dev06.log:Finishing main test [ts=1436255745000, date=Tue Jul 07 13:25:45 IST 2015]
logs_drivers/132002_2_pnq-gst-dev06.log:Finishing main test [ts=1436255758933, date=Tue Jul 07 13:25:58 IST 2015]
logs_drivers/132015_3_pnq-gst-dev06.log:Finishing main test [ts=1436255772103, date=Tue Jul 07 13:26:12 IST 2015]

Add mention of BenchmarkProbe interface in "Creating Yardstick Benchmarks" section of readme

Suggest to add mention of BenchmarkProbe interface in "Creating Yardstick Benchmarks" section of README.md. Text could be like:

BenchmarkProbe interface is an important building block of the framework - use standard implementations supplied by Yardstick or create your own.

I think that way it would be easier to readers to grok how framework works and what is expected when creating own benchmarks.

Current wording can leave impression that besides implementing BenchmarkServer and BenchmarkDriver interfaces nothing else is needed to integrate with the framework. And even cursory glance on yardstick and ignite-yardstick code may still keep such an impression.

Incorrect representation of Latency graph when multiple drivers are used

When mutiple drivers are used. The latencies of all drivers is added and the total number is plotted on the chart, which gives wrong impression of increased latency.

Above strategy is good for plotting throughput. But for latency, you should have plotted the average.

Thanks,
Rahul

How to configure third type of entiry (say Locator) in addition to Server and Driver

I want to benchmark our product using yardstick.
Our product have following entities:

1. Locator - for clustor management and discovery, very few in number
2. Server - stores data, scalable
3. Client - performs user operation, Driver in case of yardstick

We can configure Server and Client, please let me know how to configure Locator member?