csgillespie/benchmarkme

Crowd sourced benchmarking

System benchmarking

R benchmarking made easy. The package contains a number of benchmarks, heavily based on the benchmarks at https://mac.R-project.org/benchmarks/R-benchmark-25.R, for assessing the speed of your system.

The package is for R 3.5 and above. In previous versions R, detecting the effect of the byte compiler was tricky and produced unrealistic comparisons.

Overview

A straightforward way of speeding up your analysis is to buy a better computer. Modern desktops are relatively cheap, especially compared to user time. However, it isn’t clear if upgrading your computing is worth the cost. The benchmarkme package provides a set of benchmarks to help quantify your system. More importantly, it allows you to compare your timings with other systems.

Overview

The package is on CRAN and can be installed in the usual way

install.packages("benchmarkme")

There are two groups of benchmarks:

• benchmark_std(): this benchmarks numerical operations such as loops and matrix operations. The benchmark comprises of three separate benchmarks: prog, matrix_fun, and matrix_cal.
• benchmark_io(): this benchmarks reading and writing a 5 / 50, MB csv file.

The benchmark_std() function

This benchmarks numerical operations such as loops and matrix operations. This benchmark comprises of three separate benchmarks: prog, matrix_fun, and matrix_cal. If you have less than 3GB of RAM (run get_ram() to find out how much is available on your system), then you should kill any memory hungry applications, e.g. firefox, and set runs = 1 as an argument.

To benchmark your system, use

library("benchmarkme")
## Increase runs if you have a higher spec machine
res = benchmark_std(runs = 3)

and upload your results

## You can control exactly what is uploaded. See details below.
upload_results(res)

You can compare your results to other users via

plot(res)

The benchmark_io() function

This function benchmarks reading and writing a 5MB or 50MB (if you have less than 4GB of RAM, reduce the number of runs to 1). Run the benchmark using

res_io = benchmark_io(runs = 3)
upload_results(res_io)
plot(res_io)

By default the files are written to a temporary directory generated

tempdir()

which depends on the value of

Sys.getenv("TMPDIR")

You can alter this to via the tmpdir argument. This is useful for comparing hard drive access to a network drive.

res_io = benchmark_io(tmpdir = "some_other_directory")

Parallel benchmarks

The benchmark functions above have a parallel option - just simply specify the number of cores you want to test. For example to test using four cores

res_io = benchmark_std(runs = 3, cores = 4)
plot(res_io)

Previous versions of the package

This package was started around 2015. However, multiple changes in the byte compiler over the last few years, has made it very difficult to use previous results. So we have to start from scratch.

The previous data can be obtained via

data(past_results, package = "benchmarkmeData")

Machine specs

The package has a few useful functions for extracting system specs:

• RAM: get_ram()
• CPUs: get_cpu()
• BLAS library: get_linear_algebra()
• Is byte compiling enabled: get_byte_compiler()
• General platform info: get_platform_info()
• R version: get_r_version()

The above functions have been tested on a number of systems. If they don’t work on your system, please raise GitHub issue.

Uploaded data sets

A summary of the uploaded data sets is available in the benchmarkmeData package

data(past_results_v2, package = "benchmarkmeData")

A column of this data set, contains the unique identifier returned by the upload_results() function.

What’s uploaded

Two objects are uploaded:

1. Your benchmarks from benchmark_std or benchmark_io;
2. A summary of your system information (get_sys_details()).

The get_sys_details() returns:

• Sys.info();
• get_platform_info();
• get_r_version();
• get_ram();
• get_cpu();
• get_byte_compiler();
• get_linear_algebra();
• installed.packages();
• Sys.getlocale();
• The benchmarkme version number;
• Unique ID - used to extract results;
• The current date.

The function Sys.info() does include the user and nodenames. In the public release of the data, this information will be removed. If you don’t wish to upload certain information, just set the corresponding argument, i.e.

upload_results(res, args = list(sys_info = FALSE))

---

Development of this package was supported by Jumping Rivers

benchmarkme's Issues

Dummy benchmark run

For the parallel benchmarks, we need to do a warmup run. However, this could be quickenp by using a dummy flag in the different benchmarks.

`get_ram` doesn't work in Kaggle note books

On kaggle.com if I start a new notebook and I run get_ram()

I get

Unable to detect your RAM. # nocov
            Please raise an issue at https://github.com/csgillespie/benchmarkme

get_ram() is slow on macOS

system("system_profiler -detailLevel mini", intern = TRUE)) takes ~ 10 seconds to run on my recent Macbook Air.

substring(system("sysctl hw.memsize", intern = TRUE), 13) takes a few microseconds.

Output is a string, units are bytes. "8589934592"

Command found here:

https://stackoverflow.com/questions/583736/determine-physical-mem-size-programmatically-on-osx#comment30400379_818741

benchmark_std crashes in cross-product

benchmark_std does not work for me.

> benchmarkme::benchmark_std()
# Programming benchmarks (5 tests):
	3,500,000 Fibonacci numbers calculation (vector calc): 0.245 (sec).
	Grand common divisors of 1,000,000 pairs (recursion): 0.642 (sec).
	Creation of a 3,500 x 3,500 Hilbert matrix (matrix calc): 0.24 (sec).
	Creation of a 3,000 x 3,000 Toeplitz matrix (loops): 1.42 (sec).
	Escoufier's method on a 60 x 60 matrix (mixed): 1.02 (sec).
# Matrix calculation benchmarks (5 tests):
	Creation, transp., deformation of a 5,000 x 5,000 matrix: 0.545 (sec).
	2,500 x 2,500 normal distributed random matrix^1,000: 0.276 (sec).
	Sorting of 7,000,000 random values: 0.989 (sec).
	2,500 x 2,500 cross-product matrix (b = a' * a): 0.225 (sec).
Error in solve(crossprod(a), crossprod(a, b)) : 
  the leading minor of order 1 is not positive definite
Timing stopped at: 0.122 0.089 0.029
> sessionInfo()
R version 4.0.4 (2021-02-15)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Debian GNU/Linux bullseye/sid

Matrix products: default
BLAS:   /usr/lib/x86_64-linux-gnu/openblas-pthread/libblas.so.3
LAPACK: /usr/lib/x86_64-linux-gnu/openblas-pthread/libopenblasp-r0.3.13.so

locale:
 [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C              
 [3] LC_TIME=en_US.UTF-8        LC_COLLATE=en_US.UTF-8    
 [5] LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8   
 [7] LC_PAPER=en_US.UTF-8       LC_NAME=C                 
 [9] LC_ADDRESS=C               LC_TELEPHONE=C            
[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C       

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

loaded via a namespace (and not attached):
 [1] lattice_0.20-41       codetools_0.2-18      fansi_0.4.2          
 [4] crayon_1.4.1          utf8_1.2.1            foreach_1.5.1        
 [7] dplyr_1.0.5           grid_4.0.4            R6_2.5.0             
[10] lifecycle_1.0.0       magrittr_2.0.1        httr_1.4.2           
[13] pillar_1.5.1          rlang_0.4.10          doParallel_1.0.16    
[16] benchmarkmeData_1.0.4 Matrix_1.3-2          ellipsis_0.3.1       
[19] vctrs_0.3.6           generics_0.1.0        iterators_1.0.13     
[22] tools_4.0.4           glue_1.4.2            benchmarkme_1.0.7    
[25] purrr_0.3.4           parallel_4.0.4        compiler_4.0.4       
[28] pkgconfig_2.0.3       tidyselect_1.1.0      tibble_3.1.0         
> packageVersion("benchmarkme")
[1] ‘1.0.7’

get_ram() cannot retrieve RAM

OS and Hardware:

OS: Windows 10, 09-2023 cumulative update (KB5030211)
Laptop Model: HP ENVY Notebook - 15-ae112nl
Processor: Intel(R) Core(TM) i7-6500U CPU @ 2.50GHz
RAM: 16 GB
R version: 4.3.1 "Beagle Scouts"
benchmarkme package version: 1.0.8

Summary

get_ram() can't retrieve RAM and reports an Error that I don't recognize.

Steps to Reproduce

1. Run get_ram() in R Console.

Actual Result

The following Error is printed:

Error in inherits(ram, "try-error") || length(ram) == 0L || is.na(ram) : 'length = 3' in coercion to 'logical(1)'

Expected Result

The amount of RAM installed on my laptop (16 GB).

Reported by

Alessandro Mazza.

Parallel option not working

Hi Colin,

I was having a play around with this this morning. The standard function seems to work fine, but when I use the cores = .. argument I get an error as below. I've tried with 2, 4, and 8 cores with the same result.

# Programming benchmarks (5 tests):
# starting worker pid=10496 on localhost:11728 at 09:56:33.005
# starting worker pid=10495 on localhost:11728 at 09:56:33.005
# Error in unserialize(node$con) : error reading from connection
# Calls: <Anonymous> -> slaveLoop -> makeSOCKmaster
# Error in unserialize(node$con) : error reading from connection
# Calls: <Anonymous> -> slaveLoop -> makeSOCKmaster
# Execution halted
# Execution halted
# Error in makePSOCKcluster(names = spec, ...) : 
#  Cluster setup failed. 2 of 2 workers failed to connect.

What is really weird, is that when I created a reprex for this issue, it seemed to work...

library(benchmarkme)

benchmark_std(cores = 2L)
#> # Programming benchmarks (5 tests):
#> # Matrix calculation benchmarks (5 tests):
#> # Matrix function benchmarks (5 tests):
#>      user system elapsed          test test_group cores
#> 2   0.020  0.001   0.546           fib       prog     2
#> 3   0.004  0.001   0.383           fib       prog     2
#> 4   0.003  0.000   0.377           fib       prog     2
#> 21  0.003  0.000   1.172           gcd       prog     2
#> 31  0.003  0.000   0.831           gcd       prog     2
#> 41  0.003  0.001   0.905           gcd       prog     2
#> 22  0.004  0.001   0.694       hilbert       prog     2
#> 32  0.004  0.000   0.306       hilbert       prog     2
#> 42  0.004  0.000   0.402       hilbert       prog     2
#> 23  0.003  0.000   1.245      toeplitz       prog     2
#> 33  0.003  0.000   1.211      toeplitz       prog     2
#> 43  0.003  0.000   1.211      toeplitz       prog     2
#> 24  0.004  0.000   1.151     escoufier       prog     2
#> 34  0.004  0.000   1.098     escoufier       prog     2
#> 44  0.003  0.000   1.089     escoufier       prog     2
#> 25  0.003  0.000   0.990         manip matrix_cal     2
#> 35  0.003  0.000   0.725         manip matrix_cal     2
#> 45  0.004  0.000   0.714         manip matrix_cal     2
#> 211 0.003  0.000   0.956         power matrix_cal     2
#> 311 0.003  0.000   0.792         power matrix_cal     2
#> 411 0.003  0.000   0.715         power matrix_cal     2
#> 221 0.004  0.000   1.437          sort matrix_cal     2
#> 321 0.003  0.001   1.200          sort matrix_cal     2
#> 421 0.004  0.000   1.172          sort matrix_cal     2
#> 231 0.003  0.000   9.929 cross_product matrix_cal     2
#> 331 0.003  0.000   9.361 cross_product matrix_cal     2
#> 431 0.003  0.000   9.455 cross_product matrix_cal     2
#> 241 0.003  0.000   1.126            lm matrix_cal     2
#> 341 0.003  0.000   1.112            lm matrix_cal     2
#> 441 0.004  0.000   1.088            lm matrix_cal     2
#> 26  0.003  0.000  20.535      cholesky matrix_fun     2
#> 36  0.004  0.000  20.175      cholesky matrix_fun     2
#> 46  0.003  0.001  20.170      cholesky matrix_fun     2
#> 212 0.004  0.001   3.566   determinant matrix_fun     2
#> 312 0.003  0.000   3.493   determinant matrix_fun     2
#> 412 0.003  0.000   3.476   determinant matrix_fun     2
#> 222 0.003  0.000   0.871         eigen matrix_fun     2
#> 322 0.003  0.000   0.841         eigen matrix_fun     2
#> 422 0.004  0.000   0.821         eigen matrix_fun     2
#> 232 0.003  0.000   0.619           fft matrix_fun     2
#> 332 0.004  0.001   0.595           fft matrix_fun     2
#> 432 0.003  0.000   0.578           fft matrix_fun     2
#> 242 0.003  0.000   3.026       inverse matrix_fun     2
#> 342 0.003  0.000   3.000       inverse matrix_fun     2
#> 442 0.003  0.000   2.923       inverse matrix_fun     2

^{Created on 2020-11-18 by the reprex package (v0.3.0.9001)}

Session info

sessioninfo::session_info()
#> ─ Session info ───────────────────────────────────────────────────────────────
#>  setting  value                       
#>  version  R version 4.0.2 (2020-06-22)
#>  os       macOS  10.16                
#>  system   x86_64, darwin17.0          
#>  ui       X11                         
#>  language (EN)                        
#>  collate  en_GB.UTF-8                 
#>  ctype    en_GB.UTF-8                 
#>  tz       Europe/London               
#>  date     2020-11-18                  
#> 
#> ─ Packages ───────────────────────────────────────────────────────────────────
#>  package         * version    date       lib source                           
#>  assertthat        0.2.1      2019-03-21 [1] CRAN (R 4.0.0)                   
#>  benchmarkme     * 1.0.4      2020-05-09 [1] CRAN (R 4.0.2)                   
#>  benchmarkmeData   1.0.4      2020-04-23 [1] CRAN (R 4.0.2)                   
#>  cli               2.1.0      2020-10-12 [1] CRAN (R 4.0.2)                   
#>  codetools         0.2-16     2018-12-24 [1] CRAN (R 4.0.2)                   
#>  crayon            1.3.4      2017-09-16 [1] CRAN (R 4.0.0)                   
#>  digest            0.6.27     2020-10-24 [1] CRAN (R 4.0.2)                   
#>  doParallel        1.0.16     2020-10-16 [1] CRAN (R 4.0.2)                   
#>  dplyr             1.0.2      2020-08-18 [1] CRAN (R 4.0.2)                   
#>  ellipsis          0.3.1      2020-05-15 [1] CRAN (R 4.0.0)                   
#>  evaluate          0.14       2019-05-28 [1] CRAN (R 4.0.0)                   
#>  fansi             0.4.1      2020-01-08 [1] CRAN (R 4.0.0)                   
#>  foreach           1.5.0      2020-03-30 [1] CRAN (R 4.0.2)                   
#>  fs                1.5.0      2020-07-31 [1] CRAN (R 4.0.2)                   
#>  generics          0.0.2      2018-11-29 [1] CRAN (R 4.0.0)                   
#>  glue              1.4.2      2020-08-27 [1] CRAN (R 4.0.2)                   
#>  highr             0.8        2019-03-20 [1] CRAN (R 4.0.0)                   
#>  htmltools         0.5.0      2020-06-16 [1] CRAN (R 4.0.0)                   
#>  httr              1.4.2      2020-07-20 [1] CRAN (R 4.0.0)                   
#>  iterators         1.0.12     2019-07-26 [1] CRAN (R 4.0.2)                   
#>  knitr             1.30       2020-09-22 [1] CRAN (R 4.0.2)                   
#>  lattice           0.20-41    2020-04-02 [1] CRAN (R 4.0.2)                   
#>  lifecycle         0.2.0      2020-03-06 [1] CRAN (R 4.0.0)                   
#>  magrittr          2.0.1      2020-11-17 [1] CRAN (R 4.0.2)                   
#>  Matrix            1.2-18     2019-11-27 [1] CRAN (R 4.0.2)                   
#>  pillar            1.4.6      2020-07-10 [1] CRAN (R 4.0.2)                   
#>  pkgconfig         2.0.3      2019-09-22 [1] CRAN (R 4.0.0)                   
#>  purrr             0.3.4      2020-04-17 [1] CRAN (R 4.0.0)                   
#>  R6                2.5.0      2020-10-28 [1] CRAN (R 4.0.2)                   
#>  reprex            0.3.0.9001 2020-11-18 [1] Github (tidyverse/reprex@d3fc4b8)
#>  rlang             0.4.8      2020-10-08 [1] CRAN (R 4.0.2)                   
#>  rmarkdown         2.5        2020-10-21 [1] CRAN (R 4.0.2)                   
#>  rstudioapi        0.11       2020-02-07 [1] CRAN (R 4.0.0)                   
#>  sessioninfo       1.1.1      2018-11-05 [1] CRAN (R 4.0.0)                   
#>  stringi           1.5.3      2020-09-09 [1] CRAN (R 4.0.2)                   
#>  stringr           1.4.0      2019-02-10 [1] CRAN (R 4.0.0)                   
#>  tibble            3.0.4      2020-10-12 [1] CRAN (R 4.0.2)                   
#>  tidyselect        1.1.0      2020-05-11 [1] CRAN (R 4.0.0)                   
#>  vctrs             0.3.4      2020-08-29 [1] CRAN (R 4.0.2)                   
#>  withr             2.3.0      2020-09-22 [1] CRAN (R 4.0.2)                   
#>  xfun              0.19       2020-10-30 [1] CRAN (R 4.0.2)                   
#>  yaml              2.2.1      2020-02-01 [1] CRAN (R 4.0.0)                   
#> 
#> [1] /Library/Frameworks/R.framework/Versions/4.0/Resources/library

...but still doesn't work without doing a reprex. I'm guessing a dependency issue? Although I can't figure out what might be missing.

Apologies if this me being stupid... but I can't figure it out.

Missing import: Error in globalVariables(c("a", "b", "ans")) : could not find function "globalVariables"

Hi. The

benchmarkme/R/benchmark_matrix_calculations.R

Line 1 in 8ee0d3a

globalVariables(c("a", "b", "ans"))

lacks a @importFrom utils globalVariables or needs to be specified as:

utils::globalVariables(...)

I'm getting errors on this when revdep checking several packages, e.g. R CMD check on data.frame reports:

Error in globalVariables(c("a", "b", "ans")) : 
  could not find function "globalVariables"
Error: unable to load R code in package ‘benchmarkme’
Execution halted

PS. I've spotted many other packages that forget to import globalVariables(), setMethod(), ...

`get_ram` not working on Windows 10

When I run get_ram it returns NA B.

Here is my sessionInfo()

R version 3.5.3 (2019-03-11)
Platform: x86_64-w64-mingw32/x64 (64-bit)
Running under: Windows 10 x64 (build 17763)

Matrix products: default

locale:
[1] LC_COLLATE=English_United States.1252  LC_CTYPE=English_United States.1252    LC_MONETARY=English_United States.1252
[4] LC_NUMERIC=C                           LC_TIME=English_United States.1252    

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

loaded via a namespace (and not attached):
[1] compiler_3.5.3 tools_3.5.3   

Incorrect/outdated docs: get_ram() says it uses memory.size() on Windows

This does not seem to be true.

> memory.size()
[1] 308.74
> memory.size(TRUE)
[1] 316.81
> benchmarkme::get_ram()
8.2 MB

Strange timing results

Hi, I am getting very strange results for 2 and 4 processes. The 4 process run is taking slightly longer than the 2 process run:

inverse	FFT	eigen	cholesky
3.71	0.801	1.22	28.84
4.008	0.854	1.256	29.33

The R command I used to run the benchmark is res_io = benchmark_std(runs = 3, cores = 4). It doesn't look like that this benchmark is multi-threaded.

get_cpu() and get_platform_info() do not work correctly on PowerPC

Benchmarks run fine, however CPU and other system info are not picked:

> get_cpu()
$vendor_id
character(0)

$model_name
character(0)

$no_of_cores
[1] 2

> get_platform_info()
$OS.type
[1] "unix"

$file.sep
[1] "/"

$dynlib.ext
[1] ".so"

$GUI
[1] "X11"

$endian
[1] "big"

$pkgType
[1] "source"

$path.sep
[1] ":"

$r_arch
[1] ""

> get_r_version()
$platform
[1] "powerpc-apple-darwin10.0.0d2"

$arch
[1] "powerpc"

$os
[1] "darwin10.0.0d2"

$system
[1] "powerpc, darwin10.0.0d2"

$status
[1] ""

$major
[1] "4"

$minor
[1] "2.3"

$year
[1] "2023"

$month
[1] "03"

$day
[1] "15"

$`svn rev`
[1] "83980"

$language
[1] "R"

$version.string
[1] "R version 4.2.3 (2023-03-15)"

$nickname
[1] "Shortstop Beagle"

Problem when uploading benchmark results on OSX El Capitan

MacBook Pro (Retina, 13-inch, Early 2015)
OSX 10.11.1 (15B42)

upload_results(res)
I'm having trouble detecting your RAM. So try a number of things to help future versions
awk: can't open file /proc/meminfo
source line number 1
sh: wmic: command not found
Error in system("wmic MemoryChip get Capacity", intern = TRUE) :
error in running command
In addition: Warning message:
running command 'awk '/MemFree/ {print $2}' /proc/meminfo' had status 2 `

Different number of machines in comparison of plot(res)

This package is great and exactly what I was looking for.

I am just wondering about one thing, when I run the test of different machines, and run plot(res), I get a different number of total machines. For example, on my MacBook I get a rank out of 749 machines, but on a linux VM I get a rank out of 93 machines, so I am just wondering what machines I am being ranked against, exactly? Is it grouping by OS or computer type or something else? I just want to interpret it correctly

Furthermore, and this is a much more minor issue but I just want to understand it correctly, after I upload results and redo the test, I still get ranked out of the same number. I would have expected the number to go up by 1, since I have contributed my machines results. I imagine there is something to prevent duplicate entries, but it didn't go up even after the first time I uploaded results.

get_ram() fails on mac os x 10.9.5

The issue is the following line:

 ram = system("system_profiler -detailLevel mini | grep \"  Memory:\"", 
+        intern = TRUE)[1]

which in my case is:

[1] "      Memory: 8 GB"

This is strsplit on the space and passed to to_Bytes which expects the first element to be 8 and the second "GB". Which is evidently not the case.

Here is my sessionInfo() :

R version 3.2.1 (2015-06-18)
Platform: x86_64-apple-darwin13.4.0 (64-bit)
Running under: OS X 10.9.5 (Mavericks)

locale:
[1] fr_FR.UTF-8/fr_FR.UTF-8/fr_FR.UTF-8/C/fr_FR.UTF-8/fr_FR.UTF-8

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] benchmarkme_0.1.3 devtools_1.9.1   

loaded via a namespace (and not attached):
 [1] lattice_0.20-33  digest_0.6.8     grid_3.2.1       R6_2.1.1        
 [5] jsonlite_0.9.19  magrittr_1.5     httr_1.0.0       stringi_1.0-1   
 [9] curl_0.9.4       rstudioapi_0.3.1 DT_0.1           Matrix_1.2-2    
[13] tools_3.2.1      stringr_1.0.0    htmlwidgets_0.5  yaml_2.1.13     
[17] compiler_3.2.1   memoise_0.2.1    htmltools_0.2.6 

Release benchmarkme 1.0.2

Prepare for release:

• devtools::check()
• devtools::check_win_devel()
• rhub::check_for_cran()
• revdepcheck::revdep_check(num_workers = 4)
• Polish NEWS
• Polish pkgdown reference index

Submit to CRAN:

• usethis::use_version('patch')
• Update cran-comments.md
• devtools::submit_cran()
• Approve email

Wait for CRAN...

• Accepted 🎉
• usethis::use_github_release()
• usethis::use_dev_version()
• Tweet

Bug in benchmark::rank_results

The number of rank is larger than the total number of sample, after running the rank_results function. Here is how to reproduce the bug:

library(benchmarkme)
res_cpu <- benchmark_std(runs = 3)
rank_results(res_cpu)

# You are ranked 70 out of 45 machines. (the number may vary across different machines)

A closer look at the rank_results function reveals that the bug lies in the last few lines:

    if (verbose)
        message("You are ranked ", ben_rank, " out of ", nrow(results),
            " machines.")

where nrow(results) should be nrow(pas_res).

Another question I want to confirm is that does the past performance dataset benchmarkmeData::past_results change/accumulate as time passes by? If so, then perhaps I may yield different benchmarks for the same machine?

Thank you.
Yu

Possibility to access the benchmark dataset

Is it possible to acces the benchmark dataset, so the results of the system and that of the systems which it is compared against can be downloaded/accessed. This way it gives a better understanding where the system actually is ranked. Just dots on a plot don't say that much.

Vignette describing benchmarks

I think it would be really helpful to have a package vignette that explains each benchmark in detail using the details from the original source of the benchmark. I know this would make it easier for labs and system builders to include write-ups of these benchmarks in their discussion of machine/architecture performance differences.

Would this be something you think is appropriate? I could write something up and PR it.

System details fails when package not loaded

I try to run benchmarkme without loading the package and it faild

benchmarkme::get_sys_details()

> benchmarkme::get_sys_details()
Error in get("benchmark_std", envir = globalenv()) : 
  object 'benchmark_std' not found

Also why not use 1Mb=1024KB, or is there any place where I can set the option?

CRAN issue:

If the runjags message

'Xgrid is not available on this machine. Xgrid functions are only
available on machines running Mac OS X 10.5 (Leopard), OS X 10.6 (Snow
Leopard) or OS X 10.7 (Lion) and with access to an Xgrid controller'

is accurate (and it seems so) this is ancient history and the
non-functionality should be removed.

Please correct before 2021-02-23 to safely retain your package on CRAN.

---

Follow up email

Package benchmarkme has not done as you were asked, to look for sysctl
(not just check if it is on the path).

As a result,

Name: get_cpu

Title: CPU Description

Aliases: get_cpu

** Examples

Return the machine CPU

get_cpu()
$vendor_id
[1] NA

$model_name
[1] NA

$no_of_cores
[1] 8

Name: get_ram

Title: Get the amount of RAM

Aliases: get_ram

** Examples

Return (and pretty print) the amount of RAM

get_ram()
Unable to detect your RAM. # nocov
Please raise an issue at
https://github.com/csgillespie/benchmarkme
NA B

Display using iec units

print(get_ram(), unit_system = "iec")
Unable to detect your RAM. # nocov
Please raise an issue at
https://github.com/csgillespie/benchmarkme
NA B

Name: get_sys_details

Title: General system information

Aliases: get_sys_details

** Examples

Returns all details about your machine

get_sys_details()
Unable to detect your RAM. # nocov
Please raise an issue at
https://github.com/csgillespie/benchmarkme

and if also fails (in an ugly way) if sysctl is on the path:

example(get_cpu)

get_cp> ## Return the machine CPU
get_cp> get_cpu()
sysctl: unknown oid 'machdep.cpu.vendor'
$vendor_id
character(0)
attr(,"status")
[1] 1

$model_name
[1] "Apple M1"

$no_of_cores
[1] 8

Warning message:
In system(paste(sysctl, "-n machdep.cpu.vendor"), intern = TRUE) :
running command '/usr/sbin/sysctl -n machdep.cpu.vendor' had status 1

errors benchmarking with multiple cores?

Hey Colin,

Thanks for this cool package. I'm having trouble whenever I try to benchmark with multiple cores, e.g.:

benchmarkme::benchmark_std(cores = 2)
# Programming benchmarks (5 tests):
starting worker pid=10077 on localhost:11653 at 20:04:34.058
starting worker pid=10106 on localhost:11653 at 20:04:34.473
Error in get(name, envir = envir) : object 'bm_prog_fib' not found

Any ideas?

sessionInfo()
R version 3.6.0 (2019-04-26)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Debian GNU/Linux 9 (stretch)

Matrix products: default
BLAS/LAPACK: /usr/lib/libopenblasp-r0.2.19.so

locale:
 [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C               LC_TIME=en_US.UTF-8       
 [4] LC_COLLATE=en_US.UTF-8     LC_MONETARY=en_US.UTF-8    LC_MESSAGES=C             
 [7] LC_PAPER=en_US.UTF-8       LC_NAME=C                  LC_ADDRESS=C              
[10] LC_TELEPHONE=C             LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C       

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

loaded via a namespace (and not attached):
 [1] Rcpp_1.0.1            lattice_0.20-38       codetools_0.2-16      crayon_1.3.4         
 [5] packrat_0.5.0         foreach_1.4.4         dplyr_0.8.1           assertthat_0.2.1     
 [9] grid_3.6.0            R6_2.4.0              magrittr_1.5          httr_1.4.0           
[13] pillar_1.4.0          rlang_0.3.4           rstudioapi_0.10       doParallel_1.0.14    
[17] benchmarkmeData_1.0.1 Matrix_1.2-17         iterators_1.0.10      tools_3.6.0          
[21] glue_1.3.1            benchmarkme_1.0.0     purrr_0.3.2           parallel_3.6.0       
[25] compiler_3.6.0        pkgconfig_2.0.2       tidyselect_0.2.5      tibble_2.1.1

get_ram() returns NA B on Windows 10/11 Pro machines

get_ram() returns NA B but benchmarkme:::get_windows_ram() gives "17179869184 \r" "17179869184 \r" "17179869184 \r" "17179869184 \r" "\r"

sessionInfo()
R version 4.1.2 (2021-11-01)
Platform: x86_64-w64-mingw32/x64 (64-bit)
Running under: Windows 10 x64 (build 22000)

Matrix products: default

locale:
[1] LC_COLLATE=English_United States.1252  LC_CTYPE=English_United States.1252   
[3] LC_MONETARY=English_United States.1252 LC_NUMERIC=C                          
[5] LC_TIME=English_United States.1252    
system code page: 65001

Use rhub-v2

https://www.r-bloggers.com/2024/04/r-hub-v2/

Problem to install benchemarkme

Thank you for this promising package,
however, I was not able to install the package (windows 64, R studio 0.99.436, R version 3.1.2 (2014-10-31)), and I get the following message :

> install.packages("benchmarkme")
Warning in install.packages :
  cannot open: HTTP status was '404 Not Found'
Warning in install.packages :
  cannot open: HTTP status was '404 Not Found'
Warning in install.packages :
  unable to access index for repository http://csgillespie.github.io/drat/bin/windows/contrib/3.1
Installing package into ‘C:/Users/ULB/Documents/R/win-library/3.1’
(as ‘lib’ is unspecified)
Warning in install.packages :
  cannot open: HTTP status was '404 Not Found'
Warning in install.packages :
  cannot open: HTTP status was '404 Not Found'
Warning in install.packages :
  unable to access index for repository http://csgillespie.github.io/drat/bin/windows/contrib/3.1

   package ‘benchmarkme’ is available as a source package but not as a binary

Warning in install.packages :
  package ‘benchmarkme’ is not available (for R version 3.1.2)

Thank you for your help

Error at benchmarkme

https://jumpingrivers.shinyapps.io/benchmarkme/ returns an error when a benchmark package is uploaded.

An error has occurred. Check your logs or contact the app author for clarification.

The upload file was produced by this code...

bm_results <- benchmark_std()
create_bundle(bm_results, filename = "hvbmR.rds")

I've attached hbbmR.rds
hvbmR.zip

I use this as part of the LinkedIn Learning course titled "R for Data Science: High Volume Data" ( https://www.linkedin.com/learning/r-programming-in-data-science-high-volume-data/code-how-fast-is-your-computer )

problem with get_byte_compiler

byte_compile.pdf

get_ram() reports incorrect RAM on Windows

On my Windows machine with 8GB RAM I'm getting the following

> benchmarkme::get_ram()
8.2 MB

This is either the algorithm being way off, or perhaps it's a units issue and that 8.2 is just supposed to be GB instead of MB

How to retrieve and compare two different results

Hi,

compliments for the package! It's great 😄 I was looking for a set of benchmarks to compare different configurations (libraries, multithreading, etc.) of my R setup, when I found about your package, which does all I need and much more. I have a question: I uploaded two sets of results with upload_results(res, args = list(sys_info=FALSE)), with IDs "2018-10-18-46512099" and "2018-10-18-14122665". Is there a way to retrieve them and compare them? I tried to use the benchmarkmeData package, but I couldn't find any results from 2018.

Error when package is not attached

Hi Gilles, I saw you at LondonR yesterday, it was a good talk.

I have installed the package benchmarkme but there is a light issue:
you have to attach the library and can't just access the function through the double colon.

install.packages("benchmarkme") # ok

 benchmarkme::benchmark_std()
# Programming benchmarks (5 tests):
Error in FUN(X[[i]], ...) : could not find function "bm_prog_fib"

library(benchmarkme)
benchmark_std() # ok

Setting MB options

"why not use 1Mb=1024KB, or is there any place where I can set the option?"

From #21

Parallel documentation

Following #11

• Update associated plot methods
• Update README
• Update vignette
• Update shiny app

Should plot use log="y" ?

Those atom results pretty much destroy the useful information in the plot.

get_linear_algebra() does not detect Intel MKL

Hello,

It looks like get_linear_algebra() does not detect Intel MKL blas/lapack when R is compiled to use that - it just returns character(0) for $blas and $lapack.

Looks like it should be an easy fix to detect one of the MKL libraries though:

This is using R 3.4.2-beta, Intel MKL 2015.6.233, and gcc 4.8.5.

(Here are the instructions I used to compile R with MKL on my cluster: https://github.com/ck37/savio-xsede-notes/blob/master/compile-R.md)

Cheers,
Chris

Matrix size in bm_matrix_cal_lm

In the code the matrix is of size 2000x2000 but the verbose result describes the matrix as being of size 3000x3000.

Also, for linear regression it is more common to have a rectangular model matrix with more rows than columns. You may want to use, say, 10000 x 1000.

package ‘benchmarkme’ is not available (for R version 3.4.4)

I am trying to install the benchmarkme package on the R that installs on Ubuntu 16.04 from the cran repository:

> install.packages("benchmarkme")
Installing package into ‘/home/alex/R/x86_64-pc-linux-gnu-library/3.4’
(as ‘lib’ is unspecified)
Warning message:
package ‘benchmarkme’ is not available (for R version 3.4.4)

I was able to install it on the R version (3.5.2) that comes from cran for Ubuntu 18.04.

Is the benchmarkme package supposed to install on R 3.4.4, or is 3.5.x required?
If it is, it should be clearly mentioned in the README.

Multicore/Multithread Performance

Hi,

Great package -- I am so glad someone has done this.

For some workflows it might be nice to have a sense of multicore/multithread performance.

Any plans to add that?

If it could be done it would be a great addition to the package. If not, might you accept a pull request if I can find time to do it?

Error in select_results(test_group, byte_optimize = byte_optimize, blas_optimize = blas_optimize) : unused argument (byte_optimize = byte_optimize)

Microsoft R Open 3.5.1
The enhanced R distribution from Microsoft
Microsoft packages Copyright (C) 2018 Microsoft Corporation

Using the Intel MKL for parallel mathematical computing (using 4 cores).

Default CRAN mirror snapshot taken on 2018-08-01.
See: https://mran.microsoft.com/.

upload_results(res)
Creating temporary file
Getting system specs. This can take a while on Macs
Uploading results
Upload complete
Tracking id: 2019-01-29-61350276
[1] "2019-01-29-61350276"
plot(res)
Error in select_results(test_group, byte_optimize = byte_optimize, blas_optimize = blas_optimize) :
unused argument (byte_optimize = byte_optimize)
sessionInfo()
R version 3.5.1 (2018-07-02)
Platform: x86_64-w64-mingw32/x64 (64-bit)
Running under: Windows >= 8 x64 (build 9200)

Matrix products: default

locale:
[1] LC_COLLATE=English_United States.1252 LC_CTYPE=English_United States.1252 LC_MONETARY=English_United States.1252
[4] LC_NUMERIC=C LC_TIME=English_United States.1252

attached base packages:
[1] stats graphics grDevices utils datasets methods base

other attached packages:
[1] benchmarkme_0.6.0 RevoUtils_11.0.1 RevoUtilsMath_11.0.0

loaded via a namespace (and not attached):
[1] Rcpp_1.0.0 benchmarkmeData_1.0.0 rstudioapi_0.9.0 bindr_0.1.1 magrittr_1.5 tidyselect_0.2.5
[7] doParallel_1.0.14 lattice_0.20-35 R6_2.3.0 rlang_0.3.1 foreach_1.4.4 httr_1.4.0
[13] dplyr_0.7.8 tools_3.5.1 parallel_3.5.1 grid_3.5.1 iterators_1.0.10 yaml_2.2.0
[19] assertthat_0.2.0 tibble_2.0.1 crayon_1.3.4 Matrix_1.2-14 bindrcpp_0.2.2 purrr_0.2.5
[25] codetools_0.2-15 curl_3.3 mime_0.6 glue_1.3.0 compiler_3.5.1 pillar_1.3.1
[31] pkgconfig_2.0.2