mageec / beebs Goto Github PK
View Code? Open in Web Editor NEWA set of benchmarks chosen to show the energy consumption of embedded devices under different conditions
License: GNU General Public License v3.0
A set of benchmarks chosen to show the energy consumption of embedded devices under different conditions
License: GNU General Public License v3.0
E.g. if REPEAT_FACTOR
is 4096, setting:
#define SCALE_FACTOR (REPEAT_FACTOR >> 4)
in a benchmark seems to have no effect - the benchmark()
function is still called 4096 times.
I let sum result printed on each iteration. And adjust REPEAT_FACTOR to 5.
Run dtoa.elf (compiled with -O2)
$ ./dtoa.elf
262390.355982 <= not expected result
262390.355982 <= not expected result
267945.911538
267945.911538
267945.911538
And its final result is correct.
Run dtoa.elf (compiled with -Os)
$ ./dtoa.elf
262390.355982 <= not expected result
262390.355982 <= not expected result
262390.355982 <= not expected result
262390.355982 <= not expected result
262390.355982 <= not expected result
And its final result is wrong. (the result got from every repeat are all wrong)
The main cause is "5555.5555555555555555" transfered to 0.0 on first and second repeat in compiled with -O2.
And always are 0.0 on every repeat in compiled with -Os.
In src/dtoa/libdtoa.c. local variable word0(&rv)
in function strtod doesn't be initilized well. It make getting different result on each repeat. If we add word0(&rv) = 0;
to intilize word0(&rv)
to 0. "5555.5555555555555555" are always transfererd to 0.0. So the result are not expected (compiled with Os or O2).
I am not sure the intilization for word0(&rv)
should be added. and "5555.5555555555555555" -> 0.0 is a reasonable value.
Or other issue made...
My gcc version:
Using built-in specs. COLLECT_GCC=gcc COLLECT_LTO_WRAPPER=/usr/libexec/gcc/x86_64-redhat-linux/11/lto-wrapper OFFLOAD_TARGET_NAMES=nvptx-none OFFLOAD_TARGET_DEFAULT=1 Target: x86_64-redhat-linux Configured with: ../configure --enable-bootstrap --enable-host-pie --enable-host-bind-now --enable-languages=c,c++,fortran,lto --prefix=/usr --mandir=/usr/share/man --infodir=/usr/share/info --with-bugurl=https://bugs.rockylinux.org/ --enable-shared --enable-threads=posix --enable-checking=release --enable-multilib --with-system-zlib --enable-__cxa_atexit --disable-libunwind-exceptions --enable-gnu-unique-object --enable-linker-build-id --with-gcc-major-version-only --with-linker-hash-style=gnu --enable-plugin --enable-initfini-array --without-isl --enable-offload-targets=nvptx-none --without-cuda-driver --enable-gnu-indirect-function --enable-cet --with-tune=generic --with-arch_64=x86-64-v2 --with-arch_32=x86-64 --build=x86_64-redhat-linux --with-build-config=bootstrap-lto --enable-link-serialization=1 Thread model: posix Supported LTO compression algorithms: zlib zstd gcc version 11.3.1 20220421 (Red Hat 11.3.1-2) (GCC)
Benchmark qsort
and select
use array length as index, causing global buffer overflow.
Buggy code: src/qsort/libqsort.c#107
float arr[20] = {
5, 4, 10.3, 1.1, 5.7, 100, 231, 111, 49.5, 99,
10, 150, 222.22, 101, 77, 44, 35, 20.54, 99.99, 88.88
};
...
void sort(unsigned long n)
{
unsigned long i,ir=n,j,k,l=1; // NOTE: ir is initialized as the length of arr (i.e., 20)
int jstack=0;
float a,temp;
for (;;) {
if (ir-l < M) {
...
} else {
k=(l+ir) >> 1;
SWAP(arr[k],arr[l+1])
if (arr[l] > arr[ir]) { // Line 107: arr[ir] causes global buffer overflow
SWAP(arr[l],arr[ir])
}
...
}
}
}
Buggy code: src/select/libselect.c#63
float arr[20] = {
5, 4, 10.3, 1.1, 5.7, 100, 231, 111, 49.5, 99,
10, 150, 222.22, 101, 77, 44, 35, 20.54, 99.99, 888.88
};
...
float select(unsigned long k, unsigned long n)
{
unsigned long i,ir,j,l,mid;
float a,temp;
int flag, flag2;
l=1;
ir=n; // NOTE: ir is initialized as the length of arr (i.e., 20)
flag = flag2 = 0;
while (!flag) {
if (ir <= l+1) {
...
} else if (!flag) {
mid=(l+ir) >> 1;
SWAP(arr[mid],arr[l+1])
if (arr[l+1] > arr[ir]) { // Line 63: arr[ir] causes global buffer overflow
SWAP(arr[l+1],arr[ir])
}
...
}
}
return arr[k];
}
The main
function is, or should be, the same in every single benchmark. We should really have a single main
in a common file somewhere that is built for every benchmark.
Would be nice to have an improved benchmark scaling mechanism.
Currently we have:
#define REPEAT_FACTOR ((BOARD_REPEAT_FACTOR) >> (CALIB_SCALE))
Where BOARD_REPEAT_FACTOR
is either defined in boardsupport.h
(per board) or is the constant 4096
. The CALIB_SCALE
is defined per benchmark, per board in a file calibration
within the boards config file, it will be a small number to shift by, if CALIB_SCALE
is not otherwise defined, it defaults to 0
.
Finally, in each benchmark we do:
#define SCALE_FACTOR (REPEAT_FACTOR >> NUMBER)
Where NUMBER
is an actual integer, specific to each benchmark.
It was suggested that we should replace this whole mechanism with a simpler scheme where on a per board / chip / benchmark basis we specify a single number for each benchmark. The details still need to be worked out a little more.
First reported by @LubomirBogdanov. Since #51 was merged the configure script no longer uses CFLAGS and other influential variables from the board and chip configuration, so the configuration it generates is based on passing no flags to the compiler, and linker, etc.
This will create a plethora of issues, but one concrete example is when using the ARM GNU Embedded toolchain (from https://developer.arm.com/open-source/gnu-toolchain/gnu-rm/downloads) to build for the Taiyo Yuden EYSGJN board:
$ ./configure --host=arm-none-eabi --with-chip=nrf51822 --with-board=ty_eysgjn
checking build system type... x86_64-unknown-linux-gnu
checking host system type... arm-none-eabi
checking target system type... arm-none-eabi
checking for a BSD-compatible install... /usr/bin/install -c
checking whether build environment is sane... yes
checking for arm-none-eabi-strip... arm-none-eabi-strip
checking for a thread-safe mkdir -p... /usr/bin/mkdir -p
checking for gawk... gawk
checking whether make sets $(MAKE)... yes
checking whether make supports nested variables... yes
checking whether to enable maintainer-specific portions of Makefiles... no
checking whether make supports nested variables... (cached) yes
checking how to print strings... printf
checking for style of include used by make... GNU
checking for arm-none-eabi-gcc... arm-none-eabi-gcc
checking whether the C compiler works... no
configure: error: in `/home/graham/tmp/beebs':
configure: error: C compiler cannot create executables
See `config.log' for more details
With the following in config.log:
configure:3832: checking whether the C compiler works
configure:3854: arm-none-eabi-gcc conftest.c >&5
/home/graham/tmp/gcc-arm-none-eabi-8-2018-q4-major/bin/../lib/gcc/arm-none-eabi/8.2.1/../../../../arm-none-eabi/bin/ld: /home/graham/tmp/gcc-arm-none-eabi-8-2018-q4-major/bin/../lib/gcc/arm-none-eabi/8.2.1/../../../../arm-none-eabi/lib/libc.a(lib_a-exit.o): in function `exit':
exit.c:(.text.exit+0x2c): undefined reference to `_exit'
collect2: error: ld returned 1 exit status
In this case, _exit
cannot be linked because the toolchain only includes Binutils and GCC, and no C library. Prior to the merge of #51, the test looked more like (from config.log again):
configure:3509: checking whether the C compiler works
configure:3531: arm-none-eabi-gcc --static -mcpu=cortex-m0 -mthumb -mabi=aapcs --std=gnu99 -Wall -mfloat-abi=soft -ffunction-sections -fdata-sections -fno-strict-aliasing -fno-builtin --short-enums -O0 -g -gdwarf-3 -gstrict-dwarf -T /home/graham/tmp/beebs/config/arm/chips/nrf51822/chip.ld -Wl,--start-group -mthumb -mabi=aapcs -mcpu=cortex-m0 -lc -lm -specs=nano.specs -specs=nosys.specs -nostartfiles -Wl,--end-group conftest.c >&5
I believe that this issue should be resolved by fixing the autotools setup to once again use variables from the board and chip configuration.
When running make check, the following errors occur:
ERROR: avrdude.exp: /home/george/beebs/build/src/ctl/ctl to be downloaded does not exist.
ERROR: avrdude.exp: /home/george/beebs/build/src/matmult/matmult to be downloaded does not exist.
ERROR: avrdude.exp: /home/george/beebs/build/src/sglib-arraysort/sglib-arraysort to be downloaded does not exist.
ERROR: avrdude.exp: /home/george/beebs/build/src/trio/trio to be downloaded does not exist.
The actual binaries in these directories do not get run. The files listed as not existing in the errors are still added to energy.csv, but are left to timeout.
Using GCC 4.9, -O2/-O3:
ctl-string
expint
sglib-dllist
sglib-listsort
sglib-rbtree
trio-sprintf
trio-sscanf
Using arm-none-eabi-gcc, -O2:
expint
frac
minver
rijndael
wikisort
Some of the benchmarks don't do much and should be deleted.
Candidates for deletion:
fibcall
gdb-*
lcdnum
prime
ns
Benchmarks folders that contain multiple entries in $bin_PROGRAMS (e.g. ctl) break the calibration system, as it tries to grep for the combined benchmarks name e.g. (ctl-vector ctl-stack 0).
Hello guys,
I've contributed to BEEBs a couple of years ago when there were only .o .axf (.elf) and .bin files. I was able to flash them and use openocd with my microcontrollers.
However at some point the build system changed and now there are .la .lo .o and .bin files. If I try to flash any of these files I see that my program is stuck at some random address 0xffff.ffa0 and GDB cant find any symbols even though I have -g flag.
Could you give me some kind of guidance, or a keyword, so that I can understand why you use .la and .lo files?
I'd like to contribute some more, because I have access to lots of boards at my university.
Regards,
L. B.
First reported by @jeremybennett when building for RISC-V:
$ make V=1 2>&1 | tee /tmp/1.log
...
/bin/bash ../libtool --tag=CC --mode=compile riscv32-unknown-elf-gcc -DHAVE_CONFIG_H -I. -I../../../beebs/support -I.. -g -O2 -O2 -march=rv32imc -fdata-sections -ffunction-sections -Wl,-gc-sections -MT dummy-libc.lo -MD -MP -MF .deps/dummy-libc.Tpo -c -o dummy-libc.lo ../../../beebs/support/dummy-libc.c
libtool: compile: riscv32-unknown-elf-gcc -DHAVE_CONFIG_H -I. -I../../../beebs/support -I.. -g -O2 -O2 -march=rv32imc -fdata-sections -ffunction-sections -Wl,-gc-sections -MT dummy-libc.lo -MD -MP -MF .deps/dummy-libc.Tpo -c ../../../beebs/support/dummy-libc.c -o dummy-libc.o
In file included from ../../../beebs/support/dummy-libc.c:29:
../../../beebs/support/dummy-libc.c:248:1: error: expected identifier or '(' before 'int'
ferror (FILE *stream __attribute__ ((unused)))
^~~~~~
../../../beebs/support/dummy-libc.c:248:1: error: expected ')' before '(' token
ferror (FILE *stream __attribute__ ((unused)))
^~~~~~
Makefile:500: recipe for target 'dummy-libc.lo' failed
make[2]: *** [dummy-libc.lo] Error 1
and then by @LubomirBogdanov when building for the Taiyo Yuden EYSGJN board:
I'm investigating a problem where to configure BEEBS I have to do
./configure CFLAGS=--nosys.specs --host=arm-none-eabi --with-chip=nrf51822 --with-board=ty_eysgjn
but then when I try to make it, it fails as you've added a replacement (I think) for nosys.specs. Then I decided to remove this flag and add:
./configure CFLAGS=-nostdlib --host=arm-none-eabi --with-chip=nrf51822 --with-board=ty_eysgjn
which configures and builds successfully, but the linking now is wrong and I get empty object files. I will try to see what's the problem further.
However one thing is sure - I always get:
dummy-libc.c:248:1: error: expected identifier or '(' before 'int'
ferror (FILE *stream __attribute__ ((unused)))
^~~~~~
dummy-libc.c:248:1: error: expected ')' before '(' token
ferror (FILE *stream __attribute__ ((unused)))
until I comment this empty function out. This looks mysterious as all the other functions, having the same attribute parameters, pass.
The arguments to tolower, toupper and isxdigit should be unsigned char or int, not signed char. This causes several benchmarks to fail to build on targets with newlib as part of the toolchain
ctl-string
slre
Found by googling: http://www.cygwin.com/ml/cygwin-patches/2011-q4/msg00038.html
$ make --debug --directory=src/cubic
...
File 'all' does not exist.
File 'all-am' does not exist.
File 'cubic' does not exist.
File '-lm' does not exist.
Must remake target '-lm'.
make: *** No rule to make target '-lm', needed by 'cubic'. Stop.
Why can make not find -lm
?
I saw something related to VPATH
in this issue?
Whetstone and dhrystone are still missing triggers, thus they can't be used for energy measurement. Oddly, adding the calls to the relevant support functions don't appear to work.
The two trio benchmarks fail to compile for AVR (shrimping kit) due to the following errors:
../../../src/trio-snprintf/trio.c:477:33: error: integer overflow in expression [-Werror=overflow]
../../../src/trio-sscanf/trio.c:477:33: error: integer overflow in expression [-Werror=overflow]
The large amount of stack space needed by the self verify functionality of this benchmark means the program silently fails.
support/chip.c
should include chipsupport.c
from the board directory, because it contains:
#ifdef HAVE_CHIPSUPPORT_H
#include "chipsupport.c"
#endif
However, the way it is invoked doesn't define HAVE_CHIPSUPPORT_H
, e.g.:
arm-none-eabi-gcc -DHAVE_CONFIG_H -I. -I.. -I../config/arm/boards/ty_eysgjn -I../config/arm/chips/nrf51822 --static -mcpu=cortex-m0 -mthumb -mabi=aapcs --std=gnu99 -Wall -mfloat-abi=soft -ffunction-sections -fdata-sections -fno-strict-aliasing -fno-builtin --short-enums -O0 -g -gdwarf-3 -gstrict-dwarf -MT libsupport_la-chip.lo -MD -MP -MF .deps/libsupport_la-chip.Tpo -c chip.c -o libsupport_la-chip.o
and it doesn't include anything else, so chipsupport.h is never included.
This breaks any chip that uses chipsupport.c.
This might result in different results (and failing the verify_benchmark() check) on different platforms.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.