See over test and their run time

[kronbichler]

See over test and their run time

[peterrum]

I have create the following bash script:

#!/usr/bin/env sh

# helper function: run one case and check result
function run_test() {
  echo -n "./$1 $2.prm"

  mpirun -np $3 ./$1 $2.prm > $2.screen; 

  cat $2.output | egrep -v '^\|' | egrep -v '^--' | sed 's/\[/\[ /g' | sed 's/\/[0-9]*\]/ \]/g' | sed 's/\// \//g' > $2.output.notime
  cat $2.screen | egrep -v '^\|' | egrep -v '^--' | sed 's/\[/\[ /g' | sed 's/\/[0-9]*\]/ \]/g' | sed 's/\// \//g' > $2.screen.notime

  numdiff -a 1e-5 -r 1e-8 -s ' \t\n:,' $2.output.notime $2.screen.notime | egrep -v '^$' | grep -v '+++'  > $2.diff

  if [ -s "$2.diff" ]
  then
    echo " has FAILED " 
    #echo "  cp $2.screen $2.output"
    #cp $2.screen $2.output
  else
    echo " has SUCCEEDED"
  fi
}

# helper function: run all cases for an application
function run_tests()
{
  for tt in $3
  do
    run_test "$2" $tt $1
  done
}

# default parameter
n_procs=8
cases="all"

# read input parameter
if [ "$1" ]
then
  n_procs=$1
fi

if [ "$2" ]
then
  cases=$2
fi

make -j30

for t in $cases
do
  if [ "$t" = "ns" ] || [ "$t" = "all" ]
  then
      run_tests $n_procs "beltrami"                  "beltrami_2d beltrami_2d_augp beltrami_2d_augp_proj beltrami_2d_proj beltrami_3d beltrami_3d_augp"
      run_tests $n_procs "flow_past_cylinder"        "flow_past_cylinder"
      run_tests $n_procs "flow_past_square_cylinder" "flow_past_square_cylinder"
      run_tests $n_procs "poiseuille"                "poiseuille_ns poiseuille_ns_proj poiseuille_stokes"
  fi

  if [ "$t" = "ls" ] || [ "$t" = "all" ]
  then
    run_tests $n_procs "rising_bubble" "rising_bubble_ls rising_bubble_ls_expl rising_bubble_ls_imex rising_bubble_ls_picard rising_bubble_ls_adap rising_bubble_ls_augp rising_bubble_ls_q3"

    run_tests $n_procs "spurious_currents" "spurious_currents_ls"
  fi
 
  if [ "$t" = "pf" ] || [ "$t" = "all" ]
  then
    run_tests $n_procs "rising_bubble"     "rising_bubble_pf"
    run_tests $n_procs "spurious_currents" "spurious_currents_pf"
  fi
 
  if [ "$t" = "simplex" ] || [ "$t" = "all" ]
  then
    run_tests 1 "simplex_channel" "simplex_channel"
  fi
done

which runs the specified applications (+cases) after each other (the criterion for success is more or less the same as in ctest). Each simulation is run in parallel and the applications are compiled in release mode. This is not an optimal approach but allows to run all applications in an acceptable time.

Regarding the brittle output: on what hardware, with which configuration have you run it? I have run the tests with SSE, AVX, and AVX-512 but non of them gave precisely the same results as specified in the provided output files. For now, I (re)generate the output files before making changes to the code.

[kronbichler]

The output depends on the version of Trilinos and deal.II and was some version a while ago (maybe 9.0 and Trliinos 12.2). The problem boils down to a different order in which roundoff errors are accumulated, leading to one iteration more or less, which is the main difference when I run the test locally. Especially I have seen Trilinos slightly change some things that lead to one more or less iteration. In a sense, the problem we see here is why deal.II has ranges of iterations displayed in some tests. We would need something similar here.

The right approach would be to be tolerant on iteration counts but more strict on other parts. At the same time, I do not want to completely ignore iteration counts because they tell a lot about the correctness in the linear solver. An immediate action to make is to manually go through a few systems and create multiple approved output files, similar to deal.II. That is of course not sustainable, but I have not found a good default solution without excessive scripting yet.