Single-region runs started via config file report OUT_OF_MEMORY

[remindmodel]

Single-region runs started via config file report OUT_OF_MEMORY

[0UmfHxcvx5J7JoaOhFSs5mncnisTJJ6q]

I did some testing locally, but don't know who representative it is for things on the cluster and for calls from within start.R:

for s in reporting.R validation.R emulator.R reportCEScalib.R rds_report.R \
         validationSummary.R dashboard.R
do 
	/usr/bin/time --verbose --output=time_log.txt --append \
	Rscript ./scripts/output/single/$s outputdir=./output/testOneRegi/
done

$ grep -E "(Command|Maximum)" time_log.txt

        Command being timed: "Rscript ./scripts/output/single/reporting.R outputdir=./output/testOneRegi/"
        Maximum resident set size (kbytes): 1969544
        Command being timed: "Rscript ./scripts/output/single/validation.R outputdir=./output/testOneRegi/"
        Maximum resident set size (kbytes): 4322472
Command exited with non-zero status 1
        Command being timed: "Rscript ./scripts/output/single/emulator.R outputdir=./output/testOneRegi/"
        Maximum resident set size (kbytes): 342160
Command exited with non-zero status 1
        Command being timed: "Rscript ./scripts/output/single/reportCEScalib.R outputdir=./output/testOneRegi/"
        Maximum resident set size (kbytes): 319512
        Command being timed: "Rscript ./scripts/output/single/rds_report.R outputdir=./output/testOneRegi/"
        Maximum resident set size (kbytes): 415060
        Command being timed: "Rscript ./scripts/output/single/validationSummary.R outputdir=./output/testOneRegi/"
        Maximum resident set size (kbytes): 582184
        Command being timed: "Rscript ./scripts/output/single/dashboard.R outputdir=./output/testOneRegi/"
        Maximum resident set size (kbytes): 335196

So about 2 GB for the reporting and above 4 GB for the validation. Maybe it's enough to tell everybody to deactivate the validation when setting up a scenario_config.csv for testOneRegi?

[giannou]

if you can rule out any random cluster error (i.e. you can reproduce the error), try using more CPU's. testoneregi needs one to run GAMS but I think the R post-processing needs more total memory

[giannou]

which means that the user has to be advised by the interactive start scripts to use more than one CPU's even if they choose testoneregi

[Loisel]

What puzzles me is that runs started with --testOneRegi, even in the queues with one core, just finish reliably and report normally via email, see, e.g., p/tmp/aloisdir/remind/output/testOneRegi @giannou

[giannou]

was this run supposed to write a .mif file?

[Loisel]

Good point, no, this seems not to happen by default. And the runs with the parameter set in the config file executed on multiple nodes seem to finish normally. It looks like the suggested advise in the interactive start script as you suggested would thus solve the issue.

[dklein-pik]

Hi Alois, you closed this issue. Is there anything left to be changed in the code?

[Loisel]

Actually there is, see post by @giannou above:

the user has to be advised by the interactive start scripts to use more than one CPU's even if they choose testoneregi

I was not sure whether it is a good idea to keep this open or open a more specific issue. But in hindsight I closed it to early.

[0UmfHxcvx5J7JoaOhFSs5mncnisTJJ6q]

What puzzles me is that runs started with --testOneRegi, even in the queues with one core, just finish reliably and report normally via email, see, e.g., p/tmp/aloisdir/remind/output/testOneRegi

Because they don't produce any output, which probably is meaningless for testing/debugging runs anyhow.

remind/start.R

Lines 246 to 255 in 62e032b

	# testOneRegi settings
	if (testOneRegi) {
	cfg$title <- 'testOneRegi'
	cfg$gms$optimization <- 'testOneRegi'
	cfg$output <- NA
	cfg$results_folder <- 'output/testOneRegi'
	# delete existing Results directory
	cfg$force_replace <- TRUE
	}

So the advise should be specific to the output scripts to be run. No idea about their individual memory footprint, though.

remind/config/default.cfg

Line 436 in 62e032b

cfg$output <- c("reporting","validation","emulator","reportCEScalib","rds_report","validationSummary","dashboard")

[orichters]

Fixed by increasing the memory for slurmConfig with only one task-per-node in PR #829.