nikolailms/mastersthesis-experimental-data

mastersthesis-experimental-data

This repository serves as a documentation for the experiments described in the Master's Thesis "Pruning Techniques for Lifted SAT-Based Hierarchical Planning" (https://github.com/NikolaiLMS/lilotane).

Used Software

The softwares we evaluated are located here:

• Our modified Lilotane: https://github.com/NikolaiLMS/lilotane (forked from https://github.com/domschrei/lilotane)
• PandaPIParser: https://github.com/panda-planner-dev/pandaPIparser
• PandaPIGrounder: https://github.com/panda-planner-dev/pandaPIgrounder
• PandaPIEngine: https://github.com/panda-planner-dev/pandaPIengine
• HyperTensioN: https://github.com/Maumagnaguagno/HyperTensioN and https://gitlab.anu.edu.au/u1092535/ipc2020-competitor-4

Additionally we used the following software to help with test execution:

• Runwatch: https://github.com/domschrei/runwatch

Execution of Planners

• Hypertension was executed with the commands shown in runwatch_commands.txt, we did not use the the IPC optimizations as was later discovered and described in #1
• PandaGBFS was executed with the commands shown in runwatch_commands.txt and the execute_pandagbfs.sh script
• PandaSAT was executed with the commands shown in runwatch_commands.txt and the execute_pandasat.sh script

Content

• instances/ contains our training and test data, which consists of the total-order instances of the IPC 2020 (https://github.com/panda-planner-dev/ipc2020-domains/tree/master/total-order)

• evaluations/ contains the experimental data for the Parameter Evaluation, the Per Domain Evaluation and State of The Art Evaluation sections of the thesis, consisting of the logs for each instance, aswell as additional files that contain already computed metrics like IPC-Score, PAR2-Score, Coverage etc.

• figures/ contains our created plots using the evaluation data

• util/ contains multiple scripts we created to aid our experiments

mastersthesis-experimental-data's Issues

HyperTensioN configuration

Hello, I am the author of HyperTensioN, the planner you compared against.
I found odd that no Entertainment instances were solved in your tests, as I remember fixing a parsing bug after the IPC that made the first 5 instances easily solvable (less than a second) in an old machine, repeated the tests today and they remain solvable.

Upon inspection I noticed that you are using a newer version (based on the hash in the folder name) but not the same configuration used during the IPC.
The commands from runwatch_commands.txt lack the preprocessing stages triggered by parameters provided to Hype.
HyperTensioN was deployed in the IPC with ruby Hype.rb <domain.hddl> <problem.hddl> typredicate pullup dejavu run, and it appears you applied just run, which is the same as unoptimized.
You can see more about it here.

This is probably the main reason for your comment in 4.5.1: the difference for the named planners performance should still be noted and possibly investigated in future work.

I also find it odd that you have rigid/fluent precondition preprocessing stages to prune possible paths and never compared to HyperTensioN's Pullup approach.
Elements from Fact analysis are far too similar to Pullup from my perspective.

Also, Lavindra's name is all caps in your Thesis bibliography [16].