Non-deterministic results - certain groups of nodes always missing

[AnnabellaM]

Non-deterministic results - certain groups of nodes always missing

[AnnabellaM]

Hello Soot team, I'm following up on this issue and would greatly appreciate any insights you could provide. Thank you!

[AnnabellaM]

Hi team, any feedback would be really helpful! Thank you so much in advance!

[StevenArzt]

Sorry for the late reply, we're all quite busy at the moment.

Can you elaborate a bit on the problem? If I understand you correctly, you are using the SPARK callgraph algorithm (albeit with a weird command line - why is SPARK enabled twice?). There are some fairly non-standard options with on-the-fly CG being disabled and an explicit setting for pre-jimplification. I wonder whether you observe the same behavior when enabling SPARK but otherwise sticking to the default options.

Further, I'm not sure I understand the issue. You write that some nodes are "always missing together". Does this mean that these nodes are always missing? Or does it mean that sometimes all nodes of a group are there and sometimes all nodes in the group are missing? That would be the difference between inconsistent behavior and simply an incomplete CG.

[AnnabellaM]

Hi Steven, thank you so much for your time and valuable feedback!

We've conducted testing on 30 sampling configurations, encompassing 28 options, which were generated using the 2-way covering array methodology. Consequently, certain option combinations may appear somewhat non-standard. This non-deterministic behavior has been observed in a total of 20 configurations.

Because the original configurations we generated are quite extensive, each comprising 28 options. The configuration I used in this particular example was derived from an original configuration through bisection. Despite this refinement process, it still produces inconsistent results.

I wonder whether you observe the same behavior when enabling SPARK but otherwise sticking to the default options.

I have tried running the same test case while enabling SPARK, not disabling on-the-fly CG, but keeping all other settings at their default values. And the results I obtained with this configuration are deterministic.

Further, I'm not sure I understand the issue. You write that some nodes are "always missing together". Does this mean that these nodes are always missing? Or does it mean that sometimes all nodes of a group are there and sometimes all nodes in the group are missing?

My apologies for any confusion. The statement means that all nodes of a group sometimes are there and sometimes are missing, within a particular group, all nodes occasionally appear/disappear together.

That would be the difference between inconsistent behavior and simply an incomplete CG.

The analysis that led to these inconsistent results all finished successfully. We have excluded the timed-out runs.

@StevenArzt