External function derivative detection in CasADi 3.6.x

[PeterMeisrimelModelon]

External function derivative detection in CasADi 3.6.x

[jaeandersson]

The error is in the second derivative. It generates a first order derivative function (fwd1_EXTERNAL_FUNCTION) using forward mode AD or or finite differencing. But then fails to differentiate fwd1_EXTERNAL_FUNCTION.

[modelonrobinandersson]

Hi @jaeandersson thank you for your quick reply. Do you have any suggestions for how we could debug this further? Especially since this was working with CasADi 3.5.5. Is there anything you are aware of from the previous version that:

1. changes how the derivative function is generated, or
2. changes how the differentiation is done?

[jaeandersson]

Not off the top of my head. You could look exactly at what kind of derivative functions were generated in 3.5.5 to figure out when it starts diverging from 3.6. Once you've figured out why it's behaving different, you can think of a way to resolve the issue. In any case, second derivatives using FD only is not really recommended, so I'm not even so sure it really worked for you in 3.5.5 either.

[PeterMeisrimelModelon]

Thanks for the input, maybe some clarifications:

1. Both EXTERNAL_FUNCTION and fwd1_EXTERNAL_FUNCTION are external functions, fwd1_EXTERNAL_FUNCTION is not generated via forward AD.
2. The expected behavior here is that by explicitly setting opts["enable_fd"] = true, the second derivative will be calculated using finite differences on fwd1_EXTERNAL_FUNCTION. We are aware that finite differences are not really the best option, but due to the nature of the external functions, it is the only option.
3. We could not find any changes in the ReleaseNotes resp. diffs since 3.5.5 with respect to enable_fd, yet the assert implies that the explicit opts["enable_fd"] = true has been discarded. The offending assert not being a new addition either.
4. Also for completeness, the error is generated when (trying) to create the NLP solver.

/Peter

[modelonrobinandersson]

Hi @jaeandersson. After some additional debugging we believe the error we are observing is due to the changes in #2848, among them I see that some options (in particular enable_fd) are no longer propagated due to the added if-statement (6b7dc1d#diff-3f30bada163de7ee828d1f3dd0fe431e8233b97aef17d841b8498f1df2507415R385).

Could you comment on the following:

1. Is it intentional to no longer propagate enable_fd?
2. If yes, do you have any proposal for what we can change such that the second derivative gets calculated using fd?

[jaeandersson]

1. Is it intentional to no longer propagate enable_fd?

Yes, it's intentional. In your case, you don't want to enable FD for the function itself, but for its derivative.

2. If yes, do you have any proposal for what we can change such that the second derivative gets calculated using fd?

You should be able to pass enable_fd to to the first order derivative function directly. Try to set

opts["forward_options"] = dict(enable_fd = True)

[PeterMeisrimelModelon]

1. Is it intentional to no longer propagate enable_fd?

Yes, it's intentional. In your case, you don't want to enable FD for the function itself, but for its derivative.

2. If yes, do you have any proposal for what we can change such that the second derivative gets calculated using fd?

You should be able to pass enable_fd to to the first order derivative function directly. Try to set

opts["forward_options"] = dict(enable_fd = True)

We could resolve the issue on our side by additionally adding enable_fd as explicit forward option as you suggested.

Thanks for the help!