Define ForwardDiffNumber <: Real

[juliadiff]

Define ForwardDiffNumber <: Real

[mlubin]

I'm not opposed. Worst case we reverse the decision later if it turns out that this is a bad idea, but so far I don't know of any examples where this would be dangerous or harmful.

[goretkin]

I am in strong favor of this change! I (unfortunately) haven't followed closely the development of ForwardDiff, but I think that to be consistent with this change, we need the definition like real(x::ForwardDiffNumber)=x, especially if one day ForwardDiffNumber is to support taking directional derivatives of complex functions. I gave this a little bit of thought a while ago: https://github.com/goretkin/ComplexDualNumbers.jl

[jrevels]

Just FYI, I'm working on this here and man...the number of ambiguity warnings this causes is bonkers. Unsurprising, but bonkers.

It might take a while before this is ready for a proper PR.

[papamarkou]

Hi @jrevels, I reply here as you suggested. I can see the merits of defining dual numbers as a special case of real, and it makes sense in many ways indeed. I just want to be sure we don't move towards this direction lightheartedly and make sure we have weighted the pros and cons... for example, nested types should be ok, I agree. How about the real(x::ForwardDiffNumber)=x definition, are we all happy about this too? I think @goretkin is right in that we should have a differently named function for accessing the "real" component of a dual, as now real will mean the actual whole dual.

I am not against the overall change, just to be clear.

[jrevels]

I think @goretkin is right in that we should have a differently named function for accessing the "real" component of a dual, as now real will mean the actual whole dual.

The method currently used for this is ForwarDiff.value.

[goretkin]

To clarify, the comment that @scidom is referring to I made in the context of a discussion that I thought was about Dual<:Real.

@jrevels, do you agree that if T<:Real, then the method real should be defined as real(x::T)=x?

EDIT: I guess this will actually already be true because of https://github.com/JuliaLang/julia/blob/f8a4340548e7d6be31fede13cdc4e0f5f434f33f/base/complex.jl#L40, as long as we don't define a more specialized version of it.

[papamarkou]

Ok, value sounds good - in this case we don't really need to worry about defining real() at all I think.

[papamarkou]

Oups, @goretkin, we posted simultaneously. Your suggestion for real() is ok, but may I ask, why do we need it?

About duals I agree... if we make forward diff numbers subtypes of real then same should be done for duals :)

[jrevels]

If ForwardDiffNumber <: Real, then the existing method real(::Real) will already work.

About duals I agree... if we make forward diff numbers subtypes of real then same should be done for duals

I don't agree with this, if you're talking about making DualNumbers.Dual <: Real. That should be its own separate issue, in any case.

Just to avoid confusion - especially for any lurkers who might be concerned about this issue - here is a summary of my key points (some of which were posted in the other thread).

• ForwardDiff.jl does not implement dual numbers, it implements ForwardDiffNumbers. ForwardDiffNumbers are constructions of dual/hyper-dual number "ensembles", whose sole reason for existence is automatic differentiation. The algebraic properties of the ForwardDiffNumber type are only relevant to the extent that these properties apply to practical AD of Julia code.
• DualNumbers.jl implements algebraic dual numbers as the Dual type. The Dual type should conform to the algebraic properties of mathematical dual numbers as much as implementation allows. Mathematical dual numbers are not real numbers, and do not even constitute a field due to the presence of zero divisors, and it is on this basis that I claim that Dual <: Real is a bad idea.
• ForwardDiffNumber <: Real is a good idea because it will resolve technical issues with AD in Julia, and is programmatically correct in that all ForwardDiffNumber instances should behave as drop-in replacements for Real instances in order to support forward-mode AD.

[papamarkou]

I am not sure how you have created this impression about DualNumbers.jl :) The sole reason of existence of DualNumbers.jl has been to support forward AD in the simple case of hyper-duals with a single epsilon component. In fact, the only usage case of DualNumbers.jl you will find online is that of AD, and none computer algebra-oriented. If we decide to invoke algebraic arguments as more important than programming ones, then there is absolutely no reason to apply this only on DualNumbers.jl given the way the package is utilised.

All in all, we are talking about the exact same concept, hyper-duals (and duals are a subset of this), so it makes no sense to say that one package should use duals as a subset of real and another shouldn't; at the very least, there should be a level of consistency between DualNumbers.jl and ForwardDiff.jl. Indeed the only line of counter-argument is that of use case, but there it is clear from their use in the wider community that both packages serve the very same purpose, that of AD.

I am ok to go for (hyper)duals turning to subsets of reals after having heard some convincing arguments in favour for it, but clearly can accept this change as long as it happens as a whole to any (hyper)dual numbers used for AD in Julia, including DualNumbers.jl.

[goretkin]

I also am not convinced that ForwardDiffNumber{T<:Real}<:Real should imply Dual<:Real.

@jrevels made a comment about nested ForwardDiffNumbers working. As far as I understand, it's not possible to make the DualNumbers implementation handle nested derivatives, because you cannot distinguish between the different epsilon parts. This is also described here: http://www.bcl.hamilton.ie/~barak/papers/HOSC-forward-nesting.pdf

I think that's further evidence that DualNumbers.Dual<:Real might be a bad idea.

[jrevels]

I am not sure how you have created this impression about DualNumbers.jl :) The sole reason of existence of DualNumbers.jl has been to support forward AD in the simple case of hyper-duals with a single epsilon component.

My points were more of a proposal for the future rather than an interpretation of the current packages

[jrevels]

I just wanted to point out that Dual numbers do more than strictly AD in the space of practical numerics outside the realm

Exactly (also you might be interested in https://github.com/dpsanders/ValidatedNumerics.jl). I think this point favors the separation of implementation of ForwardDiff.jl and DualNumbers.jl. There is no formal dependency between DualNumbers.jl and ForwardDiff.jl, just an overlap of the functionality they provide.

Thus, my belief is that the DualNumbers.Dual <: Real issue is totally separate from the ForwardDiff.ForwardDiffNumber <: Real issue.

[goretkin]

(re-add deleted comment since it's quoted. I had deleted it because I panicked about not finding a reference to Dual numbers as intervals and forgetting exactly how they work.)

I hope this comment doesn't distract from the main discussion, but I want to mention that you can also make Dual numbers to work as interval arithmetic (where instead of maintaining upper and lower bounds, the non-epsilon part is the center of the interval, and the epsilon part is the radius). I think it is nifty that the same algebraic properties gives you both an "Interval Number" and a "Forward Diff" number of order 1. I don't know if a hyperdual number behaves in any other interesting ways, but I just wanted to point out that Dual numbers do more than strictly AD in the space of practical numerics outside the realm.

What that's an argument for, I'm not totally sure. But this same discussion probably would happen again regarding Interval arithmetic. Except that AD/differentiation is used a ton ton more than interval arithmetic.

[papamarkou]

Thanks for pointing this out about interval arithmetic @goretkin, wasn't aware of such use of dual numbers.

@jrevels, ok, wasn't aware that you were mostly projecting to the future. I wouldn't have any serious objections to not make duals a subtype of reals, if you could convince me that there is a useful future direction. Apart from the internals of implementation, what are the benefits of not turning duals to be a subtype of reals? If I understand well, the one example mentioned so far is that of interval arithmetic, correct?

[mlubin]

Could we limit the discussion here to ForwardDiffNumber <: Real and move Dual <: Real to an issue at DualNumbers. jl?

[papamarkou]

But these two are (at the very least mathematically and thus conceptually) highly overlapping, so a high level discussion about their overall direction in association with their similarities and differences is needed in my opinion...

[mlubin]

But is anyone opposed to ForwardDiffNumber <: Real? I'm trying to ensure monotone progress in the discussion. As @jrevels argued, this lets us apply AD to any function defined over reals. There are no examples of this change producing undesirable results. Implementing the change in ForwardDiff will take some effort to resolve ambiguity warnings and possibly other issues, let's keep the discussion here focused on getting this working.

[papamarkou]

Ok, @mlubin, let's keep the momentum and our focus on getting ForwardDiffNumber <: Real up and running, as you pointed. I have had some very minor resistance still till today, but ok, I think the argument of Stefan convinced me, let's just consider the numbers in ForwardDiff to be subtypes of Real.

I think we should have a discussion about the future of DualNumbers in relation to AD and interval arithmetic; if anything else, it will clarify matters and possibly future development (it may even inform package naming for DualNumbers). To avoid distractions, we can discuss this in DualNumbers.jl, I can open a separate issue there.