Geometry fit uncertainty

[rbooth200]

Geometry fit uncertainty

[jeffjennings]

Just looking back at this, I've found with mock data in the last couple days that the uncertainty on the inc, PA and phase center as returned by the least squares optimizer in FitGeometryGaussian and FitGeometryFourierBessel can grossly underestimate the error (the true values being >> 3 sigma from the fitted values). Maybe not surprising, but just to note.

[jeffjennings]

A prior flat in cos(inc) is just prior = 1 / inc unless I'm mistaken.

[rbooth200]

HI Jeff, thanks for this. Do you know whether the very large errors are associated with the fit getting stuck in a local minimum?

[jeffjennings]

I don't suspect so, only because the tendency to underestimate the errors is consistent across several initializations of a guess for the geometry (though all for the same mock dataset). But I could be wrong, and I haven't looked into this exhaustively.

I've also varied diff_step in least_squares (https://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.least_squares.html) from 1e-3 to 1e2, which hasn't altered the results.

There might be some unique quirk to this dataset that makes the geometry fit difficult, though I don't see what that would be. I'm generating mock visibilities using galario and using estimate_weights though, so there could be potentially be contributions to the error upstream of the actual geometry fit. I added a couple plots to slack to show.

[rbooth200]

It might be, but I suspect that the errors are just bad. That's sort of expected though

[jeffjennings]

Hey @rbooth200 do you want to apply the prior in cos(inc) to the geometry fitting routines? I think it should just be prior = 1 / inc.

In #164 I added a note to the docs about how the geometry fit won't be accurate for a disc that's highly non-Gaussian, like one with a cavity. For an uncertainty estimate on ~Gaussian discs, I can check if draw_bootstrap_sample will give a realistic uncertainty estimate. Shouldn't we also be able to get an estimate straight from FitGeometryGaussian and FitGeometryFourierBesselthough, since they both use least_squares?

[rbooth200]

Can you check whether the bootstrap gives a reasonable uncertainty estimate?

However, I wonder if the best thing to do might be to note that these automated procedures are not perfect. We could edit the docs to discuss this better, suggest typical uncertainties and point to the appropriate papers that discuss it instead.