Intermittent Segmentation faults on Ubuntu-20.04 LTS on WSL2

[urschrei]

Intermittent Segmentation faults on Ubuntu-20.04 LTS on WSL2

[urschrei]

I won't be able to reproduce this under WSL2 as I don't have access to a windows machine, but I can try to reproduce it under Linux and / or macOS – I'll need an example dataset that causes the failure, though.

[urschrei]

Just as a matter of interest, are you using the (slower!) ctypes functions for a reason? Switching to

from convertbng.cutil import convert_bng causes the test to pass on my machine.

[urschrei]

No prob! I'm going to leave this open for now as there's clearly something wrong with my ctypes code.

[joshuanunn]

To follow up, I reproduced the dev environment (including Python 3.8.5) on a native on-metal Ubuntu-18.04 machine, and I still get the same Segfault using ConvertBNG 0.6.25, so doesn't appear related to WSL2 specifically. Note that the CPUs for both machines were i7 based.

I then switched to Python 3.6.9, which was already on the machine and installed everything in a venv again - in contrast, this worked fine. After switching back to Python 3.8.5 and experimenting, I can see that it is specifically fast repeated calls that seems to trigger it (or makes more likely).

Here is a minimum working example that consistently fails:

from convertbng.util import convert_bng
lots_of_conversions = [convert_bng([-1.89983], [52.48142]) for x in range(10000)]

I had to relax the library versions to get everything installed in Python 3.6.9 - the key difference in terms of requirements for convert_bng was that the numpy version (numpy==1.19.5 for Python 3.6.9 and numpy==1.21.2 for Python 3.8.5). However, doing a minimum install of numpy and convert_bng in Python 3.8.5 still segfaulted for both versions of numpy.

Hence, I'm guessing there's an issue in the latest convert_bng wheels for python 3.8?

Happy to provide more info if needed - thank you.

[urschrei]

Well, I can trigger a segfault with that data…

[joshuanunn]

Perfect, your suggested change also makes everything work fine again on my side for both the test and the processing scripts!

No reason at all - in fact speed in this application isn't too important, more just the library functionality. It just hadn't occurred to me to try, thank you for the suggestion!

[joshuanunn]

Great, I imagine this issue won't affect too many users anyway, as it's only likely to occur when making a lot of repeated smaller calls, rather than larger batches.

Thank you for your time looking at this earlier - it's much appreciated!

[tmladek]

Hi, I can confirm the intermittent segfaults on a completely different setup (native x86_64 Arch Linux, Python 3.10.9).

Just like joshuanunn, switching to cutil fixed this - but it was a real head scratcher until then!