UBSan `-fsanitize=function` reports issue with `E3ClassInfo` vs `E3Root`

[seanm]

UBSan `-fsanitize=function` reports issue with `E3ClassInfo` vs `E3Root`

[jwwalker]

E3Root is a subclass of E3ClassInfo, so even though inClass is typed as E3ClassInfo *, it could legitimately be a pointer to E3Root.

When I use your value of QUESA_EXTRA_DEBUG_C, I'm getting an access violation, not a UBSAN message, so that's a puzzle.

[seanm]

Wouldn't lldb print it's actual type though?

Yes, an access violation is the expected result with the flags I gave. It basically crashes to indicate an undefined behaviour has occurred.

[jwwalker]

No, I don't think lldb would know the real type. There are no virtual methods in these classes, so C++ can't do runtime type identification.

[jwwalker]

If I just say QUESA_EXTRA_DEBUG_C = -fsanitize=undefined, I get a more understandable error message. I still haven't figured out why the function pointer type isn't essentially the same. The pointer type is TQ3Status (*)(OpaqueTQ3Object *, void *, void *)' and the actual function has type TQ3Status (*)( E3Shared*, void*, void* ), where E3Shared is a subclass of OpaqueTQ3Object. I'll keep playing with it.

[jwwalker]

The UBSAN message is Call to function e3shader_new(OpaqueTQ3Object*, void*, void const*) through pointer to incorrect function type 'TQ3Status (*)(OpaqueTQ3Object *, void *, void *)'. But when I look at e3shader_new, there's no const! Very confusing.

[jwwalker]

I also note that if I turn on the Undefined Behavior Sanitizer using Edit Scheme, then no problem is detected.

[seanm]

but e3shader_new does have a const:

Quesa/Development/Source/Core/System/E3Shader.cpp

Line 160 in 74061f8

e3shader_new(TQ3Object theObject, void *privateData, const void *paramData)

Perhaps the checkbox in edit scheme only enables a subset of UBSan features?

[jwwalker]

When I try it again, it says Call to function e3shared_new through pointer to incorrect function type 'TQ3Status (*)(OpaqueTQ3Object *, void *, void *)'. Not sure how I managed to change e3shared_new to e3shader_new. Looks like most blah_new functions do have a const on the 3rd parameter, but e3shared_new and e3root_new are exceptions. If I continue running, I get a similar warning in E3Group::createPosition: Call to function e3group_positionnew(TQ3XGroupPosition**, OpaqueTQ3Object*, void const*) through pointer to incorrect function type 'TQ3Status (*)(void *, OpaqueTQ3Object *, const void *)'. In that case, as far as I know void const* is the same as const void *, so it must be complaining about the first parameter. I'll look into it some more.

[jwwalker]

I pushed changes to fix a slew of these warnings. There are still a few that come up if you use a Mesh geometry, but the Mesh implementation is so arcane that I can't figure out how to fix it.

[seanm]

Thanks. I confirm this fixes the case I gave when creating this ticket.

Alas, even after all your changes, I still get 9 UBSan errors in my own 3dmf unit tests, all about function pointer mismatches. I took a fast look, but as you say it's hard to untangle.

[jwwalker]

Do the errors (warnings, really) you still get involve Mesh geometries? I suppose all the fancy macro use may date back to someone trying to do what C++ templates can do, but without a good C++ compiler. That stuff should be rewritten, but I never use Meshes, so it's never been a priority for me.

[seanm]

Actually, error is the better word, since undefined behaviour is, well, undefined, and thus a bug. But in practice, the compiler ends up doing something that works I guess.

Yeah, it does look a lot like C code trying to implement some kind of class hierarchy / template stuff.

Here's the entire list:

Source/Core/System/E3IO.cpp: runtime: Undefined Behavior: Call to function e3fformat_3dmf_bin_get_formattype(E3File*) through pointer to incorrect function type 'unsigned int (*)(OpaqueTQ3Object *)'

Source/Core/System/E3IO.cpp: runtime: Undefined Behavior: Call to function e3fformat_3dmf_bin_get_nexttype(E3File*) through pointer to incorrect function type 'int (*)(OpaqueTQ3Object *)'

Source/Core/System/E3IO.cpp: runtime: Undefined Behavior: Call to function e3fformat_3dmf_bin_is_end_of_container(E3File*) through pointer to incorrect function type 'TQ3Boolean (*)(OpaqueTQ3Object *)'

Source/Core/System/E3IO.cpp: runtime: Undefined Behavior: Call to function e3fformat_3dmf_bin_read_header(E3File*) through pointer to incorrect function type 'TQ3Status (*)(OpaqueTQ3Object *)'

Source/Core/System/E3IO.cpp: runtime: Undefined Behavior: Call to function e3fformat_3dmf_bin_readobject(E3File*) through pointer to incorrect function type 'OpaqueTQ3Object ()(OpaqueTQ3Object *)'

Source/Core/System/E3IO.cpp: runtime: Undefined Behavior: Call to function e3storage_path_close through pointer to incorrect function type 'TQ3Status (*)(OpaqueTQ3Object *)'

Source/Core/System/E3IO.cpp: runtime: Undefined Behavior: Call to function e3storage_path_open through pointer to incorrect function type 'TQ3Status (*)(OpaqueTQ3Object *, TQ3Boolean)'

Source/Core/System/E3IOData.cpp: runtime: Undefined Behavior: Call to function E3FileFormat_GenericReadBinary_32 through pointer to incorrect function type 'TQ3Status (*)(OpaqueTQ3Object *, float *)'

Source/Core/System/E3IOData.cpp: runtime: Undefined Behavior: Call to function E3FileFormat_GenericReadBinaryArray_32 through pointer to incorrect function type 'TQ3Status (*)(OpaqueTQ3Object *, unsigned int, float *)'

Source/Core/System/E3Storage.cpp: runtime: Undefined Behavior: Call to function e3storage_path_getsize through pointer to incorrect function type 'TQ3Status (*)(OpaqueTQ3Object *, unsigned int *)'

Source/FileFormats/E3IOFileFormat.cpp: runtime: Undefined Behavior: Call to function e3storage_path_read through pointer to incorrect function type 'TQ3Status (*)(OpaqueTQ3Object *, unsigned int, unsigned int, unsigned char *, unsigned int *)'

Source/FileFormats/Readers/3DMF/E3FFR_3DMF_Bin.cpp: runtime: Undefined Behavior: Call to function e3fformat_3dmf_attributearray_read(E3File*) through pointer to incorrect function type 'OpaqueTQ3Object ()(OpaqueTQ3Object *)'

Source/FileFormats/Readers/3DMF/E3FFR_3DMF_Bin.cpp: runtime: Undefined Behavior: Call to function e3storage_path_read through pointer to incorrect function type 'TQ3Status (*)(OpaqueTQ3Object *, unsigned int, unsigned int, unsigned char *, unsigned int *)'

Source/FileFormats/Readers/3DMF/E3FFR_3DMF.cpp: runtime: Undefined Behavior: Call to function e3read_3dmf_bin_readnextelement(OpaqueTQ3Object*, E3File*) through pointer to incorrect function type 'void (*)(OpaqueTQ3Object *, OpaqueTQ3Object *)'

[jwwalker]

OK, it's not meshes, it's 3dmf reading. I'll try to get to that some time.

[seanm]

Cool, thanks. All we use Quesa for anymore is reading our application's old file format, which has .3dmf files embedded in the bundle.

[jwwalker]

I fixed another bunch of these issues.

[seanm]

Thanks, that seemed to fix all the ones that affect me!