ctest tests report Passed yet they are actually failing

[luxonis]

ctest tests report Passed yet they are actually failing

[themarpe]

Retested the without USB on Windows - issue stems from 'Debug' configuration halting the execution because uncaught exception is raised. 'Release' configuration terminates before timeout and tests fail.

WRT your previous point - I did get some cases of #79 - but haven't been able to pin it down yet.
Never had a case where all 18 cases wouldn't show UI. Some in between yes, but not all of them.

So I think its a combination of both - Device getting into odd USB state, where it cannot be found by OS, therefore this latter observation comes into play as you are testing 'Debug' configuration.

Will add exception handling for examples, to both solve this and show first hand that some functions may throw.

[diablodale]

@themarpe I recommend we close this issue. There are other issues (and fixes) that are more specific and actionable.
🤔

[themarpe]

@diablodale Can you try increasing timeout using TEST_TIMEOUT environment variable to eg. 10 seconds.

As tests are executed and then sent a termination signal, sometimes UI doesn't have enough time to show it yet.
A test of example can still be successful even if UI wasn't presented yet.

[themarpe]

Correct, testing the examples doesn't actually test UI visualization but rather error messages, exceptions or device misconfiguration (pipeline, etc...).

If UI doesn't show up even with larger timeout, then something else might be wrong (and test still saying passed), otherwise the tests are deemed successful.

[diablodale]

I can change that env var. What are you asking me to test?

The issue I'm reporting is multiple tests failed yet the test report was Passed. Have all the tests been designed to not need a UI. And therefore, me not seeing the UI with my eyes is not an indication that some part of the tests failed?
If yes, then ok...this issue is not an issue. I'll close it.
If no, then the test cases/harness have a bug that are reporting "passed" yet they actually failed.

[diablodale]

I suspect it is the latter. Something in test#4 caused the OAK-D and/or USB to be in a state that caused all the following tests to silently fail. I acknowledge that it is possible that my laptop was busy doing something else that it could not show UI within 5 seconds. But, I doubt it because...

• it repro'd on an 8-core intel, 32gb ram, rtx2070 super, two month old laptop that was running VSCode, chrome, and running the test case. Massive CPU and memory unused.
• happened with 18 tests...and only the 18 tests after the one that failed. It wasn't a random fail here and there which is more typical of timing issues. It is mostly impossible for my laptop to be 100% cpu bound for (18*5 sec) of this test.
• ..and I saw this whole OP scenario more than once. Its just the last time that I recognized the pattern and captured it for this issue.

I have no data other than what I collected and the repro steps. I will definitely be watching for similar behavior again when I get to heavy development in H2 this year.

Did you try my repo steps yourself? Multiple runs of the the extended ctest suite? And all of those runs reported passed and showed UI?

[themarpe]

Possible although usually device being in an unconnectable state would fail one way or another.

Did you hear autofocus clicks between those 18 test runs?

Will also give this a test on my end now.

[diablodale]

I setup, build debug, and run ctest as in the OP. However, I forgot 🍀 to connect the OAK-D into the usb3 port.
The result was 18 tests "Passed" and 4 tests "Failed". 😆
All should fail.

I think this is another method to expose something errant in the way the test cases/harness is evaluating pass/fail. The OP exposed it in test#4. Today, I exposed it by not connecting the usb3 cable.

[proc] Executing command: ctest -j1 -C Debug -T test --output-on-failure
[ctest]    Site: DALE-XMG
[ctest]    Build name: Win32-ninja
[ctest] Test project C:/njs/depthai-core/build
[ctest]       Start  1: color_camera_node_test
[kit] OK running C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Auxiliary\Build\vcvarsall.bat amd64 10.0.19041.0, env vars: [["CL",""],["_CL_",""],["INCLUDE","C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\Community\\VC\\Tools\\MSVC\\14.28.29910\\ATLMFC\\include;C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\Community\\VC\\Tools\\MSVC\\14.28.29910\\include;C:\\Program Files (x86)\\Windows Kits\\10\\include\\10.0.19041.0\\ucrt;C:\\Program Files (x86)\\Windows Kits\\10\\include\\10.0.19041.0\\shared;C:\\Program Files (x86)\\Windows Kits\\10\\include\\10.0.19041.0\\um;C:\\Program Files (x86)\\Windows Kits\\10\\include\\10.0.19041.0\\winrt;C:\\Program Files (x86)\\Windows Kits\\10\\include\\10.0.19041.0\\cppwinrt"],["LIBPATH","C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\Community\\VC\\Tools\\MSVC\\14.28.29910\\ATLMFC\\lib\\x64;C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\Community\\VC\\Tools\\MSVC\\14.28.29910\\lib\\x64;C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\Community\\VC\\Tools\\MSVC\\14.28.29910\\lib\\x86\\store\\references;C:\\Program Files (x86)\\Windows Kits\\10\\UnionMetadata\\10.0.19041.0;C:\\Program Files (x86)\\Windows Kits\\10\\References\\10.0.19041.0;C:\\Windows\\Microsoft.NET\\Framework64\\v4.0.30319"],["LINK",""],["_LINK_",""],["LIB","C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\Community\\VC\\Tools\\MSVC\\14.28.29910\\ATLMFC\\lib\\x64;C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\Community\\VC\\Tools\\MSVC\\14.28.29910\\lib\\x64;C:\\Program Files (x86)\\Windows Kits\\10\\lib\\10.0.19041.0\\ucrt\\x64;C:\\Program Files (x86)\\Windows Kits\\10\\lib\\10.0.19041.0\\um\\x64"],["PATH","C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\Community\\Common7\\IDE\\\\Extensions\\Microsoft\\IntelliCode\\CLI;C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\Community\\VC\\Tools\\MSVC\\14.28.29910\\bin\\HostX64\\x64;C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\Community\\Common7\\IDE\\VC\\VCPackages;C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\Community\\Common7\\IDE\\CommonExtensions\\Microsoft\\TestWindow;C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\Community\\Common7\\IDE\\CommonExtensions\\Microsoft\\TeamFoundation\\Team Explorer;C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\Community\\MSBuild\\Current\\bin\\Roslyn;C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\Community\\Team Tools\\Performance Tools\\x64;C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\Community\\Team Tools\\Performance Tools;C:\\Program Files (x86)\\Microsoft Visual Studio\\Shared\\Common\\VSPerfCollectionTools\\vs2019\\\\x64;C:\\Program Files (x86)\\Microsoft Visual Studio\\Shared\\Common\\VSPerfCollectionTools\\vs2019\\;C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\Community\\Common7\\Tools\\devinit;C:\\Program Files (x86)\\Windows Kits\\10\\bin\\10.0.19041.0\\x64;C:\\Program Files (x86)\\Windows Kits\\10\\bin\\x64;C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\Community\\\\MSBuild\\Current\\Bin;C:\\Windows\\Microsoft.NET\\Framework64\\v4.0.30319;C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\Community\\Common7\\IDE\\;C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\Community\\Common7\\Tools\\;C:\\Windows\\system32;C:\\Windows;C:\\Windows\\System32\\Wbem;C:\\Windows\\System32\\WindowsPowerShell\\v1.0\\;C:\\Windows\\System32\\OpenSSH\\;C:\\Program Files\\Microsoft VS Code\\bin;C:\\Strawberry\\c\\bin;C:\\Strawberry\\perl\\site\\bin;C:\\Strawberry\\perl\\bin;C:\\Program Files\\NVIDIA GPU Computing Toolkit\\CUDA\\v10.1\\bin;C:\\Program Files\\NVIDIA GPU Computing Toolkit\\CUDA\\v10.1\\libnvvp;C:\\Program Files (x86)\\NVIDIA Corporation\\PhysX\\Common;C:\\Program Files\\NVIDIA Corporation\\Nsight Compute 2019.4.0\\;C:\\Program Files\\Docker\\Docker\\resources\\bin;C:\\ProgramData\\DockerDesktop\\version-bin;C:\\Users\\dale\\AppData\\Local\\Microsoft\\WindowsApps;C:\\njs\\wslgit;;C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\Community\\VC\\Tools\\Llvm\\x64\\bin;C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\Community\\Common7\\IDE\\CommonExtensions\\Microsoft\\CMake\\CMake\\bin;C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\Community\\Common7\\IDE\\CommonExtensions\\Microsoft\\CMake\\Ninja;C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\Community\\Common7\\IDE\\VC\\Linux\\bin\\ConnectionManagerExe"],["TMP","C:\\Users\\dale\\AppData\\Local\\Temp"],["FRAMEWORKDIR","C:\\Windows\\Microsoft.NET\\Framework64\\"],["FRAMEWORKDIR64","C:\\Windows\\Microsoft.NET\\Framework64\\"],["FRAMEWORKVERSION","v4.0.30319"],["FRAMEWORKVERSION64","v4.0.30319"],["UCRTCONTEXTROOT",""],["UCRTVERSION","10.0.19041.0"],["UNIVERSALCRTSDKDIR","C:\\Program Files (x86)\\Windows Kits\\10\\"],["VCINSTALLDIR","C:\\Program Files (x86)\\Microsoft Visual Studio\\2019\\Community\\VC\\"],["VCTARGETSPATH",""],["WINDOWSLIBPATH","C:\\Program Files (x86)\\Windows Kits\\10\\UnionMetadata\\10.0.19041.0;C:\\Program Files (x86)\\Windows Kits\\10\\References\\10.0.19041.0"],["WINDOWSSDKDIR","C:\\Program Files (x86)\\Windows Kits\\10\\"],["WINDOWSSDKLIBVERSION","10.0.19041.0\\"],["WINDOWSSDKVERSION","10.0.19041.0\\"],["VISUALSTUDIOVERSION","16.0"]]
[ctest]  1/22 Test  #1: color_camera_node_test ..............***Failed    5.13 sec
[ctest] 
[ctest]       Start  2: image_manip_node_test
[ctest]  2/22 Test  #2: image_manip_node_test ...............***Failed    5.11 sec
[ctest] 
[ctest]       Start  3: pipeline_test
[ctest]  3/22 Test  #3: pipeline_test .......................   Passed    0.46 sec
[ctest]       Start  4: neural_network_test
[ctest]  4/22 Test  #4: neural_network_test .................***Failed    5.23 sec
[ctest] 
[ctest] ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
[ctest] neural_network_test.exe is a Catch v2.11.0 host application.
[ctest] Run with -? for options
[ctest] 
[ctest] -------------------------------------------------------------------------------
[ctest] Neural network node data checks
[ctest] -------------------------------------------------------------------------------
[ctest] ..\tests\src\neural_network_test.cpp(41)
[ctest] ...............................................................................
[ctest] 
[ctest] ..\tests\src\neural_network_test.cpp(41): FAILED:
[ctest] due to unexpected exception with message:
[ctest]   No available devices
[ctest] 
[ctest] ===============================================================================
[ctest] test cases: 1 | 1 failed
[ctest] assertions: 1 | 1 failed
[ctest] 
[ctest] 
[ctest]       Start  5: camera_preview
[ctest]  5/22 Test  #5: camera_preview ......................   Passed    5.03 sec
[ctest]       Start  6: camera_video
[ctest]  6/22 Test  #6: camera_video ........................   Passed    5.03 sec
[ctest]       Start  7: mono_camera
[ctest]  7/22 Test  #7: mono_camera .........................   Passed    5.03 sec
[ctest]       Start  8: camera_mobilenet
[ctest]  8/22 Test  #8: camera_mobilenet ....................   Passed    5.04 sec
[ctest]       Start  9: webcam_mobilenet
[ctest]  9/22 Test  #9: webcam_mobilenet ....................   Passed    5.04 sec
[ctest]       Start 10: mjpeg_encoding
[ctest] 10/22 Test #10: mjpeg_encoding ......................   Passed    5.03 sec
[ctest]       Start 11: h264_encoding
[ctest] 11/22 Test #11: h264_encoding .......................   Passed    5.04 sec
[ctest]       Start 12: stereo
[ctest] 12/22 Test #12: stereo ..............................   Passed    5.06 sec
[ctest]       Start 13: image_manip
[ctest] 13/22 Test #13: image_manip .........................   Passed    5.04 sec
[ctest]       Start 14: image_manip_warp
[ctest] 14/22 Test #14: image_manip_warp ....................   Passed    5.04 sec
[ctest]       Start 15: color_camera_control
[ctest] 15/22 Test #15: color_camera_control ................   Passed    5.03 sec
[ctest]       Start 16: system_information
[ctest] 16/22 Test #16: system_information ..................   Passed    5.03 sec
[ctest]       Start 17: device_queue_event
[ctest] 17/22 Test #17: device_queue_event ..................   Passed    5.04 sec
[ctest]       Start 18: opencv_support
[ctest] 18/22 Test #18: opencv_support ......................   Passed    5.04 sec
[ctest]       Start 19: mobilenet_device_side_decoding
[ctest] 19/22 Test #19: mobilenet_device_side_decoding ......   Passed    5.04 sec
[ctest]       Start 20: tiny_yolo_v3_device_side_decoding
[ctest] 20/22 Test #20: tiny_yolo_v3_device_side_decoding ...   Passed    5.03 sec
[ctest]       Start 21: tiny_yolo_v4_device_side_decoding
[ctest] 21/22 Test #21: tiny_yolo_v4_device_side_decoding ...   Passed    5.04 sec
[ctest]       Start 22: camera_mobilenet_sync
[ctest] 22/22 Test #22: camera_mobilenet_sync ...............   Passed    5.03 sec
[ctest] 
[ctest] 86% tests passed, 3 tests failed out of 22
[ctest] 
[ctest] Total Test time (real) = 106.65 sec
[ctest] 
[ctest] The following tests FAILED:
[ctest] 	  1 - color_camera_node_test (Failed)
[ctest] 	  2 - image_manip_node_test (Failed)
[ctest] 	  4 - neural_network_test (Failed)
[ctest] Errors while running CTest
[ctest] CTest finished with return code 8
[extension] [9213] cmake.ctest finished (returned 8)

[diablodale]

Did it click? I can't remember.
I've been running the ctest repeated as in OP today. I get random failures of test cases, but I haven't seen the no UI of the OP yet.