fredericbonnet/cmake-test-explorer

CMake Test Explorer for Visual Studio Code

cmake-test-explorer's Issues

CMake Test Explorer triggers CMake Tools to configure non CMake projects.

Having CMake Test Explorer and CMake Tools installed and opening a non CMake project will trigger a configure CMake window to open.

Initial bug report with CMake Tools but then figured out that it's CMake Test Explorer that's triggering this.

microsoft/vscode-cmake-tools#3002

Search/filter for the list of tests

It would be great if this extension had a filter box that allows filtering tests based on their status and/or regular expression. Currently, it's quite difficult to navigate when there are thousands of tests in a project. A filtered list would be quite useful for looking at the status and/or running specific tests as needed.

Read cmakeExplorer.debugConfig from workspace file?

It seems that cmakeExplorer.debugConfig does not support launch configurations defined in workspace file. Could this be implemented?

security contact

Hello,

I am part of a team of academic researchers at North Carolina State University. Our research in vscode extensions found an issue. Is there a security contact we could report to?

Thanks!

Environment variables defined in debugConfig are ignored

I have a test that reads environment variables. I define those variables in debug configuration in launch.json, e.g.:

{
    "name": "myDebugConfig",
    "type": "cppdbg",
    "request": "launch",
    # ...
    "environment": [
        { "key": "MY_KEY", "value": "MY_VALUE" },    
    ]
}

I configure cmakeExplorer.debugConfig to use myDebugConfig and yet my test doesn't see MY_KEY to be defined.

Crashed tests are not reported as failures (reprise)

Hi,
I'm afraid we've just noticed a reoccurrance of this issue: #47

My guess is that at some point CTest has changed its output: in the case I'm seeing the issue, the summary line is looking like:
1/1 Test #9: MySuite.MyTest ...Subprocess aborted***Exception: 0.00 sec

Based on your explanation from the previous issue, the problem is presumably that that output again manages not to match the CTEST_FAILED_RE regex (because there's an error message between the dots and the stars). I've done a quick experiment, and if we omit the three stars from the regex:
const CTEST_FAILED_RE = /^\s*\d+\/\d+ Test\s+#(\d+): (.+) \.\.\.+/;
the test is once again picked up as failed. Any chance of including that in a quick update?

"environment is not iterable" when debugging test on macos

When I'm trying to launch a test for bebugging in the CMake Test Explorer, I get an error saying "environment is not iterable"

It says to open the launch.json, whether or not this file exists.

If it does not exists, it ask me to create one, but it doesn't solve the problem.

empty tests list for after v0.14.1

Hi. I used cmake-test-explorer in the past successfully. But now I find an empty list when using it.
So, I tried rollback into multiple versions and the more recent version in which the extension work properly is v0.14.1.
My stat is:
code 1.73.0
TestExplorer UI v2.21.1
TestAdater v0.1.6.

Any suggestion?

Fails to find test cases

When running the extension everything seems to be fine but then no test cases are loaded.

This is the logfile, is there any way to find a more detailed logfile or to debug it?

[2023-03-27 10:17:35.132] [WARN] CMake integration is enabled but the CMake Tools extension is not active
[2023-03-27 10:17:35.879] [INFO] Workspace contains CMake project files, waiting for CMake Tools extension to activate
[2023-03-27 10:17:39.392] [WARN] Test Explorer found
[2023-03-27 10:17:39.393] [INFO] Creating adapter for c:\Users\20204729\Projects\sdf3
[2023-03-27 10:17:39.393] [INFO] Initializing CMake test adapter
[2023-03-27 10:17:39.393] [INFO] Registering adapter for c:\Users\20204729\Projects\sdf3
[2023-03-27 10:17:39.394] [INFO] Loading CMake tests
[2023-03-27 10:17:39.395] [INFO] Initialization finished

Add a "Go to test" button

It would be very useful to have a "Go to test" additional button (alongside "Debug this test" and "Run this test") on the test tree view, that will open the (main) test file, in a similar fashion as the Python test explorer.

Here are the current CMake Test Explorer options:

Here are the options from the Python Test Explorer:

Cheers.

Tests fail to run when multiple configurations are available

In the log file I see the following line "Test not available without configuration. (Missing "-C "?)".

If I add to settings "cmakeExplorer.extraCtestRunArgs": "-C ${buildType}", then the tests run successfully. I expected that cmake-test-explorer will install the used configuration itself from cmakeExplorer.buildConfig.

Publish to OpenVSX?

Hi,

it would be great if this extension was also available on OpenVSX https://open-vsx.org/

Feature request: Add paths to the environment when executing tests

Would it be possible to add folder locations as extra locations where the test executables can search for libraries like dlls? The extension C++ TestMate has this feature available.

Can't see test output with native Testing UI

I've been experimenting with the new native Testing UI (setting "testExplorer.useNativeTesting": true). In many ways it's better than the older Test Explorer UI (having an option to sort by test status is really nice), and I presume you'll be planning to move over to it at some point. However, it appears that I can no longer see test output - there's a "Show test output" button, but the resulting output terminal just shows "The test run did not record any output." for each run. I assume that the mechanism for feeding output to the extension must be different in the newer environment, and that an update may be needed to the extension to support this. I could be wrong, though - hope I'm asking in the right place :)

WORKING_DIRECTORY of test not propagated when debugging a test

When you launch a test for debugging from the Test Explorer side panel (with the little bug icon), it doesn't seem to respect the WORKING_DIRECTORY set for the test otherwise. It runs from there, but debugging doesn't. I have tests that rely on DLLs built by the project and when launched from other places, they don't even hit main but instacrash in the debugger.

more granular activation events

Currently this extension is always activated which then activates the cmake-tools extensions which for non-cmake projects produces an annoying warning message about not finding a CMakeLists.txt file

I'd recommend activating on the existence on the existence of a CMakeLists.txt in the project (similar to cmake-tools)

  "activationEvents": [
    "workspaceContains:**/CMakeLists.txt",
  ]

PASS_REGULAR_EXPRESSION tests don't work

I was trying to demonstrate the use of tests using regular expressions instead of just exit codes. I was surprised to see that setting a regular expression as a test property overrides exit code, and even if the exit code is hardcoded to fail, if console output matches the regex, the test still passes as far as CTest is considered.

My guess is that you should be surprised too, because the extension doesn't handle this well. Having a test passing on a regex, ctest and the extension disagree on the test's success.

I was trying to look at the code and it seems to me as if the ctest executable would be circumvented altogether and the extension would launch the test on it's own in the working directory and args obtained from the ctest JSON output. This seems like the wrong approach to take, because there is a lot more going on under the hood of CTest which the extension should not try to reproduce. (CTest implements load balancing, processor affinity, etc.) The extension should invoke ctest and parse output and look for test names/labels and Pass/Failed.

Note: the only thing where the extension should circumvent ctest is looking for the DEPENDS property on a test, and auto-reset the state of tests where the executable or any input datafile is newer than the timestamp of the test run. CTest cannot figure this out on its own.

Crashed tests are not reported as failures

If a test process crashes altogether (for example, segfaults) the explorer appears to ignore it rather than marking it as failed. The test retains the "not run" icon, and if viewing the tests as a tree view, the icon on the containing suite ignores the result (so if all the other tests succeed and the tree is collapsed, there may be no visual indication that any test has failed).

In case it matters, our tests are mostly C++ using GTest and registered via cmake's gtest_discover_tests function, but I think it'd happen regardless of the test framework.
With a test suite like

TEST(TestSuite, TestCrashingFunction)
{
    int x = *(reinterpret_cast<int*>(0));
    (void)x;
    GTEST_SUCCEED();
}
TEST(TestSuite, TestAllIsWell)
{
    GTEST_SUCCEED();
}

CTest itself reports something like

Test project /build/Debug
    Start 125: TestSuite.TestCrashingFunction
1/2 Test #125: TestSuite.TestCrashingFunction ...***Exception: SegFault  0.00 sec
    Start 126: TestSuite.TestAllIsWell
2/2 Test #126: TestSuite.TestAllIsWell ..........   Passed    0.00 sec

50% tests passed, 1 tests failed out of 2

Total Test time (real) =   0.02 sec

The following tests FAILED:
        125 - TestSuite.TestCrashingFunction (SEGFAULT)
Errors while running CTest

But the test explorer shows

EDIT:
Sorry, forgot to mention the environment. I reproduced this in VSCode 1.63.2 on Windows, CMake Test Explorer 0.15.1 (with Test Explorer UI 2.21.1) and developing in either a WSL2+Docker environment via Remote-Containers 0.209.6 or a remote server via Remote-SSH 0.70.0). I'm told the behaviour is the same on a Mac + Docker Desktop setup. The actual dev environment is Linux (RHEL7) in all cases. I've seen the same behaviour for some time, though, so I don't think this is new or version-specific.

SyntaxError: Unexpected token T in JSON at position 0

Hi, It's a great extension, but I couldn't get it to work with my workspace. When I activate Test Explorer tab it says: Error while loading tests - click to show. And when I click I can see in the Output:
SyntaxError: Unexpected token T in JSON at position 0

Crashed tests are not showing up as failed

If a test crashes/hits an assert, the status of the test does not show up as failed, it just changes to "outdated". If the test has previously passed, it just shows as a darker shade of green, which is easy to mistake for passing (IMO).

If you use the native testing UI the crashed test is also not accounted for in the "X/Y tests passed" (example, if you have 100 tests and one crashes, it will say "99/99 test passed (100%)")

I'm using gtest if that makes any difference?

Autorun CWD Incorrect?

Everytime I rebuild my project, one of the tests fails. If I actually run it from the list manually - it works. The test depends on the CWD, and is also quite slow. I wonder what is actually failing, incorrect CWD setting or some timeout?

Thank you.

No longer require Test Explorer UI extension

VSCode has native support for the test explorer since a while now and the Test Explorer UI extension is deprecated.

Extension causes high cpu load

fredericbonnet.cmake-test-adapter-unresponsive.cpuprofile.txt

• Issue Type: Performance
• Extension Name: cmake-test-adapter
• Extension Version: 0.14.1
• OS Version: Windows_NT x64 10.0.19043
• VS Code version: 1.57.1

fredericbonnet.cmake-test-adapter-unresponsive.cpuprofile.txt

Find more details here: https://github.com/microsoft/vscode/wiki/Explain-extension-causes-high-cpu-load

My test window doesnt populate

Hi,

I can't see any of the tests in the test window.

If I run the ctest --show-only and ctest --show-only=json-v1; both commands shows all the tests.

I haven't found any log so I can't provide more information.

BTW. I'm developing using a docker container (Remote - Containers extension).

Independent CTest invocations break RESOURCE_LOCK and test fixtures

I'm developing a local CI runner implemented as a CMake module to overcome the limitations of GitHub and Gitlab which don't cater to local development by enabling running CI pipelines locally. (They were not designed for this use case.)

My module in development accepts a test matrix definition and executes them locally inside docker containers. Containers spin up as CTest fixture setups (they were designed for such use cases) and are discarded as fixture cleanups. the module also allows uses of RESOURCE_LOCK to forbid tests running in parallel when using a shared resource. (Such is the actual case of multi-configuration build systems, MSBuild or Ninja Multi-Config generators for eg. which do not tolerate parallel invocations of differing configurations (Build and Release) on the same build tree concurrently.)

The problem with the extensions current design is that both fixtures and resource locks only work as part of a single CTest invocation. When each test is run in a separate CTest invocation, multiple instances of the same resource lock and fixture may be in flight, which results in breaking compilations and random errors.

I'm not sure if it's possible with the current Test Explorer API, but I would like to ask for no more than one CTest invocation when running all (or any subset) of the tests.

No support for Ctest named configuration

Hi,

Nice work on this extension! It works very good in most cases except when user uses the named configuration feature of CTest, see https://cmake.org/cmake/help/v3.0/command/add_test.html CONFIGURATIONS.

It would be nice to be able to specify the named configuration to use (in our case we use All, Unit, Integration,... and the test explorer shows no tests).

How do I exclude some of the tests?

I have some tests that really should not be run every time. In fact, they should only be run on demand. In my CMake config I use -LE IntegrationTest as extra switches for ctest. What can I do in TestExplorer?

The same tests are listed on all folders of multi-folder workspace

In a project with multiple folders the UI shows the tests for the currently selected workspace for all folders.

Given a multi folder workspace, both folders have a cmake build system with tests.

With one of the folders selected:

If selecting the other folder:

Build before test

Maybe I'm dumb, but I can't find a way to have one keypress build the project before running tests.

I tried setting --build-and-test in "Extra Ctest Run Args", but it had no effect.

Error: Cannot spaw command 'ctest'

I tried the extension, and it fails to initialize with the error in the title. I saw there is no place to set the path to the CTest executable, so I added it to the PATH so a vanilla CLI invocation can find it. However, this did not solve the problem.

I'm using a Windows 10 machine, added a user entry to the PATH, relog, test on a vanilla command line that ctest.exe is found, but the extension within the IDE still cannot launch it, even after wiping the entire .vscode folder.

My cmakeExplorer.buildDir setting is set to ${workspaceRoot}/.vscode/build, because that's where CMake Tools is set to place its output. I've set cmakeExplorer.logfile to be the same, but then I got this error:

Couldn't write log file ${workspaceRoot}/.vscode/build: Error: ENOENT: no such file or directory, open 'C:\Kellekek\Microsoft\VisualStudio\Code\${workspaceRoot}\.vscode\build'

Is the extension handling paths correctly? Why is the VS Code install path precede an unexpanded JSON variable?

Test do not load in WIndows

Hello,

Thank you for this great plug-in!

Using the same project (https://github.com/anagno/gama) everything is working perfectly under ubuntu. The tests appear under the Test View and I am able to run them. But under windows the tests do not appear in the Test View (and I could not find any errors).

Any ideas why this might be happening ? Do I need to install something extra? Any ideas how I can debug it?

KR,

Vasileios

Can't run tests from gtest's TYPED_TEST with nested template type parameters

When I define a GoogleTest TYPED_TEST with a TypeParam that is a nested template specialization (such as ClassTemplate<ClassTemplate<int>>), the resulting test shows up correctly in the test explorer but I am unable to run it. When I click the "Run all tests" button, the status for that test does not change, nor when I run the individual test.

A full project is at https://github.com/barometz/cmake-tx-repro, but the minimal repro (minus CMake config) is:

#include <gtest/gtest.h>

template <typename T> class ClassTemplate {};
template <typename T> using TypedFixture = testing::Test;
TYPED_TEST_SUITE(TypedFixture, testing::Types<ClassTemplate<ClassTemplate<int>>>);
TYPED_TEST(TypedFixture, Test) {}

My first hunch is it might be because of the space in the resulting test name (gtest's test name generation separates the closing > >, presumably for historical reasons). I'm going to see if I can find and resolve it this afternoon, but I figured I'd write the issue up first.

custom debug configuration program is not overriden

Hi. I'm trying to use a custom debug configuration using the CodeLLDB debug adapter under MacOS.
The debug configuration I'm using is

{
      "type": "lldb",
      "request": "launch",
      "name": "Test Debug"
}

however when I try to debug a test I get the message "program" property is required for launch. If I add a program field to the configuration then it takes it as is and doesn't override it with the test program.

versions:
VSCode: 1.50.1
CodeLLDB: v1.6.0
CMake Test Explorer: v0.10.0

Extra Ctest Env vars: Also support dotenv file

Currently, CMake test explorer allows to hardcode env vars in cmakeExplorer.extraCtestEnvVars.
But there is no way to load env vars from a dotenv file like in C/C++ extension debugging configuration (https://code.visualstudio.com/docs/cpp/launch-json-reference#_environment-variable-definitions-file).

It would be convenient to add this feature.
I'm using conan (C/C++ package manager), and I need to initialize several environment variables (PATH, LD_LIBRARY_PATH or DYLD_LIBRARY_PATH basically), before running executables having runtime dependencies. conan can generate a shell to source, a .bat for windows, and a dotenv file.
The content of those environment variables can be very complex, it can change after an update of the dependency graph, or a switch between CMake build types, so hardcoding env vars in cmakeExplorer.extraCtestEnvVars is not an option here.

Test not run when using the WORKING_DIRECTORY property (add_test function)

Tests does not seems to run when they are configured in CMake with the WORKING_DIRECTORY property. This attribute was added to the add_test call in CMake.

example:
add_test( NAME ${file} COMMAND ${file} WORKING_DIRECTORY ${CMAKE_CURRENT_LIST_DIR} )

The test is successfully detected in TestExplorer, it can be 'run', it will get a green mark automatically but the test has not really run. Even a test that normally fails are marked as successful.

When looking at the Output window it looks like ctest was not started from the proper directory (the working directory is not the same location as the test binary folder)

• with the WORKING_DIRECTORY property:

Test project F:/Projets/git/Test/Template/Template

That path corresponds to the WORKING_DIRECTORY variable

• without the WORKING_DIRECTORY property:

Test project F:/Projets/git/Test/build/Visual Studio Community 2019 Release - amd64/Debug/Template/Template
    Start 1: adl
1/1 Test #1: adl ..............................   Passed    0.01 sec

100% tests passed, 0 tests failed out of 1

Total Test time (real) =   0.02 sec

The path specified is the binary location of the test application (same that will appears if ctest is run manually even with the WORKING_DIRECTORY attribute)

Running the test manually with ctest or with CMake Tools runs as expected.

configure PATH environment var for running tests

I have a cmake windows build which use msys2 libraries in C:\tools\msys64\mingw64\bin.
The test-explorer fail to load such libraries when running tests.
When I extends system PATH, the test-explorer works.

I would like to avoid extends the system PATH.
Is it possible a configure PATH when running tests?

trigger BuildBeforeRun also by running a test in the testexplorer

Hi,
If I use CMake it rebuilds before I run a test.

See BuildBeforeRun in
https://github.com/microsoft/vscode-cmake-tools/blob/develop/docs/cmake-settings.md#cmake-settings

can we trigger this BuildBeforeRun also by running a test in the testexplorer?

Not respecting fixtures

I am not sure which ctest command you use to run, but somehow it seems to skip any setup/cleanup fixtures. This results in tests failing. If I use ctest -L or -R it works fine?

Run all tests from this file

is not working, the test sources don't seem to be linked to the test cases picked up by the plugin

Feature request: A list of all the individual tests in the test explorer panel

Other extensions have this feature. To have a list of all the individual unit tests and to be able to execute each test individually instead of just having a list of cmake targets.

Debugging a test doesn't relay the environment to the debugger

I have test which is reading some environment variables but finds them unset. When running through CTest they're found, but not when debugging the target. I found that getCmakeTestDebugConfiguration() only captures the CTest property WORKING_DIRECTORY property, but not ENVIRONMENT. vscode.DebugConfiguration ought to have an env field (as far as I could tell) which should be populated with the content of said property.

Disabled tests are marked as failed

I've noticed that if we have tests which have been configured with the CTest DISABLED property, CTest correctly skips them but the tests show in the test explorer as failed (red X or equivalent icon). I imagine there's some sort of intermediate skipped/warning state that would probably be more appropriate for disabled tests, and failing that it would probably be preferable to have them marked as passed rather than failed.

I believe this applies to any test marked with set_tests_properties(<name> PROPERTIES DISABLED True) or equivalent, but in our case, the tests are defined in C++ GoogleTest, along the lines of TEST(mytestcase, DISABLED_mytest) {...}. They're then registered with gtest_discover_tests(test_target). That produces a test registration CMake file along the lines of

add_test([=[mytestcase.mytest]=] path/to/test_target.exe [==[--gtest_filter=mytestcase.DISABLED_mytest]==] --gtest_also_run_disabled_tests)
set_tests_properties( [=[--gtest_filter=mytestcase.DISABLED_mytest]=] PROPERTIES DISABLED True)

When run via ctest, that gives

Test project path/to/test_target.exe
    Start 1: mytestcase.mytest
1/1 Test #1: mytestcase.mytest ...***Not Run (Disabled)   0.00 sec
No tests were found!!!

But the test shows as failed in test explorer.

In case it's of interest, I notice that CLion has exactly the same issue!

added an option to suiteDelimiter to sort by label

Hello,

It would be nice to add an option to cmakeExplorer.suiteDelimiter to organize by ctest labels

Can't get Google Test unit tests to show up

I can't get my Google Test unit tests to show up in the Testing extension window. If I use C++ TestMate I can, but not CMake Test Explorer. I guess what are the differences between the two? Is this expected? If so, maybe I'm misunderstanding the purpose of CMake Test Explorer vs C++ TestMate. If it's not, maybe it's my usage of gtest_discover_tests() instead of add_test()? I know the former calls the later but maybe there's an incompatibility there. Any advice would be appreciated.

Usage of env variables

The cmake extension may use powershell env access in its variables, ex:
"cmake.buildDirectory": "${workspaceFolder}/_build-${buildType}-${env:OS}"
which is resolved to _build-Windows_NT on windows.

However, the cmake-tesplorer do not do the same, ex:
"cmakeExplorer.buildDir" : "${workspaceFolder}/_build-${buildType}-${env:OS}"
which is resolved to _build-${env:OS}

Configure extension using CMakePresets/'testPresets'

CMake added support for test presets in their CMakePresets.json file.
With CLion & VS Code starting to support this format now it seems like a good way to configure all build & test configurations in one place.
It would be great if this extension would be able to read those configurations as well.

Does not find any tests

Hi,

in my CMakeLists file I create some tests using the add_test function. This runs and configures fine. However, the test cases do not show up in the test explorer.

I have no idea what could be causing this or how to begin debugging it. Any help is appreciated and I'm happy to give as much info as needed.

adding a quick option to run test wit arg

adding a quick option to ru ctest with specifiq args without change cmakeExplorer.extraCtestLoadArgs option (permanent)

Support for CodeLLDB launch configs

We use CodeLLDB rather than CppTools for debugging. The launch.json configs are mostly the same, and up to version 0.14.2, this worked fine; we just point CMake Test Explorer's cmakeExplorer.debugConfig setting at an LLDB launch config and it'd work ok. As of 0.15 we're getting an error dialog undefined is not iterable (cannot read property Symbol(Symbol.iterator)) when we attempt to debug a test from the test explorer.

I believe the reason is that CodeLLDB takes its environment settings in a different format from cpptools:
CppTools:

    "environment": [
      {
        "name": "<varname>"
        "value": "<blah>"
      }
    ],

CodeLLDB:

    "env": {
        "<varname>": "<blah>"
    }

If I add an empty environment list to the launch config it no longer produces errors, but that's going to be rather confusing! Could you support both formats?

For what it's worth, the same error occurs with a cpptools debug config with no environment list: that's less of a problem since it's created by default, but I think it should still work.

Custom debug provider.

Nice extension you got there!
Seems like there is a built-in support for cppdbg only. Would be great to have some other options (lldb), or even custom build launchers.