fredericbonnet / cmake-test-explorer Goto Github PK
View Code? Open in Web Editor NEWCMake Test Explorer for Visual Studio Code
License: BSD 3-Clause "New" or "Revised" License
CMake Test Explorer for Visual Studio Code
License: BSD 3-Clause "New" or "Revised" License
Having CMake Test Explorer and CMake Tools installed and opening a non CMake project will trigger a configure CMake window to open.
Initial bug report with CMake Tools but then figured out that it's CMake Test Explorer that's triggering this.
It would be great if this extension had a filter box that allows filtering tests based on their status and/or regular expression. Currently, it's quite difficult to navigate when there are thousands of tests in a project. A filtered list would be quite useful for looking at the status and/or running specific tests as needed.
It seems that cmakeExplorer.debugConfig
does not support launch configurations defined in workspace file. Could this be implemented?
Hello,
I am part of a team of academic researchers at North Carolina State University. Our research in vscode extensions found an issue. Is there a security contact we could report to?
Thanks!
I have a test that reads environment variables. I define those variables in debug configuration in launch.json
, e.g.:
{
"name": "myDebugConfig",
"type": "cppdbg",
"request": "launch",
# ...
"environment": [
{ "key": "MY_KEY", "value": "MY_VALUE" },
]
}
I configure cmakeExplorer.debugConfig
to use myDebugConfig
and yet my test doesn't see MY_KEY
to be defined.
Hi,
I'm afraid we've just noticed a reoccurrance of this issue: #47
My guess is that at some point CTest has changed its output: in the case I'm seeing the issue, the summary line is looking like:
1/1 Test #9: MySuite.MyTest ...Subprocess aborted***Exception: 0.00 sec
Based on your explanation from the previous issue, the problem is presumably that that output again manages not to match the CTEST_FAILED_RE regex (because there's an error message between the dots and the stars). I've done a quick experiment, and if we omit the three stars from the regex:
const CTEST_FAILED_RE = /^\s*\d+\/\d+ Test\s+#(\d+): (.+) \.\.\.+/;
the test is once again picked up as failed. Any chance of including that in a quick update?
Hi. I used cmake-test-explorer in the past successfully. But now I find an empty list when using it.
So, I tried rollback into multiple versions and the more recent version in which the extension work properly is v0.14.1.
My stat is:
code 1.73.0
TestExplorer UI v2.21.1
TestAdater v0.1.6.
Any suggestion?
When running the extension everything seems to be fine but then no test cases are loaded.
This is the logfile, is there any way to find a more detailed logfile or to debug it?
[2023-03-27 10:17:35.132] [WARN] CMake integration is enabled but the CMake Tools extension is not active
[2023-03-27 10:17:35.879] [INFO] Workspace contains CMake project files, waiting for CMake Tools extension to activate
[2023-03-27 10:17:39.392] [WARN] Test Explorer found
[2023-03-27 10:17:39.393] [INFO] Creating adapter for c:\Users\20204729\Projects\sdf3
[2023-03-27 10:17:39.393] [INFO] Initializing CMake test adapter
[2023-03-27 10:17:39.393] [INFO] Registering adapter for c:\Users\20204729\Projects\sdf3
[2023-03-27 10:17:39.394] [INFO] Loading CMake tests
[2023-03-27 10:17:39.395] [INFO] Initialization finished
It would be very useful to have a "Go to test" additional button (alongside "Debug this test" and "Run this test") on the test tree view, that will open the (main) test file, in a similar fashion as the Python test explorer.
Here are the current CMake Test Explorer options:
Here are the options from the Python Test Explorer:
Cheers.
In the log file I see the following line "Test not available without configuration. (Missing "-C "?)".
If I add to settings "cmakeExplorer.extraCtestRunArgs": "-C ${buildType}", then the tests run successfully. I expected that cmake-test-explorer will install the used configuration itself from cmakeExplorer.buildConfig.
Hi,
it would be great if this extension was also available on OpenVSX https://open-vsx.org/
Would it be possible to add folder locations as extra locations where the test executables can search for libraries like dlls? The extension C++ TestMate has this feature available.
I've been experimenting with the new native Testing UI (setting "testExplorer.useNativeTesting": true
). In many ways it's better than the older Test Explorer UI (having an option to sort by test status is really nice), and I presume you'll be planning to move over to it at some point. However, it appears that I can no longer see test output - there's a "Show test output" button, but the resulting output terminal just shows "The test run did not record any output." for each run. I assume that the mechanism for feeding output to the extension must be different in the newer environment, and that an update may be needed to the extension to support this. I could be wrong, though - hope I'm asking in the right place :)
When you launch a test for debugging from the Test Explorer side panel (with the little bug icon), it doesn't seem to respect the WORKING_DIRECTORY
set for the test otherwise. It runs from there, but debugging doesn't. I have tests that rely on DLLs built by the project and when launched from other places, they don't even hit main but instacrash in the debugger.
Currently this extension is always activated which then activates the cmake-tools extensions which for non-cmake projects produces an annoying warning message about not finding a CMakeLists.txt
file
I'd recommend activating on the existence on the existence of a CMakeLists.txt
in the project (similar to cmake-tools)
"activationEvents": [
"workspaceContains:**/CMakeLists.txt",
]
I was trying to demonstrate the use of tests using regular expressions instead of just exit codes. I was surprised to see that setting a regular expression as a test property overrides exit code, and even if the exit code is hardcoded to fail, if console output matches the regex, the test still passes as far as CTest is considered.
My guess is that you should be surprised too, because the extension doesn't handle this well. Having a test passing on a regex, ctest and the extension disagree on the test's success.
I was trying to look at the code and it seems to me as if the ctest executable would be circumvented altogether and the extension would launch the test on it's own in the working directory and args obtained from the ctest JSON output. This seems like the wrong approach to take, because there is a lot more going on under the hood of CTest which the extension should not try to reproduce. (CTest implements load balancing, processor affinity, etc.) The extension should invoke ctest
and parse output and look for test names/labels and Pass/Failed.
Note: the only thing where the extension should circumvent ctest is looking for the DEPENDS property on a test, and auto-reset the state of tests where the executable or any input datafile is newer than the timestamp of the test run. CTest cannot figure this out on its own.
If a test process crashes altogether (for example, segfaults) the explorer appears to ignore it rather than marking it as failed. The test retains the "not run" icon, and if viewing the tests as a tree view, the icon on the containing suite ignores the result (so if all the other tests succeed and the tree is collapsed, there may be no visual indication that any test has failed).
In case it matters, our tests are mostly C++ using GTest and registered via cmake's gtest_discover_tests
function, but I think it'd happen regardless of the test framework.
With a test suite like
TEST(TestSuite, TestCrashingFunction)
{
int x = *(reinterpret_cast<int*>(0));
(void)x;
GTEST_SUCCEED();
}
TEST(TestSuite, TestAllIsWell)
{
GTEST_SUCCEED();
}
CTest itself reports something like
Test project /build/Debug
Start 125: TestSuite.TestCrashingFunction
1/2 Test #125: TestSuite.TestCrashingFunction ...***Exception: SegFault 0.00 sec
Start 126: TestSuite.TestAllIsWell
2/2 Test #126: TestSuite.TestAllIsWell .......... Passed 0.00 sec
50% tests passed, 1 tests failed out of 2
Total Test time (real) = 0.02 sec
The following tests FAILED:
125 - TestSuite.TestCrashingFunction (SEGFAULT)
Errors while running CTest
EDIT:
Sorry, forgot to mention the environment. I reproduced this in VSCode 1.63.2 on Windows, CMake Test Explorer 0.15.1 (with Test Explorer UI 2.21.1) and developing in either a WSL2+Docker environment via Remote-Containers 0.209.6 or a remote server via Remote-SSH 0.70.0). I'm told the behaviour is the same on a Mac + Docker Desktop setup. The actual dev environment is Linux (RHEL7) in all cases. I've seen the same behaviour for some time, though, so I don't think this is new or version-specific.
Hi, It's a great extension, but I couldn't get it to work with my workspace. When I activate Test Explorer tab it says: Error while loading tests - click to show. And when I click I can see in the Output:
SyntaxError: Unexpected token T in JSON at position 0
If a test crashes/hits an assert, the status of the test does not show up as failed, it just changes to "outdated". If the test has previously passed, it just shows as a darker shade of green, which is easy to mistake for passing (IMO).
If you use the native testing UI the crashed test is also not accounted for in the "X/Y tests passed" (example, if you have 100 tests and one crashes, it will say "99/99 test passed (100%)")
I'm using gtest if that makes any difference?
Everytime I rebuild my project, one of the tests fails. If I actually run it from the list manually - it works. The test depends on the CWD, and is also quite slow. I wonder what is actually failing, incorrect CWD setting or some timeout?
Thank you.
VSCode has native support for the test explorer since a while now and the Test Explorer UI extension is deprecated.
fredericbonnet.cmake-test-adapter-unresponsive.cpuprofile.txt
Performance
cmake-test-adapter
0.14.1
Windows_NT x64 10.0.19043
1.57.1
fredericbonnet.cmake-test-adapter-unresponsive.cpuprofile.txt
Find more details here: https://github.com/microsoft/vscode/wiki/Explain-extension-causes-high-cpu-load
Hi,
I can't see any of the tests in the test window.
If I run the ctest --show-only and ctest --show-only=json-v1; both commands shows all the tests.
I haven't found any log so I can't provide more information.
BTW. I'm developing using a docker container (Remote - Containers extension).
I'm developing a local CI runner implemented as a CMake module to overcome the limitations of GitHub and Gitlab which don't cater to local development by enabling running CI pipelines locally. (They were not designed for this use case.)
My module in development accepts a test matrix definition and executes them locally inside docker containers. Containers spin up as CTest fixture setups (they were designed for such use cases) and are discarded as fixture cleanups. the module also allows uses of RESOURCE_LOCK to forbid tests running in parallel when using a shared resource. (Such is the actual case of multi-configuration build systems, MSBuild or Ninja Multi-Config generators for eg. which do not tolerate parallel invocations of differing configurations (Build and Release) on the same build tree concurrently.)
The problem with the extensions current design is that both fixtures and resource locks only work as part of a single CTest invocation. When each test is run in a separate CTest invocation, multiple instances of the same resource lock and fixture may be in flight, which results in breaking compilations and random errors.
I'm not sure if it's possible with the current Test Explorer API, but I would like to ask for no more than one CTest invocation when running all (or any subset) of the tests.
Hi,
Nice work on this extension! It works very good in most cases except when user uses the named configuration feature of CTest, see https://cmake.org/cmake/help/v3.0/command/add_test.html CONFIGURATIONS.
It would be nice to be able to specify the named configuration to use (in our case we use All, Unit, Integration,... and the test explorer shows no tests).
I have some tests that really should not be run every time. In fact, they should only be run on demand. In my CMake config I use -LE IntegrationTest as extra switches for ctest. What can I do in TestExplorer?
Maybe I'm dumb, but I can't find a way to have one keypress build the project before running tests.
I tried setting --build-and-test
in "Extra Ctest Run Args", but it had no effect.
I tried the extension, and it fails to initialize with the error in the title. I saw there is no place to set the path to the CTest executable, so I added it to the PATH so a vanilla CLI invocation can find it. However, this did not solve the problem.
I'm using a Windows 10 machine, added a user entry to the PATH, relog, test on a vanilla command line that ctest.exe is found, but the extension within the IDE still cannot launch it, even after wiping the entire .vscode folder.
My cmakeExplorer.buildDir
setting is set to ${workspaceRoot}/.vscode/build
, because that's where CMake Tools is set to place its output. I've set cmakeExplorer.logfile
to be the same, but then I got this error:
Couldn't write log file ${workspaceRoot}/.vscode/build: Error: ENOENT: no such file or directory, open 'C:\Kellekek\Microsoft\VisualStudio\Code\${workspaceRoot}\.vscode\build'
Is the extension handling paths correctly? Why is the VS Code install path precede an unexpanded JSON variable?
Hello,
Thank you for this great plug-in!
Using the same project (https://github.com/anagno/gama) everything is working perfectly under ubuntu. The tests appear under the Test View and I am able to run them. But under windows the tests do not appear in the Test View (and I could not find any errors).
Any ideas why this might be happening ? Do I need to install something extra? Any ideas how I can debug it?
KR,
Vasileios
When I define a GoogleTest TYPED_TEST
with a TypeParam that is a nested template specialization (such as ClassTemplate<ClassTemplate<int>>
), the resulting test shows up correctly in the test explorer but I am unable to run it. When I click the "Run all tests" button, the status for that test does not change, nor when I run the individual test.
A full project is at https://github.com/barometz/cmake-tx-repro, but the minimal repro (minus CMake config) is:
#include <gtest/gtest.h>
template <typename T> class ClassTemplate {};
template <typename T> using TypedFixture = testing::Test;
TYPED_TEST_SUITE(TypedFixture, testing::Types<ClassTemplate<ClassTemplate<int>>>);
TYPED_TEST(TypedFixture, Test) {}
My first hunch is it might be because of the space in the resulting test name (gtest's test name generation separates the closing > >
, presumably for historical reasons). I'm going to see if I can find and resolve it this afternoon, but I figured I'd write the issue up first.
Hi. I'm trying to use a custom debug configuration using the CodeLLDB debug adapter under MacOS.
The debug configuration I'm using is
{
"type": "lldb",
"request": "launch",
"name": "Test Debug"
}
however when I try to debug a test I get the message "program" property is required for launch
. If I add a program
field to the configuration then it takes it as is and doesn't override it with the test program.
versions:
VSCode: 1.50.1
CodeLLDB: v1.6.0
CMake Test Explorer: v0.10.0
Currently, CMake test explorer allows to hardcode env vars in cmakeExplorer.extraCtestEnvVars
.
But there is no way to load env vars from a dotenv file like in C/C++ extension debugging configuration (https://code.visualstudio.com/docs/cpp/launch-json-reference#_environment-variable-definitions-file).
It would be convenient to add this feature.
I'm using conan (C/C++ package manager), and I need to initialize several environment variables (PATH, LD_LIBRARY_PATH or DYLD_LIBRARY_PATH basically), before running executables having runtime dependencies. conan can generate a shell to source, a .bat for windows, and a dotenv file.
The content of those environment variables can be very complex, it can change after an update of the dependency graph, or a switch between CMake build types, so hardcoding env vars in cmakeExplorer.extraCtestEnvVars
is not an option here.
Tests does not seems to run when they are configured in CMake with the WORKING_DIRECTORY property. This attribute was added to the add_test call in CMake.
example:
add_test( NAME ${file} COMMAND ${file} WORKING_DIRECTORY ${CMAKE_CURRENT_LIST_DIR} )
The test is successfully detected in TestExplorer, it can be 'run', it will get a green mark automatically but the test has not really run. Even a test that normally fails are marked as successful.
When looking at the Output window it looks like ctest was not started from the proper directory (the working directory is not the same location as the test binary folder)
Test project F:/Projets/git/Test/Template/Template
That path corresponds to the WORKING_DIRECTORY variable
Test project F:/Projets/git/Test/build/Visual Studio Community 2019 Release - amd64/Debug/Template/Template
Start 1: adl
1/1 Test #1: adl .............................. Passed 0.01 sec
100% tests passed, 0 tests failed out of 1
Total Test time (real) = 0.02 sec
The path specified is the binary location of the test application (same that will appears if ctest is run manually even with the WORKING_DIRECTORY attribute)
Running the test manually with ctest or with CMake Tools runs as expected.
I have a cmake windows build which use msys2 libraries in C:\tools\msys64\mingw64\bin.
The test-explorer fail to load such libraries when running tests.
When I extends system PATH, the test-explorer works.
I would like to avoid extends the system PATH.
Is it possible a configure PATH when running tests?
Hi,
If I use CMake it rebuilds before I run a test.
See BuildBeforeRun in
https://github.com/microsoft/vscode-cmake-tools/blob/develop/docs/cmake-settings.md#cmake-settings
can we trigger this BuildBeforeRun also by running a test in the testexplorer?
I am not sure which ctest command you use to run, but somehow it seems to skip any setup/cleanup fixtures. This results in tests failing. If I use ctest -L or -R it works fine?
is not working, the test sources don't seem to be linked to the test cases picked up by the plugin
Other extensions have this feature. To have a list of all the individual unit tests and to be able to execute each test individually instead of just having a list of cmake targets.
I have test which is reading some environment variables but finds them unset. When running through CTest they're found, but not when debugging the target. I found that getCmakeTestDebugConfiguration()
only captures the CTest property WORKING_DIRECTORY
property, but not ENVIRONMENT
. vscode.DebugConfiguration
ought to have an env
field (as far as I could tell) which should be populated with the content of said property.
I've noticed that if we have tests which have been configured with the CTest DISABLED property, CTest correctly skips them but the tests show in the test explorer as failed (red X or equivalent icon). I imagine there's some sort of intermediate skipped/warning state that would probably be more appropriate for disabled tests, and failing that it would probably be preferable to have them marked as passed rather than failed.
I believe this applies to any test marked with set_tests_properties(<name> PROPERTIES DISABLED True)
or equivalent, but in our case, the tests are defined in C++ GoogleTest, along the lines of TEST(mytestcase, DISABLED_mytest) {...}
. They're then registered with gtest_discover_tests(test_target)
. That produces a test registration CMake file along the lines of
add_test([=[mytestcase.mytest]=] path/to/test_target.exe [==[--gtest_filter=mytestcase.DISABLED_mytest]==] --gtest_also_run_disabled_tests)
set_tests_properties( [=[--gtest_filter=mytestcase.DISABLED_mytest]=] PROPERTIES DISABLED True)
When run via ctest, that gives
Test project path/to/test_target.exe
Start 1: mytestcase.mytest
1/1 Test #1: mytestcase.mytest ...***Not Run (Disabled) 0.00 sec
No tests were found!!!
But the test shows as failed in test explorer.
In case it's of interest, I notice that CLion has exactly the same issue!
Hello,
It would be nice to add an option to cmakeExplorer.suiteDelimiter to organize by ctest labels
I can't get my Google Test unit tests to show up in the Testing extension window. If I use C++ TestMate I can, but not CMake Test Explorer. I guess what are the differences between the two? Is this expected? If so, maybe I'm misunderstanding the purpose of CMake Test Explorer vs C++ TestMate. If it's not, maybe it's my usage of gtest_discover_tests() instead of add_test()? I know the former calls the later but maybe there's an incompatibility there. Any advice would be appreciated.
The cmake extension may use powershell env access in its variables, ex:
"cmake.buildDirectory": "${workspaceFolder}/_build-${buildType}-${env:OS}"
which is resolved to _build-Windows_NT on windows.
However, the cmake-tesplorer do not do the same, ex:
"cmakeExplorer.buildDir" : "${workspaceFolder}/_build-${buildType}-${env:OS}"
which is resolved to _build-${env:OS}
CMake added support for test presets in their CMakePresets.json file.
With CLion & VS Code starting to support this format now it seems like a good way to configure all build & test configurations in one place.
It would be great if this extension would be able to read those configurations as well.
Hi,
in my CMakeLists file I create some tests using the add_test
function. This runs and configures fine. However, the test cases do not show up in the test explorer.
I have no idea what could be causing this or how to begin debugging it. Any help is appreciated and I'm happy to give as much info as needed.
adding a quick option to ru ctest with specifiq args without change cmakeExplorer.extraCtestLoadArgs option (permanent)
We use CodeLLDB rather than CppTools for debugging. The launch.json configs are mostly the same, and up to version 0.14.2, this worked fine; we just point CMake Test Explorer's cmakeExplorer.debugConfig
setting at an LLDB launch config and it'd work ok. As of 0.15 we're getting an error dialog undefined is not iterable (cannot read property Symbol(Symbol.iterator))
when we attempt to debug a test from the test explorer.
I believe the reason is that CodeLLDB takes its environment settings in a different format from cpptools:
CppTools:
"environment": [
{
"name": "<varname>"
"value": "<blah>"
}
],
CodeLLDB:
"env": {
"<varname>": "<blah>"
}
If I add an empty environment list to the launch config it no longer produces errors, but that's going to be rather confusing! Could you support both formats?
For what it's worth, the same error occurs with a cpptools debug config with no environment list: that's less of a problem since it's created by default, but I think it should still work.
Nice extension you got there!
Seems like there is a built-in support for cppdbg only. Would be great to have some other options (lldb), or even custom build launchers.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.