microsoft / vstest Goto Github PK
View Code? Open in Web Editor NEWVisual Studio Test Platform is the runner and engine that powers test explorer and vstest.console.
License: MIT License
Visual Studio Test Platform is the runner and engine that powers test explorer and vstest.console.
License: MIT License
User cannot re-run a test if debugger is detached while debugging a test. Cancel doesn't cancel the the request.
Debugger should detach.
Test run should be aborted.
Run All should be active for next run.
Debugger detaches.
Test run remains in inprogress.
Run All is not active and cancel doesn't cancel the request.
VS/Preview/25719.00
VSIX: https://github.com/Microsoft/vstest/releases/tag/v15.0.0-preview-20160914-02
VSTest runner does not print help message when run without any arguments.
Run vstest.console.exe
without any args
A help message is shown.
Just copyright message is shown and the runner exits.
vstest.console.exe version from 53689b0 commit.
"dotnet.exe vstest.console.dll testDllName.dll" always return with exit-code 1
run a .net core test with vstest.console runner.
The process should exit with exit-code 0
If user has a very large project, the discovered tests should show in batches in the test explorer. Currently the test explorer just stays in progress until all tests are discovered.
Tests should start showing up in chunks.
Test explorer remains in progress until all tests are discovered.
VS: alm/25722.00
TestPlatform: d9dbd28
The object model needs to be cleaned up based on 11.0 version. Some utilities can be refactored out.
E.g. RunSettingsUtilities. See the PR for more details: https://devdiv.visualstudio.com/DevDiv/VS.in%20Agile%20Testing%20IDE/_git/MSTest/pullrequest/33962
Validate if the existing test project templates work with asp.net core.
Throw a proper message for the user to understand the state and be able to install the dotnet cli.
Describe the issue you've observed
Test run should complete and report results.
The run does not complete.
Please share additional details about the test environment.
Operating system, Build version of vstest.console
They are multiple places
communicationManager.ReceiveRawMessage();
andcommunicationManager.SendMessage
can throw IOException.
Randomly close testhost and vstest.console process
Expected behavior
Should properly handle all kind of process crashes
Process has been waiting or execption not properly handled.
For .netcore portable PDBs are the primary symbol format. OM needs to support parsing these symbols.
We observed that the call to RunTests was hanging for ever. This was likely a bug in TP that subsequently got fixed but it would be nice to no hang and return error code so that the caller can continue.
At the moment TestPlatform uses two different Newtonsoft.Json version: 7.0.1 (specified in TranslationLayer.Nuspec) and 9.0.1 (specified everywhere else). However VS15 ships with Newtonsoft.Json version 8.0.3.
What steps can reproduce the defect?
Please share the setup, commandline for vstest.console, sample project, target
framework etc.
Share the expected output
What is the behavior observed?
Please share additional details about the test environment.
Operating system, Build version of vstest.console
User gets an error when TestPlatform.vsix is installed in a VS preview.
artifacts\Debug\TestPlatform.vsix
VSIX installation should be successful.
VSIX installation shows following warning, it is successful if warning is ignored.
---------------------------
VSIX Installer
---------------------------
This extension is not compatible with Visual Studio 15 and will not be supported in RC. Installation may cause Visual Studio to be unstable. Do you want to continue?
---------------------------
Yes No
---------------------------
VS preview. VSIX from d9dbd28.
We have observed significant delay when test discovery and execution is requested through VsTestConsoleWrapper. For example, the following log shows discovery took ~7 seconds and execution took ~4 second.
[2016-08-26 09:51:55.084 Info] - TestDriver - [Workflow 1] Received TestRequestEvent
[2016-08-26 09:51:55.084 Info] - TestDriver - [Workflow 1] Start test discovery.
[2016-08-26 09:52:02.724 Info] - TestDriver - [Workflow 1] Discovered 0 tests from ClassLibrary2.dll, updating cached data.
[2016-08-26 09:52:02.724 Info] - TestDriver - [Workflow 1] Discovered 1 tests from TestLibrary.dll, updating cached data.
[2016-08-26 09:52:02.740 Info] - TestDriver - [Workflow 1] Start executing 1 tests from 1 assemblies.
[2016-08-26 09:52:06.762 Info] - TestDriver - [Workflow 1] Handle test result for TestSomething2: Passed
[2016-08-26 09:52:06.762 Info] - TestDriver - [Workflow 1] Tests execution completed successfully.
[2016-08-26 09:52:06.762 Info] - TestDriver - [Workflow 1] Notify Aggregator about completion of test run
This behavior was observed on both Preview 3 VS and latest willow based build from ML branch. Note that this doesn't happen all the time. I didn't notice this before upgrading to latest TestPlatform NuGet packages, it was usually around 1 second, not sure if this is related to the upgrade.
When the TestPlatform bits are loaded, they cause our binaries to load when they search for the ITestExtensionComponentProvider
instances.
Specifically this is done in the LiveUnitTestingClientProvider
class, where Assembly.LoadFrom
is explicitly called in the InitializeTestExtension
method.
An API should be provided where we can explicitly enable our 'component provider'/'service client' once our package is known to be loaded.
If vstest.console exits abruptly, the child test host processes must exit.
Test host should exit.
Test host keeps running.
VS: alm/25722.00
TestPlatform: d9dbd28
System.NullReferenceException: Object reference not set to an instance of an object.
at Microsoft.VisualStudio.TestPlatform.CommunicationUtilities.SocketCommunicationManager.WriteAndFlushToChannel(String rawMessage)
at Microsoft.VisualStudio.TestPlatform.Client.DesignMode.DesignModeClient.ConnectToClientAndProcessRequests(Int32 port, ITestRequestManager testRequestManager)
at Microsoft.VisualStudio.TestPlatform.CommandLine.Processors.PortArgumentExecutor.Execute()
at Microsoft.VisualStudio.TestPlatform.CommandLine.Executor.ExecuteArgumentProcessor(IArgumentProcessor processor, Int32& exitCode)
at Microsoft.VisualStudio.TestPlatform.CommandLine.Executor.Execute(String[] args)
at Microsoft.VisualStudio.TestPlatform.CommandLine.Program.Main(String[] args)
\\mlangfs1\public\kevinh\vstest.console.dmp
MSTest templates should be available for .net core. These templates should be pushed into the dotnet/sdk
repo.
First build of TestPlatform.sln on VS is failing for SmokeTests because of a relative path for packages directory.
Workaround is to do a commandline build first.
In .net core, the adapter can be picked up from the output directory of test assembly. This directory should be probed by test platform. With this change, the user will not need to specify a --testAdapterPath
with vstest
command.
TestPlatform vsix will include feature flags through an artifact. TestWindow can use testplatform features based on what is enabled.
What steps can reproduce the defect?
Please share the setup, commandline for vstest.console, sample project, target
framework etc.
Share the expected output
What is the behavior observed?
Please share additional details about the test environment.
Operating system, Build version of vstest.console
Provide a /diag option in vstest.console to capture trace logs. User may specify a log capture file with /diag:log.txt
. Traces for testhost processes should be captured as log.testhost.<id>.txt
where <id>
is a unique id such as timestamp.
Same trace logs should be available in stdout when user sets the VSTEST_TRACE
environment variable.
Test adapters added as reference to the test app are enlisted in the testapp.deps.json
. Test platform should use these adapters for discovery/execution of tests.
User should be able to Debug All
and Debug Selected
tests in a .net core project.
TranslationLayer throws an error on the communication channel if vstest.console exits
Run All should work on step 3
Following error is thrown:
System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. ---> System.IO.IOException: Unable to write data to the transport connection: An established connection was aborted by the software in your host machine. ---> System.Net.Sockets.SocketException: An established connection was aborted by the software in your host machine
at System.Net.Sockets.Socket.Send(Byte[] buffer, Int32 offset, Int32 size, SocketFlags socketFlags)
at System.Net.Sockets.NetworkStream.Write(Byte[] buffer, Int32 offset, Int32 size)
--- End of inner exception stack trace ---
at System.Net.Sockets.NetworkStream.Write(Byte[] buffer, Int32 offset, Int32 size)
at System.IO.Stream.WriteByte(Byte value)
at System.IO.BinaryWriter.Write(Byte value)
at System.IO.BinaryWriter.Write7BitEncodedInt(Int32 value)
at System.IO.BinaryWriter.Write(String value)
at Microsoft.VisualStudio.TestPlatform.CommunicationUtilities.SocketCommunicationManager.WriteAndFlushToChannel(String rawMessage) in D:\dd\gh\Microsoft\vstest\src\Microsoft.TestPlatform.CommunicationUtilities\SocketCommunicationManager.cs:line 241
at Microsoft.VisualStudio.TestPlatform.CommunicationUtilities.SocketCommunicationManager.SendMessage(String messageType, Object payload) in D:\dd\gh\Microsoft\vstest\src\Microsoft.TestPlatform.CommunicationUtilities\SocketCommunicationManager.cs:line 202
at Microsoft.TestPlatform.VsTestConsole.TranslationLayer.VsTestConsoleRequestSender.DiscoverTests(IEnumerable`1 sources, String runSettings, ITestDiscoveryEventsHandler eventHandler) in D:\dd\gh\Microsoft\vstest\src\Microsoft.TestPlatform.VsTestConsole.TranslationLayer\VsTestConsoleRequestSender.cs:line 107
at Microsoft.TestPlatform.VsTestConsole.TranslationLayer.VsTestConsoleWrapper.DiscoverTests(IEnumerable`1 sources, String discoverySettings, ITestDiscoveryEventsHandler discoveryEventsHandler) in D:\dd\gh\Microsoft\vstest\src\Microsoft.TestPlatform.VsTestConsole.TranslationLayer\VsTestConsoleWrapper.cs:line 104
Similar behaviour for Debug All Scenario is observed as well.
vstest.console from 576f536 bits.
Test description attributes like Description, CssIteration, WorkItem, CssProjectStructure do not show up in the trx log. Only Owner and Priority are part of the log.
The trx should have the description string associated with the test in the log.
The description string does not show up in the log.
I'm trying this on Dev15 prev-5 and on Windows 10.
Describe the issue you've observed
Share the expected output
During discovery, vstest.console takes up to 50% of CPU. This will slow down other tasks like editing etc. while tests are getting discovered.
Same CPU usage is observed during execution of tests.
Please share additional details about the test environment.
Operating system, Build version of vstest.console
testhost.exe
with a specific testhost.deps.json
. In this case, user's dependencies don't resolve appropriatelyWhile this behavior is fine for desktop clr, for .netcore these premises may not work.
testproject.dll
only after a dotnet publish
, so that all dependencies are copied to output directory. dotnet build
doesn't copy all dependencies to output folder.testhost.deps.json
is used, System.Xml
used by testhost
will be loaded, although user wanted to use a later version of System.Xml
. Multi-targeting will be hard(There is no change in the test platform behavior for test projects targeting net46
or desktop clr. We concentrate only on netcore
scenarios below)
In dotnet
core, set of dependencies for a project testproject
are resolved with testproject.deps.json
file available in the output path of the project. Test platform should invoke testhost.dll
with the deps file of testproject
i.e. testproject.deps.json
.
Since testhost.dll
is always invoked with testproject.deps.json
, it is not possible to reuse the testhost for running tests of all test projects. For each test project/container, a separate testhost
process will be invoked. This will have a performance impact (compared to existing implementation where a single testhost process is used for running tests from all assemblies).
Adapters are referenced in the testproject
as a exclude: compile
dependency. Test platform will probe the adapter from paths in testproject.deps.json
. The lookup convention remains unchanged (*.testadapter.dll
are considered test adapters).
Note that user may still provide a logger (or data collector) extension with a path. This will require the logger to be published.
Every test project will include a exclude: compile
type dependency on Microsoft.TestPlatform.TestHost
nuget package. During project build, nuget takes the responsibility to compute the dependencies required for both testproject
and testhost
and create the testproject.deps.json
.
testhost.dll
is always compiled with the lowest netcoreapp
runtime. This allows it to be retargeted to any higher runtime as specified in testproject.deps.json
.
If the tests are run on the same machine that has source enlisted and build artifacts, simple dotnet test
should work.
For CDP, where the build artifacts are consumed, a dotnet publish
is required while publishing the artifacts. This ensures all dependencies are available for dotnet vstest
on the test machine. In this case, user needs to provide path to testproject.dll
directly.
dotnet publish
for a netcore
target produces a netcoreapp1.0
directory with all dependencies. A runtime specific directory is created for net46
target.
asp.net
test apps?Please specify the type of issue:
Currently, TestPlatform NuGet packages contains Newtonsoft.Json.dll
, which should be specified as a NuGet dependency instead. This is causing problem for projects that referencing both Newtonsoft.Json
and TestPlatform packages.
On running "Test -> Debug All" from the VS; the execution just hangs.
All tests should run under debugger. Breakpoints hit.
Hang.
VS/Preview/25719.00
VSIX: https://github.com/Microsoft/vstest/releases/tag/v15.0.0-preview-20160914-02
In .NetCore vstest.console needs full path of the test dll to run the tests
It should run the test
no test getting discover
This will help us get test coverage. There are two items in this:
dotnet test3
dotnet test3
When user has selected a testsettings file, unit tests are not discovered by test platform.
Tests should be discovered in test explorer
Tests are not discovered.
VS: alm/25722.00
TestPlatform VSIX: d9dbd28
If the tests are already discovered and available in the test explorer. Run All should not discover them again.
Tests should directly run and results are displayed.
Tests are discovered again. This makes the user wait for two minutes (for the 10k tests project).
VS: alm/25722.00
TestPlatform: d9dbd28
Test explorer uses below config file to read features to enable for TPv2. The default vsix should include testplatform.config
.
<?xml version="1.0" encoding="utf-8"?>
<configuration>
<appSettings>
<add key="feature.net40" value="false" />
</appSettings>
</configuration>
OM has a dependency on json.net, it may lead to diamond dependency problems if the test depends on a different version of json.net.
Reevaluate the dependencies for TL, it needs to package TP.Common etc. at the moment.
Cancelling Test Debugging Session through Test Window or Toolbar menu is not working.
Debugging session should get terminated, user should be able to start run/debug tests.
Debugging session is not terminated, cannot start new debug/run session. Have to kill vstest.console or restart visual studio.
There are not templates for VB for Unit testing for .NET core.
VS IDE >> File -> New Project -> There is no Unit Test template targeting VB.
The Microsoft.TestPlatform.CommunicationUtilities binary directly references Newtonsoft.Json v9.0.0 while VS has standardized on v8.0.3.
Looking at the project, it references v8.0.3, so the project reference to v9 is indirect and NuGet or MSBuild resolves it to be v9 when the compilation actually occurs (causing the binary to have the direct reference to v9).
For a netcore test app, vstest.console must be able to run tests against net46
or any specified platform.
Test platform should provide a test
target and a task to consume the output of a test project to run it with vstest.console.
The task and targets should be available as a nuget package consumed in CLI repo.
Double click on discovered tests doesn't navigate to source code. Source information doesn't show up in the Summary pane.
Tests should be discovered. Double click on test should navigate to source file.
Tests are discovered. Double click on test doesn't navigate to source.
VS: alm/25722.00
Testplatform: d9dbd28
Not sure if related, but OM manifest file for reg free com is not there in TPv2 extension folder, they are available in TPv1 folder.
Tracking issue for blockers to enable test impact:
User should be able to run multiple test assemblies for .net core. In addition, a sln containing multiple .net core projects should have support for discover/execution of tests.
vstest.console doesn't run tests from test container with relative path.
Run the following command:
dotnet.exe c:\vstest\vstest.console.dll test\testproject.dll /platform:x64 /testAdapterPath:path\to\adapter
Tests should run.
Test run fails with following error:
Could not find file test\testproject.dll
Note that tests run if an absolute path to testproject.dll is provided.
C# Static CC is not working
It actually worked when set environment variable "TestingTools_DisableTestplatformV2" to 1.
In .net core tests, user needs to reference 3 packages to run tests:
Can we reduce this?
Test Framework needs to be separate in the options below since each test framework also provides its own console runner.
Options available:
When I replaced direct reference to checked-in assemblies with NuGet package from vs-test myget feed, run test through VSTestConsoleWrapper with current test adapters is broken.
Observed behavior is Runtests
returns as Aborted
. Below is the log from TestImpact:
[2016-08-22 16:54:47.356 Info] - TestDriver - [Workflow 1] Start test discovery.
[2016-08-22 16:54:47.989 Info] - TestDriver - [Workflow 1] Discovered 1 tests from ClassLibrary3.dll, updating cached data.
[2016-08-22 16:54:47.989 Info] - TestDriver - [Workflow 1] Start executing 1 tests from 1 assemblies.
[2016-08-22 16:54:48.460 Info] - TestDriver - [Workflow 1] Tests execution cancelled/aborted, 1 tests left unexecuted.
Please note that run test through Test Explorer doesn't work either, the behavior is run test never finish.
------ Load Playlist started ------
========== Load Playlist finished (0:00:00.002) ==========
------ Discover test started ------
========== Discover test finished: 1 found (0:00:00.686) ==========
------ Run test started ------
Here some info about my setup:
I have uploaded all the related bits to help you identify the issue to \mlangfs1\public\gel\TestAdapterIssue
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.