Running test suites remotely (via WinRM)

[microsoft]

Running test suites remotely (via WinRM)

[GSam]

I normally get this:

Error: Test host process exited unexpectedly.

Error: Failed to queue test run 'Administrator@TESTING-1 2017-10-26 03:31:19': Unable to start the agent process. 

PsExec doesn't seem to help in any way, so I'm at a loss as to how to workaround this.

[VivianTian]

Hi, GSam
Currently, we haven't test with WinRM and there's no guidance for WinRM yet. I would suggest you have a try with powershell.
Also, we would have an investigation on how to test with WinRM and back here to give you a response later.
Thanks for patience.

[GSam]

The use case is that I want to trigger the tests remotely. What protocol it uses doesn't really matter, but I've discovered a few additional details:

• Running with %vspath%"..\IDE\mstest.exe /noisolation appears to trigger the test code to run but it seems there are other issues with running without isolation (while also running under WinRM).
  I also don't know if the tests will behave any differently under isolation.

First of all, if you nest this command within another shell, the tests fail to begin in the same way originally. Furthermore, mstest.exe will always hang at the end of a run. This would be fine if I could capture the output (and kill it at the appropriate time), but tests suddenly fail upon redirection of standard output. This seems to be related to the problem with nested shells.

<Message>Class Initialization method Microsoft.Protocol.TestSuites.Kerberos.TestSuite.KILE.KileInteractiveLogonTest.ClassInitialize threw exception. System.IO.IOException: System.IO.IOException: The handle is invalid.
.</Message>
          <StackTrace>    at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
   at System.IO.__Error.WinIOError()
   at System.Console.SetBufferSize(Int32 width, Int32 height)
   at System.Console.set_BufferHeight(Int32 value)
   at Microsoft.Protocols.TestTools.TestClassBase.Initialize(TestContext testContext, String testSuiteName)
   at Microsoft.Protocols.TestTools.TestClassBase.Initialize(TestContext testContext)
   at Microsoft.Protocol.TestSuites.Kerberos.TestSuite.KILE.KileInteractiveLogonTest.ClassInitialize(TestContext testContext) in e:\Jenkins\workspace\CodeSign-TestSuites\TestSuite\TestSuites\Kerberos\src\TestSuite\KILE\KileInteractiveLogonTest.cs:line 20
</StackTrace>

Even addressing (or ignoring) these issues, once you combine WinRM and mstest.exe, the tests suddenly fail again for no clear reason. It appears to be related to environment issues.

I've found one particular workaround by using Win32-OpenSSH and python pexpect (with mstest.exe) to control a remote shell. The tests don't seem to fail surreptitiously, it still hangs, but the standard output can be retrieved correctly. So far this seems the most viable solution for us.

• Triggering a test run via Team Foundation server and remote calls seems entirely possible. Unfortunately this requires significantly more setup time (part of which is probably manual or at least hard to automate). I would prefer not to have to do this to trigger a test run remotely because having a long-term Windows instance for this sole purpose is not practical for us (and neither is setting it up every time), but it does appear the most standard way to do so.

• There might be a task scheduler based solution. It appears that you can trigger the tests in this way without the nested shell issue occurring. But it needs to run under the correct user context and without an open session, which seems a bit bothersome to configure (assuming it will still work once it's been setup).

[JiajunW]

Hi Sam,

The guide for using PowerShell Core to remotely trigger test suite is documented in our wiki. https://github.com/Microsoft/WindowsProtocolTestSuites/wiki/Remotely-Trigger-Test-Suites

This solution leverages PowerShell Core for its cross platform flexibility, and it uses a task scheduler based solution to avoid the issue you may have encountered when directly executing the test suite in a remote shell session.