Unittest failures in build which don't seem related to code changes

[microsoft]

Unittest failures in build which don't seem related to code changes

[qubeat]

May be related?: H2 simulation works with error after other tests (or if you simply run it few times) https://quantum.uservoice.com/forums/906946-samples-and-documentation

[anpaz]

I noticed this problem too, and I have trying to reproduce it with no success. As you point out is not the seed. Could be that we're not releasing resources. My intuition, though, is that this is a multi-threading problem, however you should be able to create multiple instances of a Simulator and run each one in parallel (which is what the tests would do), so I'm confused about what the actual problem is.
@qubeat, the H2 problem could be related, however the h2 simulation is random in nature, and we are not providing a seed, so it's not too surprising to see random failures in it.
If you guys find out anything, please let us know.

[qubeat]

@anpaz-msft The failures are too big and not quite random (most are around -0.4). They also seem never appear at first run of dotnet. Seems the more runs, the more errors. Below rather an illustrative example:

[anpaz]

The H2 simulation algorithm is probabilistic at different levels and it's not unexpected to see it failing; the reason for the jumps you see there is that the algorithm is finding not the ground state but a higher excitation level, which is an expected result. If you look carefully at the code, you'll notice that for each bond length we calculate the energy 3 times and select the minimum value, because, again the result may give you any excitation level and not just the ground state.

[RolfHuisman]

Same issues to try to rerpo it here. I tried doing a diff of the log files of passed and failed builds of the same release, and the only thing I noticed was is that that the parallism between a failed build and a passed one differs. But haven't found a correlation yet. Running parallel tests hasn't failed locally either, so it can indeed be a disposing issue.

[qubeat]

@anpaz-msft I change code to repeat 5 times and it seems work better, at least no errors after couple runs in VS Code. However I am not sure about excited states, it may be some calculation artifact as well - there is a "pit" on graphs

[RolfHuisman]

Starting to suspect the init or destroy method in Microsoft.Quantum.Simulator.Runtime.dll. Validating hypothesis. (Nope)

[RolfHuisman]

Had a linux machine run

while dotnet test Samples/UnitTesting; do :; done;

and

while dotnet test LibraryTests; do :; done;

for a number of hours on a 2 vcpu, ubuntu machine but can't seem to repro this.

[RolfHuisman]

Hmm.. LibraryTest seems to lock up with

Measured: [One;Zero;Zero;One]
measured: [Zero;One;Zero;One]
measured: [One;One;Zero;One]
measured: [Zero;Zero;One;One]
measured: [One;Zero;One;One]
measured: [Zero;One;One;One]
measured: [One;One;One;One]"

as the last output when I run these two parallel.
Can't get this to repro on windows. Getting the feeling this is something specific to the linux version.

[RolfHuisman]

After installing a nostalgia version of ubuntu (14.02) and running the tooling on a 2 vcpu vm, I can finally get the issue reproed. @anpaz-msft which os and compiler are you using to build the the Linux runtime dll ? Would it be possible to staticly link the openmp library ?

[RolfHuisman]

Testing ExpIZArcTan2NC against Exp{}. Attempt: 0
Trying ...
[xUnit.net 00:00:15.6653463] RepeatUntilSuccessCircuitsTest [FAIL]
Failed RepeatUntilSuccessCircuitsTest
Error Message:
Microsoft.Quantum.Simulation.Core.ExecutionFailException : Error:the probability to measure |+⟩ in the first ancilla must be 0.5
Expected: 0.5
Actual: 0.166666666666667
Stack Trace:
at Microsoft.Quantum.Simulation.Simulators.QuantumSimulator.QSimAssertProb.<get_Body>b__6_0(ValueTuple6 _args) at Microsoft.Quantum.Simulation.Core.Operation2.Apply(I a)
at Microsoft.Quantum.Simulation.Core.Operation2.Microsoft.Quantum.Simulation.Core.ICallable<I,O>.Apply(I args) at Microsoft.Quantum.Samples.UnitTesting.ExpIZArcTan2NC.<get_Body>b__54_0(Qubit __in) in /home/reddog/Quantum/Samples/UnitTesting/RepeatUntilSuccessCircuits.qs:line 88 at Microsoft.Quantum.Simulation.Core.Operation2.Apply(I a)
at Microsoft.Quantum.Simulation.Core.Operation2.Microsoft.Quantum.Simulation.Core.ICallable.Apply(Object args) at Microsoft.Quantum.Canon.ApplyToFirstQubit.<>c.<get_Body>b__7_0(ValueTuple2 __in) in /home/reddog/Quantum/Microsoft.Quantum.Canon/Combinators/ApplyToFirst.qs:line 28
at Microsoft.Quantum.Simulation.Core.PartialApplication3.<get_Body>b__20_0(P a) at Microsoft.Quantum.Simulation.Core.Operation2.Apply(I a)
at Microsoft.Quantum.Simulation.Core.Operation2.Microsoft.Quantum.Simulation.Core.ICallable.Apply(Object args) at Microsoft.Quantum.Extensions.Testing.AssertOperationsEqualReferenced.<get_Body>b__27_0(ValueTuple3 __in)
at Microsoft.Quantum.Simulation.Core.Operation2.Apply(I a) at Microsoft.Quantum.Simulation.Core.Operation2.Microsoft.Quantum.Simulation.Core.ICallable<I,O>.Apply(I args)
at Microsoft.Quantum.Samples.UnitTesting.RepeatUntilSuccessCircuitsTest.<get_Body>b__34_0(QVoid __in) in /home/reddog/Quantum/Samples/UnitTesting/RepeatUntilSuccessCircuitsTests.qs:line 42
at Microsoft.Quantum.Simulation.Core.Operation`2.Apply(I a)
at Microsoft.Quantum.Simulation.XUnit.TestOperation.<get_TestOperationRunner>b__6_0(IOperationFactory sim) in C:\build-agent_work\7\s\src\simulation\Testing\TestOperation.cs:line 40
at Microsoft.Quantum.Samples.UnitTesting.SimulatorTestTargets.QuantumSimulatorTarget(TestOperation operationDescription) in /home/reddog/Quantum/Samples/UnitTesting/QuantumSimulatorTestTargets.cs:line 59
Standard Output Messages:
The seed used for this test is 763837676
Testing ExpIZArcTan2NC against Exp{}. Attempt: 0
Trying ...

Total tests: 15. Passed: 12. Failed: 3. Skipped: 0.
Test Run Failed.
Test execution time: 17.2773 Seconds
rolf@Blade:~/Quantum$

[RolfHuisman]

Running a simple test;

export OMP_NUM_THREADS=1
dotnet test Samples/UnitTesting

Runs correctly,

export OMP_NUM_THREADS=2
dotnet test Samples/UnitTesting

Failes on Ubuntu 14.04 with G++-4.6 from ubuntu-toolchain-r-test.
So it implies there is a race condition i.c.w. OpenMP.
Given I can't repro it with later ubuntu versions, it might be caused by this specific version of openmp.

[RolfHuisman]

Repro steps I used
Spin up on Azure a "Ubuntu Server 14.04 LTS" from Canonical (Standard B2s (2 vcpus, 4 GB memory))

wget -q https://packages.microsoft.com/config/ubuntu/14.04/packages-microsoft-prod.deb
sudo dpkg -i packages-microsoft-prod.deb
sudo apt-get install apt-transport-https
sudo apt-get update
sudo apt-get install dotnet-sdk-2.1
sudo apt-get install git
sudo add-apt-repository ppa:ubuntu-toolchain-r/test
sudo apt-get update
sudo apt-get install g++-4.9
git clone https://github.com/Microsoft/Quantum.git
cd Quantum
dotnet build
dotnet test Samples/UnitTesting

Should get:

Total tests: 15. Passed: 12. Failed: 3. Skipped: 0.
Test Run Failed.

[anpaz]

A bug on OpenMP is right on the money of where we were betting the problem would be.
Because this appears to be solved on the latest version and you found a workaround for travis, I don't want to do any changes on our end (like statically linking OpenMP) to fix it.
I will accept the change to reduce the number of threads on travis to 1 so we don't see this failures anymore.

[anpaz]

Thanks @RolfHuisman, I've merged the changes.