microsoft/virtualclient

Benchmarking and monitoring automation, designed for cloud, built with .NET.

Virtual Client

---

The following links provide additional information on the Virtual Client project.

• Overview
• Getting Started + How to Build

Getting Started

You will follow the Tutorial to benchmark your system with a quick workload: OpenSSL Speed - SHA256.

Contributing

We welcome your contribution, and there are many ways to contribute to VirtualClient:

• Just say Hi. It inspires us to know that there are fellow performance enthusiatics out there and VirtualClient made your work a little easier.
• Feature Requests. It helps us to know what benchmarks people are using.
• Submit bugs. We apologize for the bug and we will investigate it ASAP.
• Review source code changes. You likely know more about one workload than us. Tell us your insights.
• Review the documentation and make pull requests for anything from typos to new content.
• We welcome you to directly work in the codebase. Please take a look at our CONTRIBUTING.md. Start here and contact us if you have any questions.

Thank you and we look forward to your contribution.

This project welcomes contributions and suggestions. Most contributions require you to agree to a Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us the rights to use your contribution. For details, visit https://cla.opensource.microsoft.com.

When you submit a pull request, a CLA bot will automatically determine whether you need to provide a CLA and decorate the PR appropriately (e.g., status check, comment). Simply follow the instructions provided by the bot. You will only need to do this once across all repos using our CLA.

This project has adopted the Microsoft Open Source Code of Conduct. For more information see the Code of Conduct FAQ or contact [email protected] with any additional questions or comments.

Telemetry Notice

Data Collection.

The software may collect information about you and your use of the software and send it to Microsoft. Microsoft may use this information to provide services and improve our products and services. You may turn off the telemetry as described in the repository. There are also some features in the software that may enable you and Microsoft to collect data from users of your applications. If you use these features, you must comply with applicable law, including providing appropriate notices to users of your applications together with a copy of Microsoft’s privacy statement. Our privacy statement is located at https://go.microsoft.com/fwlink/?LinkID=824704. You can learn more about data collection and use in the help documentation and our privacy statement. Your use of the software operates as your consent to these practices.

VirtualClient Does Not Collect Your Data by Default

VirtualClient does not collect any of your benchmark data and upload to Microsoft. When benchmarking at scale, and leveraging VC's telemetry capabilities, users need to explicitly provide a connection string, that points to a user-owned Azure Data Explorer cluster. VirtualClient does host a Azure storage account to host the benchmark binaries or source. The only information VirtualClient team could infer from usage, is the download traces from Azure storage account.

About Benchmark Examples in Source

VirtualClient has example benchmark outputs in source, for unit-testing purpose, to make sure our parsers work correctly. Those runs might or might not be ran on Azure VMs. The results have also been randomly scrubbed. These examples do not represent Azure VM performance. They are in the source purely for unit testing purposes.

Trademarks

This project may contain trademarks or logos for projects, products, or services. Authorized use of Microsoft trademarks or logos is subject to and must follow Microsoft's Trademark & Brand Guidelines. Use of Microsoft trademarks or logos in modified versions of this project must not cause confusion or imply Microsoft sponsorship. Any use of third-party trademarks or logos are subject to those third-party's policies.

virtualclient's Issues

[VC_1.11.0] Failed to parse SPECjvm metrics from results because index was outside the bounds of the array if run more than 2 times.

Describe the bug
Failed to parse SPECjvm metrics from results because index was outside the bounds of the array if run more than 2 times.

To Reproduce
Steps to reproduce the behavior:

1. All VirtualClient versions has this issue(1.0-1.11);
2. VirtualClient --profile=PERF-CPU-COREMARK.json --iterations=3 --parameters="CompilerVersion=11" --packages=https://virtualclient.blob.core.windows.net/packages
3. System under test configuration (OS, VM size, architecture, etc)
   Failed on both Ubuntu 20.04 & 22.04 & redhat8.2, failure rate 100%

Expected behavior
Parse passed for multiple SPECjvm iterations.

Screenshots
See as the attachment.

Additional context
Only 1st time result could be parsed successfully;
It will be still failed if delete the old VC version and then enable a new one;

Onboard MySQL OLTP workload

Inconsitent presence of UTF-8 BOM at start of files with default git configuration on Linux

Describe the bug
Inconsistent use of BOM at start of files. For example, 1078 .cs files in this repo, 868 have the BOM, the others do not. 920 .json files overall, 132 with the BOM

To Reproduce
find . -type f | grep '.cs$' | xargs file
and then examine output of file command for "BOM" (or not).

Expected behavior
Consistency.

I would remove the BOM altogether, but this will touch many files, and may introduce craziness when outstanding branches attempt to merge. The BOM is not recognized by the Linux file-system per-se, but is handled on a case by case basis. For example, the python json parser will fail on json files starting with the BOM, unless one specifies a different code point than default.

Build scripts build DEBUG versions, not RELEASE

Describe the bug
The build scripts build.cmd and build-packages.cmd build with the Debug configuration, not Release.

1. Optimization. The release configuration enables the compiler to perform optimizations that are not done in debug mode. These optimizations include advanced code restructuring and removal of unnecessary instructions, which can lead to faster execution times. For example, in release the JIT can perform inlining, where it replaces a method call with the method's body to save the overhead of the call.
2. No code contracts. Release mode often exclude code contracts and assertions that are used for checking conditions during development. These checks can slow down the execution because they add extra instructions at runtime.
3. Non-Optimized Libraries. When building in debug, the referenced libraries might also be the non-optimized versions, which can further degrade performance.
4. Non-optimized data structures. Data structures and algorithms might not be optimized for space and time efficiency in debug mode, as the focus is on readability and debugability for humans.

To Reproduce
Inspect the files build.cmd and build-packages.cmd. The arguments to dotnet build and dotnet pack are hard coded to Debug.

Expected behavior
If building for debug configurations is necessary, then there should be a command line argument to set the version to be built or both should be built by default.

Screenshots
See build results for #201. Example: Build

  VirtualClient.Monitors -> D:\a\VirtualClient\VirtualClient\out\bin\Debug\AnyCPU\VirtualClient.Monitors\net6.0\VirtualClient.Monitors.dll
  VirtualClient.Api -> D:\a\VirtualClient\VirtualClient\out\bin\Debug\AnyCPU\VirtualClient.Api\net6.0\VirtualClient.Api.dll
  VirtualClient.TestFramework -> D:\a\VirtualClient\VirtualClient\out\bin\Debug\AnyCPU\VirtualClient.TestFramework\net6.0\VirtualClient.TestFramework.dll
  VirtualClient.ExampleWorkload -> D:\a\VirtualClient\VirtualClient\out\bin\Debug\AnyCPU\VirtualClient.ExampleWorkload\net6.0\ExampleWorkload.dll
  VirtualClient.Actions -> D:\a\VirtualClient\VirtualClient\out\bin\Debug\AnyCPU\VirtualClient.Actions\net6.0\VirtualClient.Actions.dll

Build inconsistent with different .NET SDK versions installed (example: .NET 8 RC2)

While running a build with .NET 8 RC2 SDK installed, the build.cmd can fail. Specifying the SDK tool chain in global.json to use .NET 7 SDK resolves this issue. See #211 for fix.

C:\Program Files\dotnet\sdk\8.0.100-rc.2.23502.2\Sdks\Microsoft.NET.Sdk\targets\Microsoft.NET.Sdk.FrameworkReferenceResolution.targets(106,33): error MSB4030: "true -p:TrimUnusedDependenc
ies=true" is an invalid value for the "PublishTrimmed" parameter of the "ProcessFrameworkReferences" task. The "PublishTrimmed" parameter is of type "System.Boolean". [E:\src\GitHub\rjmur
illo\VirtualClient\src\VirtualClient\VirtualClient.Dependencies\VirtualClient.Dependencies.csproj]
C:\Program Files\dotnet\sdk\8.0.100-rc.2.23502.2\Sdks\Microsoft.NET.Sdk\targets\Microsoft.NET.Sdk.FrameworkReferenceResolution.targets(106,33): error MSB4030: "true -p:TrimUnusedDependenc
ies=true" is an invalid value for the "PublishTrimmed" parameter of the "ProcessFrameworkReferences" task. The "PublishTrimmed" parameter is of type "System.Boolean". [E:\src\GitHub\rjmur
illo\VirtualClient\src\VirtualClient\VirtualClient.Contracts\VirtualClient.Contracts.csproj]
C:\Program Files\dotnet\sdk\8.0.100-rc.2.23502.2\Sdks\Microsoft.NET.Sdk\targets\Microsoft.NET.Sdk.FrameworkReferenceResolution.targets(106,33): error MSB4030: "true -p:TrimUnusedDependenc
ies=true" is an invalid value for the "PublishTrimmed" parameter of the "ProcessFrameworkReferences" task. The "PublishTrimmed" parameter is of type "System.Boolean". [E:\src\GitHub\rjmur
illo\VirtualClient\src\VirtualClient\VirtualClient.Monitors\VirtualClient.Monitors.csproj]
C:\Program Files\dotnet\sdk\8.0.100-rc.2.23502.2\Sdks\Microsoft.NET.Sdk\targets\Microsoft.NET.Sdk.FrameworkReferenceResolution.targets(106,33): error MSB4030: "true -p:TrimUnusedDependenc
ies=true" is an invalid value for the "PublishTrimmed" parameter of the "ProcessFrameworkReferences" task. The "PublishTrimmed" parameter is of type "System.Boolean". [E:\src\GitHub\rjmur
illo\VirtualClient\src\VirtualClient\VirtualClient.Api\VirtualClient.Api.csproj]
C:\Program Files\dotnet\sdk\8.0.100-rc.2.23502.2\Sdks\Microsoft.NET.Sdk\targets\Microsoft.NET.Sdk.FrameworkReferenceResolution.targets(106,33): error MSB4030: "true -p:TrimUnusedDependenc
ies=true" is an invalid value for the "PublishTrimmed" parameter of the "ProcessFrameworkReferences" task. The "PublishTrimmed" parameter is of type "System.Boolean". [E:\src\GitHub\rjmur
illo\VirtualClient\src\VirtualClient\VirtualClient.Common\VirtualClient.Common.csproj]
C:\Program Files\dotnet\sdk\8.0.100-rc.2.23502.2\Sdks\Microsoft.NET.Sdk\targets\Microsoft.NET.Sdk.FrameworkReferenceResolution.targets(106,33): error MSB4030: "true -p:TrimUnusedDependenc
ies=true" is an invalid value for the "PublishTrimmed" parameter of the "ProcessFrameworkReferences" task. The "PublishTrimmed" parameter is of type "System.Boolean". [E:\src\GitHub\rjmur
illo\VirtualClient\src\VirtualClient\VirtualClient.Actions\VirtualClient.Actions.csproj]
C:\Program Files\dotnet\sdk\8.0.100-rc.2.23502.2\Sdks\Microsoft.NET.Sdk\targets\Microsoft.NET.Sdk.FrameworkReferenceResolution.targets(106,33): error MSB4030: "true -p:TrimUnusedDependenc
ies=true" is an invalid value for the "PublishTrimmed" parameter of the "ProcessFrameworkReferences" task. The "PublishTrimmed" parameter is of type "System.Boolean". [E:\src\GitHub\rjmur
illo\VirtualClient\src\VirtualClient\VirtualClient.Core\VirtualClient.Core.csproj]
C:\Program Files\dotnet\sdk\8.0.100-rc.2.23502.2\Sdks\Microsoft.NET.Sdk\targets\Microsoft.NET.Sdk.FrameworkReferenceResolution.targets(106,33): error MSB4030: "true -p:TrimUnusedDependenc
ies=true" is an invalid value for the "PublishTrimmed" parameter of the "ProcessFrameworkReferences" task. The "PublishTrimmed" parameter is of type "System.Boolean". [E:\src\GitHub\rjmur
illo\VirtualClient\src\VirtualClient\VirtualClient.Main\VirtualClient.Main.csproj]

Write documentations for commercial workloads