dotnet/jitutils

Dotnet JIT code gen utilities - jitutils

This repo holds a collection of utilities used by RyuJIT developers to automate tasks when working on CoreCLR.

Summary

Current tools include:

1. Assembly diffs: jit-diff, jit-dasm, jit-dasm-pmi, jit-analyze, jit-tp-analyze.
2. CI jobs information: cijobs.
3. JIT source code formatting: jit-format.
4. General tools: pmi
5. Experimental tools: performance-explorer
6. BenchmarkDotNet Analysis

Getting started

1. Clone the jitutils repo:

    git clone https://github.com/dotnet/jitutils

2. Install a recent .NET Core SDK (including the dotnet command-line interface, or CLI) from here.

3. Build the tools:

    cd jitutils
    bootstrap.cmd

(on non-Windows, run bootstrap.sh. NOTE: On Mac, you need to first use ulimit -n 2048 or the dotnet restore part of the build will fail.)

4. Optionally, add the built tools directory to your path, e.g.:

    set PATH=%PATH%;<root>\jitutils\bin

jitutils's Issues

SuperPMI Replay

A simple way to replay a superPMI collection.

Expand the SuperPMI Collection to include more than CoreCLR/FX

Some potential ideas are:

• Roslyn Build and Test
• Asp.NET build/test

PMI: options for using local .Net Core builds

As part of dev inner loop workflow it will be common to want to run PMI on local .Net Core builds. Here are some initial ideas on various approaches we could take.

For diffing and correctness checking we would like to use a checked build of the jit. So we either need a jit build that is compatible with the host CLI, or else we need a way to invoke a simpler locally built runner (say corerun).

For robustness we initially will likely use DRIVEALL to run the locally built jit in a child process, so we can push past asserts.

Once a particular method becomes interesting (from an assert or asm diff) we can use PREPONE to look at just that method. Currently that does not use a child process. So perhaps we need a DRIVEONE to parallel DRIVEALL.

If we go the CLI route, we can use the instructions in coreclr for using local nuget packages. This perhaps is most simply done by cloning the project file and adding new entries for the runtime. Or we can publish and then patch. It might make sense to always run the jit under test as an altjit. This would mean the cross-arch and self-arch jitting workflows would be the same.

Benchmark Dot Net seemingly solves similar issues and it might be interesting to consider adopting their approach; this would conceptually allow cross-runtime diffing (say comparing .net core vs .net framework codegen, or .net vs mono).

We probably want custom complus settings to apply only to the methods jitted via PMI. Even if we prejit/crossgen PMI.exe there will still be some residual jitting from SIMD methods in the core library. So we might need an assembly-inclusive altjit filter (currently there's only an exclusion list).

Jit-dasm should use /fragile crossgen

Jit-dasm uses crossgen to avoid having to execute the code, but crossgen has several limitations compared to jitting code. Crossgen has some sort of /fragile option to lift those constraints, and using this mode would mean the generated diffs would be more representative of diffs that will arise in jitted code. Presumably it will be interesting to be able to select whether /fragile is desired or not when generating disasm/diffs, and I think that /fragile should be the default.

SuperPMI fix SuperPMI Asserts and NYIs

Currently under mac/linux there are a few SuperPMI errors that happen.

Assertions

• SuperPMI assertion 'EmbedGenericHandle != nullptr' failed ("Encountered an empty LWM while looking for ...")
• SuperPMI assertion 'GetFieldType != nullptr' failed ("Didn't find anything for 000001C688F60C20")
• SuperPMI assertion 'CanSkipMethodVerification != nullptr' failed ("Didn't find anything for 00000238DAA9B9F8")

As well, while crossgenning the system frameworks SuperPMI NYIs consistantly with a few helpers not being implemented.

NYIs

• ERROR: getReadyToRunHelper NYI

jit-diff handling of errors (e.g., JIT asserts) should be improved

I ran diffs with:

jit-diff diff -o c:\diffs -t 4 -a -f -v --test_root c:\gh\coreclr\bin\tests\Windows_NT.x86.Release --core_root c:\gh\coreclr\bin\tests\Windows_NT.x86.checked\Tests\Core_Root -b e:\gh\coreclr2\bin\Product\Windows_NT.x86.checked -d c:\gh\coreclr\bin\Product\Windows_NT.x86.checked

It ran all baselines (as far as I know), but two failed with asserts. It then output:

Dasm commands returned with total of 2 failures

and stopped.

This message is not useful. It should print out the file paths of the *.err files that were generated, capturing the errors.

It should also continue, running the diff compiler as well, so I get as many diffs as possible.

Implement support for most common workflow (RELEASE assemblies and CHECKED jit)

We care mostly about measurements on RELEASE builds of the core library and framework assemblies.

Unfortunately jit dasm won't work in RELEASE builds of CoreCLR, so we need to blend in some parts of CHECKED builds (ideally just the clrjit.dll). Would be nice if the tool handled this automatically and warned if necessary parts were missing.

PMI Usage message is incorrect

The dotnet CLI-version of the usage message prints "dotnet.exe" as the tool, which is confusing:

c:\gh\jitutils>where pmi
c:\gh\jitutils\bin\pmi.bat

c:\gh\jitutils>pmi -?
Usage:

  dotnet.exe Count PATH_TO_ASSEMBLY
      Count the number of types and methods in an assembly.

  dotnet.exe PrepOne PATH_TO_ASSEMBLY INDEX_OF_TARGET_METHOD
      JIT a single method, specified by a method number.

  dotnet.exe PrepAll PATH_TO_ASSEMBLY [INDEX_OF_FIRST_METHOD_TO_PROCESS]
      JIT all the methods in an assembly. If INDEX_OF_FIRST_METHOD_TO_PROCESS is specified, it is the first
      method compiled, followed by all subsequent methods.

  dotnet.exe DriveAll PATH_TO_ASSEMBLY
      The same as PrepAll, but is more robust. While PrepAll will stop at the first JIT assert, DriveAll will
      continue by skipping that method.

Environment variable PMIPATH is a semicolon-separated list of paths used to find dependent assemblies.

For Prepall and Prepone, optional suffixes will change output behavior:
   -Quiet will suppress in-progress messages for type and method exploration
   -Time will always show elapsed times, even in -Quiet mode

This is from:

private static int Usage()
{
    string exeName = System.Diagnostics.Process.GetCurrentProcess().MainModule.FileName; // get the current full path name of PMI.exe
    exeName = System.IO.Path.GetFileName(exeName); // strip off the path; just use the EXE name.
    Console.WriteLine(
        "Usage:\r\n"
        + "\r\n"
        + "  " + exeName + " Count PATH_TO_ASSEMBLY\r\n"

jit-diff should show path of base and diff

To allow for easy cut-and-paste to a directory diff program. e.g.,:

To examine diffs, use a command like:
   <diff-program> c:\diffs\base\4 c:\diffs\diff\4

e.g., on Windows:
    windiff c:\diffs\base\4 c:\diffs\diff\4

On Windows, our desktop testing infrastructure also automatically detects if the paths are shared, giving a UNC path. This is useful for copy-and-paste of the command to email to others to look at your diffs, e.g.:

   windiff \\brucefo1\mydiffs\base\4 \\brucefo1\mydiffs\diff\4

jit-format crash: Cannot create a file when that file already exists

D:\gh\coreclr3>jit-format --verbose -a x86 -o Windows -b Debug -c d:\gh\coreclr3
Formatting jit directory.
Formatting dll project.
Neither compile_commands.json exists, nor is there a build log. Running CMake to generate compile_commands.json.
Starting Build at 16:36:01.27
BUILD: Commencing CoreCLR Repo build
BUILD: Checking prerequisites
BUILD: Using environment: "C:\Program Files (x86)\Microsoft Visual Studio 14.0\Common7\Tools\\VsDevCmd.bat"
Tools are already initialized.
Running: d:\gh\coreclr3\Tools\dotnetcli\dotnet.exe d:\gh\coreclr3\Tools\run.exe d:\gh\coreclr3\config.json build -Project=d:\gh\coreclr3\build.pr
oj -generateHeaderWindows -NativeVersionHeaderFile="d:\gh\coreclr3\bin\obj\_version.h" -BuildOS=Windows_NT -BuildType=Debug -BuildArch=x86
Running: d:\gh\coreclr3\Tools\msbuild.cmd /nologo /verbosity:minimal /clp:Summary /maxcpucount /nodeReuse:false  d:\gh\coreclr3\build.proj /p:__B
uildType=Debug /p:__BuildArch=x86 /p:__BuildOS=Windows_NT  /t:GenerateVersionHeader /p:GenerateVersionHeader=true /p:NativeVersionHeaderFile=d:\g
h\coreclr3\bin\obj\_version.h
Command execution succeeded.
BUILD: Commencing build of native components for Windows_NT.x86.Debug
BUILD: Using environment: "C:\Program Files (x86)\Microsoft Visual Studio 14.0\Common7\Tools\\..\..\VC\vcvarsall.bat" x86
BUILD: Regenerating the Visual Studio solution
-- VS_PLATFORM_TOOLSET is
-- VS_PLATFORM_NAME is
-- Configuring done
CMake Error: Error required internal CMake variable not set, cmake may be not be built correctly.
Missing variable is:
CMAKE_RC_CREATE_SHARED_LIBRARY
CMake Error: Error required internal CMake variable not set, cmake may be not be built correctly.
Missing variable is:
CMAKE_RC_CREATE_SHARED_LIBRARY
CMake Error: Error required internal CMake variable not set, cmake may be not be built correctly.
Missing variable is:
CMAKE_RC_CREATE_SHARED_LIBRARY
CMake Error: Error required internal CMake variable not set, cmake may be not be built correctly.
Missing variable is:
CMAKE_RC_CREATE_SHARED_LIBRARY
-- Generating done
-- Build files have been written to: D:/gh/coreclr3/bin/nmakeobj/Windows_NT.x86.Debug
BUILD: Repo successfully built.  Finished at 16:36:13.10
BUILD: Product binaries are available at d:\gh\coreclr3\bin\Product\Windows_NT.x86.Debug

Unhandled Exception: System.IO.IOException: Cannot create a file when that file already exists
   at System.IO.Win32FileSystem.MoveFile(String sourceFullPath, String destFullPath)
   at System.IO.File.Move(String sourceFileName, String destFileName)
   at ManagedCodeGen.jitformat.Main(String[] args)

Update framework assembly list

Update framework assembly list. There are two options here. Update the baked in list based on what shipped in v1 or make it dynamic based on what's available in the specified core_root directory.

Implement support for TP measurements

Since there is already support for crossgenning a set of assemblies, it should be straightforward to measure the time it takes for the crossgen.

jit-format: Investigate clang-format indentation failures

Occasionally, clang-format will format our code wrong around ifdefs. See this as an example.

Investigate “error: Can't find --crossgen tool.”

This error was showing up when running the command:
C:\git\tools\jitutils>bin\jit-diff.cmd diff --analyze --frameworksonly --crossgen C:/git/coreclr/bin/Product/Windows_NT.x64/crossgen --output C:/git/out --core_root C:/git/tools/jitutils/fx --base C:/git/coreclr/bin/Product/Windows_NT.x64.Debug --diff C:/git/coreclr/bin/Product/Windows_NT.x64.Debug_base

jit-diff should report number of unchanged methods

At the same time jit-diff reports the numbers of improved and regressed methods it should output the unchanged numbers.

Original ask is here:

dotnet/coreclr#9479 (comment)

jit-diff and jit-diff-pmi verbose output should include exact set of environment variable settings

When trying to reproduce a failure of the tools to run, using the --verbose switch should give all the information you need. This should include all of the COMPlus and other environment variables that are set before a command is run, including their values.

Currently, only some "ambient" variable values that are passed through are output, plus the not-very-useful "Setting AltJit" message.

The output should be in a form that can by copy-pasted on the platform, e.g., set COMPlus_JitDisasm=*, etc.

Jit-dasm should clean up generated .ni.exe files

Failure to do so leaves them in the test directories (if you were diffing tests), which means that from then on when you run tests you'll be picking up the crossgened code and not testing what you intended.

jit-format output could be more friendly when errors are detected

1. It shouldn't print “Clang-Format needs to be rerun in fix mode”. It should say "Run jit-format -fix -no-tidy" or leave out the -no-tidy part if there are tidy errors.
2. It could maybe print the first X lines of specific error message even in non-verbose mode

jit-format should try to infer the location of coreclr

So we don't have to specify it on the command line if things are laid out in the "standard" way, however we define that.

jit-diff should run base and diff jit-dasm in parallel

This way you don't have to wait until the entire base dasm is generated to inspect the diff for System.Private.CoreLib (which comes first when doing a fx diff).

Besides, crossgen appears to be single threaded so running 2 crossgens in parallel should speed up things.

Any downsides in doing this?

config.json file is missing from the jitutils repo

From the jitutils get started website: https://github.com/dotnet/jitutils/blob/master/doc/getstarted.md, at the bottom of the page in configuration section, it mentioned config.json that is included in the jitutils repo, however, the config.json file is actually missing from the repo:

"A sample config.json is included in the jitutils repo as an example that can be modified for a developers own context. "

click config.json hyper link leads to "Page not found" on GitHub

Initial proposal to automate SuperPMI Collection

Allow SuperPMI collection to be done automatically over the CoreCLR and CoreFX Repos. Below was the initial proposal for the tool.

Overview

A managed application to automatically run superPMI using an already built coreclr.

Example Usage

SuperPMiCollect.exe x86 checked

Storage

The superPMI information collected will be stored in Azure Blob Storage. It will be compressed so that the information be stored efficiently. The directory structure will be emulated below in Azure Blob Storage.

Directory Structure

Blob Storage

|--Git Hash
    |--OS
        |--Arch
            |--JITS.zip
            |-- MCH
                  |--Mchfile1.zip
                  |--Mchfile2.zip
                  |--   …
            |--MCH_ASM
                  |--Mchfile1.zip
                  |--MchFile2.zip
                  |--   …
            |--CORE_ROOT.zip

Finding a Sync Point

It will be almost guaranteed to be too wasteful to collect on each checkin, therefore, we need to find a way to sync to a certain changeset, given a git hash. Basically, with some git log parsing we can find the date of both hashes and compare the amount of changes between them. We can walk through the list of hashes we have stored and find the one that is closest to the current hash you are on.

Then you can pull down all of the information for that arch and os.

Collecting

Collecting can already be done with the collection tool Bruce wrote. We can run it over the over the crossgen of the CoreFX Frameworks, coreclr test run and Roslyn testing to start. See #34 for information on expanding this.

Describe usage of JIT-diff with protonjit

It is not clear form the documentation how to use JIT diff for arm64.

Dotnet CLI tools are not available so presumably it is intended to be run on amd64 Unix/Windows.

This may be a s simple as defining AltJIT, but it would be nice to have explicit instructions.

jit-diff: improve reconciliation when there are multiple methods with the same name

See dotnet/coreclr#14244 -- the "savings" seen here is misleading:

-516 : System.Private.CoreLib.dasm - InterfaceIsCheckWorkaround`1:.cctor() (7 methods)

The diff version had only 3 of the 7 copies -- the other 4 instantations were no longer needed because of changes in codegen in other methods.

Reconciliation should have noted that the base version had 4 extra methods instead of making it look like both versions had these 7 methods and they shrank a lot.

SuperPMI Work Items

SuperPMI

We need to create SuperPMI collections for a large amount of code to help with quick assertion checking and asm diffs. The work here is:

Task	Issue	Time Expected
Automate the collection	#33	PR
Easy superPMI Replay	#35
Add collection to DotnetCI	#44
Expand collection resources	#34	TBD
Fix SuperPMI to use coredis tools, to enable asm diffs	6942	TBD

jit-diff should change order of work

It looks like jit-diff does all base asm generation, followed by all diff asm generation, e.g.:

for each (case in {base, diff}) do
   for each (file in asm-list) do
       jit-asm // generate asm for {case, file}

This should be inverted to:

for each (file in asm-list) do
   for each (case in {base, diff}) do
       jit-asm // generate asm for {case, file}

such that you can start looking at base/diff immediately after the first file is processed.

Create jit-config command

Create a new tool, jit-config, for handling all management and manipulation of the jitutils config.json file. Move the jit-diff "install", "list", and "uninstall" commands to jit-config. This would slightly simplify jit-diff, as it would have exactly one "command", so jit-diff diff could become just jit-diff.

Ubuntu 16.04 ./bootstrap.sh errors

[sdmaclea@EASTPOINT-1604 jitutils (master)]$ ./bootstrap.sh 

Welcome to .NET Core!
---------------------
Learn more about .NET Core @ https://aka.ms/dotnet-docs. Use dotnet --help to see available commands or go to https://aka.ms/dotnet-cli-docs.
Telemetry
--------------
The .NET Core tools collect usage data in order to improve your experience. The data is anonymous and does not include commandline arguments. The data is collected by Microsoft and shared with the community.
You can opt out of telemetry by setting a DOTNET_CLI_TELEMETRY_OPTOUT environment variable to 1 using your favorite shell.
You can read more about .NET Core tools telemetry @ https://aka.ms/dotnet-cli-telemetry.
Configuring...
-------------------
A command is running to initially populate your local package cache, to improve restore speed and enable offline access. This command will take up to a minute to complete and will only happen once.
Decompressing 100% 2050 ms
Expanding 100% 2678 ms
warn : The folder '/home/sdmaclea/git/jitutils' does not contain a project to restore.
'./src/jit-dasm' does not contain a project.json file
cp: cannot create regular file '/home/sdmaclea/git/jitutils/bin/jit-dasm': No such file or directory
'./src/jit-diff' does not contain a project.json file
cp: cannot create regular file '/home/sdmaclea/git/jitutils/bin/jit-diff': No such file or directory
'./src/jit-analyze' does not contain a project.json file
cp: cannot create regular file '/home/sdmaclea/git/jitutils/bin/jit-analyze': No such file or directory
'./src/jit-format' does not contain a project.json file
cp: cannot create regular file '/home/sdmaclea/git/jitutils/bin/jit-format': No such file or directory
'./src/cijobs' does not contain a project.json file
cp: cannot create regular file '/home/sdmaclea/git/jitutils/bin/cijobs': No such file or directory
Specify --help for a list of available options and commands.
error: Unrecognized option '--runtime'
'./src/packages' does not contain a project.json file
Downloading clang-format to bin directory
bin/clang-format: No such file or directory
chmod: cannot access 'bin/clang-format': No such file or directory
Downloading clang-tidy to bin directory
bin/clang-tidy: No such file or directory
chmod: cannot access 'bin/clang-tidy': No such file or directory
Either Clang-tidy or clang-format was not installed. Please install and put them on the PATH to use jit-format.
Tools can be found at http://llvm.org/releases/download.html#3.8.0
Adding /home/sdmaclea/git/jitutils/jitutils/bin to PATH
Done setting up!

Jit-diff should look for specific name of dlls instead of globbing

Currently jit-diff looks for "*clrjit*" as the jit tool, which causes trouble when there are two files in the base or diff directory matching that pattern. Jit-diff should only look for files named exactly clrjit.dll (libclrjit.so on linux). The same fix should be made for crossgen as well.

Modify jit-dasm to take base/diff jits rather than crossgen

Crossgen has a feature where it always looks for the System.Private.CoreLib in the directory where it resides. Even if it is passed a full path.

Modify jit-dasm to take a crossgen parameter which points to a single crossgen/System.Private.CoreLib pair and then uses the crossgen -JitPath argument to produce output from a base/diff jit.

This will remove the potential for diffs in System.Private.CoreLib that don't result from the JIT.

"Command line is too long" error

There have been a number of reports of "command line too long" coming out of the sub tools that jit-diff uses. Below is one report but there have been others. Fix would be to find a way to work around this.

I tried running the following from PowerShell:

jit-diff diff --analyze --crossgen \git\coreclr\bin\Product\Windows_NT.x64.Checked.master\crossgen.exe --base \git\coreclr\bin\Product\Windows_NT.x64.Checked.master --diff \git\coreclr\bin\Product\Windows_NT.x64.Checked --output \git\Diffs --core_root D:\git\coreclr\bin\tests\Windows_NT.x64.Checked\Tests\Core_Root --test_root D:\git\coreclr\bin\tests\Windows_NT.x64.Checked

I still get the “The command line is too long.” Error.

Issues running jit-diffs on windows

Hit a couple of issues using this:

• Seems like JIT_DASM_ROOT must be set or app crashes.
• Windows requires .exe suffix for tools
• App should validate git is on path up front instead of crashing after running base set
• Framework DLL list seems outdated. Seems like you could just mine the test root.

jit-analyze: create filtered and optionally markdown-friendly results file

The current approach requires on run a file diff on rather large text files (eg 80MB). Since jit-analyze knows which methods have size diffs it should be able to create filtered views that just focus on the methods with diffs. These could be prioritized in various orders, eg largest regressions, largest percent regressions, etc.

Furthermore it would be great if this view could also be written out in some markdown friendly fashion. I've tried two different approaches here, with mixed results.

First is to just create patch files. These often don't have enough context, but they get rendered reasonably well by github.

Second is to create tables with side by side views. Alignment can be tricky and you might lose syntax and diff highlighting but I find this easier to parse, at least in some cases.

Creation of either of these manually is laborious, so automating it would be really nice.

PMI as a tool for measuring jit throughput

Most users see the impact of jit throughput primarily when jitting, especially at application startup.

Current practice for measuring jit throughput in CoreCLR relies on prejitting, using crossgen to jit large numbers of methods. PMI offers an alternative and complementary view of jit throughput.

This issue tracks exploring how to make PMI be a robust tool for exploring jit throughput.

Fix cijobs build warnings

These warnings need to be fixed.

D:\gh\jitutils\src\cijobs\cijobs.cs(513,38): warning CS1998: This async method lacks 'await' operators and will run synchronously. Consider using
 the 'await' operator to await non-blocking API calls, or 'await Task.Run(...)' to do CPU-bound work on a background thread.
D:\gh\jitutils\src\cijobs\cijobs.cs(415,38): warning CS1998: This async method lacks 'await' operators and will run synchronously. Consider using
 the 'await' operator to await non-blocking API calls, or 'await Task.Run(...)' to do CPU-bound work on a background thread.
D:\gh\jitutils\src\cijobs\cijobs.cs(541,39): warning CS1998: This async method lacks 'await' operators and will run synchronously. Consider using
 the 'await' operator to await non-blocking API calls, or 'await Task.Run(...)' to do CPU-bound work on a background thread.
D:\gh\jitutils\src\cijobs\cijobs.cs(239,32): warning CS0649: Field 'cijobs.JobBuilds.builds' is never assigned to, and will always have its defau
lt value null
D:\gh\jitutils\src\cijobs\cijobs.cs(210,29): warning CS0649: Field 'cijobs.Action.lastBuiltRevision' is never assigned to, and will always have i
ts default value
D:\gh\jitutils\src\cijobs\cijobs.cs(221,27): warning CS0649: Field 'cijobs.Job.name' is never assigned to, and will always have its default value
 null
D:\gh\jitutils\src\cijobs\cijobs.cs(200,27): warning CS0649: Field 'cijobs.Artifact.fileName' is never assigned to, and will always have its defa
ult value null
D:\gh\jitutils\src\cijobs\cijobs.cs(201,27): warning CS0649: Field 'cijobs.Artifact.relativePath' is never assigned to, and will always have its
default value null
D:\gh\jitutils\src\cijobs\cijobs.cs(76,28): warning CS0169: The field 'cijobs.Config._privateBranchName' is never used
D:\gh\jitutils\src\cijobs\cijobs.cs(214,33): warning CS0649: Field 'cijobs.BuildInfo.actions' is never assigned to, and will always have its defa
ult value null
D:\gh\jitutils\src\cijobs\cijobs.cs(222,27): warning CS0649: Field 'cijobs.Job.url' is never assigned to, and will always have its default value
null
D:\gh\jitutils\src\cijobs\cijobs.cs(232,24): warning CS0649: Field 'cijobs.Build.number' is never assigned to, and will always have its default v
alue 0
D:\gh\jitutils\src\cijobs\cijobs.cs(71,28): warning CS0169: The field 'cijobs.Config._forkUrl' is never used
D:\gh\jitutils\src\cijobs\cijobs.cs(227,30): warning CS0649: Field 'cijobs.ProductJobs.jobs' is never assigned to, and will always have its defau
lt value null
D:\gh\jitutils\src\cijobs\cijobs.cs(216,27): warning CS0649: Field 'cijobs.BuildInfo.result' is never assigned to, and will always have its defau
lt value null
D:\gh\jitutils\src\cijobs\cijobs.cs(215,35): warning CS0649: Field 'cijobs.BuildInfo.artifacts' is never assigned to, and will always have its de
fault value null
D:\gh\jitutils\src\cijobs\cijobs.cs(233,27): warning CS0649: Field 'cijobs.Build.url' is never assigned to, and will always have its default valu
e null
D:\gh\jitutils\src\cijobs\cijobs.cs(206,27): warning CS0649: Field 'cijobs.Revision.SHA1' is never assigned to, and will always have its default
value null

jit-format crashes given bad --coreclr path

e.g.:

D:\gh\coreclr3>jit-format -a x86 -o Windows -b Debug -c d:\gh\corclr3

Unhandled Exception: System.ComponentModel.Win32Exception: The system cannot find the file specified
   at System.Diagnostics.Process.StartCore(ProcessStartInfo startInfo)
   at System.Diagnostics.Process.Start()
   at Microsoft.DotNet.Cli.Utils.Command.Execute()
   at ManagedCodeGen.jitformat.TryCommand(String name, IEnumerable`1 commandArgs, Boolean capture)
   at ManagedCodeGen.jitformat.Config.validate()
   at ManagedCodeGen.jitformat.Main(String[] args)

That's not a helpful error message. It should check for existing path.

Overview of work envisioned for PMI

Here's a high-level summary of work envisioned for PMI. Will open child issues as needed.

• Coverage of generic types (#128, #159, #160)
• Coverage of generic methods (#160)
• Fix DRIVEALL for CLI (#130)
• How to use local builds with PMI (CoreCLR dev innerloop workflow) (see #129)
• Integration into jit-diffs for PMI based diffing (#134)
• How to best enable altjit for PMI (#136, #142)
• Experiment with running normal jit as an altjit when testing default jit via PMI
• CI tooling for CoreCLR to use PMI (for correctness)
• CI tooling for CoreCLR to use PMI (for diffs)
• Add tests for PMI to jitutils CI
• Assess generic strategy and see how well we're covering methods in generic types
• Tooling for using PMI to assess jit thoughput (#138)
• Approach to measuring internal jit phase timings with PMI
• Multithreading across / within assemblies for correctness and diffing runs
• Update documentation (#135)

jit-diff: handle output better

jit-diff outputs quite a lot of text (such as the command lines for every call to jit-dasm). If any warnings or errors are output before or during a run, they can be overwhelmed by the "normal" output. This can be solved in one of several ways:

1. Display so much less output by default that any warning/error output is easily seen. (All output could be displayed by passing a --verbose switch).
2. Save all warning/error output, and re-display it at the very end (like the way msbuild does)

Integrate jit-diff into CI job to allow lab run baseline/diff across lab jobs.

Speed up diff analysis but including it in the lab jobs.

• Run .dasm baseline for commit trigger jobs (Linux, Windows x x64, x86, arm64)
• PR jobs run diff jobs and compare against merge point where there should be data from the commit job.

Automate collection by adding to dotnetCI

Running `jit-diff diff` crashes

c:\gh\coreclr>jit-diff diff
Environment variable JIT_UTILS_ROOT not found - no configuration loaded.

Unhandled Exception: System.MissingMethodException: Method not found: 'Microsoft.DotNet.Cli.Utils.Command Microsoft.DotNet.Cli.Utils.Command.Create(System.String, System.Collections.Generic.IEnumerable`1<System.String>, NuGet.Frameworks.NuGetFramework)'.
   at ManagedCodeGen.jitdiff.TryCommand(String name, IEnumerable`1 commandArgs, Boolean capture)
   at ManagedCodeGen.jitdiff.Config.SetRID()
   at ManagedCodeGen.jitdiff.Config..ctor(String[] args)
   at ManagedCodeGen.jitdiff.Main(String[] args)

Maybe this should output the "help" screen, but it shouldn't crash.

Add defaults for cijobs

Add --arch, --os, and --build arguments to cijobs. They would automatically map to Jenkins job ids. They could be defaulted using the config.json file. Then, you could do:

cijobs list --arch x86 // only show the x86 jobs
cijobs list --os ubuntu // only show Ubuntu jobs
cijobs list --arch x86 --os ubuntu // only show x86 ununtu jobs
cijobs copy -l --arch x64 --os windows --build checked // copy the latest x64/Windows/checked build
cijobs copy -l // same as above, if the standard defaults or config.json file specify x64/Windows/checked

Also, jit-diff install should match cijobs copy functionality.

jit-diff: automatically discover configuration

This addition would allow for automatic discovery of a typical configuration for diff'ing. When jit-diff is run in this mode it builds up a context and fills out the config file. At the high level this looks in some default locations, finds the current branch and fork-point, downloads the baseline build via 'cijobs', and writes out the config file to track this point. As of this writing I think that all the features we need are currently in the tool and just need to be wrapped up together.

Steps:

• Look for default locations for coreclr adjacent to the jitutils folder.
• Set up default output directory
• Use "git name-rev HEAD" or the like to discover the branch that the repo is on.
• Use "git merge-base $feature master" to discover the fork point from the mainline
• Download build associated with the fork-point via cijobs from the ci and install it in the tools area of the output directory.
• Set all defaults into the config file.
• Run diff.

jit-format shouldn't print a message about missing JIT_UTILS_ROOT

Unless we expect almost everyone to set JIT_UTILS_ROOT, it shouldn't be mentioned in the default output if no problems were detected.

Target netcoreapp2.0 by default

At some point we should update jitutils to default to targeting netcoreapp2.0, so that the build scripts "just work" with a .NET Core 2.0 cli install.

This will be a "breaking change" to the build steps, requiring an update to the 2.0 cli, and will probably require minor changes to the jit-format jobs.

See #115 for some discussion.

Scan for asserts in jit-analyze

Jit-analyze runs should scan for asserts in dasm output. This functionality should be exposed both in the full analysis for code size/diffs but also as a stand alone quick "find the asserts" pass (separate flag).

[question] Getting total generated code size from a JitDump

Hi, I was wondering if it's possible to get the total number of bytes of code the JIT produces for a given program, or in other words the sum of the code size of all of the methods it generates. Is there a CLR configuration knob for this, or if not is it possible with the current tooling in this repo?

Thanks!

Add % impact to jit-analyze results.

For code size differences compute % of the improvement or regression. Should cover method, file, and all up.

Display total method count in addition to changed method count

See #9230 (comment)

jit-format: crash with -a x86

Either make it clear that x86 is broken or fix it somehow.