dotnet/sdk

Core functionality needed to create .NET Core projects, that is shared between Visual Studio and CLI

Welcome to dotnet sdk

This repository contains core functionality needed to create .NET projects that are shared between Visual Studio and the .NET CLI.

• MSBuild tasks are under /src/Tasks/Microsoft.NET.Build.Tasks/.

See dotnet/project-system for the project system work that is specific to Visual Studio.

Common project and item templates are found in template_feed.

Build status

Windows x64

Installing the SDK

Official builds

Latest builds

How do I engage and contribute?

We welcome you to try things out, file issues, make feature requests and join us in design conversations. Be sure to check out our project documentation

This project has adopted the .NET Foundation Code of Conduct to clarify expected behavior in our community.

How do I build the SDK?

Start with the Developer Guide.

How do I test an SDK I have built?

To test your locally built SDK, run eng\dogfood.cmd after building. That script starts a new Powershell with the environment configured to redirect SDK resolution to your build.

From that shell your SDK is available in:

• any Visual Studio instance launched via & devenv.exe
• dotnet build
• msbuild

How do I determine the timeline I must follow to get my changes in for a specific version of .NET?

Please see the Pull Request Timeline Guide.

How we triage and review PRs

With the SDK repository being the home for many different areas, we've started trying to label incoming issues for the area they are related to using Area- labels. Then we rely on the codeowners to manage and triages issues in their areas. Feel free to contact the owners listed in that file if you're not getting a response on a particular issue or PR. Please try to label new issues as that'll help us route them faster.

For issues related to the central SDK team, typically they are assigned out to a team member in the first half of each week. Then each member is asked to review and mark those needing further discussion as "needs team triage" and otherwise setting a milestone for the issue. Backlog means we will consider it in the future if there is more feedback. Discussion means we have asked for more information from the filer. All other milestones indicate our best estimate for when a fix will be targeted for noting that not all issues will get fixed. If you are not getting a quick response on an issue assigned to a team member, please ping them.

The example query used for triage of .NET SDK issues can be viewed here

For PRs, we assign a reviewer once a week on Wednesday, looking only at PRs that are green in the build. If you are contributing:

• Get the PR green.
• Include a test if possible.
• Mention @dotnet-cli if you want to raise visibility of the PR.

sdk's Issues

dotnet-native package and command

A developer should be able to:

apt-get dotnet-compile-native
dotnet compile-native

and the result should be a single-file compiled native executable for the IL in their directory.

Enable xunit test execution

https://github.com/piotrpMSFT/dotnet-test-samples/

That will show you how to start playing with the prototype xunit runner. If you find issues, please feel free to open them here:

https://github.com/dotnet/coreclr.xunit

Remove Mono Dependency from CLI branch

Enable CI for dotnet/cli

Let's tie into our Jenkins server for now.

Figure out how Nuget and CLI interact

potential example:
dotnet install XXXX {VVVVV}

add the nuget package XXXX with optional version VVVVV to the project.json in the current directory.

Package creation should consume build output based on directory conventions

Let's try to avoid having to touch the packaging scripts when adding stuff to the package layout

Need to establish and create tests for user scenarios benchmarks

We need to set up goals for each of the commands and write user scenario perf tests for those commands.

man pages for dotnet

dotnet executable should support printing its own usage instructions. The same should be printed when calling 'man dotnet'

Remove the FrameworkReferenceResolver

It's ugly and should be deleted and doesn't work xplatform. Requires https://github.com/dotnet/corefx/issues/3934

/cc @Petermarcu @ericstj @piotrpMSFT

Enable workflow orchestration between dotnet driver invocations

For example - use json files to pass information between dotnet driver invocations.

Remove reference to Microsoft.NETCore.Runtime

Because I don't like downloading the internet everytime 😄

Cross-Gen or Native Compile managed exes

Do we need to do this on Windows? What are requirements? How soon can we have real Compile-To-Native?

Compiler extensibility

We should add an additional --compiler argument to dotnet compile. dotnet compile --compiler {compiler}. In absence of any compiler we'll use csc or sniff some sensible default. It can also be specified in the project.json file

{
    "version": "1.0.0-*",
    "compileFiles": [
       "Something.fs",
       "Program.fs"
     ],
    "compilationOptions": {
        "emitEntryPoint": true,
        "compiler": "fsc"
    },
    "dependencies": {
        "Microsoft.NETCore.ConsoleHost": "1.0.0-*",
        "Microsoft.NETCore.Runtime": "1.0.1-*",

        "System.Console": "4.0.0-*",
    },
    "frameworks": {
        "dnxcore50": { }
    }
}

The above project would be an f# project and we would execute the equivalent of dotnet compile --framework dnxcore50 --compiler fsc. The command line would override whatever was in the file.

/cc @jaredpar @anurse

compile and publish commands need handle native compilation in its various forms

Today we provide the option to include native images with your output:

--native Build and include native images. User must provide targeted CoreCLR runtime versions

We need to reconcile this with all the other scenarios and provide the right knobs/defaults. Should this be a compile or publish option (or both).

Examples of some of the scenarios:
Ngen:
$ dotnet compile --aot=partial
Full AOT:
$ dotnet compile --aot=full
Leave as IL:
$ dotnet compile --aot=none

Define approach for identifying Mac vs. Linux vs. Windows

We currently depend on ComSpec env var. Is this the right check?

Consider complement to dotnet install

dotnet install will install a package to your project.json

Let's consider whether we should include a complement to that command which removes the package.
The same command could potentially have an option to remove unused packages.

Recommendation for Command Line Parser

Can someone make a recommendation on what CLI parser to use? some options:

• ASP.NET
• CoreFx
• docopt

dotnet-run

dotnet-run

NAME
dotnet-run -- Runs source code 'in-place' without any explicit compile or launch commands.

SYNOPSIS
dotnet run [options]

DESCRIPTION
The run command provides a convenient option to run source code with one command. It compiles source code, generates an output program and then runs that program. This command is useful for fast iterative development and can also be used to run a source-distributed program (e.g. website).

This command relies on the compile command to compile source inputs to a .NET assembly, before launching the program. The requirements for and handling of source inputs for this command are all inhereted from the compile command. The documentation for the compile command provides more information on those requirements.

The native command uses compile in the following way:

dotnet compile -o dotnetapp.exe [overriden defaults from dotnet run]

Output files, including temporary files, are written to the child bin folder, which will be created if it doesn't exist. Files will be overwritten as needed.

Options

-v, --verbose
Prints verbose logging information, to follow the flow of execution of the command.

CLI Roadmap

The dotnet CLI is a driver for a set of distinct and standalone commands. We need to prioritize these commands so that people can use them E2E for common tasks (like conference demos). These are broken into priorities.

Commands

This document just lists the command and doesn't explain what they do. It links to other issues that go into depth on those commands, as available. Some of these commands may not be the final syntax. We'll fix that.

Demo (--native)

• dotnet #4301
• dotnet init dotnet/cli#192
• dotnet help #4301
• dotnet compile dotnet/cli#48

Converge with DNX

• dotnet publish
• dotnet publish-web
• dotnet restore dotnet/cli#79
• dotnet run dotnet/cli#59
• dotnet test
• dotnet pack

Additional Features

• dotnet install-package dotnet/cli#76
• dotnet clean #4294
• dotnet api dotnet/cli#106
• dotnet format
• dotnet repl

Open Issues for toolchain convergence:

• Resource/Satellite Assemblies (resgen + compile)
• Global commands
• NuGet integration (remove DNU Restore)
• Build Events in Project.json
• Performance
• OSSSigning
• Acquisition of toolchain
• VB/F# Support

Hosting convergence:

• Finding and installing shared runtimes (DNVM, shared hosting, servicing)
• Assembly
• Azure performance

Corefx usage of cli:

• Using dotnet cli in corefx
• Custom .NET directory structures
• Integration with MSBuild

VSIntegration:

• csproj and xproj
• debugging
• launching
• Many more
• VB/F# Support

Languages

P0

• C#

P1

• F#
• VB

Sign the debian packages before shipping

dotnet-compile

dotnet-compile

NAME
dotnet-compile -- Compiles source files to a binary format and saves to a target file.

SYNOPSIS
dotnet compile [options]

DESCRIPTION
The compile command compiles source files to a binary file, either IL byte code or native machine code, depending on the options provided. The default option is compilation to IL byte code, but may change to native compilation as that toolchain matures.

The default IL [--il] output is a PE32 exe [exe], with the default extension of ".exe" on all OSes. The exe must include a static main entry point, or it is an error. The dll [dll] output option has the default extension of ".dll".

The IL exe output type needs a runtime host to execute. The IL exe output type also copies a host to the output directory. The host is renamed to the name of the exe. For example, if the file is intended to be "foo" (-o foo), then the host will be called foo, with the appropriate default native file extension fo the OS (see the native file extensions, below). The PE32 exe will be called "[filename]"-app.exe". In this case, it would be called "foo-app.exe".

The default native [--native] output is a native exe that conforms to the architecture of the underlying operating system (i.e. running on 64-bit OS will produce a native 64-bit exe). This can be overriden via the --arch switch and specifying the wanted architecture. The executable has a default extension of "" on Linux and OS X and ".exe" on Windows. The source must include a static void main entry point, or it is an error, unless otherwise specified in the project.json. The dynamic library [dylib] output option has the default extension of ".so" on Linux/UNIX, ".dynlib" on OS X and ".dll" on Windows. The static library [staticlib] option has the default extension of ".a" on Linux, UNIX and OS X and ".lib" on Windows.

This command relies on the following artifacts: source files, project.json project file, restore.json temporary file and restored NuGet dependencies. In the case that an app depends on only the .NET Standard Library, then the project.json and restore.json file can be inferred and no additional NuGet dependencies are required. Other cases where these files are not provided or they do not match (more on that later) are error states.

The project.json file represents and describes the project. It can contain several setting, which are described in the Build. The most important information in the project.json file are the root (not transitive) NuGet dependencies and the files to be compiled. By default, this is a wildcard -- "*.cs". It supports both inclusion and exclusion semantics.

The restore.json file is expanded form of the project.json file. It includes the transitive closure of the project, per framework. It is produced by a NuGet client, typically by using the dotnet restore command. The restore.json file can be used by tools to safely determine the closure of dependencies, without having to manually calculate them. This file is only intended for tools, is temporary and should not be checked into source control. It should be present in .gitignore files.

It is important to know that a restore.json is invalid given that a project.json has been changed. The restore.json has enough information to determine this state given a project.json. The compile command validates this state and will error if the restore.json is invalid.

The compile command relies on NuGet dependencies for compilation, as references. These are expected to be found in the user-local NuGet cache (typically location here). It is an error state if a given NuGet package is not found.

Output files, including temporary files, are written to the child bin folder, which will be created if it doesn't exist. Files will be overwritten as needed.

Options

-n, --native [exe | dynlib | lib]
Compiles source to native machine code, for the local machine. The default is a native executable. The default exe extension is no extension and ".exe" on Windows. The default dynlib extension is ".a", ".dynlib" on OS X and ".dll" on Windows.

-a, --arch [x86 | x64]
Determines the architecture of the binary that --native produces. Valid values are x86 for 32-bit and x64 for 64-bit.

--il [exe | dll]
Compiles source to IL byte code, which is (typically) portable across machine types. The default output is a PE32 exe, with the default extension of ".exe" on all OSes. The exe must include a static main entry point, or it is an error. The DLL output option has the default extension of ".dll".

-o, --output filename
Specifies the filename to be used. It is an error not to specify an output filename. If no extension is provided, the default one is provided for the output type.

-r, --restore
Restores the project before attempting to compile the project.

-v, --verbose
Prints verbose logging information, to follow the flow of execution of the command.

Final content for MAN/--help

Stop over building with dotnet-compile

We should sort (topo sort) project references and compile them non recursively. We'll need a flag to disable building project dependencies.

Top-level build & test commands

let's have simple commands to build & test the repo

dotnet-native

dotnet-native

NAME
dotnet-native -- Compiles source files to a native executable.

SYNOPSIS
dotnet native [options]

DESCRIPTION
The native command compiles source files into a native executable. The source files are textual language source files. The default output is a native exe, with the default extension of ".exe" on Windows and "" on other OSes. The exe must include a static void main entry point, or it is an error, unless otherwise specified in the project.json. The dynamic library output option has the default extension of ".dll" on Windows and ".dylib" on OS X and ".so" on Linux.

This command relies on the compile command to compile source inputs to a .NET assembly, before compiling to native code. The requirements for and handling of source inputs for this command are all inhereted from the compile command. The documentation for the compile command provides more information on those requirements.

The native command uses compile in the following way:

dotnet compile -o dotnetassembly.dll [overriden defaults from dotnet compile]

Output files, including temporary files, are written to the child bin folder, which will be created if it doesn't exist. Files will be overwritten as needed.

Options

-t, --target [exe | library | staticlibrary]
Specifies the output type to generate. The default is exe.

-o, --output filename
Specifies the filename to be used. If no extension is provided, the default one is provided for the output type.

-v, --verbose
Prints verbose logging information, to follow the flow of execution of the command.

Disable csc warnings about binding redirects

C:\dev\git\apps\WebApplication1\src\WebApplication1\Program.cs(81,24): warning CS1701: Assuming assembly reference 'System.Reflection, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' used by 'System.Reflection.Extensions' matches identity 'System.Reflection, Version=4.1.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' of 'System.Reflection', you may need to supply runtime policy

We did this programmatically with roslyn APIs in dnu:

https://github.com/aspnet/dnx/blob/dev/src/Microsoft.Dnx.Compilation.CSharp.Common/CompilerOptionsExtensions.cs#L26

Publish should copy content - other files

We do this in dnx today to copy other files (like config files) to the output directory.

Infer RID from running OS/Platform

We should infer the default RID based on the running OS/bitness. This affects restore and publish

Get Package lintian-clean

Figure out which packages the dotnet- commands will live in.

These commands have interwoven dependencies on each other, and we need to figure out which package each will be deployed as a part of.

The dotnet-run command is a good example. It has a dependency on dotnet-publish, which has a dependency on dotnet-compile and dotnet-resolve-references. Should these all be part of the package which contains csc, since they depend on it? If so, what commands will the base dotnet package support?

In the event that dotnet-publish automates the restore step, where then will dotnet-restore live? If it lives in the base dotnet package, does whatever package dotnet-publish is in have a dependency on the base dotnet package?

Find the right way to test the script files shipped as part of the debian package

dotnet-clean command

We need to determine what this does. Is it project specific?

dnu parity

In the next 2 weeks, we'd like to be able to change the ASP.NET build infrastructure to use dotnet cli. To do that we need to make sure we have parity for most of the commands. This is an issue to track the bare minimum required for us to switch over:

• - Running prebuild, postbuild events in project.json
• - Running prepublish, postpublish events in project.json
• - Compile module support in the compiler (TBD)
• - Support for resources https://github.com/dotnet/cli/issues/68
  • - Embedded resources
  • - Resx files
• - Support for all compilation options defined in project.json
  • - Support for oss signing (see aspnet/Announcements#86) depends on dotnet/roslyn#1171
  • - Support for producing pdbs
  • - Support for producing portable pdbs
  • - Support for language version
• - Support for all 3 OSes (linux, osx, windows)
• - .NET Framework support (as a compilation output)
  • - Support for compiling with mono reference assemblies - This may or may not be a big issue. (this is enabled right now via an env variable). There's still work needed here. See https://github.com/dotnet/corefx/issues/3934
• - Publish to support copying content files to the output folder (aka msbuild's copy to output) - see dotnet/cli#61
• - Showing dependency diagnostics on the command line
• - Disable binding redirect warnings https://github.com/dotnet/cli/issues/47
• - Add support for generating binding redirects for .NET Framework https://github.com/dotnet/cli/issues/397
• - Globbing support for dotnet compile
• - Support generating dll metadata based on project.json metadata - https://github.com/dotnet/cli/issues/398
• - Satellite resource assembly generation (today we use roslyn but maybe we need a resgen port?) https://github.com/dotnet/cli/issues/403

Commands

• - dotnet run see https://github.com/dotnet/cli/issues/59
• - dotnet pack (dotnet compile -> nuget pack)
  • Generate symbol packages https://github.com/dotnet/cli/issues/495
• - dotnet test This one is a little fuzzy. The DNX handles testing very differently but we can do something here.

Engineering

• - Performance needs to be as good as dnu build - Was using in roslyn in memory, so saw all the benefits of caching metadata refs and syntax trees (for cross compile)

Implement new shared content pattern in NuGet v3

https://github.com/NuGet/Home/wiki/%5BSpec%5D-Content-v2-for-project.json

Packaging tests

• Upgrade
• Remove
• Purge
• Install

Try to validate that the packages install a consistent payload.

Packaging Story

What will we ship as Debian packages?
What will we ship as NuGet packages?
What are the packages we expect to pull in during apt-get dotnet?

Fix command line execution quoting

It's inconsistent and broken when running cmd vs when running an exe directly.

Move package testing to Docker containers

PROPOSED: dotnet CLI command layering

Proposed dotnet CLI command layering

Commands in the dotnet CLI can represent atomic tasks, such as "Compile source to IL with the csc compiler" or more complicated workflows like "dotnet-compile" which should restore dependencies, compile to IL, then compile that IL to native.

We intend for workflows to be decomposable into their base atomic components, allowing for easy modification and integration into other workflow or build management systems.

We would also like for atomic commands to provide users with a common style so they can reason about the commands easily.

Finally, we expect that a single executable can provide the implementation of several commands [consider NuGet restore, install, list]. We shouldn't need multiple copies of the executable to expose these commands, though we prefer to represent them as individual items on the path.

Requirements

The design of command layering, then, should satisfy the following requirements:

1. End-to-end commands like "dotnet-compile" should be composed of atomic commands
2. Atomic commands should have a single purpose
3. Atomic commands should have a common style
4. Atomic commands should follow the dotnet- naming convention
5. A given tool should be able to expose multiple commands

Design [proposed]

The base of the layering design is formed of the individual tools provided to dotnet developers. These tools are complemented by scripts representing atomic commands, wherein the scripts follow a similar style and each expose a single responsibility. These commands are then connected with one another via end-to-end scripts.

Layering Diagram

dotnet-compile
dotnet-restore	dotnet-compile-csc	dotnet-compile-il2native
nuget	csc	il2native

Sample Scripts

The below snippets are trimmed down representations of the aforementioned concepts. They are not intended to capture the full implementation.

dotnet-compile

dotnet restore
dotnet compile-csc
dotnet compile-il2native

dotnet-compile-csc

csc /target:library /out:Something.xyz *.cs

Setup publishing of Debian packages

Get dotnet.exe working on Linux/Mac

See dotnet/cli#8 (comment)

The prototype assumes a few Windows things right now, they should be minor to fix but we should do that ASAP.

dotnet --help command

should support both --help and invalid input.

Debian Source package should contain Source Code

Typically, the entire source code is included as a part of a debian source package, and the build of this source code is orchestrated as part of the build of the overall package. This orchestration is handled in a special makefile (debian/rules).

Clang, and git both follow this convention, and it will likely be necessary for us to do so if we ever want to have our package as part of an official debian distribution.

There are some complications with this. For example, if our projects are building using msbuild, we would have to include it as part of the source package or make an msbuild debian package to list as a build dependency.

For the moment, it should not affect user experience, or the demo. Discussed with @Sridhar-MS and marking Post-November.

Final metadata for debian packages

Build a "ProjectModel" library to avoid raw lock file parsing

DNX has a concept of a Project Context which allows the runtime and restore process to query various data about a project (what libraries are required from which packages, where the source files are for a project etc.). Right now we manually read that out of the lock file and copy reference DNX runtime assemblies to resolve source files, but we should build a proper public API to support that.

As part of this, I expect to remove the separate dotnet-resolve-dependencies and dotnet-resolve-sources tools. I think having those in separate exes might be a little overkill ;)

'dotnet commands' command

dotnet-commands should enumerate all executables on the path that match the 'dotnet-' preamble. The preamble should not be displayed in the output. So, if we have:

dotnet-restore
dotnet-commands
dotnet-compile

then we should see:

$ dotnet commands
  compile
  commands
  restore

Output Verbosity

How do we want to handle verbosity of what is written by the CLI?

build.sh is failing in the CI builds

http://dotnet-ci.cloudapp.net/job/Private/job/dotnet_cli_linux_debug_prtest/2/console

mkdir: cannot create directory '/.local': Permission denied
10:12:01 % Total % Received % Xferd Average Speed Time Time Time Current
10:12:01 Dload Upload Total Spent Left Speed
10:12:01
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0Warning: Failed to create the file /.local/share/nuget.latest.exe: No such
10:12:01 Warning: file or directory
10:12:01
0 3656k 0 16383 0 0 152k 0 0:00:24 --:--:-- 0:00:24 152k
10:12:01 curl: (23) Failed writing body (0 != 16383)
10:12:01 curl: (3) malformed
10:12:01 cp: cannot stat '/.local/share/nuget.latest.exe': No such file or directory
10:12:01 Cannot open assembly '.nuget/nuget.exe': No such file or directory.
10:12:01 Cannot open assembly '.nuget/nuget.exe': No such file or directory.
10:12:01 ./build.sh: line 33: packages/KoreBuild/build/dnvm.sh: No such file or directory
10:12:01 ./build.sh: line 37: dnvm: command not found
10:12:01 Cannot open assembly 'packages/Sake/tools/Sake.exe': No such file or directory.
10:12:01 Build step 'Execute shell' marked build as failure
10:12:01 [BFA] Scanning build for known causes...
10:12:01 [BFA] No failure causes found
10:12:01 [BFA] Done. 0s
10:12:01 Setting status of 619050ffe27dda00f6b4d55220c4c1dd675e4d56 to FAILURE with url http://dotnet-ci.cloudapp.net/job/Private/job/dotnet_cli_linux_debug_prtest/2/ and message: 'Build finished. No test results found.'
10:12:01 Using context: Linux Debug Build
10:12:02 Finished: FAILURE

Unified console host

We need to design/create a unified console host for running dotnet apps using

• app local fx
• or Shared fx
• or custom fx

dotnet command --> dotnet-command

DotNet driver should interpret the first parameter passed on the command line as the name of an executable it should call. If an executable file with that name is found on the path then any subsequent parameters are passed through to that executable:

dotnet compile A B C

calls

dotnet-compile A B C

If the executable is not found, DotNet cleanly notifies the calling script and returns 1.

dotnet: 'compile' is not a dotnet command. See 'dotnet --help'.