Publish selfcontained Artifacts

[adamralph]

Publish selfcontained Artifacts

[adamralph]

Hey @enzian, thanks for raising this. Out of interest, do you currently have a use case for this?

[enzian]

Yes, we have repositories that contain documentation which is versioned. We could use minver to publish this documentation versioned!

[adamralph]

@enzian how would you actually do that? Would you have some kind of CI build to publish when pushing a tag?

[enzian]

@adamralph yes, this feature would make sense in CI build environments!

[adamralph]

@enzian given that you're doing other .NET work, do you not already have the .NET SDK installed on your build agents?

[enzian]

given that you're doing other .NET work

Well I'm not only doing .net work!

[adamralph]

@enzian sure, I wasn't suggesting otherwise. 😉

But do you actually have the situation where you would like to use a CI system to publish these docs but the relevant agents/images/etc. do not have the .NET SDK installed?

[enzian]

I'm actually using hosted agents that are created on demand and have absolutely nothing if I don't bring it ;-)

[adamralph]

OK, so now I believe you that you have a valid use case. 😉

I'm not opposed to this if it doesn't add too much maintenance burden. I guess we'd have to build several versions for different platforms and publish them somewhere?

[bording]

I'm not opposed to this if it doesn't add too much maintenance burden. I guess we'd have to build several versions for different platforms and publish them somewhere?

Yes, because each self-contained deployment would only work for the RID specified, win-x64, linux-x64, etc. That means you'll have to publish multiple large zip files somewhere.

Then, you'll have to manage acquiring and extracting them on your CI systems, likely with manually written code/scripts.

Or you can get the .NET Core SDK for the machine, and do a tool install, and then you have the CLI tool readily available.

On Linux systems, acquiring the SDK is trivial since it's available through package managers, and it's not that much more difficult to get for other OSes.

Personally, it sounds like getting the SDK and installing the command line tool is less work for everyone.

Also, exposing the command line tool as a separate artifact instead of keeping it internal does also complicate doing #112 when it makes sense.

[adamralph]

Thanks @bording. Some good arguments there.

Also, exposing the command line tool as a separate artifact instead of keeping it internal does also complicate doing #112 when it makes sense.

I assumed that, if this goes ahead, it would be an alternative way of distributing the CLI tool, which is already public.

[bording]

I assumed that, if this goes ahead, it would be an alternative way of distributing the CLI tool, which is already public.

Fair enough, though I suspect you'd likely end up needing a separate project in the solution for it regardless.

[adamralph]

Yeah, that's probably true.

But in any case, before going any further with this, I need to be convinced, as per your arguments above, that getting the self-contained artifact onto the build agents is easier than getting the .NET SDK and installing minver-cli. @enzian what are your thoughts on that?

[enzian]

My approach would be to add a build pipeline on azure devops that builds the cli project for multiple targets, zip all of them and drops them into the respective github release!

This means, that if I want to use it, i can just wget/curl, unzip and be done! That seems much easier to me then downloading the entire SDK (which takes time and time is an issue when building...) then downloading the tool, which takes time again, not to speak of the disk space and traffic used in this whole process! So, it takes fewer steps and takes less time and resources.

Fair enough, though I suspect you'd likely end up needing a separate project in the solution for it regardless.

That is not the case: You can either build all your targets by adding them to the .csproj or by adding them to the build/publish commands of the dotnet cli. This will then build the self-contained packages and drop them in a folder with their target mnemonic!

[adamralph]

@enzian as I understand it, the self-contained apps will also contain the entire .NET Core runtime. Is that the case?

I want to be sure there is enough benefit to offset the extra maintenence burden. Bear in mind that right now the Azure pipelines build only exists to serve as a smoke test for the pipelines specific console pallette. The Appveyor build is the CI build. I'd prefer to keep everything in the build script rather than have CI server config performing specific parts of the build. Also bear in mind that there are no GitHub releases currently. The only release is to NuGet.

[enzian]

the self-contained apps will also contain the entire .NET Core runtime

yes, some you don't need are stripped out though. I tested this quickly: Building minver to as a self contained artifact makes for a 75Mb package. The dotnet core sdk (2.1.something) is ~210 Mb and that's not counting the nuget dependencies that need to be pulled additionally!

I know, that 75Mb sounds huge but considering it contains everything needed for running a tool and can be downloaded as one file with a single HTTP GET as opposed to installing 200Mb of sdk and pulling in dozens (or much more) nuget packages is not exactly more efficient and definitely takes more time! On hosted build agents, time = money, storage is not an issue!

PS: As a side note because I know it's not ready yet: using IL-Linker when publishing I'm able to decrease this to about 30Mb!

                                                             Before linking (B)   After linking (B)    Size decrease
-----------                                                  -----------          -----------          -----------
Total size of assemblies                                     50'361'448           28'913'640           42.59%
-----------                                                  -----------          -----------          -----------
minver-cli.dll                                               9'216                9'216                0.00%
LibGit2Sharp.dll                                             420'352              420'352              0.00%
McMaster.Extensions.CommandLineUtils.dll                     140'776              140'776              0.00%
MinVer.Lib.dll                                               16'384               16'384               0.00%
Microsoft.CSharp.dll                                         773'200              0                    100.00%
Microsoft.VisualBasic.dll                                    465'992              0                    100.00%
Microsoft.Win32.Primitives.dll                               22'096               12'288               44.39%
Microsoft.Win32.Registry.dll                                 85'584               49'152               42.57%
SOS.NETCore.dll                                              54'864               0                    100.00%
System.AppContext.dll                                        14'416               0                    100.00%
System.Buffers.dll                                           14'416               5'120                64.48%
System.Collections.Concurrent.dll                            205'384              112'128              45.41%
System.Collections.Immutable.dll                             2'636'384            205'824              92.19%
System.Collections.NonGeneric.dll                            97'360               71'680               26.38%
System.Collections.Specialized.dll                           89'680               57'344               36.06%
System.Collections.dll                                       329'800              256'000              22.38%
System.ComponentModel.Annotations.dll                        149'064              92'672               37.83%
System.ComponentModel.DataAnnotations.dll                    15'440               0                    100.00%
System.ComponentModel.EventBasedAsync.dll                    34'888               0                    100.00%
System.ComponentModel.Primitives.dll                         53'840               31'744               41.04%
System.ComponentModel.TypeConverter.dll                      671'304              481'792              28.23%
System.ComponentModel.dll                                    16'464               6'144                62.68%
System.Configuration.dll                                     18'000               0                    100.00%
System.Console.dll                                           160'632              129'024              19.68%
System.Core.dll                                              20'856               0                    100.00%
System.Data.Common.dll                                       2'833'504            979'968              65.41%
System.Data.dll                                              24'136               0                    100.00%
System.Diagnostics.Contracts.dll                             15'440               5'632                63.52%
System.Diagnostics.Debug.dll                                 14'928               5'632                62.27%
System.Diagnostics.DiagnosticSource.dll                      67'152               53'760               19.94%
System.Diagnostics.FileVersionInfo.dll                       29'264               0                    100.00%
System.Diagnostics.Process.dll                               236'624              187'904              20.59%
System.Diagnostics.StackTrace.dll                            33'888               22'016               35.03%
System.Diagnostics.TextWriterTraceListener.dll               32'336               0                    100.00%
System.Diagnostics.Tools.dll                                 15'968               6'656                58.32%
System.Diagnostics.TraceSource.dll                           124'496              0                    100.00%
System.Diagnostics.Tracing.dll                               35'920               14'848               58.66%
System.Drawing.Primitives.dll                                101'968              68'608               32.72%
System.Drawing.dll                                           19'552               0                    100.00%
System.Dynamic.Runtime.dll                                   15'440               0                    100.00%
System.Globalization.Calendars.dll                           15'440               0                    100.00%
System.Globalization.Extensions.dll                          14'928               0                    100.00%
System.Globalization.dll                                     14'712               0                    100.00%
System.IO.Compression.Brotli.dll                             66'128               0                    100.00%
System.IO.Compression.FileSystem.dll                         13'896               0                    100.00%
System.IO.Compression.ZipFile.dll                            34'376               0                    100.00%
System.IO.Compression.dll                                    264'056              156'160              40.86%
System.IO.FileSystem.AccessControl.dll                       59'464               0                    100.00%
System.IO.FileSystem.DriveInfo.dll                           38'480               0                    100.00%
System.IO.FileSystem.Primitives.dll                          14'944               0                    100.00%
System.IO.FileSystem.Watcher.dll                             61'008               0                    100.00%
System.IO.FileSystem.dll                                     213'072              114'176              46.41%
System.IO.IsolatedStorage.dll                                78'200               0                    100.00%
System.IO.MemoryMappedFiles.dll                              64'080               46'080               28.09%
System.IO.Pipes.AccessControl.dll                            15'952               0                    100.00%
System.IO.Pipes.dll                                          129'096              0                    100.00%
System.IO.UnmanagedMemoryStream.dll                          14'928               0                    100.00%
System.IO.dll                                                14'712               0                    100.00%
System.Linq.Expressions.dll                                  1'604'704            593'920              62.99%
System.Linq.Parallel.dll                                     1'296'968            0                    100.00%
System.Linq.Queryable.dll                                    173'648              0                    100.00%
System.Linq.dll                                              432'208              266'752              38.28%
System.Memory.dll                                            246'352              103'936              57.81%
System.Net.Http.dll                                          1'128'016            1'015'808            9.95%
System.Net.HttpListener.dll                                  643'664              0                    100.00%
System.Net.Mail.dll                                          546'896              0                    100.00%
System.Net.NameResolution.dll                                85'584               70'144               18.04%
System.Net.NetworkInformation.dll                            180'304              126'976              29.58%
System.Net.Ping.dll                                          88'656               52'224               41.09%
System.Net.Primitives.dll                                    199'248              174'080              12.63%
System.Net.Requests.dll                                      345'680              308'736              10.69%
System.Net.Security.dll                                      533'072              454'144              14.81%
System.Net.ServicePoint.dll                                  33'888               16'384               51.65%
System.Net.Sockets.dll                                       515'656              372'224              27.82%
System.Net.WebClient.dll                                     158'800              0                    100.00%
System.Net.WebHeaderCollection.dll                           66'144               50'176               24.14%
System.Net.WebProxy.dll                                      26'704               0                    100.00%
System.Net.WebSockets.Client.dll                             71'240               0                    100.00%
System.Net.WebSockets.dll                                    140'152              0                    100.00%
System.Net.dll                                               15'952               0                    100.00%
System.Numerics.Vectors.dll                                  156'752              55'808               64.40%
System.Numerics.dll                                          13'904               0                    100.00%
System.ObjectModel.dll                                       85'064               35'840               57.87%
System.Private.DataContractSerialization.dll                 2'017'360            1'341'440            33.51%
System.Private.Uri.dll                                       236'624              210'432              11.07%
System.Private.Xml.Linq.dll                                  368'504              206'848              43.87%
System.Private.Xml.dll                                       8'310'352            5'686'272            31.58%
System.Reflection.DispatchProxy.dll                          68'176               0                    100.00%
System.Reflection.Emit.ILGeneration.dll                      14'928               5'632                62.27%
System.Reflection.Emit.Lightweight.dll                       14'928               5'120                65.70%
System.Reflection.Emit.dll                                   14'928               5'632                62.27%
System.Reflection.Extensions.dll                             14'712               0                    100.00%
System.Reflection.Metadata.dll                               1'068'112            593'408              44.44%
System.Reflection.Primitives.dll                             14'928               5'632                62.27%
System.Reflection.TypeExtensions.dll                         29'560               0                    100.00%
System.Reflection.dll                                        15'440               0                    100.00%
System.Resources.Reader.dll                                  14'920               0                    100.00%
System.Resources.ResourceManager.dll                         14'928               5'632                62.27%
System.Resources.Writer.dll                                  43'088               0                    100.00%
System.Runtime.CompilerServices.VisualC.dll                  16'976               0                    100.00%
System.Runtime.Extensions.dll                                394'616              282'112              28.51%
System.Runtime.Handles.dll                                   14'920               0                    100.00%
System.Runtime.InteropServices.RuntimeInformation.dll        26'192               14'336               45.27%
System.Runtime.InteropServices.WindowsRuntime.dll            15'944               0                    100.00%
System.Runtime.InteropServices.dll                           46'160               26'112               43.43%
System.Runtime.Loader.dll                                    14'416               0                    100.00%
System.Runtime.Numerics.dll                                  192'080              116'224              39.49%
System.Runtime.Serialization.Formatters.dll                  295'504              125'440              57.55%
System.Runtime.Serialization.Json.dll                        14'920               0                    100.00%
System.Runtime.Serialization.Primitives.dll                  27'208               15'360               43.55%
System.Runtime.Serialization.Xml.dll                         15'944               6'656                58.25%
System.Runtime.Serialization.dll                             15'432               0                    100.00%
System.Runtime.dll                                           49'232               38'400               22.00%
System.Security.AccessControl.dll                            213'368              0                    100.00%
System.Security.Claims.dll                                   88'136               49'664               43.65%
System.Security.Cryptography.Algorithms.dll                  356'944              178'688              49.94%
System.Security.Cryptography.Cng.dll                         264'784              201'216              24.01%
System.Security.Cryptography.Csp.dll                         175'184              125'440              28.40%
System.Security.Cryptography.Encoding.dll                    68'168               50'176               26.39%
System.Security.Cryptography.OpenSsl.dll                     31'312               0                    100.00%
System.Security.Cryptography.Primitives.dll                  88'656               47'104               46.87%
System.Security.Cryptography.X509Certificates.dll            371'280              284'160              23.46%
System.Security.Principal.Windows.dll                        139'336              108'544              22.10%
System.Security.Principal.dll                                15'440               6'144                60.21%
System.Security.SecureString.dll                             14'928               0                    100.00%
System.Security.dll                                          16'464               0                    100.00%
System.ServiceModel.Web.dll                                  15'440               0                    100.00%
System.ServiceProcess.dll                                    14'408               0                    100.00%
System.Text.Encoding.Extensions.dll                          14'920               5'120                65.68%
System.Text.Encoding.dll                                     14'928               0                    100.00%
System.Text.RegularExpressions.dll                           368'200              332'800              9.61%
System.Threading.Overlapped.dll                              15'952               6'656                58.27%
System.Threading.Tasks.Dataflow.dll                          479'608              0                    100.00%
System.Threading.Tasks.Extensions.dll                        15'432               0                    100.00%
System.Threading.Tasks.Parallel.dll                          109'152              0                    100.00%
System.Threading.Tasks.dll                                   15'944               6'144                61.47%
System.Threading.Thread.dll                                  41'552               23'040               44.55%
System.Threading.ThreadPool.dll                              14'928               5'120                65.70%
System.Threading.Timer.dll                                   14'200               5'120                63.94%
System.Threading.dll                                         75'856               34'816               54.10%
System.Transactions.Local.dll                                346'192              0                    100.00%
System.Transactions.dll                                      14'928               0                    100.00%
System.ValueTuple.dll                                        15'440               0                    100.00%
System.Web.HttpUtility.dll                                   43'600               0                    100.00%
System.Web.dll                                               13'904               0                    100.00%
System.Windows.dll                                           13'920               0                    100.00%
System.Xml.Linq.dll                                          14'416               0                    100.00%
System.Xml.ReaderWriter.dll                                  21'584               12'288               43.07%
System.Xml.Serialization.dll                                 14'928               0                    100.00%
System.Xml.XDocument.dll                                     15'440               5'632                63.52%
System.Xml.XPath.XDocument.dll                               16'464               0                    100.00%
System.Xml.XPath.dll                                         14'928               5'632                62.27%
System.Xml.XmlDocument.dll                                   15'224               0                    100.00%
System.Xml.XmlSerializer.dll                                 17'488               8'192                53.16%
System.Xml.dll                                               23'120               0                    100.00%
System.dll                                                   53'328               0                    100.00%
WindowsBase.dll                                              14'432               0                    100.00%
mscorlib.dll                                                 63'240               0                    100.00%
netstandard.dll                                              106'576              97'280               8.72%
System.Private.CoreLib.dll                                   13'033'008           11'167'744           14.31%

Built like this, the self-contained CLI comes in at 38Mb.

[enzian]

oh and, BTW: I don't want to be the guy that annoys everyone with his craziness: So if this is too much - a "thanks but no thanks" is fine for me ;-)

[adamralph]

Not at all @enzian. 🙂

I'm simply weighing up the pros and cons of this. At the moment I'm seeing:

Pros

• 75MB vs 200MB (the size of dotnet-sdk-2.2.105-win-x64.exe) to download on a fresh build agent (assuming it doesn't need to do anything else .NET related)

Cons

• More involved version selection. With the SDK approach, you just have to choose between https://dot.net/v1/dotnet-install.sh and https://dot.net/v1/dotnet-install.ps1 and then run dotnet install minver-cli. With self contained apps you have to choose the correct RID, and there is entire catalog of RIDs to choose from.
• Increased build time, due to producing multiple apps for many RIDs.
• Increased complexity in the MinVer source solution/projects.
• The need to publish the apps somewhere. This will involve a new deployment step, whether it's to a GitHub release or somewhere else.

Am I missing anything?

[enzian]

Am I missing anything?

Not really!
Though I would argue that:

Increased build time, due to producing multiple apps for many RIDs.

is not really an issue, since all targets could be built in parallel and:

Increased complexity in the MinVer source solution/projects.

is basically one change in one csproj xml node.

I'm OK, not going down this road, and if I really want it, I can still do this myself ;-) Do we close?

[adamralph]

@enzian I'm not opposed to doing this as long as the added maintenance burden is acceptable.

all targets could be built in parallel and:
is basically one change in one csproj xml node

Could you please send a PR demonstrating this?

Regarding releasing, it looks like I can set up a GitHub releases deployment in Appveyor to do it, so that should be a low burden.

[bording]

If you wanted to support all the operating systems with these self-contained builds that you can with the existing tool, you'll have to account for the fact that LibGit2Sharp has a native dependency, and provide a separate build for each of them. At the moment, that would mean you'd need:

• alpine-x64
• alpine.3.9-x64
• debian.9-x64
• fedora-x64
• linux-x64
• osx
• rhel-x64
• ubuntu.18.04
• win-x64
• win-x86

And it would be on the consumers of these self-contained builds to understand the RID fallback graph so they know which of them to use based on the linux distro they're using.

That list also changes over time, so that would have to be maintained as well.

[enzian]

That list also changes over time, so that would have to be maintained as well.

@bording: that is very much true, and something I have not really considered! Maintaining this would either mean to limit the targets the minver project provides binary packages for (say for example win-x64, alpine-x64, linux-x64, osx). This however will force a lot of people installing the runtime anyways...

We also lack the manpower to ensure the provided artifacts actually work on all platforms...

[adamralph]

So if I understand correctly, building native packages for all these platforms would be quite involved, yes?

[enzian]

building native packages for all these platforms would be quite involved

Yes, the question is, if this outweighs the advantages...
I'd shelve this issue for now, and reopen when there is more demand for self-contained artifacts!

[adamralph]

@enzian OK, thanks. In that case, I'll go ahead and close it. We can always re-open if the requirement comes up again.