soupbuild/soup

Soup is a build system that guarantees incremental build correctness and aims to simplify many aspects of developing shared code while maintaining best in class performance.

Soup Build

Quick Links

• Official Site
• Getting Started
• Command Line Interface (CLI)
• Samples

Overview

Soup is a cross platform build system for developing software as a community, with a primary focus on developer velocity. Soup was built from the ground up to support guaranteed minimum incremental builds while allowing for reusability of both build logic and runtime code.

The system utilizes a three phased approach to building code. A declarative frontend Recipe tells the build what to build. A generate layer uses Extensions to define how to build. And finally, the evaluate engine executes the build Operations to generate the desired output.

The clean isolation individual projects, coupled with the ability to write custom build logic work together to allow for easy authoring of a package that can be shared with other teams or organizations through a public source based Package Manager.

The entire system is programming language agnostic and allows for easy integration with any language. This allows for developers to learn one solution and apply that knowledge to any project regardless of the languages it uses.

More Information:

• Design Requirements and Goals - The core goals and requirements for the system.
• Architecture - An up to date breakdown of the system Runtime and Design.
• Command Line Interface (CLI) - The help documentation for the Soup Command line application.
• Simple Minimal Language (SML) - The documentation for the Declarative language used by Soup Recipes.

Release Status

Soup is currently in an Beta stage. This means that anyone is welcome to play around with the source code or the releases on GitHub, however there may be breaking changes in the internal structures and definitions as work is completed. It is therefor recommended not to be used in any production capacity. The design is stabilizing over time and Version 1 will be released when we believe there will be no more breaking changes or security concerns that would block a first release.

Contributing

Soup is currently in active prototyping and testing. If you are interested in contributing to the project feel free to submit a PR or download the latest release and file an issue with suggestions or bugs. 😄.

• Getting Started
• Samples
• Developer Setup

soup's Issues

Integrage VSWhere

It is a pain to have to specify the location of the MSVC binaries/tools and headers

Binary Build History File Format

Is your feature request related to a problem? Please describe.
A large percentage of the build time is spent parsing the build history json file. This bad.

Describe the solution you'd like
Investigate an binary format that can be read from disk fast.

Describe alternatives you've considered
Maybe a database?

Build History should clean out old operations

Describe the bug
When the build is successful the build history should be cleared to remove old operations. It should still keep un-run operations for failed incremental builds, but when an operation replaces another we should not keep them both since only one will be valid.

To Reproduce
Steps to reproduce the behavior:

1. Build with a set of macros predefined
2. Build with a different set

Expected behavior
Only one operation should be tracked, but instead it has both versions

Pretty Toml

CppToml does not support pretty printing the toml files and throws away comments. This causes edits to the recipe file to blow away formatting and comments. May want to look into a different toml parser.

Operation with no output does not run when exe has changed

Describe the bug
Because there is no output files to check the timestamp against the test harness executables do not get run when they are updated.

To Reproduce
Steps to reproduce the behavior:

1. Run Build with a Test Extension
2. Change a test
3. Runt build again

Expected behavior
Test should run

Investigate BuildXL

The current incremental build is slow and not perfect. Soup should rely on another well implemented build execution engine.

BuildXL could be that project.

Windows Path Normalize Improvements

Windows path normalizing currently uses character toupper conversions. This is ugly, does not handle utf8 multicharacter values and will not work well on Linux.

Within the runtime soup should require exact file path matching, however when monitoring operation evaluations other executables may use non unique paths on windows (ie. C:/Windows/ and C:/WINDOWS). This means on windows we have to have a file path comparison logic that ignores case in utf8 strings in the SystemAccessMonitor but during get attributes calls we need to add extra checks to ensure the file paths are exact to help improve interoperability between linux and windows.

Should ValueLists be sorted when generating output folder

Unique file hashing does not sort ValueLists. This may cause different orders to generate different folders. It is not as important as sorting the ValueTable since those have many different use cases in the unique input to a package however it may become an issue.

Support Linux

There is currently blocking bugs in the Clang bugs that are preventing the linux compilation of the command line exe. This task will track that work once it is ready to be supported.

Pass VCToolsInstallDir into clang to tell it where link.exe is

Allow for Out of Source building

Is your feature request related to a problem? Please describe.
The default location of a build should probably remain inside the package folder in a single "out" folder, however building inside the source tree is not best practice. Ideally we should be able to specify a root folder to place all of the build output.

Describe the solution you'd like
One solution would be to use directory config files that are used by the CLI to look for a root directory specification for all projects under that folder. Otherwise a user config could override it for an entire user account, or a exe config could override it for the installation on a given machine.

Directories should not use timestamp for incremental

Describe the bug
Directories created should use the fact that the directory exists to check incremental builds. Otherwise if the mkdir.exe is newer than the directory it will always run.

Windows App Container

To provide a higher level of security when executing arbitrary code that was compiled during the build evaluation and execution phase we should instrument the Tasks and Operation invocations to run under a Windows App container to prevent access to OS functionality. This can then be relaxed by allowing build tasks to request special permissions.

Filtered Dependencies

Is your feature request related to a problem? Please describe.
Add ability to filter dependencies directly in the recipe declaration. For example add ability to have windows only dependency.

Describe the solution you'd like
TBD

Describe alternatives you've considered
TBD

Additional context
TBD

DLLs incremental builds not working for MSVC

The root issue here is that MSVC link.exe does not always reset the timestamp for the implib. A fix for this could be to always delete the output files for a build node to ensure they are always created.

Scope Loading All File Write Times

Instead of loading all file write times we should check to see how likely this is to improve performance. As is there are a ton of files that are processed that are not included in the builds.

Could we limit this to known folder locations that have a higher likelihood to be used or preprocess folder heirarchy from loaded operation results to perform an initial rehydrate pass.

Recipe folder paths are not normalized

running a build with or without the trailing directory slash will make the build definition appear to be different and incremental builds will not trigger.

Replace TimeStamp Based Incremental Builds

Is your feature request related to a problem? Please describe.
Replace time-stamp based builds with a better solution.

Describe the solution you'd like
Time stamps are not a true indication of when a file does not match up with the output of a build. We need to has the contents of a file to determine if they match up with the original source that was used to generate a build output to have a true indication if an incremental build needs to re-run an operation

Windows Installer

Is your feature request related to a problem? Please describe.
Create a windows installer (msi) to make setup easier for windows users.

Describe the solution you'd like
Probably Wix

Generate Binary Version from package version on windows.

The build task should auto inject the package version into the binary targets.

This will most likely be implemented by generating a resource file during compilation.
https://docs.microsoft.com/en-us/windows/win32/menurc/about-resource-files

Investigate Known Folder Macros

It may be useful for distributed builds and moving package builds around to allow for source and target folders to be macro values that are only expanded during evaluate time. This way the build extensions do not know the actual location of files so a build can be generated on one machine and evaluated on another.