mbtproject/mbt

The most flexible build tool for monorepo

mbt's Issues

Upgrade libgit2 reference to latest

libgit2 0.27 is out! 😄

Build local changes without committing

At the moment all mbt commands operate on changes that are committed to the repository.

While developing, it could be quite handy to be able to build changes in your workspace, therefore it would be good to have this as a supported workflow as follows.

mbt build local

Discovers the modules based on the local workspace state and builds the ones that are affected by current (uncommitted) changes

mbt apply local --to <path to template>

Applies the local repository state to the specified path in working directory

mbt describe local

Displays all discoverable modules in topological order

[bug] mbt always return exit code 0

Which mbt command are you having troubles with?

• run-in

What's the current behaviour?

• Currently mbt doesn't retain cmd's exit code & always return exit code 0

What's the expected behaviour?

• Should return sub-command's exit code probably

Environment

• mbt version: 0.19.0
• os: linux
• arch: amd64

`mbt build branch` doesn't honor dependencies

Hi,
We've noticed that when we're building project via:
mbt build branch master --in .
mbt builds modules one by one in alphabet order:

~ $ mbt build branch master --in . | grep INFO
INFO[0000] BUILD module-a in module-a for 4d7de1733c7b2b231e50ca197de714a6385916ed
INFO[0000] BUILD module-b in module-b for 6908e3e689b72ffd64070c281997c0c9d60271b5
INFO[0000] BUILD module-c in module-c for 68af707b99e9864f71b1f61be870307e48d0f1c3
INFO[0000] BUILD module-d in module-d for 024c073bf4e6efe31aed8281b48a998753d3214c

Nevertheless build diff --from <sha> --to <sha> --in . works as expected: modules building with honoring dependencies info:

~ $ mbt build diff --from <sha> --to <sha> --in . | grep INFO
INFO[0000] BUILD module-a in module-a for 4d7de1733c7b2b231e50ca197de714a6385916ed
INFO[0000] BUILD module-d in module-d for 024c073bf4e6efe31aed8281b48a998753d3214c
INFO[0000] BUILD module-c in module-c for 68af707b99e9864f71b1f61be870307e48d0f1c3
INFO[0000] BUILD module-b in module-b for 6908e3e689b72ffd64070c281997c0c9d60271b5

Add the ability to run commands from any directory inside the repo

Currently, mbt xxx commands can only be executed from the repository root dir. Being able to run these commands from any directory inside will simplify the usage for the developers.

Commands resulting in errors display usage

This is likely due to this issue in Cobra
spf13/cobra#340

Local commands don't consider dependants

A depends on B. You make a change to B and run
mbt xxx local

Resulting operation is performed only on B.

Resulting operation (xxx) should be performed on A as well.

• mbt version: 0.14.1
• os: osx
• arch: x64

mbt does not restore a detached head after the build

mbt build | run-in xxx when repository head is detached.
Then run git status.

What's the current behaviour?
git reports that there are changes in the index.

What's the expected behaviour?
git should report that nothing to commit and working tree is clean.

Environment
All environments.

• mbt version: 0.22.0
• os: all
• arch: all

Make --from commit option to be inclusive

Suggestion

Make --from option for command mbt build commit to include commit into the build.

Such behavior will be more git-like, plus we will be able to use same command build command when you have only one commit in your repository \ branch.

Remove empty lines in apply command output

Visualise the entirety of a filtered sub-graph

When used with --name and --graph options in describe command, this feature will emit a dot graph containing the dependents and dependencies of the filtered modules.

They should be colour coded and grouped for easy identification of relationship.

mbt xxx local commands do not detect the newly added modules in deeper levels

To reproduce:
From repository root run:

mkdir -p a/b
echo "name: new-module" > a/b/.mbt.yml

Then try a local command, for example:

mbt describe local

Result: "no modules found"
Expected: Modules table with a single entry for newly added module.

Change the shape of generated graph

When using mbt describe xxx --graph to generate a graph, it uses the default graphviz shape ellipse. This makes it hard to read when module names are long. Using a box will improve the visual appearance of the generated graphs.

Changes to root module are not detected correctly

Which mbt command are you having troubles with?
all commands

What's the current behaviour?
When we have a module at the repository root (not a very common scenario), changes to the content in the root directory does not cause the root module to be treated as a modified.

To reproduce

mkdir test
cd test
git init 
echo "name: test" > .mbt.yml
git add -A
git commit -m "first"
git checkout -b "branch"
echo "hello" > foo
git add -A
git commit -m "second"
mbt describe pr --src branch --dst master
# returns an empty results set

What's the expected behaviour?
should display test module as modified

Environment
all environments

Changes to file dependencies are not reflected in version number

Create a module with a file dependency. Run mbt describe head and notice the version of the module. Make a modification to the dependent file. Run mbt describe head again.

What's the current behaviour?
Module version is the same

What's the expected behaviour?
Should have a new module version

Environment
All environments

• mbt version: 0.15.1
• os: All
• arch: All

Improve the test coverage of apply functionality.

Not very helpful error on misconfiguration

On misconfiguring .mbt.yml file by naming a module and referencing itself in dependencies in the same configuration, the following error is thrown:

Error: An unexpected error occurred. See below for more details.
For support, create a new issue at https://github.com/mbtproject/mbt/issues

not a dag
inner error: not a dag
call stack:
github.com/mbtproject/mbt/e.Wrap /home/travis/gopath/src/github.com/mbtproject/mbt/e/e.go(90)
github.com/mbtproject/mbt/lib.toModules /home/travis/gopath/src/github.com/mbtproject/mbt/lib/discover.go(176)
github.com/mbtproject/mbt/lib.(*stdDiscover).ModulesInWorkspace /home/travis/gopath/src/github.com/mbtproject/mbt/lib/discover.go(126)
github.com/mbtproject/mbt/lib.(*stdManifestBuilder).ByWorkspaceChanges /home/travis/gopath/src/github.com/mbtproject/mbt/lib/manifest_builder.go(105)
github.com/mbtproject/mbt/lib.(*stdSystem).ManifestByWorkspaceChanges /home/travis/gopath/src/github.com/mbtproject/mbt/lib/manifest.go(43)
github.com/mbtproject/mbt/cmd.glob..func13 /home/travis/gopath/src/github.com/mbtproject/mbt/cmd/describe.go(119)
github.com/mbtproject/mbt/cmd.buildHandler.func1 /home/travis/gopath/src/github.com/mbtproject/mbt/cmd/build_handler.go(15)
github.com/mbtproject/mbt/vendor/github.com/spf13/cobra.(*Command).execute /home/travis/gopath/src/github.com/mbtproject/mbt/vendor/github.com/spf13/cobra/command.go(698)
github.com/mbtproject/mbt/vendor/github.com/spf13/cobra.(*Command).ExecuteC /home/travis/gopath/src/github.com/mbtproject/mbt/vendor/github.com/spf13/cobra/command.go(783)
github.com/mbtproject/mbt/vendor/github.com/spf13/cobra.(*Command).Execute /home/travis/gopath/src/github.com/mbtproject/mbt/vendor/github.com/spf13/cobra/command.go(736)
main.main /home/travis/gopath/src/github.com/mbtproject/mbt/main.go(10)
runtime.main /home/travis/.gimme/versions/go1.9.4.linux.amd64/src/runtime/proc.go(195)
runtime.goexit /home/travis/.gimme/versions/go1.9.4.linux.amd64/src/runtime/asm_amd64.s(2337)

(0xbba140,0xc420399bc0)

mbt should be able to identify which module is misconfigured and should throw an appropriate message with it.

Display build summary

It would be useful to know how long each module takes to build when diagnosing slow builds. At the end of mbt build ... command, display a summary of all modules built and the time taken for each.

Add "default command" behavior

It will be great to add default command handler into mbt configs, with possibility to determine name of command invoked:

name: UniversalCommandModule
commands:
  default:
    cmd: sh
    args: 
      - "$MBT_REPO_PATH/scripts/$MBT_COMMAND_NAME.sh"

Acting like that you will be able to add commands without changes in config files. You just need to add new script to the correct location, and that's it.
Even if your projects are not unified, you still can use default command to handle missing commands in your way.

Add possibility to invoke list of commands

Idea

It seems reasonable to be able to invoke list of commands like:
mbt build local --all --command build,collect,release-notes

This approach allows to re-use information about projects and conditions of a call.

run-in support to run platform specific cmd

Which mbt command are you having troubles with?
Run-in

What's the current behaviour?
I cannot have one command name with different cmd on different platforms. the os property on the command just prevents the command from being executed on non-related platforms

What's the expected behaviour?
One command name with different cmd for each platform just like build command

Add option to switch to external git tool (git.exe)

We currently trying to use mbt to build our monorepo on Windows and we faced few problems:

1. Libgit2 does not work correctly with long paths and it is not going to be fixed.
2. Libgit2 does not work correctly in docker container for some reason, and always returns empty request results for repository.

In both cases git.exe has correct behavior and could be used as backup tool.
Can we consider such option? It could be useful in a lot of other corner cases, and definitely can make tool more usable.

Document how internals work

Passing command line arguments to `build` or any command

Is it possible to pass command line arguments when calling mbt build or mbt run-in

I am looking to use mbt for but currently we would call, for example,
npm run dev:eap -- --DEV_API <custom host domain>
or to pass arguments in during a build that configure how the build executes. Is there any way to do this with mbt I couldn't see anything in the documentation that indicated it was possible?

Add mbt build commit xxx command

mbt build commit xxx where xxx is a valid commit SHA, should build all modules as of that commit.

Clean up workspace after build

Which mbt command are you having troubles with?
mbt build commit|pr|diff|branch

What's the expected behaviour?
During build, mbt checks out the relevant commit and then at the end of the build it switches back to whatever is pointed by head. However, in the process, it leaves modified files in workspace. Instead, it should leave the workspace in it's pristine state.

Environment
all environments

Add a whatif command to find out the impact of changing a module

For developers, sometimes it's useful to find out the impact of changing a module to it's dependency graph. We should be able to invoke this command as mbt whatif <module-name> and the output should be a list of impacted module names in topological order.

We should also be able to specify multiple module names.

e.g.
mbt whatif module-a module-b

[enhancement] Ability to run custom functions

It would be great if mbt could run custom functions as defined in .mbt.yml, similar to yarn/npm run x where x is defined in package.json

Supports Maven and/or Gradle (Java build tools)?

Version is written to stderr

Which mbt command are you having troubles with?
mbt version

What's the current behaviour?
Outputs the version information to stderr

What's the expected behaviour?
Should output version to stdout

Environment
All

• mbt version: 0.16.0
• os: Mac
• arch: x64

Prevent two modules from using the same name

Module name is used as the key of internal maps. During discovery phase (discover.go), we should return an error if the module name is already taken by another module.

Error message should return the details of conflicting modules so that the user can easily locate them to fix it.

Use current working directory as the default input for --in parameter

It's a common pattern to change working directory to the repository root before executing mbt. By using this as the default value for --in parameter, we can save a few key strokes.

Add support for shortened SHAs

Current commands only accept the full SHA. It would be nice to be able to use the shortened version just like git.

.NET core project native support

Scenario

User has a git repository with multiple .NET core applications. Same repository also contains shared libraries used by those applications.

- lib/foo
- lib/bar
- web-app
- console-app

Typically, in a setup like this, each application directory is marked as an mbt module by placing a .mbt.yml file. However, if user wants to rebuild/re-version applications when their dependencies change, all library directories must also be marked as mbt modules and correct dependencies must be established in .mbt.yml files.
This feature intends to simplify this process by being able to understand the dependencies defined within .csproj files.

Specification

• A directory with a .csproj file is recognised as an mbt module (conveniently called csproj module).
• Module name is .csproj file name without extension.
• All project references in .csproj becomes dependencies of that module.
• When a directory contains both .mbt.yml and .csproj files, module is generated based on .mbt.yml and augmented with dependencies found in .csproj.
• If a particular directory contains more than one .csproj files, behaviour of this feature is not specified (this can be re-visited in a future version)

Design Notes

• Modify discover.go to with an extensible model that can be used to support other types of projects (e.g. maven pom files, sbt projects) in the future.

• In order to make sure backward compatibility, this feature is activated via --detect-csproj switch.

Add ability to skip building module

With a differential build flow set up, sometimes there is a need to not build a particular module. It would be good to have flag / file based token to indicate that this module although defined and with changes, there is no need to build it.

A possible implementation would be to add a --skip flag but that would require changing the CI script everytime. An alternative is to check for a .mbt_skip_module file at the module root and if the file exists, skip building or to define a global .mbt_skip_modules at the repo root with a list of modules not to build.

If any modules that depended on a skipped module, it should either skip or fail gracefully, optimally it should be configurable I guess.

What do you think?

Display a simple error message when user enters an invalid commit sha

e.g. mbt describe commit fjldaf should display a simple error message than:
encoding/hex: invalid byte: U+006A 'j'

Generate a dot file representing the entire dependency graph

mbt describe xxx commands should take --graph flag and emit the appropriate dot file with the appropriate dependency graph.

We should use the directed graph format specified here:
https://en.wikipedia.org/wiki/DOT_(graph_description_language)

Improve documentation structure

• Remove unwanted links from "see also" section.
• Review documentation

Test the distro before publishing

Currently build agents run mbt version command at the end of the build. We should add a smoke test for each command (mbt describe | build | apply) to this list for release builds.

Build specific modules with mbt build local command

At the moment, mbt build local and mbt build local --all builds workspace changes or all modules in the workspace respectively.

It would be nice to be able to mbt build local --name <module name> where it builds the modules matching the specified pattern.

Pattern matching could something like sub-sequence matching in order to specify the desired module(s) with the least number of keystrokes.

Expose the hashes based on content for locally-modified modules.

Which mbt command are you having troubles with?
mbt describe local

What's the current behaviour?
It returns 'local' for any modules with local modifications. This is fine but I'd like to also build some optimization in our pipeline whereby it doesn't unnecessarily rebuild things. So, exposing the "content hash" per module perhaps via mbt hash module would be a nice feature to have.

What's the expected behaviour?
Access to the content hashes (apart from committing local changes and then calling mbt describe head could be made available.

Create describe diff command

Ideally we should have describe counterpart for every build xxx command. Based on that theory, it would useful to have describe diff --from <from sha> --to <to sha> option.

Add easier way to build last commit in current branch

Can we add command that will build last commit in current branch?
Like one of the following:

mbt build last
mbt build commit last
mbt build commit HEAD

Alternatively we can just resolve aliases like HEAD, HEAD^, HEAD~N in all execution scenarios.

having repository absolute root path as an env variable on build

Hi,
It's good to have access the mono repo absolute root path in build/custom-command runtime as an environment variable (e.g. MBT_REPO_PATH or MBT_ROOT_PATH) just like MBT_MODULE_NAME and MBT_MODULE_VERSION
Also we can have MBT_MODULE_PATH to get the current module path (relative to root path) as well.

Read information about projects from configuration file

Suggestion

Add possibility to cache list of paths to project configurations to make mbt work faster.

On big a repository operation to find all .mbt.yml files can take really long time.
Because of that we have quite long wait time for scenarios when user want to perform operation for the whole repository. For example:
mbt describe local --all
of just
mbt build local --name MyProject

For such cases it will be useful to support non-interactive scenario of work, when we can:

1. Perform cache operation on mbt to find all projects in repository and store it to configuration file that can be source controled or used locally:
   mbt cache-projects local
2. Run mbt in non-interactive mode that will use configuration file to gather information about projects in the repository.

Universal build command in the MBT spec file

Not sure if this already exists or not, but is there a way, and if not would it make sense, to have a default version of the build command in the spec file.

What I mean is in the .mbt.yml file, instead of defining the build command for each platform, to define it once as a default command, to run on all platforms unless otherwise specified.

# as an example something like this
name: my-module
build:
  default: # regardless of the platform, run this command on build
    cmd: some-platform-agnostic-command

This could be useful in case there's no need to specify the same build command for multiple platforms, think of an NPM run script for example as it's platform independent in many cases as an example.

It could potentially offer a way to override it for specific platforms, for example if there's a custom build ran on Linux for any platform specific reason,

name: my-module
build:
  default: # runs on any system
    cmd: my-generic-command
  linux: # runs if the system is linux based
    cmd: a-linux-specific-build-script

Ability to build changes in a commit

Current mbt build commit <commit sha> builds all modules found in that commit tree. It would be nice to be able to run a variation of that command mbt build commit <commit sha> --changes where it would build only the modules changed in that commit.

Adding Any helper method to apply command template

Having any helper method in templates to extend contains method to check for more than one item at a time in a list.

sample usage in template file

_#add serviceName if any of "iis" or "database" exists in deployment.tags_
{{- if (any (property $module "deployment.tags") ("iis", "database")) }}
    "serviceName": "{{property $module "deployment.serviceName"}}",
{{- end }}

Versions of other folders as environment variables to current build

I find it necessary to know the name of the artifacts of dependencies when building a folder.

for example:
It could be convenient to use MBT_MODULE_VERSION for tagging all the artifacts, but when module B is built after module A, the build script no longer has access to MBT_MODULE_VERSION of module A.

The best plan I had was using something like

mbt describe branch $(git rev-parse --abbrev-ref HEAD) --json | jq .$1.Version | sed 's/"//g'

which uses jq to parse the json response of the current versions in this git branch.

What ideas for workarounds do you have? It would be convenient having environment variables like
MBT_{MODULE_NAME}_VERSION
for all the projects.

mbt build local on an empty repository does not build anything

Steps to reproduce:

• Create a new git repository
• Create a new mbt module
• Run mbt build local

Expected: New module to be built
Result: Nothing is built

Also related:

• Run mbt describe local

Expected: New module to be displayed in the modules list
Result: No modules found