Minor change to custom system triggers full rebuild

[nerves-project]

Minor change to custom system triggers full rebuild

[mobileoverlord]

@niklasnisbeth your memory is pretty correct!

However, there are times where its not smart enough, and you just need to start over to ensure that things get built properly. I am totally open to discussing this though considering I too have been bit by it. Make a tiny change, massive rebuild, sadness.

Things to Consider

1. Changes to dependencies in the parent project.
   For example, nerves_system_br gets updated, or, the package containing the defconfig and related files gets updated.

I think in this scenerio a full rebuild is required.

2. Changes to files within the system like defconfig or kernel config

Here we could employ a default strategy to attempt to call make and not force a full rebuild. We would also want to add some API commands that allow you to force the full rebuild, but, maybe mix clean could handle that if we shim it.

What we currently do

The problem actually stems from the way we determine if the build needs to happen again, and where the files for the build are stored. Currently, we determine if the source files have diverged from the build product by means of hashing the hashes of all the files tracked in the checksum files list declared in the Nerves package config. here is an example of which from nerves_system_rpi3 https://github.com/nerves-project/nerves_system_rpi3/blob/master/nerves.exs#L21-L30

The overall hash of the hashes (the checksum) is then placed in the artifact in a file CHECKSUM. Whenever a mix compile is called, these values are compared and if they do not match, the nerves_package compiler is invoked for the package.

When using the docker provider, we use the checksum as part of the name of the container so we can have a way of linking build environments to source files. This is where we need to have part of the discussion. There are problems with the current setup like wasted containers and space. Clean up is inevitably needed and currently unhandled.

@GregMefford @tmecklem Chime in if you have ideas for how to better checksums with docker containers but still allow partial rebuilds.

[GregMefford]

I think we need to rework the way the Docker build system works because I agree that it's messy and prone to error/confusion. One ides is to just have one container for Nerves builds and just manage all the build directories in there, syncing the source files over and running make just like you would if you were in a Linux environment. Then, like you said, we'd need a command to make it easy to proxy a make clean into the appropriate directory within the container for when you want to do a full re-build.

This is on my list of things I'd like to give some attention to when I can free up some time to work on it.

[mobileoverlord]

This has been fixed in nerves 0.6.0. Docker containers will no longer trigger full builds and you can clean them using mix nerves.clean Let us know if you have any issues with it. :)

[GregMefford]

Also fixed for Linux by #148, which will be released in the next version of Nerves. Sorry we missed that change when building directly on Linux.