output build directory / tracking BUILD.py

[jachris]

output build directory / tracking BUILD.py

[jachris]

Hello velkyel,

rules may write to arbitrary files in the filesystem. Only the outputs declared to the build system (the ones metioned in core.publish(outputs=...)) are required to be inside the build directory. This is currently enforced here:

cook/cook/core/rules.py

Lines 69 to 70 in 74e79da

	elif not phony and not misc.is_inside(output, system.build('.')):
	raise ValueError('output outside of build directory')

For example you may install any files in the appropriate system directories. Just don't mention them inside core.publish. This means the files will not be tracked by the system — for example, modification or deletion will not trigger a rebuild. Tasks like "install" should be created by using core.publish(phony=True, outputs=['install'], force=True, ...), which means that they will be executed uncondictionally.

Do you have use-case where you require an output outside of the build directory that should be tracked by the build-system? If so, sharing it would help very much. We can change this behaviour if reasonable.

[velkyel]

Hello,

my simplified use-case is following: I have two directories a/ and b/ in project root. a/ contains files and I want convert/transform them (based on quite complicated rules) into b/ (b/ contains other files too - not only conversions from a/). Changes in a/ or deletion in b/ should trigger rebuild.

Another issue: cook doesn't track BUILD.py itself like scons do with SConstruct. (forced rebuild is invoked on build script modifications)

[jachris]

Alright,

the restriction regarding output file paths is disabled for now (b58b54e). It might be enabled again in the future, but I guess having the option to disable this on individual rules might be useful then. For example, one could write core.publish(strict=False, ...) or something like that, we will have to see.

Regarding your other point: Cook does not automatically track the BUILD.py because of granularity: Tiny changes which might only affect one single task would lead to rebuilding every single task defined in that script. Instead, rules are advised to mention any data they depend on using core.publish(check=XYZ, ...). However, I just pushed a new commit (c19e094) which allows you to do the following:

core.publish(
    inputs=[__file__],
    outputs=...,
    message=...
)

This will track the entire BUILD.py script. __file__ resolves to the path of the file containg it.

In the future, we might add means of automatically tracking the function defining the rules (i.e. hashing the functions __code__ attribute or something like that).

[velkyel]

great. For this patch ae1f48b too. strict keyword switch will be ok. Issue about tracking BUILD.py script changes is not so important for me so file solution is good enough.
Thank you!

[jachris]

Great to hear this!

If you have any other comments / questions, feel free to open issues / message me. 👍