hrdag/makr

tools for recursive make in principled data processing projects

makr

tools for recursive make in projects organized by principled data processing.

Install from this directory with python setup.py install.

Run the tests from this directory with python setup.py test.

Then use the tool by cd'ing into your favorite principled data processing task, and saying $ makr. That will run make (whether or not the Makefile is in the task or src/ directory). If you give the command as makr -r, it will extract the dependencies from the Makefile, sort them topologically, and make the tasks in order to update the current task.

makr's Issues

convert paths from strings to Path objects

see path object.

reorganize makr as python package

maybe split library to init and leave just the cmd line utility in bin. This will happen in the py-pkg branch.

move os.chdir from inside functions to decorator

following this StackOverflow

make all functions handle relative paths + absolute paths + project paths

add function for os.chdir() that can deal with each kind

`makr` should echo `make` stdout and stderr when executing

other than when makr is printing the database for get_deps_from_make, it should echo make's stdout and stderr. Otherwise the user gets confused about what's happening inside of make.

create test-rule in test-task with more than one target

create test-task with more than one rule

Makefile parsing might be harder if there's more than one to find.

better tests for symlinked dependencies

sometimes make refuses to report the database if a symlinked dependency is missing. This can happen if the upstream task has make clean. The bug is inconsistent and I don't know exactly why it's happening.

add req, test for python >= 3.7

This should fail at install and at runtime if python is not >= 3.7

new command line features: -c, -f, -d

With -c, run make clean. -f to clean and run. With -d, show the sequence of dependencies upstream from the current dir.

What should -v do?

get task from dependency

For each dependency:

• is this dependency a file in the current task?
• a symlink to another task?
• a relative path to another task?

Follow and return the task name.

add dependency checker for R and python scripts

once a script has setup an argparse parser, and run it, it should check that the Makefile had all the arguments listed as dependencies and warn the user if something is missing.

create ProjPath class

subclass pathlib.Path. Move get_task_path and get_git_root to class. Make taskpath an attribute, is_a_task a method.

`test_remake_1change` fails bc `makr` does not recurse

Not sure why, fails for @philneff.

>       assert all([pre_times[p] < post_times[p] for p in OUTPUTS])
E       assert False
E        +  where False = all([False, False, False, False, False, False, ...])

Recursion also failing for real project. But the various get_deps tests pass. Hmm.

recurse over list of dependencies, adding each layer

create command line tool

with at least -r capability