opensourceeconomics/econ-project-templates

Templates for reproducible economics projects

econ-project-templates's Issues

Prepare for release 0.2

• Make sure RTD versioning works as expected - point to stable, possibility to get to master (https://docs.readthedocs.io/en/stable/versions.html) with relevant differences in where cookiecutter points to.
• Add release notes MD / RST file (finally a place to adequately thank all those helpers), link in docs (possible if in main folder? else just in docs?)
• put bld/examples under version control
• Move docs of 0.1 release to RTD, too, if possible.
• Tag release
• Get specific Zenodo DOI (IIUC, see http://help.zenodo.org/#versioning) and force push to tag with that citation for front page / license of docs.
• Put in redirect only on gh-pages
• Make sure everything from current README is on RTD landing page, put link to RTD in README and add some notes that this part is mainly for developers.

pre-commit hooks fail in windows

Potentially related to Issue #57
Committing fails in my setting (Windows 10 machine), unless I circumvent the commit hook. Apparently, some pip routine fails. See the log below.

It is striking that there is no py_env-python3.7 folder in z:\.cache\pre-commit\repod5i_j8ft\. I don't think is is a permission issue, since I have full permissions on the Z: drive.

(dynamod)` W:\izamod\dyn\izadynmod>git commit -m "some message"
[INFO] Locking pre-commit directory
[INFO] Installing environment for https://github.com/ambv/black. 
[INFO] Once installed this environment will be reused.
[INFO] This may take a few minutes...
An unexpected error has occurred: CalledProcessError: Command: ('Z:\\.cache\\pre-commit\\repod5i_j8ft\\py_env-python3.7\\Scripts\\pip.exe', 'install', '.')

Return code: 1
Expected return code: 0
Output:
Processing z:\.cache\pre-commit\repod5i_j8ft
  Installing build dependencies: started
  Installing build dependencies: finished with status 'error'
Errors:
ERROR: Complete output from command 'z:\.cache\pre-commit\repod5i_j8ft\py_env-python3.7\scripts\python.exe' 'z:\.cache\pre-commit\repod5i_j8ft\py_env-python3.7\lib\site-packages\pip' install --ignore-installed --no-user --prefix 'C:\Users\iza6354\AppData\Local\Temp\pip-build-env-6o1cigzy\overlay' --no-warn-script-location --no-binary :none: --only-binary :none: -i https://pypi.org/simple -- 'setuptools>=40.8.0' wheel:

  ERROR: Collecting setuptools>=40.8.0

    WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.VerifiedHTTPSConnection object at 0x0000026A3668BE80>: Failed to establish a new connection: [Errno 11003] getaddrinfo failed')': /simple/setuptools/

    WARNING: Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.VerifiedHTTPSConnection object at 0x0000026A366A50B8>: Failed to establish a new connection: [Errno 11003] getaddrinfo failed')': /simple/setuptools/

    WARNING: Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.VerifiedHTTPSConnection object at 0x0000026A366A50F0>: Failed to establish a new connection: [Errno 11003] getaddrinfo failed')': /simple/setuptools/

    WARNING: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.VerifiedHTTPSConnection object at 0x0000026A366A5390>: Failed to establish a new connection: [Errno 11003] getaddrinfo failed')': /simple/setuptools/

    WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.VerifiedHTTPSConnection object at 0x0000026A366A5400>: Failed to establish a new connection: [Errno 11003] getaddrinfo failed')': /simple/setuptools/

    ERROR: Could not find a version that satisfies the requirement setuptools>=40.8.0 (from versions: none)

  ERROR: No matching distribution found for setuptools>=40.8.0

  ----------------------------------------

ERROR: Command "'z:\.cache\pre-commit\repod5i_j8ft\py_env-python3.7\scripts\python.exe' 'z:\.cache\pre-commit\repod5i_j8ft\py_env-python3.7\lib\site-packages\pip' install --ignore-installed --no-user --prefix 'C:\Users\iza6354\AppData\Local\Temp\pip-build-env-6o1cigzy\overlay' --no-warn-script-location --no-binary :none: --only-binary :none: -i https://pypi.org/simple -- 'setuptools>=40.8.0' wheel" failed with error code 1 in None
Check the log at Z:\/.cache\pre-commit\pre-commit.log

Very (very) minor: Comma in page headers of Sphinx PDF documentation

The default page header line is defined in sphinx.sty is

 \fancyhead[LE,RO]{{\py@HeaderFamily @title, \py@release}} 

so the header becomes "title, release".

In the conf.py of the template, you set the release to empty string in the latex_elements:

 # Remove the "Release ..." subtitle from the LaTeX frontpage.
    "releasename": "" 

This way the default page header is "title, " with a lost comma, which is ugly. To maintain flexibility in case someone wants the release in the header, one could add the following preamble to the latex_elements:

# Comment out if you want the release displayed in the header
    'preamble': r"""
        \makeatletter 
            \fancypagestyle{normal}{ 
                \fancyhf{} 
                \fancyfoot[LE,RO]{{\py@HeaderFamily\thepage}} 
                \fancyfoot[LO]{{\py@HeaderFamily\nouppercase{\rightmark}}} 
                \fancyfoot[RE]{{\py@HeaderFamily\nouppercase{\leftmark}}} 
                \fancyhead[LE,RO]{{\py@HeaderFamily \@title}}
             } 
        \makeatother"""

Build failing with R project template

I'm trying to get the R project template to run as is. So I forked this git repo and then cloned just the R branch using

git clone -b R [email protected]:RoyalTS/econ-project-templates.git

Followed the instructions in the README, ran

python waf.py configure
python waf.py build

and eventually ran into the following error:

Waf: Leaving directory `/Users/tobiasschmidt/Dropbox/research/econ-project-templates/bld'
Build failed
Traceback (most recent call last):
  File "/Users/tobiasschmidt/Dropbox/research/econ-project-templates/.mywaflib/waflib/Runner.py", line 287, in task_status
    return tsk.runnable_status()
  File "/Users/tobiasschmidt/Dropbox/research/econ-project-templates/.mywaflib/waflib/extras/sphinx_build.py", line 346, in runnable_status
    ret = super(sphinx_build_task, self).runnable_status()
  File "/Users/tobiasschmidt/Dropbox/research/econ-project-templates/.mywaflib/waflib/Task.py", line 594, in runnable_status
    new_sig = self.signature()
  File "/Users/tobiasschmidt/Dropbox/research/econ-project-templates/.mywaflib/waflib/Task.py", line 572, in signature
    self.sig_implicit_deps()
  File "/Users/tobiasschmidt/Dropbox/research/econ-project-templates/.mywaflib/waflib/Task.py", line 763, in sig_implicit_deps
    (nodes, names) = self.scan()
  File "/Users/tobiasschmidt/Dropbox/research/econ-project-templates/.mywaflib/waflib/extras/sphinx_build.py", line 254, in scan
    doc_name + suffix for suffix in suffixes)))
WafError: Could not find Sphinx document node at any of: 11 added, 0 changed, 0 removed.rst

What am I doing wrong?

Make environments consistent (set-env.sh / set-env.bat / .environment.(linux|osx|windows).yml

• In the Python branch, this should be more or less up-to-date, at least for Linux)
• Make sure versions of files are at Python 3.6 and latest version of everything else
• Port to the other branches, make sure things work for other OSs

Update references in docs

E.g., Gaudecker JF is still listed as "forthcoming" ...

Documentation improvements

• Note that on Windows, people should use the Anaconda prompt unless they know what they are doing
• Note among prerequisites that they need to execute "conda install python-graphviz"

Improvement of automatically generated DAGs

• add source files, e.g. schelling.py
• Different colour for task names.

bld/project_paths.py is not created

After cleaning waf, bld.project_paths is not re-created.
After running
python waf.py distclean
python waf.py configure

python waf.py build fails. This is not due to setting $PATH or $PYTHONPATH wrongly, the file is just not there.
What might be causing this?

Re-enable conda_environment creation once conda 4.6 is out

See: conda/conda#7206.

Disabled in 18e2fcf, will need to revert that.

Structure this using releases rather than just cloning the repo

• Makes it clearer which version we are talking about
• Allows development without affecting users.

Make auto-formatting more convenient with pre-commit

Hi,

I had two issues with the format_python_files.py approach which let me to change this in my template.

1. It is annoying to execute the formatting when you already agreed that files should look this way.
2. isort does not really work when used inside tox as it misrepresents first, second, third party applications which caused tests to fail.

Therefore, I now use a different approach using pre-commit. The tool allows you to define several git hooks which will be execute when trying to commit. In case any of the defined hooks catches an error, the commit is aborted.

All you need is a config file and you need to install all packages under repo: local. Then, type pre-commit install and the hooks will run before each commit. The posted config runs black, black on documentation, flake8 and doc8. Furthermore, isort is replaced with reorder_python_imports which is simpler and easier to implement.

There are a lot of other hooks available under https://pre-commit.com/hooks.html as yaml checks, syntax upgrades, etc..

Best

Get Python Example to Run, EPP Course

After following the steps from Getting Started, I am still unable to get the python example to run. I am using Windows 10 Pro and the Anaconda Prompt. python waf.py configure works. However, python waf.py build gives the following error message: No module named 'sphinxcontrib.bibtex'. (I did not check the box for sphinx, as far as I can tell)

I already tried the following:

• Updating Sphinx
• Updating Miktex

pip install sphinxcontrib-bibtex did the trick.

Get Julia template to run

Should be simple:

• Add a function for Julia to .mywaflib/waflib/extras/write_project_headers.py
• Adjust the path in src/data_management/get_simulation_draws.jl
• Write a Julia plugin for pytask
• Convert the remaining steps of the analysis from Python to Julia (there is a version in the 0.3 releases for get_simulation_draws.jl)
• Convert the documentation -- this plugin should work well enough: http://bastikr.github.io/sphinx-julia/

Associating git with another default editor

You might want to incorporate this link into your slides / lectures to help out students not familiar with vim.
It shows how to set up editors mate, atom or sublime as your default git editor for commit messages etc.

For sublime:

git config --global core.editor "subl -n -w"

https://help.github.com/articles/associating-text-editors-with-git/

Clear instructions for setting up everything upfront & using PowerShell on Windows

• Insert a step 2 to make sure everybody has exactly the same setup:
  • Terminal on Mac / Linux, Powershell on Windows.
  • In correct location. No spaces / special characters in path (see prog-econ repo, slides on Shell Introduction).
  • All paths to required programmes should be correct (explain that we will need to launch them from the shell), i.e., pdflatex and, if required, anything that is required like stata, matlab, r should start the respective programme (check the files {{cookiecutter.project_slug}}/.mywaflib/waflib/extras/run_*_script.py). Assume atom is installed and atom launches editor from shell. Similar for git.
  • Links to how to set paths on MacOS, Windows. Explain that new shell needs to be launched./
• New step 3 (currently 2) should read
  $ conda deactivate $ conda install cookiecutter pre-commit
  First part just to be sure. Second because conda init often fails (Mac, Windows) for reasons I have not fully understood.
• Add a step 7 (currently between 5 and 6) on what to do if anything fails:
  1. Understand the error
  2. Type
  $ atom ~/.cookiecutter_replay/econ-project-template-[version].json
  Fix the faulty setting if something went wrong there, alternatively fix paths etc. When done, launch a new shell if necessary and type:
  $ cookiecutter --replay https://github.com/hmgaudecker/econ-project-templates/archive/[version].zip

Retain old versions of docs

Starting with 0.3, we should keep the documentation associated with those versions.

Stata project paths not fully sorted

For those (still) using Stata:
Stata project paths are not fully sorted. This might generate differing project_paths.do-files after re-running write_project_paths, even if there are essentially no changes to the project paths. This is because items in Python dictionaries are not consistently sorted over time. It causes unnecessary re-runs if Stata analysis files depend on the project paths.

From file waflib/extras/write_project_headers.py:

def run(self):
        with open(self.outputs[0].abspath(), 'w') as out_file:
            out_file.write(PROJECT_PATHS_STATA_COMMENT)
            for name in sorted(self.env.PROJECT_PATHS.keys()):
                val = self.env.PROJECT_PATHS[name]

                if isinstance(val, Node.Node):
                    out_file.write(
                        'global PATH_{n} "{p}/"\n'.format(
                            n=name,
                            p=val.abspath()
                        )
                    )
                elif name == 'ADO' and isinstance(val, dict):
                    self._write_ado_paths(val, out_file)

While some sorting takes place, projects paths with keys 'ADO' are not sorted. This can lead to changes in the order in which the paths contained in 'ADO' occur in project_paths.do.

Stata branch fails with custom Stata starting directory

When running a fresh template of the Stata branch with a Stata installation that uses a custom starting directory (Changing the path in a profile.do), then waf fails. The reason is that the "include project_paths" line cannot be executed, the file is not found. It is a minor irritation, but maybe it would be possible to write code which detects that behavior through running a polit .do file which just checks the current path and if it is not the waf directory, throws a message notifying the user of the problem. A more thorough solution would be to reset the Stata path in every .do file.

Build failed Runner.py sphinx_build.py Task.py

In the python branch I get the following error using a fresh anaconda (python 3) installation in Gnu/Linux Ubuntu 17.04.

I get this error in the R and Python branches. Do you know a solution? I haven't changed anything - it's a fresh clone of the econ-project-templates repository.

Waf: Leaving directory `/home/robert/econ-project-templates/bld'
Build failed
Traceback (most recent call last):
  File "/home/robert/econ-project-templates/.mywaflib/waflib/Runner.py", line 270, in task_status
    return tsk.runnable_status()
  File "/home/robert/econ-project-templates/.mywaflib/waflib/extras/sphinx_build.py", line 356, in runnable_status
    ret = super(sphinx_build_task, self).runnable_status()
  File "/home/robert/econ-project-templates/.mywaflib/waflib/Task.py", line 659, in runnable_status
    new_sig = self.signature()
  File "/home/robert/econ-project-templates/.mywaflib/waflib/Task.py", line 640, in signature
    self.sig_implicit_deps()
  File "/home/robert/econ-project-templates/.mywaflib/waflib/Task.py", line 799, in sig_implicit_deps
    (bld.node_deps[key], bld.raw_deps[key]) = self.scan()
  File "/home/robert/econ-project-templates/.mywaflib/waflib/extras/sphinx_build.py", line 242, in scan
    app,
TypeError: update() takes 4 positional arguments but 5 were given

WARNING: Build failed
Traceback (most recent call last):
  File "/home/robert/econ-project-templates/.mywaflib/waflib/Runner.py", line 270, in task_status
    return tsk.runnable_status()
  File "/home/robert/econ-project-templates/.mywaflib/waflib/extras/sphinx_build.py", line 356, in runnable_status
    ret = super(sphinx_build_task, self).runnable_status()
  File "/home/robert/econ-project-templates/.mywaflib/waflib/Task.py", line 659, in runnable_status
    new_sig = self.signature()
  File "/home/robert/econ-project-templates/.mywaflib/waflib/Task.py", line 640, in signature
    self.sig_implicit_deps()
  File "/home/robert/econ-project-templates/.mywaflib/waflib/Task.py", line 799, in sig_implicit_deps
    (bld.node_deps[key], bld.raw_deps[key]) = self.scan()
  File "/home/robert/econ-project-templates/.mywaflib/waflib/extras/sphinx_build.py", line 242, in scan
    app,
TypeError: update() takes 4 positional arguments but 5 were given

Testing with Azure on Windows

Implement at least subbattery of tests that do not rely on latex distribution.
Do some more research on whether there is any possibility to get latex in azure on windows.

Errors in Python-variant

Dear Mr. Gaudecker,

recently I cloned a fresh version of the Python module and when running it now, several errors occurred.

First, the first Python file to be executed did not find the project_paths.py file, which I solved by adding an empty __init__.py file in the same directory. Maybe waf should do that by itself?

Second, it seems to me that there is a small coding mistake. In schelling.py and plot_locations.py, it seems that the line
model = json.load(open(ppj("IN_MODEL_SPECS", model_name + ".json"), encoding="utf-8"))
should actually be
model = json.load(open(ppj("IN_MODEL_SPECS", model_name + ".json")), encoding="utf-8")

The third error, which I am not able to solve right now, is that waf sends the following message:
WafError: Could not find Sphinx document dependency node: C:\Users\...\pythontemplate\src\paper\refs.bib
Is there some fix for this? I don't really know where to start debugging it as the file exists and the path to the file is correct. Of course, when I remove the reference to the bibliography, everything works just fine.

I would highly appreciate any help. Thank you!

Best regards,
Felix

Change getting the code how-to (Getting started)

• Currently it refers to using Git, then deleting .git etc.
• Should be: Download the version for Python / ..., save where you want to start your project

Add DAG generation to all templates

Currently only in Python

Fix paths in all templates

The project paths set in the main dictionary were different across branches and Tobias made changes relative to what the other languages expect. This is a huge mess currently and needs to be cleared up very soon.

Add sphinx autodoc for Matlab

• install sphinxcontrib-matlabdomain via pip in environment if both configure_running_matlab_from_waf and configure_running_sphinx_from_waf are set to True.
• Then make documentation of functions etc exactly the same as in Python example
• See the code by @gehringl for how to do that.

Add index.rst to Stata directory

set-env.sh / .bat should fail on error

E.g.:

$ source set-env.sh update
Executing: 
source activate prefs-pf-choice
conda update cython ipython-notebook matplotlib numpy pandas patsy pip pytables scipy sphinx

discarding /Users/xxx/miniconda3/envs/prefs-pf-choice/bin from PATH
prepending /Users/xxx//miniconda3/envs/prefs-pf-choice/bin to PATH
Error: package 'ipython-notebook' is not installed in /Users/xxx//miniconda3/envs/prefs-pf-choice
Executing: 
source activate prefs-pf-choice
pip install -U cloud_sptheme sphinxcontrib-bibtex

This should fail after the Error.

Same for giving unknown options (add help: Supported options are "install" (...) and "update" (...), if no option is given, ...)

Cross-platform environments

This is following up on #16 and to finally settle all issues related to environments.

Instead of the one file

.environment.yml

we want three:

.environment.linux.yml
.environment.osx.yml
.environment.windows.yml

set-env.bat reads from the last, set-env.sh needs a switch based on the return value of uname. Everything else is just too painful to work with.

As for the best way of exporting things, check for the resolution of https://groups.google.com/a/continuum.io/forum/#!topic/conda/OlFk6HbC57E

Update docs on project-specific environments.

They still talk about picky.

Once #16 and #18 are solved.

Re-enable --shell-escape for LaTeX

Similar to f7bb61b, but for main wscript. Was there any reason to disable it that I cannot remember? Leaving it on is crucial for the Stata template because the only thing Stata consistently produces are eps-files and they need conversion to pdf before being useful to pdflatex.

Update documentation for R

Removed the attempt to have self-contained packages in 416e2a7. Proably the docs need an update for that, also that requirements for running things are to have the following libraries installed:

• foreign
• zoo
• car
• rjson
• sandwich
• lmtest
• AER
• ivpack

Use Waf 2.0

set-env.sh / .bat should just activate the environment as default

(environment already existing)

$ source set-env.sh 
No arguments supplied. Continuing with 'create' and 'install' as default

The typical case is to just activate the environment, i.e., the Python script should not do anything, I guess.

The Yield an error if it does not exist (conda will probably do that automatically).

Update howto-release

There is a branch "z-howto-release", this should probably just be a file in the gh-pages branch which is not shown publicly among the rendered pages. No idea whether it is up-to-date.

After moving over, the branch can be deleted.

As noted in dbe9ca6, please add that one step will be to add a set of versions to the environment.yml file which is up-to-date and known to be working at the time.

cookiecutter-version: Allow for non-ASCII characters in author name

Seems like trying to use an "ö" broke things. Any more details, @bfischer1990?

Should be possible and should become a test case.

Minor: add dependency graph to the html-part of the documentation

I noticed that this is missing and did not see why, it can be added by something similar to

.. raw:: html

    <object data="../dependency_graph.pdf" type="application/pdf" style="width:100%; height:500px">
        alt : <a href="../dependency_graph.pdf">DAG.pdf</a>
    </object>

combine conda/pip package management

More of a question than an issue perhaps: Why is it you're separating the management of packages installed via conda from those installed via pip? E.g. in set-env.sh you first read in a txt generated via conda list --export > conda_versions.txt and then pip install the packages listed in requirements.txt).

Why not let conda handle all of it? Couldn't you export the entire environment using conda env export > environment.yml (http://conda.pydata.org/docs/commands/env/conda-env-export.html), which outputs not just the packages installed via conda but also those installed via pip, and re-create that environment using conda env create -f environment.yml?

Make graphs for documentation of Stata branch

Same as for the current Python branch docs, main graph here:

http://hmgaudecker.github.io/econ-project-templates/_images/schelling_dependencies.png

Instead of the Schelling example, use the AJR / Albouy example as detailed in the code documentation of the Stata branch (will need to follow what the wscript files are doing).

Needs to be done before @hmgaudecker gets started on #28

Conda environment has problems installing the pip packages upon first creation

Right now, conda enivronment is automatically installed on the system. But somehow, during this first installment, the packages listed under pip are not installed. I always have to update the conda environment from file for them to be properly installed.

Remove underscores from title in documentation

Introduce testing with Azure

Comprehensive test suite

We need a test suite that actually checks whether waf builds succeed. This should work via Travis, I suppose.

This will of course be difficult for Matlab & Stata, but it should be doable for Python, Julia, and R.

Add documentation for Stata / R

Add pyupgrade to pre-commit hooks.

I find it extremely useful to automatically upgrade my code to newer syntax. You can pass different flags to pyupgrade, --py3-plus or even --py36-plus, if you limit yourself to specific Python versions.

ValueError: too many values to unpack

Dear all,

I get the following error. Does anyone know how I can fix this?
This happens also with a freshly pulled template on an OsX yosemite macbook pro. I already tried to reinstall anaconda (python 3) as well as sphinx+sphinxcontrib-bibtex. It creates a LaTeX document and a presentation. Any help is highly appreciated.
If you need further information from me, please tell me. Thanks a lot!

Best, Robert

Waf: Leaving directory `/Users/rbirkelbach/Documents/econ-project-templates/bld'
Build failed
Traceback (most recent call last):
File "/Users/rbirkelbach/Documents/econ-project-templates/.mywaflib/waflib/Runner.py", line 287, in task_status
return tsk.runnable_status()
File "/Users/rbirkelbach/Documents/econ-project-templates/.mywaflib/waflib/Task.py", line 590, in runnable_status
new_sig = self.signature()
File "/Users/rbirkelbach/Documents/econ-project-templates/.mywaflib/waflib/Task.py", line 568, in signature
self.sig_implicit_deps()
File "/Users/rbirkelbach/Documents/econ-project-templates/.mywaflib/waflib/Task.py", line 759, in sig_implicit_deps
(nodes, names) = self.scan()
File "/Users/rbirkelbach/Documents/econ-project-templates/.mywaflib/waflib/extras/sphinx_build.py", line 37, in scan
msg, dummy, iterator = s.env.update(s.config, s.srcdir, s.doctreedir, s)
ValueError: too many values to unpack (expected 3)

Waf build raises: "RuntimeError: generateor raised StopIteration" for Python 3.7

Waf build failed for Python 3.7 but worked after downgraded to 3.6.7.

Reason: PEP 479 is enabled for all code in Python 3.7, meaning that StopIteration exceptions raised directly or indirectly in coroutines and generators are transformed into RuntimeError exceptions. (https://docs.python.org/3/whatsnew/3.7.html)

The exact location of the problem can be found when waf building in 3.7 environment.

When specifying configure_running_sphinx_from_waf = False, building the documentation fails

If that is False, we should not copy the documentation folder at all and not recurse into it from src/wscript

Mysterius error in running waf.py build and tex file is not compiled.

Error Message:

source not found: 'r.tex' in bld(source='r.tex', target='', meths=['apply_tex', 'process_rule', 'process_source'], features=['tex'], path=C:\Users\Radost\Documents\Doktor\Research\patience_compulsory\src\paper, idx=1, tg_idx_count=13, prompt=1, _name='r', posted=True, type='pdflatex') in C:\Users\Radost\Documents\Doktor\Research\patience_compulsory\src\paper

My wscript in src.paper:

def build(ctx):

    for s in 'research_paper':
        ctx(
            features='tex',
            source=s + '.tex',
            prompt=1,
            name=s
        )
        # Install to the project root.
        ctx.install_files(
            ctx.env.PROJECT_PATHS['PROJECT_ROOT'].abspath(),
            s + '.pdf'
        )
        # Running LaTeX processes in parallel leads to
        # too many potential problems.
        ctx.add_group()

My main wscript:

#! python


import os

from collections import OrderedDict

# The project root directory and the build directory.
top = "."
out = "bld"


def set_project_paths(ctx):
    """Return a dictionary with project paths represented by Waf nodes."""

    pp = OrderedDict()
    pp["PROJECT_ROOT"] = "."
    pp["IN_DATA"] = "src/original_data/"
    pp["IN_MODEL_CODE"] = "src/model_code"
    pp["IN_MODEL_SPECS"] = "src/model_specs"
    pp["LIBRARY"] = "src/library"
    pp["IN_DATA_MANAGEMENT"] = "src/data_management"
    pp["BLD"] = ""
    pp["OUT_DATA"] = f"{out}/out/data"
    pp["OUT_ANALYSIS"] = f"{out}/out/analysis"
    pp["OUT_FINAL"] = f"{out}/out/final"
    pp["OUT_FIGURES"] = f"{out}/out/figures"
    # OUT_MODEL_SPECS is only required for using Stata with JSON and
    # can be safely deleted otherwise
    pp["OUT_MODEL_SPECS"] = f"{out}/src/model_specs"
    pp["OUT_TABLES"] = f"{out}/out/tables"


    # Stata's adopaths get special treatment.
    lib = pp["LIBRARY"]
    pp["ADO"] = {}
    pp["ADO"]["PERSONAL"] = os.path.join(lib, "stata/ado_ext/personal")
    pp["ADO"]["PLUS"] = os.path.join(lib, "stata/ado_ext/plus")
    pp["ADO"]["LOCAL"] = os.path.join(lib, "stata")


    # Convert the directories into Waf nodes.
    for key, val in pp.items():
        if not key == "ADO":
            pp[key] = ctx.path.make_node(val)
        else:
            for adokey, adoval in val.items():
                pp[key][adokey] = ctx.path.make_node(adoval)
    return pp


def path_to(ctx, pp_key, *args):
    """Return the relative path to os.path.join(*args*) in the directory
    PROJECT_PATHS[pp_key] as seen from ctx.path (i.e. the directory of the
    current wscript).

    Use this to get the relative path---as needed by Waf---to a file in one
    of the directory trees defined in the PROJECT_PATHS dictionary above.

    We always pretend everything is in the source directory tree, Waf takes
    care of the correct placing of targets and sources.

    """

    # Implementation detail:
    #   We find the path to the directory where the file lives, so that
    #   we do not accidentally declare a node that does not exist.
    dir_path_in_tree = os.path.join(".", *args[:-1])
    # Find/declare the directory node. Use an alias to shorten the line.
    pp_key_fod = ctx.env.PROJECT_PATHS[pp_key].find_or_declare
    dir_node = pp_key_fod(dir_path_in_tree).get_src()
    # Get the relative path to the directory.
    path_to_dir = dir_node.path_from(ctx.path)
    # Return the relative path to the file.
    return os.path.join(path_to_dir, args[-1])


def configure(ctx):
    ctx.env.PYTHONPATH = os.getcwd()
    # Disable on a machine where security risks could arise
    ctx.env.PDFLATEXFLAGS = "-shell-escape"
    ctx.load("run_py_script")
    ctx.load("run_do_script")
    ctx.load("run_r_script")
    ctx.load("sphinx_build")
    ctx.load("write_project_headers")
    # ctx.find_program("dot")
    ctx.load("tex")



def build(ctx):
    ctx.env.PROJECT_PATHS = set_project_paths(ctx)
    ctx.path_to = path_to
    # Generate header file(s) with project paths in "bld" directory
    ctx(features="write_project_paths", target="project_paths.py")
    ctx(features="write_project_paths", target="project_paths.do")
    ctx(features="write_project_paths", target="project_paths.r")
    ctx.add_group()
    ctx.recurse("src")

Note that, while I am using biblatex with backend biber on Windows, I am quiet certain, that that is not the issue. Error still pops up when I comment out the bibliography stuff.

I also ran distclean before running waf again. Error still appears.

Default Stata path on Mac

run_do_script seems to expect the Stata executable to be on the PATH, which on the Mac seems to me to be unnecessarily complicated. Can't it simply look for the executable in the one place it's likely to be, namely at /Applications/Stata/StataSE.app/Contents/MacOS/ for StataSE (I assume the whole thing looks similar for the other Stata versions)?