copier-org/copier

Library and command-line utility for rendering projects templates.

copier's Issues

Use some keys from copier.yml as default values for `exclude`, `include`, etc.

The values should be the overwriteable by those passed as arguments to copier.copy(), because they are conveniences, not rules.

Minor Bug: Version in tests.helpers

Shouldn't that be 2.4.2 by now? If so maybe dynamizing by sourcing the current version from a single source of truth would seem appropriate.

https://github.com/jpscaletti/copier/blob/302dde9b17a8198e08fe8c8cf034ed226112f752/tests/helpers.py#L17

Enhancement - Simplify creation of configuration object

This issue relates solely to improving the architecture it does not address any bug or suggests any new functionality.

Creating the configuration object

copier needs to take in account options coming from:

1. defaults, Like DEFAULT_EXCLUDE and DEFAULT_INCLUDE
2. files - the copier.yml
3. user input - CLI args, prompt

It also needs do some basic validation and ensure default values for certain options like, extra_paths defaulting to [] instead of None.

The entirety of the configuration options I herein call the configuration object, albeit this is in the current implementation is not strictly correct as we deal with independent variable not brought under the umbrella of a single dict yet.

The problem

The current approach to creating the configuration object as used in main.copy_local does not strike me as DRY or pythonic.

Consider for example that we duplicated the same command sequence four times:
https://github.com/jpscaletti/copier/blob/18b2c70c3d582a9d3fc241dec7df34c45dcdd1b3/copier/main.py#L203-L216

Also, default values are not defined in a single place:
https://github.com/jpscaletti/copier/blob/18b2c70c3d582a9d3fc241dec7df34c45dcdd1b3/copier/main.py#L132-L133

A cleaner way

While working on SublimeLinter 4 I learned that the problem of creating a single configuration object from different precursor objects is better solved by sequential merger of dicts.

Such a pattern not only allows to define default values in a single place but also easily sets the order of precedence. In this hierarchy of precedence, the default options would reside at the lowest tier, being overridden by options sourced from a copier.yml and those provided at the command line.

conf_obj = {**default_opts, **yaml_opts, **cli_opts}

As an additional advantage, this approach would also enable a much more straightforward validation of the args.

Related: #45

Are skip and force mutually exclusive?

In my understanding, the skip and force options represent two mutually exclusive resolution strategies.

If so we should reflect this fact in:

• the cli arg parsing (argparse provides ways to specify mutual exclusivity)
• in the Flags config object (pydantics validators are very easy to use)
• the docs of course

EDIT:
We should validate in a single place. The best place is the config submodule. It will not make a difference to the user if error messages are properly generated.

Enhancement - Add requirements.txt

Having a requirements.txt or possibly even a requirements-dev.txt would help when developing copier, by allowing easier installation of dependencies. This is of particular importance as we will not include modern package files such as pyproject.toml or Pipfile (See: #40).

• How come we do not have those files included?
• Am I doing something wrong here?

(Side info: I am using pipenv for development as I have not been able to get my dev setup running with poetry.)

Enhancement - Consider donation option

With the most recent changes copier has become an incredibly valuable tool for myself and most certainly many other users.

I feel such a project deserves support.

If you like add a donation option, possibly even via adding a FUNDING.yml (allowing to tip directly via GitHub).

Enhancement: Allow running pytest

I'd like to run tests with pytest (vs mere tox).
Unfortunately, pytest complaints that it can not find the copier module.

Removing
https://github.com/jpscaletti/copier/blob/bfaefdb1e8d2fbf1270497af1641d52f710a7d52/copier/__init__.py#L5

allows running tests by calling:

> pytest

Maybe I am calling pytest the wrong way or this is indeed an issue.

Bug: Does not copy empty folders

I have not been able to get empty folders getting copied to a generated project.
Once the folder is endowed with just a single file it is respected by the copying process.

This behavior I can reliably reproduce using the latest version of copier 2.1.0.

Template file with filename also templated?

Is there any way to have a filename renamed based on the data? For instance, directories can be templated using syntax like:

src/{{project_name}}

But, is there anyway to get a file also renamed?

src/{{project_name}}/{{project_name}}.py.tmpl

I didn't see this supported in the cursory src code browsing I did, but maybe there is another way.

Thanks for providing this; I found it much simpler and more elegant than cookiecutter!

launch merge tool to solve conflicts

Excellent work!

It would be good if the script has the possibility of asking the user whether it wants to solve the conflicts manually.

Currently it asks:

    conflict  project/docker/docker-compose.yml
  Overwrite /home/yucer/src/tests/project/docker/docker-compose.yml? (y/n) [y]

This would be nice:

    conflict  project/docker/docker-compose.yml
  Overwrite /home/yucer/src/tests/project/docker/docker-compose.yml? (y/n) [y] n
  Solve conflict /home/yucer/src/tests/project/docker/docker-compose.yml? (y/n) [y] y

and then launch a merge-tool.

If you don't have time, I can try to do it. Just give me a clue of where I have to look first.

Question on `voodoo.json`

That file provides variables whose, values are prompted for.
But how can I just declare variables with already provided values?
In that case prompting does not make any sense.

Refurbish README.md

Ideas:

• gif for showing copy process
• "Why not cookiecutter?"
• advertise type hints
• advertise high test coverage
• credit jpscaletti for his awesome original work

Advertise Type Hints

Our code base is completely tape annotated.
Many python devs may consider this a good sign of code quality.

We should advertise this fact, including a sentence in our README.md.

Sharing identical files between templates

I share many identical files between different template folders.
I'd really like to keep my templates DRYer and include those files by reference only.

@jpscaletti
Do you get an idea of how we could achieve this?

Some ideas:

• We allow for a multi-level inheritance whereby a template folder inherits templates from a parent template.
• We allow specifying folder whose contents or a list of file paths to be copied over to the destination folder.

Relates to: #58

Enhancement - Support plugin architecture

Eventually, copier might support extension via plugins.

Such plugins would alter existing functionality or provide new functionality altogether.
Possibly avenues for new plugins are:

• Jinja2 filters, tags and functions
• ...

Plugins might be installed via pip via the common syntax for installing extensions:

pip install copier[plugin_name]

By supporting plugins copier will become additive in nature while still maintaining a lean core.

Actually I am negative about going down this road. Likely copier is good and (almost) complete in its current incarnation.
Still, I have opened the issue to gain clarity/ stimulate discussion about what copier is about to become or remain.

Bug: copier.json not excluded

I always end up with a dest folder containing my copier.json file, the reason seems it not being added of the default exclusion list. Is this behaviour intended?

https://github.com/jpscaletti/copier/blob/8e5948db6a9475ba66d790d87fce073746426242/copier/main.py#L28-L38

Related: #30

Reminder: Catch TypeError on empty copier.yml

An invalid config file raises ValueError, while the latter raises a TypeError.
The latter of which we currently do not catch to handle them the same way we with ValueError...

A test case with an empty copier.yml should also be included...

Example skeleton from README is missing

Quote from README

$  voodoo [email protected]:lucuma/voodoo-flask.git myapp

The abovementioned repo does not exist. Also, there are actually quite few of them on GitHub. I managed to find only two — one embedded into Clay, and one in virantha/ voodoo_templates.

Anyway, thanks for this piece of code! Refreshingly simple after Pyramid scaffolds and Paste templates

Repository transfer name collision with fork.

@jpscaletti

I have renamed my own copier fork to pykong:copier_fork in an attempt to circumvent the name collision whilst keep the open PR #79 alive.

Please have another try at transfering.

Final newline stripped from templated files

If a template file ends with a new line voodoo will strip that new line when outputting the rendered template. This causes flake8 to complain about W292 no newline at end of file for all templated python files. New lines at the end of a template file should be preserved by voodoo.

Bug - Root level __init__.py

I have not been able to either user pyannotate or MonkeyType to add type hints to copier's code.

In the issue I opened on MonkeyType it was suggested this being due to the __init__.py
present at the root level of copier:
Instagram/MonkeyType#139

Similar problems exist for pyannotate:
dropbox/pyannotate#61

I do not possess the experience with developing python packages to judge whether the root level __init__.py should be there or not. However, it should be considered in light of the problems I have experienced.

Enhancement: Improve output for tasks to be run

Instead of one single line:

   _tasks? [['cd [[ project_name ]] && git init', 'cd [[ project_name ]]', 'pipenv install']] 

List commands and already substitute variables:

   Tasks to be run:
   - cd MyProject && git init
   - cd MyProject && pipenv install

As a bonus: Some progress indication while running those tasks would be very useful.
Like:

Runnings task (2/5) --- Done

Enhancement: Better support for template inheritance

One of Jinja2's prime features is the support of template inheritance.
http://jinja.pocoo.org/docs/2.10/templates/#template-inheritance

The problem I see with Voodoo's implementation is that parent files will also get copied over to the project folder.

Wouldn't it be better to reserve the .tmpl suffix only for those parent files?
Those could still be used as base templates, but will be omitted from copying over.
This inverts the selection process compared to the status quo.

Voodoo could then try to render all non-suffixed text files present under the template path.
Any non-text file (e.g. image files) would simply be copied. (Likely this could be achieved by catching exceptions that arise when trying to open/render non-text files. Hence not requiring an extensive list of file extensions for discerning.)

Enhancement: Official integration of pipenv

It is reasonable to assume that the one thing most users creating a new python project via copier may want to do is to initialize a virtual environment and install packages.

Hence an official way to support pipenv would be very welcomed.

Enhancement: Allow configuring template folder

How? Configuration via settings file and/or environment variable.
Why? Simplifying command to: copier init (Will then prompt for which template to be used to create the new project from a list of folders found in the template folder.)

Example: Passing function to data

@jpscaletti

The readme states that you can even pass a function as extra context via the data argument.

Use the data argument to pass whatever extra context you want to be available in the templates. The arguments can be any valid Python value, even a function.

Can you provide an example of how to do so? I seem not to be able to make it work, nor have I found an example in the docs or tests.

Further, can we call such a function in our Jinja2 templates then? (That would be extremely useful!)

Transfer: How to link PyPI?

@jpscaletti

How do we fare about the uplink of this repo to PyPI?
After all, people will want to be able to install copier via pip I presume.

Do you have got any idea/instructions on that part of the transfer?

Enhancement: Add pyproject.toml

CONTRIBUTING.md suggests poetry for development.
How then does it come that no pyprojct.toml is included?
Shouldn't we add one? Should we maybe replace setup.py entirely?

Is this project still alive?

@jpscaletti

Could you please give me a life sign whether you are still interested in developing copier any further? The issues and PRs are turning stale. I know very well that developing open source in your free time can be quite a time demanding but copier I believe fills a void for me and other users, being a simple more usable alternative to cookiecutter.

In case you do no longer want to actively maintain copier consider sharing push access, transferring ownership or encouraging friendly fork. In any case state the status of the project, so users can make an informed decision on using copier in accordance with good open-source practices.

Enhancement: Cleanup on failure

To prevent defunct remnants of a failed attempt to create a project an already created folder should be deleted when the creation process fails (e.g. task runner experiences error).

Copy updates as 1st-class citizens

The scaffolding landscape in Python currently is pretty bad, thanks for bringing to life this nice project.

Handling updates is pretty bad still. Imagine:

1. You have a scaffolding.
2. You copy it and start a new project.
3. Your new project evolves in a separate git repo.
4. You update your scaffolding.
5. You want to apply your scaffolding to your copied project again, but don't want the default answers; you want the ones you used last time.
6. You don't want the updates to touch the files that you already touched, unless they've been touched also in those lines in the source scaffolding.

cruft handles this pretty well, as it stores all answers in a json file, which also happens to include the git commit of the source when it was copied, and this allows it to extract the diff since that commit to the latest and apply only that diff to the destination.

How should copier copy (lol) this behavior?

First, when copying:

1. When a scaffolding is copied, store all answers in a file. It could be something like .copied.yml.
2. Store in .copied.yml also the source of the copy (e.g. _copied_from: gh:my/scaffolding).
3. When the scaffolding was a git repo, store also the copied commit. It could be a hidden key in .copied.yml (e.g.: _copied_commit: o3ho4ho5hoho2hoh2o).

Then, support running copier without arguments, which:

1. Fails if .copied.yml is absent.
2. Fails if _copied_from is absent in that file.
3. If _copied_commit is absent, it should be the equivalent of copier $_copied_from ..
4. If _copied_commit is present, it should be the equivalent of copied --from-commit $_copied_commit --to-commit HEAD $_copied_from ..

As you probably have guessed, we need to add new flags --from-commit and --to-commit (which defaults to HEAD), which:

1. Fails if source is not a git repo.
2. If present, will generate a patch from --from-commit to --to-commit.
3. It will only apply that diff.
4. Does nothing if both flags are equal.

Why all of this?

1. We need to be able to just execute copier in a pre-copied scaffolding and let it get only the diff since it was copied last time.
2. We need to be able to ensure a project is updated with its scaffolding, i.e. to raise an error in CI when that's not the case.

_exclude, _include and other special keys in the config file `copier.[yml|toml|json]` are being ignored

The get_user_data just filter them out!

Transfer - Relationship with mastermold

@jpscaletti

Admittedly, I do not understand what mastermold is.
How is it related to copier?
What is the mm.py file for?

Enhancement - Let invalid config_data not pass silently.

load_config_data when encountering an invalid file config will silently return an empty dictionary instead of raising an exception.

IMHO this is in stark violation of The Zen of Python which proclaims:

Errors should never pass silently.
Unless explicitly silenced.

@jpscaletti
I do not know whether this is by design. In case it is not we might make load_config_data to raise.

Bug - Missing dependency in mm.py

It appears pydantic is missing in the mastermold file.
Am I correct?

https://github.com/jpscaletti/copier/blob/69363a5944111e22733ea536fdab4ae825711d9c/mm.py#L24-L28

Consider adding task runner

Thank You

@jpscaletti First of all thank you very much for putting up this awesome package.
I was looking for a simpler to use alternative to cookiecutter and believe I have found it in Voodoo.

Add a TaskRunner

I believe there is a low hanging fruit to give Voodoo superpowers:
You may want to add a basic command execution capability to this already awesome package.
This would allow to endow a newly minted project to be supplemented with additional properties.
Such additional properties could for example be a virtual environment associated with the particular projects, the initiation of a git repository or fetching certain files from the internet.

Implementation:

Keep it simple.
User puts a tasks.json file into the template dir. That file contains an array of shell command.
Which are successively ran against python's subprocess. In its simplest form that file could look like:

[
    "git init .",
    "pipenv shell"
]

This means that in the newly created project folder a git repository and a python virtual environment will be created, once the main templating process has finished.

Additional nodes

• exit codes should be checked and non-zero ones leading to termination of the command chain
• piping the stdout of a ran command into the stdin of its successor might also be added.
• This Sublime Text plugin essentially already implements all the code needed:
  https://github.com/bit101/ProjectMaker/blob/b2e82189226cbf13383aedf6e4177307af35a50f/projectmaker.py#L296 (just adapt function to suit a non Sublime Text context)
• To be more clear:

def exec_tasks(self):
    task_file = os.path.join(self.template_path, "tasks.json")
    if os.path.exists(task_file):
        with open(task_file, "r") as f:
            tasks = json.loads(f)
            for t in tasks:
                try:
                    subprocess.run(t, shell=True, check=True)
                except subprocess.CalledProcessError as e:
                    # handling non-zero exit codes
                    print(e) 
                    break

Question: Configuring copy() arguments in copier.toml?

Can we specify any argument as required by the copier() function via the copier.toml file?
Except maybe for src_path and dst_path.

From the code, it seems only _include and _exclude are treated as arguments:
https://github.com/jpscaletti/copier/blob/3edbaabe70a294b1bab838d13c04de297573ff4e/copier/user_data.py#L86

Still, the readme mentions specifying the data argument in the copier.toml.

Enhancement - Sharing identical portions of copier.yaml

My copier.yaml files do share a lot of content.
e.g. I have specified the same basic sequence of tasks to be run.

I would like to share some content between different copier.yaml files.

@jpscaletti
Do you get an idea of how we could achieve this?

One idea:

• We may somehow leverage Jinja2's inheritance mechanism like we already do for other templates.

Relates to: #58

Suggestion - Submit to awesome list

TL:DR copier is awesome!
I see it offering the most functionality that cookiecutter possess without the bloat. Hence we should consider submitting it to an awesome list to make it known to more users.

It could be one of those:
https://github.com/sindresorhus/awesome

Most likely the python list:
https://github.com/vinta/awesome-python

Add CONTRIBUTORS.md

To honor the diligent contributors of this project it is a tradition to list their names in a special file.

A special mention would, of course, go to @jpscaletti as the original author of this project without his diligence and vision copier would not be a thing right now.
@jpscaletti if you do not object I am going to have your name listed there.

Enhancement: Dir name = project name

I want to run all tasks in the freshly created dir.
Hence, is there a way to specify the working dir for the tasks to run in?

# Commands to be executed after the copy
_tasks:
    - cd [[ project_name ]]
    - git init

A simple cd command does not do the trick.
(Platform: Linux Mint)

Test Coverage Badge

We may want to include a badge displaying our code coverage.
As we have fairly good coverage we should advertise this fact as another sign of good repo quality.

I have no experience with GitHub badges and hence can not recommend any service.
One that I found through googling though is codecov.io who provides badges like that:

Add tests for the `prompt` and `prompt_bool` helpers

Dealing with multiple config files.

Having more than one config file in a source directory is an erroneous condition.
Maybe it is bikeshedding. Or maybe a corner case we may want to address.

Possible solutions:

1. Stick to the current precedence mechanism ignoring those files with lower precedence.
2. Resort to only supporting one file type (likely yaml).
3. Throw an error on encountering this situation.

@jpscaletti
What do you think?

Enhancement - Configuration via Environment Variables

Should we allow copier to source configuration from environment variables?
One advantage would be to make the same set of values available for all our templates.
Like _extra_paths would likely point to the same folder always, independent of the template.

The configuration via env vars would take the lowest precedence so that we get a hierarchy of precedence of:

user input > copier.yml > env var

Update Readme: Configuration file format

@jpscaletti

Three config file formats (TOML, YAML and JSON) find mention in the readme and the code base. As of the most recent version copier.yaml does not seem to work for me any longer, forcing me to switch to json.

Could you therefore maybe clarify which configuration file formats we can actually use?
Does the README.md require update in this regard?

Syntax error in Makefile

I'm not entirely sure whether this intentional or not, but "voodoo" is spelt "vodoo".

I haven't got the time right this second to see if the other code puts stuff in "vodoo" or "voodoo", or maybe I'm just completely wrong here, if so I do apologize. In a bit of a rush.

Bug: Specifying _extra_paths in copier.yml does not work

Neither specifying _extra_paths nor _extra-paths will result in those paths being respected.

Example:

_extra_paths:
- ~/Projects/.templates/.templates/

From debugging I found that the extra_paths variable never showes up in config_data:
https://github.com/jpscaletti/copier/blob/bfaefdb1e8d2fbf1270497af1641d52f710a7d52/copier/main.py#L183
copier version: 2.4.2

Enhancement - Type annotate code base

@jpscaletti What do you think of hardening copier with endowing all functions with type annotations?

Like the central copy function for example:

def copy(
    src_path: str,
    dst_path: str,
    data: Optional[dict] = None,
    *,
    exclude: Optional[List[str]] = None,
    include: Optional[List[str]] = None,
    tasks: Optional[List[str]] = None,
    envops: Optional[dict] = None,
    extra_paths: Optional[List[str]] = None,
    pretend: bool = False,
    force: bool = False,
    skip: bool = False,
    quiet: bool = False
) -> None:

Enhancement - Parent template folders

I believe it would be very helpful to every user of copier to allow for DRYer templates. Common redundancies across templates which can not be addressed via the current functionality of copier are:

• sets of files (e.g. all my web templates feature the same set of static files)
• contents of copier.yaml (e.g. I always run the same seven tasks, regardless of the project)

Potential solutions:

• allow to inherit an entire folder: all folders of a given path will be copied over to the new rendered folder
• allow inheritance of copier.yaml files where a parent template can be included

EDIT:

I believe this issue is not clear enough. In part because I wasn't sure what I was looking for.
I am working on making the idea clearer.