Allow finer control over execution of tasks

[pawamoy]

I would like to discuss about a way to improve the usability of tasks. In short, I think it would be great to have control over when (and if) a task is executed.

Related to #196 and #238.

[yajo]

Yes, I think that it should work similar to migrations: pre and post. We could even have pre-copy, pre-update, post-copy, post-update. That should be enough, right?

[yajo]

@baurmatt had a good idea at #238 (comment): add $OPERATION={copy,update} to the environment. With that, you can simply include any needed if in your task code and perform it or not, without needing to overcomplicate the copier.yml file. Also it'd be fully backwards compatible.

IMHO the path to follow.

[pawamoy]

pre-copy, pre-update, post-copy, post-update

I'd love to have those indeed. Would this be accomplished with both STAGE and OPERATION env vars then? The four combinations would be mutually exclusive, meaning each one is only executed when the conditions are met.

STAGE / OPERATION	copy	update
before
after

[yajo]

Yes, well... definitely this couldn't be backwards-compatible after all.

[pawamoy]

I'm OK with implementing this incrementally if you'd like to make it backwards-compatible first, then add functionality like more stages.

[pawamoy]

Hey @yajo, I'd like to start tackling this. Basically, I want to add an OPERATION environment variable when running tasks, as was discussed previously.

Do you already have an idea of how this could be implemented?
I was wondering if it would make sense to set the environment variable with os.environ["OPERATION"] = "update" to later on pass it in the extra_env argument:

run_tasks(
        conf, render, [{"task": t, "extra_env": {"STAGE": "task", "OPERATION": os.environ["OPERATION"]}} for t in conf.tasks]
    )

...but I'm not sure because it's like using a global variable 😅
Also, maybe we would not even need to pass it in the extra_env argument since we set it with os.environ?

What do you think?

[yajo]

What if we apply a logic like the one explained at #326 but for tasks too?

[pawamoy]

Isn't it exactly like what is discussed here 🙂 ? Passing everything needed (env vars) to tasks when executing them?

[yajo]

Lol yes...

I've got pretty clear now that copier 6 will break all things. It's gonna be fun 😆

[KristianTanggaard]

Is there a preferred way of handling things in terms running commands only on creation of a new project vs. updating them now we're up on copier 7?

For example: Wanting to just run git init and doing an initial commit on creation, but not on any updates afterwards?

[yajo]

FWIW if you git init on an existing repo, it does nothing.

Other than that, right now there's no official way to distinguish it. Thus, this issue exists. 😁

You have to use your imagination. One workaround for you would be to check if .git exists. If it doesn't you know you're copying. If it does, you're updating.

[KristianTanggaard]

That is true. I just gave it as an example. Considering I might then want to do an initial commit, and add a second or third branch, it seems like something that might get a bit iffy.

I will probably do it with a python script that checks the state before running anything.

[yajo]

You may consider also pushing a PR here to enhance this feature and close the issue. 😃

[lyz-code]

Hi all, first thank you for making and maintaining this awesome project <3.

I'm having this same question:

Is there a preferred way of handling things in terms running commands only on creation of a new project vs. updating them now we're up on copier 7?

For example: Wanting to just run git init and doing an initial commit on creation, but not on any updates afterwards?

And I've thought of creating a tag 0.0.0 on the first valid commit of the template and then have an initial migration script for version 0.1.0. That way instead of doing copier copy src dest you can do:

copier copy -r 0.0.0 src dest
copier update

It will run over all the migrations steps you make in the future. A way to tackle this is to eventually release a 1.0.0 and move the 0.1.0 migration script to 1.1.0 using copier copy -r 1.0.0 src dest.

As I'm new to the tool I was wondering if you think this is a good idea, or if it can backfire

Thanks!

PS: Our paths cross again @pawamoy :)

[pawamoy]

Hey @lyz-code, happy to cross paths again 😄
I think this is a good idea, however this can indeed backfire, for this very simple reason: all the versions of your template will not be backward compatible with 0.0.0. If they are now, they probably won't be in the future. This might be because of the template itself, or because of extensions it uses, or because of the version of Copier it required at the time of each version release. So I think this is doomed to fail at some point 😕 This will be OK for existing projects, but not when trying to generate new ones.

[lyz-code]

Thanks @pawamoy for your quick answer. The next "solution" I can think of is to embed inside the generated project's Makefile an init target that runs the init script. The user will then need to:

copier copy src dest
cd dest
make init

Not ideal but it can be a workaround until we have the pre-copy tasks

[bswck]

What is the status of this issue? Any news so far?

[bswck]

Also, I would honestly call it TASK_STAGE or EVENT, not OPERATION.

[bswck]

For those who need a workaround, you might look into my template—https://github.com/bswck/skeleton/blob/master/handle-task-event.sh.

I made a template shell script that is rendered and run every time copier operates. It's included in copier.yml with {% include handle-task-event.sh %} as a single task.

It has some caveats and requires Redis to communicate between task stages aka "operations", but maybe that works for you at the moment.

Not sure if my shell script interacts well with existing projects, but I will be fixing it to incorporate the template into all of my Python projects.

[yajo]

Thanks for providing that workaround. For the time being, the only progress is that now we require the --trust flag to execute any tasks or migrations.

[hahuang65]

Newcomer here, wondering if this suggestion is naive or preposterous...

But in the interest of preserving backwards-compatibility... Could we simply add the new operations/stages as new categories?

We could add a pre-run, pre-update, post-run, post-update... and then simply alias tasks to post-run and post-update?

This would maintain backwards compatibility and have the benefit of not having overly complex logic branches in the tasks yaml, given many combinations of stage and operation.

It would allow the yaml document to flow nicely as well, allowing users to order the headers as they want.

For what it's worth, I'd be extremely eager to work on this. It's the only thing preventing me from migrating to copier. I need a pre-check that will prevent the project from being generated if the user doesn't have a certain tool installed.

[bswck]

My favorite approach would be a single declarative Python script that would allow to register callbacks for each event; with overall appearance similar to noxfile.py.
Something more or less like:

# hooks.py
from copier import after_copy, after_update, session

@after_update
def teardown_update() -> None:
    session.run("somecmd")

@after_copy
def create_repo() -> None:
    should_create_repo: bool = ...
    if should_create_repo:
        session.run("gh", "repo", "create", ...)

[sisp]

@bswck That would only work for Python scripts, but tasks can use executables written in any language.

[bswck]

@bswck That would only work for Python scripts, but tasks can use executables written in any language.

Why? I just showed an example of invoking the GitHub CLI.

If you're referring to rendering scripts and then running them, session.run could provide facilities for it.

A Python layer for registering callbacks would not only make things easier, but also extend the overall ability to control tasks.

If you are, however, opposing the idea of having to define a Python script, I would argue then that the potential benefit justifies the trade-off.

[pawamoy]

@bswck, @sisp meant that providing this feature through Python decorators like after_copy would make the feature, well, Python-only. It would require to write a Python script to run a Bash task or run any other executable that is not Python. A language-agnostic implementation would probably use environment variables instead of Python decorators, since you check env vars in any language and act accordingly. Then only, Copier could maybe provide such decorators to facilitate writing tasks in Python 🙂 So your idea is not incompatible with the feature 👍

[yajo]

FWIW I was scared away from yeoman because I didn't want to write js just for a template. In the same fashion I don't want to scare away js devs by requiring writing hooks in Python. The truth is that we can just use env variables and command calls, and that just gives the dev all possibilities. Copier has remained as lang-agnostic as possible and I wouldn't like to loose that.

[pescobar]

in case it's helpful for others, I am testing this workaround which seems to work to only execute the tasks during the first execution of copier to initialize my project (create the required gitlab repos and extra stuff I don't want to repeat when doing copier update)

in copier.yml :

_tasks:
  - bash .project_init.sh

and in .project_init.sh :

#!/bin/bash

if [[ $PWD == /tmp* ]]; then
    echo "We are /tmp/. This is copier update. Exiting"
    exit 0
fi

if [ -d .git/ ]; then
  echo "This project is already initialized. Exiting"
  exit 0
fi

git init
git add .
git commit -a -m 'First commit by copier'

# here more initilization stuff

Script .project_init.sh` is another template in my copier project.

I need to do more testing but so far it seems to work for me

[sevdesk-ryanp]

I just saw the 'good for newbies' label and could potentially set aside some time to work on this. Has there been any progress yet, or would it pretty much be a clean slate to work from

[pawamoy]

@sevdesk-ryanp thanks for offering your help! There is an unfinished PR here: #1511, if you want to take a look (and maybe take over).

[eckelsjd]

Another workaround I have been using to avoid running intialization tasks on copier update uses an extensions.py file to define an is_copier_update variable:

# copier.yml

_jinja_extensions:
  - copier_templates_extensions.TemplateExtensionLoader
  - extensions.py:TemplateDefaultExtension
...

_tasks:
  - command: python setup_github.py  # Or any other "initialization-only" tasks
    when: "{{ not is_copier_update }}"

And then the extension defines is_copier_update:

# extensions.py

from jinja2.ext import Extension
from pathlib import Path

class TemplateDefaultExtension(Extension):
    def __init__(self, environment):
          super().__init__(environment)
          environment.globals["is_copier_update"] = Path(".copier-answers.yml").exists()

The .copier-answers.yml file will only exist in project root before invoking copier if the project was already made with copy, thus it signals when a copier update is happening.

This will require the --trust flag since it uses extensions, but I believe you already would have to using tasks. See extensions.

[woile]

For small tasks I use this:

_tasks:
  - test "${PWD#'/tmp'}" = "$PWD" && git init # on first time only

[KalleDK]

I don't know if this goes here. I was looking for a way to populate some of the defaults by "proding" the system.

Either directly in the questions, or as a task recording variables to be used during the questions.

email:
  type: str
  help: What is your email address?
  default_cmd:
    command: git config user.email

[sisp]

@KalleDK You can create a Jinja extension for this. See, e.g., https://github.com/pawamoy/copier-pdm/blob/adff9b64887d0b4c9ec0b42de1698b34858a511e/extensions.py#L23-L27 & https://github.com/pawamoy/copier-pdm/blob/adff9b64887d0b4c9ec0b42de1698b34858a511e/copier.yml#L11.

[KalleDK]

@KalleDK You can create a Jinja extension for this. See, e.g., https://github.com/pawamoy/copier-pdm/blob/adff9b64887d0b4c9ec0b42de1698b34858a511e/extensions.py#L23-L27 & https://github.com/pawamoy/copier-pdm/blob/adff9b64887d0b4c9ec0b42de1698b34858a511e/copier.yml#L11.

Thank you - works like a charm.. Just needed to inject the extension loader :)

[bhuynhdev]

I don't think this issue should be closed. Seems like #1733 didn't fully resolve this issue.

The good thing is that the good work in #1510 and #1733 paved the way to implement a PR for this more easily.

#1510 gave Tasks the ability to use the _stage variable in its when condition, which is of value "before"/"after".
However it didn't set those stage values for tasks, only for migrations. Additionally, in the main:Worker.run_copy function, _execute_tasks was only called once after _render_template has been called, making tasks currently only a "post-copy" feature.

As a result, a way to resolve this issue would be:

1. make the Template.tasks function includes a stage parameter like in Template.migration_tasks,
2. then in main.py calls _execute_tasks twice, before and after the call the _render_template

I can make a PR for this some time this week.

However this has the implication that Phase.TASKS is kinda no longer valid, because there is no singular "Tasks" phase - there is now pre-copy tasks, and post-copy tasks. Also I am not sure how this would work with the _copier_phase variable