Tasks run from wrong folder on update with 6.0.0a6

[chris-sanders]

I switched from using copier 5.1.0 to the latest 6.0.0a6 to avoid copier deleting a folder that it didn't create. However with the v6 version I'm having an issue with tasks getting executed by invoke during a copier update.

With version 5 the tasks run as expected, but with v6 during an update tasks that call invoke don't run from the destination folder but from the template sub-folder, where it can't find the expected files.

The template can be found at the v6 branch here: https://github.com/charmed-kubernetes/pytest-operator-template/tree/v6

You can run the tox -e integration and you'll see that it fails when it invokes one of the scripts because it can't find the file, which is in the destination folder the task is just not being run in the correct location.

[yajo]

Thanks for testing the alpha and for reporting, I'll add this to the v6 roadmap.

[chris-sanders]

I've been looking at this more and I've now triggered it on the v5 aslo. But I think I see what's actually happening. As part of the update process the template is cloned into an empty directory and then diffed against the actual final directory. During the initial copy into the tmp directory it's running the tasks and I have tasks which expect files in the destination directory which fail.

I'm not sure how to best handle that. I can catch the errors and just exit clean, but I actually wanted to fail if the destination folder doesn't match expectation. I'm applying this template as a delta on top of another folder structure and If the structure doesn't match what I expect I would like it to fail.

[yajo]

There are a couple of ideas that come to my mind.

1. Use invoke's --search-root to specify where to look for it. I use it in one of our templates and it works fine.

2. I actually wanted to fail if the destination folder doesn't match expectation

   Currently Copier only runs tasks after the copy. What you want is probably a better system that runs them before, to see if the project expectations are matched. That progress is being tracked in Allow finer control over execution of tasks #240 and I hope it's ready for v6 final.

[yajo]

You can run the tox -e integration and you'll see that it fails when it invokes one of the scripts because it can't find the file, which is in the destination folder the task is just not being run in the correct location.

Tests fail with something that doesn't seem related to copier:

╰─ pipx run tox -e integration
integration create: /tmp/tmp.srxrxoudz4/pytest-operator-template/.tox/integration
integration installdeps: invoke, copier>=6.0.0a6, pytest, black, pyyaml
integration installed: appdirs==1.4.4,attrs==20.3.0,black==20.8b1,click==7.1.2,colorama==0.4.4,copier==6.0.0a6,iniconfig==1.1.1,invoke==1.5.0,iteration-utilities==0.10.1,Jinja2==2.11.3,MarkupSafe==1.1.1,mypy-extensions==0.4.3,packaging==20.9,pathspec==0.8.1,pluggy==0.13.1,plumbum==1.7.0,prompt-toolkit==3.0.18,py==1.10.0,pydantic==1.8.1,Pygments==2.8.1,pyparsing==2.4.7,pytest==6.2.2,PyYAML==5.4.1,pyyaml-include==1.2.post2,questionary==1.9.0,regex==2021.3.17,toml==0.10.2,typed-ast==1.4.2,typing-extensions==3.7.4.3,wcwidth==0.2.5
integration run-test-pre: PYTHONHASHSEED='2696812564'
integration run-test: commands[0] | pytest -x -v /tmp/tmp.srxrxoudz4/pytest-operator-template/tests/integration/ --provider machine
============================================================================================================ test session starts =============================================================================================================
platform linux -- Python 3.9.2, pytest-6.2.2, py-1.10.0, pluggy-0.13.1 -- /tmp/tmp.srxrxoudz4/pytest-operator-template/.tox/integration/bin/python
cachedir: .tox/integration/.pytest_cache
rootdir: /tmp/tmp.srxrxoudz4/pytest-operator-template, configfile: tox.ini
collected 8 items                                                                                                                                                                                                                            

tests/integration/test_template.py::TestTemplate::test_pytest ERROR                                                                                                                                                                    [ 12%]

=================================================================================================================== ERRORS ===================================================================================================================
_________________________________________________________________________________________________ ERROR at setup of TestTemplate.test_pytest _________________________________________________________________________________________________

session_folder = PosixPath('/tmp/pytest-of-yajo/pytest-0/session0')

    @pytest.fixture(scope="session")
    def charm_dir(session_folder):
        charm_dir = session_folder / "charm-dir"
        tmp_dir = Path("charm-dir")
        tmp_dir.mkdir()
>       subprocess.check_call(
            ["charmcraft", "init", "--author", "Pytest Conftest"], cwd=tmp_dir
        )

tests/integration/conftest.py:55: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
/usr/lib64/python3.9/subprocess.py:368: in check_call
    retcode = call(*popenargs, **kwargs)
/usr/lib64/python3.9/subprocess.py:349: in call
    with Popen(*popenargs, **kwargs) as p:
/usr/lib64/python3.9/subprocess.py:951: in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <Popen: returncode: 255 args: ['charmcraft', 'init', '--author', 'Pytest Con...>, args = ['charmcraft', 'init', '--author', 'Pytest Conftest'], executable = b'charmcraft', preexec_fn = None, close_fds = True, pass_fds = ()
cwd = PosixPath('charm-dir'), env = None, startupinfo = None, creationflags = 0, shell = False, p2cread = -1, p2cwrite = -1, c2pread = -1, c2pwrite = -1, errread = -1, errwrite = -1, restore_signals = True, gid = None, gids = None
uid = None, umask = -1, start_new_session = False

    def _execute_child(self, args, executable, preexec_fn, close_fds,
                       pass_fds, cwd, env,
                       startupinfo, creationflags, shell,
                       p2cread, p2cwrite,
                       c2pread, c2pwrite,
                       errread, errwrite,
                       restore_signals,
                       gid, gids, uid, umask,
                       start_new_session):
        """Execute program (POSIX version)"""
    
        if isinstance(args, (str, bytes)):
            args = [args]
        elif isinstance(args, os.PathLike):
            if shell:
                raise TypeError('path-like args is not allowed when '
                                'shell is true')
            args = [args]
        else:
            args = list(args)
    
        if shell:
            # On Android the default shell is at '/system/bin/sh'.
            unix_shell = ('/system/bin/sh' if
                      hasattr(sys, 'getandroidapilevel') else '/bin/sh')
            args = [unix_shell, "-c"] + args
            if executable:
                args[0] = executable
    
        if executable is None:
            executable = args[0]
    
        sys.audit("subprocess.Popen", executable, args, cwd, env)
    
        if (_USE_POSIX_SPAWN
                and os.path.dirname(executable)
                and preexec_fn is None
                and not close_fds
                and not pass_fds
                and cwd is None
                and (p2cread == -1 or p2cread > 2)
                and (c2pwrite == -1 or c2pwrite > 2)
                and (errwrite == -1 or errwrite > 2)
                and not start_new_session
                and gid is None
                and gids is None
                and uid is None
                and umask < 0):
            self._posix_spawn(args, executable, env, restore_signals,
                              p2cread, p2cwrite,
                              c2pread, c2pwrite,
                              errread, errwrite)
            return
    
        orig_executable = executable
    
        # For transferring possible exec failure from child to parent.
        # Data format: "exception name:hex errno:description"
        # Pickle is not used; it is complex and involves memory allocation.
        errpipe_read, errpipe_write = os.pipe()
        # errpipe_write must not be in the standard io 0, 1, or 2 fd range.
        low_fds_to_close = []
        while errpipe_write < 3:
            low_fds_to_close.append(errpipe_write)
            errpipe_write = os.dup(errpipe_write)
        for low_fd in low_fds_to_close:
            os.close(low_fd)
        try:
            try:
                # We must avoid complex work that could involve
                # malloc or free in the child process to avoid
                # potential deadlocks, thus we do all this here.
                # and pass it to fork_exec()
    
                if env is not None:
                    env_list = []
                    for k, v in env.items():
                        k = os.fsencode(k)
                        if b'=' in k:
                            raise ValueError("illegal environment variable name")
                        env_list.append(k + b'=' + os.fsencode(v))
                else:
                    env_list = None  # Use execv instead of execve.
                executable = os.fsencode(executable)
                if os.path.dirname(executable):
                    executable_list = (executable,)
                else:
                    # This matches the behavior of os._execvpe().
                    executable_list = tuple(
                        os.path.join(os.fsencode(dir), executable)
                        for dir in os.get_exec_path(env))
                fds_to_keep = set(pass_fds)
                fds_to_keep.add(errpipe_write)
                self.pid = _posixsubprocess.fork_exec(
                        args, executable_list,
                        close_fds, tuple(sorted(map(int, fds_to_keep))),
                        cwd, env_list,
                        p2cread, p2cwrite, c2pread, c2pwrite,
                        errread, errwrite,
                        errpipe_read, errpipe_write,
                        restore_signals, start_new_session,
                        gid, gids, uid, umask,
                        preexec_fn)
                self._child_created = True
            finally:
                # be sure the FD is closed no matter what
                os.close(errpipe_write)
    
            self._close_pipe_fds(p2cread, p2cwrite,
                                 c2pread, c2pwrite,
                                 errread, errwrite)
    
            # Wait for exec to fail or succeed; possibly raising an
            # exception (limited in size)
            errpipe_data = bytearray()
            while True:
                part = os.read(errpipe_read, 50000)
                errpipe_data += part
                if not part or len(errpipe_data) > 50000:
                    break
        finally:
            # be sure the FD is closed no matter what
            os.close(errpipe_read)
    
        if errpipe_data:
            try:
                pid, sts = os.waitpid(self.pid, 0)
                if pid == self.pid:
                    self._handle_exitstatus(sts)
                else:
                    self.returncode = sys.maxsize
            except ChildProcessError:
                pass
    
            try:
                exception_name, hex_errno, err_msg = (
                        errpipe_data.split(b':', 2))
                # The encoding here should match the encoding
                # written in by the subprocess implementations
                # like _posixsubprocess
                err_msg = err_msg.decode()
            except ValueError:
                exception_name = b'SubprocessError'
                hex_errno = b'0'
                err_msg = 'Bad exception data from child: {!r}'.format(
                              bytes(errpipe_data))
            child_exception_type = getattr(
                    builtins, exception_name.decode('ascii'),
                    SubprocessError)
            if issubclass(child_exception_type, OSError) and hex_errno:
                errno_num = int(hex_errno, 16)
                child_exec_never_called = (err_msg == "noexec")
                if child_exec_never_called:
                    err_msg = ""
                    # The error must be from chdir(cwd).
                    err_filename = cwd
                else:
                    err_filename = orig_executable
                if errno_num != 0:
                    err_msg = os.strerror(errno_num)
>               raise child_exception_type(errno_num, err_msg, err_filename)
E               FileNotFoundError: [Errno 2] No such file or directory: 'charmcraft'

/usr/lib64/python3.9/subprocess.py:1823: FileNotFoundError
========================================================================================================== short test summary info ===========================================================================================================
ERROR tests/integration/test_template.py::TestTemplate::test_pytest - FileNotFoundError: [Errno 2] No such file or directory: 'charmcraft'
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
============================================================================================================== 1 error in 0.42s ==============================================================================================================
ERROR: InvocationError for command /tmp/tmp.srxrxoudz4/pytest-operator-template/.tox/integration/bin/pytest -x -v tests/integration --provider machine (exited with code 1)
__________________________________________________________________________________________________________________ summary ___________________________________________________________________________________________________________________
ERROR:   integration: commands failed

So I ran the test manually:

╰─ copier --version
copier 6.0.0a5.post28+386e50e

╰─ cd (mktemp -d)

╰─ git clone https://github.com/charmed-kubernetes/pytest-operator-template -b v6 
Clonando en 'pytest-operator-template'...
remote: Enumerating objects: 193, done.
remote: Counting objects: 100% (193/193), done.
remote: Compressing objects: 100% (98/98), done.
remote: Total 193 (delta 84), reused 165 (delta 62), pack-reused 0
Recibiendo objetos: 100% (193/193), 29.41 KiB | 792.00 KiB/s, listo.
Resolviendo deltas: 100% (84/84), listo.

╰─ copier copy ./pytest-operator-template/ dst

No git tags found in template; using HEAD as ref
🎤 The class name for the charm, must be a valid python class name. This is in src/charm.py for existing charms.
class_name? Format: str MyClass
🎤 charm_type? Format: str container
    create  .
    create  tox.ini
    create  tests
    create  tests/unit
    create  tests/unit/test_charm.py.tmpl
    create  tests/unit/empty
    create  tests/integration
    create  tests/integration/test_charm.py
    create  tests/data
    create  tests/data/empty
    create  tasks.py
    create  src
    create  src/charm.py.tmpl
    create  [[_copier_conf.answers_file]].tmpl
    create  .github
    create  .github/workflows
    create  .github/workflows/tests.yaml.tmpl

 > Running task 1 of 3: invoke check-yaml
Traceback (most recent call last):
  File "/usr/bin/invoke", line 33, in <module>
    sys.exit(load_entry_point('invoke==1.4.1', 'console_scripts', 'invoke')())
  File "/usr/lib/python3.9/site-packages/invoke/program.py", line 384, in run
    self.execute()
  File "/usr/lib/python3.9/site-packages/invoke/program.py", line 566, in execute
    executor.execute(*self.tasks)
  File "/usr/lib/python3.9/site-packages/invoke/executor.py", line 129, in execute
    result = call.task(*args, **call.kwargs)
  File "/usr/lib/python3.9/site-packages/invoke/tasks.py", line 127, in __call__
    result = self.body(*args, **kwargs)
  File "/tmp/tmp.srxrxoudz4/dst/tasks.py", line 40, in check_yaml
    config = yaml.safe_load(config_file.read_text())
  File "/usr/lib64/python3.9/pathlib.py", line 1255, in read_text
    with self.open(mode='r', encoding=encoding, errors=errors) as f:
  File "/usr/lib64/python3.9/pathlib.py", line 1241, in open
    return io.open(self, mode, buffering, encoding, errors, newline,
  File "/usr/lib64/python3.9/pathlib.py", line 1109, in _opener
    return self._accessor.open(self, flags, mode)
FileNotFoundError: [Errno 2] No such file or directory: 'config.yaml'
Traceback (most recent call last):
  File "/var/home/yajo/mydevel/copier/.venv/bin/copier", line 5, in <module>
    CopierApp.run()
  File "/var/home/yajo/mydevel/copier/.venv/lib/python3.9/site-packages/plumbum/cli/application.py", line 614, in run
    inst, retcode = subapp.run(argv, exit=False)
  File "/var/home/yajo/mydevel/copier/.venv/lib/python3.9/site-packages/plumbum/cli/application.py", line 609, in run
    retcode = inst.main(*tailargs)
  File "/var/home/yajo/mydevel/copier/copier/cli.py", line 40, in _wrapper
    return method(*args, **kwargs)
  File "/var/home/yajo/mydevel/copier/copier/cli.py", line 251, in main
    self.parent._worker(
  File "/var/home/yajo/mydevel/copier/copier/main.py", line 568, in run_copy
    self._execute_tasks(
  File "/var/home/yajo/mydevel/copier/copier/main.py", line 189, in _execute_tasks
    subprocess.run(task_cmd, shell=use_shell, check=True, env=local.env)
  File "/usr/lib64/python3.9/subprocess.py", line 528, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command 'invoke check-yaml' returned non-zero exit status 1.

The interesting part is:

File "/tmp/tmp.srxrxoudz4/dst/tasks.py", line 40, in check_yaml
  config = yaml.safe_load(config_file.read_text())

Which shows clearly that invoke finds the tasks.py file and that it gets executed inside the destination directory.

Also, the failure:

FileNotFoundError: [Errno 2] No such file or directory: 'config.yaml'

Is correct:

╰─ env LANG=C ls dst/config.yaml
ls: cannot access 'dst/config.yaml': No such file or directory

So, AFAICS, all these are bugs in the template itself, not something related to copier.

Regarding the issue at hand, our CI asserts that the tasks are executed in the dst directory always. You can check the test here, and see that it's passing. Unless there's a bug in the test, this issue is fixed in our side:

copier/tests/test_tasks.py

Lines 8 to 43 in 93d95a6

	@pytest.fixture(scope="module")
	def demo_template(tmp_path_factory):
	root = tmp_path_factory.mktemp("demo_tasks")
	build_file_tree(
	{
	root
	/ "copier.yaml": f"""
	_templates_suffix: {SUFFIX_TMPL}
	_envops: {BRACKET_ENVOPS_JSON}
	other_file: bye
	# This tests two things:
	# 1. That the tasks are being executed in the destiantion folder; and
	# 2. That the tasks are being executed in order, one after another
	_tasks:
	- mkdir hello
	- cd hello && touch world
	- touch [[ other_file ]]
	"""
	}
	)
	return str(root)
	def test_render_tasks(tmp_path, demo_template):
	copier.copy(demo_template, tmp_path, data={"other_file": "custom"})
	assert (tmp_path / "custom").is_file()
	def test_copy_tasks(tmp_path, demo_template):
	copier.copy(demo_template, tmp_path, quiet=True, force=True)
	assert (tmp_path / "hello").exists()
	assert (tmp_path / "hello").is_dir()
	assert (tmp_path / "hello" / "world").exists()
	assert (tmp_path / "bye").is_file()

Feel free to reopen if needed 😊