Document the order of precedence of environment variable sources

[happz]

Environment variables visible to test - but not just tests, prepare scripts or Ansible playbooks must be considered too! - are composed from various sources:

External sources

• run: --environment and --environment-file CLI options
• plan: environment and environment-file fmf keys
• importing plan: environment fmf key of the importing plan
• provision phases ("per-guest environment"): environment fmf key, --environment CLI option
• test: environment fmf key
• recipe: saved run environment replacing current run environment

Internal sources

• plugin action itself - test name, test data, plan data, ...
• test framework
• invocation contexts - reboot, restart, abort, pidfile, restraint, ...

Environment variables recognized by tmt

• command variables: https://tmt.readthedocs.io/en/stable/overview.html#variables
• step variables: https://tmt.readthedocs.io/en/stable/overview.html#step-variables
• test variables: https://tmt.readthedocs.io/en/stable/overview.html#test-variables
• plugin variables: https://tmt.readthedocs.io/en/stable/overview.html#plugin-variables

(Did I miss any source?)

The question is: in which order shall they be applied? We are not 100 % sure, and then there are users asking.

The goal is to put together a documentation similar to https://docs.ansible.com/ansible/latest/playbook_guide/playbooks_variables.html#variable-precedence-where-should-i-put-a-variable

---

Status

Untangling the environment, one step at a time. At this moment, the best we can do is to document the order, but it's far from being clean, nice, and desirable.

• untangle plan environment from test environment, and add plan environment to the final environment in the execute plugin
  • Do not inject plan environment into test environment #4455
• drop the duplication of "run" environment, run.environment and "plan CLI environment" seems to be the same thing. refactor run environment to use the same composition approach so we can better track the sources:
  • Refactor how tmt run collects its environment #4415
• move TMT_SOURCE_DIR addition elsewhere - when it's needed, who needs to add it, do we need another context, etc.
  • Move TMT_SOURCE_DIR envvar setting out of the discover step #4616
• plan environment file - when it's pulled from guests, is it saved into the same runner path? Plan may prepare different content on different guests, sharing single local path might be incorrectly enforcing one version to all guests. Possibly a bug waiting to be noticed.
  • Make plan environment file guest-specific #4772
• environment for guest-targeting commands was based on tmt process environment
  • dropped by accident in Fix prepare/shell and provision/local missing the plan environment #4852
  • Do not populate the default command environment with tmt process envvars #4900
• assorted changes to improve our control over environment use
  • Attach test environment to its invocation #4452
  • Move test check invocation under the umbrella of test invocation #4453
  • Drop plan environment from the default command environment #4613
  • Rename various env parameters to environment #4614
• Split plan environment into fmf and CLI parts
  • Separate ingestion of environment from fmf keys and CLI options #4968
• Remove duplicity of CLI input (**self._environment_from_cli in Plan.environment, and **self._environment_from_cli in Run.environment
  • Separate ingestion of environment from fmf keys and CLI options #4968
• Separate user-owned and tmt-owned envvars
  • Separate intrinsic environment variables #5054
• Implement environment builder to make sure the same process is followed
  • Adding methods for building environment in the right order #5057
• Review plugins like prepare/install or prepare/artifact - do they expose variables as they should?
• Document plugins that do not expose the environment like other plugins - namely ansible
• Announce the ordering changes, and, well, make the changes
• Make plan script file guest-specific (same problem we had with plan environment file)
  • Plan source script shall be guest-specific file #5034
• Expose TMT_SOURCE_DIR to prepare and finish phases Probably not reasonable, as prepare and finish phases are not tied to one discover phase like execute phases are.

---

Assorted notes

Current (rough) ordering

AFAICT, the rough order of precedence, as of now, is something like the following:

• provision[].environment
• test[].environment
• "plan environment"
  • $TMT_PLAN_ENVIRONMENT_FILE
  • plan environment-file key
  • plan environment key
  • importing plan "native" environment
    • importing plan environment-file key
    • importing plan environment key
    • importing plan importing plan ...
  • run environment saved in the workdir (--environment and --environment-file from previous invocations)
  • tmt run --environment
  • tmt run --environment-file
  • plan intrinsic variables: TMT_VERSION, TMT_TREE, ...
• plugin intrinsic variables
  • abort context
  • reboot context
  • restart context
  • pidfile context
  • restraint context
  • test framework
  • ...

Desired state

This should be the desired order of precedence, from the weakest to strongest sources:

1. test[].environment
2. plan environment-file key
3. plan environment key
4. importing plan "native" environment
   1. importing plan environment-file key
   2. importing plan environment key
   3. importing plan importing plan ...
5. provision[].environment
6. $TMT_PLAN_ENVIRONMENT_FILE
7. command-line input:
   1. tmt run
      • run environment saved in the workdir (--environment-file and --environment from previous invocations)
      • tmt run --environment-file
      • tmt run --environment
   2. tmt * export
      • tmt * export --environment-file
      • tmt * export --environment
   3. tmt try
      • tmt try --environment-file
      • tmt try --environment
8. plugin intrinsic variables
   1. abort context
   2. reboot context
   3. restart context
   4. pidfile context
   5. restraint context
   6. test framework
   7. guest topology
   8. plan intrinsic variables: TMT_VERSION, TMT_TREE, ...

[HouMinXi]

Give a more example
I plan start two tests, the first test need BUILDID parameter, the second test need OVS_URL/DPDK_URL/NIC_DRIVER as parameters.

on plan level, I define environment as below:

  environment:
    OVS_URL: "https://download.eng.pek2.redhat.com/brewroot/packages/openvswitch3.5/3.5.1/34.el10fdp/x86_64/openvswitch3.5-3.5.1-34.el10fdp.x86_64.rpm"
    DPDK_URL: "https://download.devel.redhat.com/brewroot/packages/dpdk/24.11.1/2.el10/x86_64/dpdk-24.11.1-2.el10.x86_64.rpm"
    BUILDID: $@{build_id}
    NIC_DRIVER: ice,mlx5_core

Q1: Currently, all parameters can be seen on all tests. BUILDID/OVS_URL/DPDK_URL/NIC_DRIVER exist on the first test and the second test. This can easily cause parameter confusion. In fact, task1 only needs the BUILDID parameter.

Q2: NIC_DRIVER is a user-define parameter that used for specify which nic should be used in corresponding server. One server can have muti-nics. So I need to send NIC_DRIVER parameter to plan. So it would exist in plan-level environment, but I didn't know how to change to NIC_DRIVER in provision-level environment.