State management redesign for multi-host control

[ehelms]

User Stories

US1: Deploy a server and add proxies over time
A sysadmin deploys a Foreman server from their workstation, then days or weeks later adds 2-5 content proxies one at a time. Each deploy-proxy invocation must not inherit or overwrite the server's persisted parameters, and vice versa. Ongoing operations (upgrades, config changes, feature enablement, certificate renewal, backup) target one host at a time without affecting other hosts' state.

US2: Certificate bundle flow from a single control node
When adding a proxy, the sysadmin generates a certificate bundle -- either on the server or directly on the control node -- and deploy-proxy picks it up automatically from the control node's state without manual file transfer between machines.

US3: Rebuild a failed server or proxy
When a server dies, the sysadmin restores it from backup on the control node and reconnects existing proxies. When a proxy dies, the sysadmin redeploys just that proxy from the control node using its stored state (secrets, parameters, certificates) without affecting the server or other proxies.

US4: Manage identical but independent Foreman servers
A sysadmin maintains multiple independent Foreman servers (e.g., dev/staging/prod) with the same features, tuning, and configuration but different hostnames and separate data. They may define the config once and apply it to each server, or deploy one and replicate its settings to others -- all from a single control node or from separate control nodes sharing a configuration template.

US5: Development environment alongside production
A developer uses forge for a dev environment and foremanctl for a test deployment on the same machine. Running one must not contaminate the other's persisted parameters or secrets. This includes the case where the developer runs a dev Foreman server via forge and then deploys a proxy to connect to it.

US6: Multiple development environments
A developer runs multiple forge environments simultaneously (e.g., two VMs with different configurations, or a dev server and a dev proxy). Each environment's state must be isolated.

Problem

foremanctl currently uses a single flat state directory (OBSAH_STATE, defaulting to .var/lib/foremanctl/) on the control node for two distinct categories of state:

1. Parameter persistence - obsah writes parameters.yaml to OBSAH_STATE after every successful run, and loads it as defaults on the next run
2. Secret/credential files - Ansible password lookups generate and cache secrets (DB passwords, OAuth keys, CA passwords, encryption keys) in obsah_state_path

Both foremanctl deploy and foremanctl deploy-proxy share the same OBSAH_STATE path and therefore the same parameters.yaml file. This creates problems when the control node manages multiple targets.

Current Architecture

How state flows today

foremanctl (bash wrapper)
  └─ sets OBSAH_STATE=.var/lib/foremanctl
  └─ sets OBSAH_PERSIST_PARAMS=true
  └─ exec obsah "$@"
         │
         ├─ Loads parameters.yaml as argparse defaults
         ├─ Runs ansible-playbook with -e (extra vars including obsah_state_path)
         └─ On success: merges current args into parameters.yaml and saves

The obsah_state_path variable is passed to Ansible, where roles use it to resolve secret file paths:

• foreman.yml: foreman_admin_passwd_file: "{{ obsah_state_path }}/foreman-admin-init-passwd"
• database.yml: foreman_database_password_file: "{{ obsah_state_path }}/foreman-db-password"
• base.yaml: certificates_ca_password_file: "{{ obsah_state_path }}/certificates-ca-password"
• post_install: post_install_done_flag: "{{ obsah_state_path }}/.installed"

Certificate artifacts go to /var/lib/foremanctl/certs on the target host (hardcoded in certificates.yml, not using obsah_state_path).

The forge wrapper shares the same state path

Both foremanctl and forge set OBSAH_STATE=${OBSAH_BASE}/.var/lib/foremanctl. This means development workflows also share the same parameter file.

Problems

1. Parameter persistence clashes across commands

parameters.yaml is a single flat file. When a user runs:

foremanctl deploy --tuning large --certificate-source custom_server ...
foremanctl deploy-proxy --certificate-bundle /tmp/proxy.tar.gz --foreman-fqdn server.example.com

The second command inherits all persisted parameters from the first (e.g. tuning, certificate_source, flavor, features). More critically, it overwrites the file with deploy-proxy's parameter set, so the next foremanctl deploy run picks up deploy-proxy values like foreman_name (FQDN of the server) and flavor: foreman-proxy-content.

This is already partially mitigated by persist: false on some parameters (e.g. certificates_bundle), but the core problem remains: parameters from different commands and different target hosts are co-mingled in one file.

2. Backups capture target state but miss control state

The backup role archives obsah_state_path from the target:

- name: Backup foremanctl state directory
  community.general.archive:
    path: "{{ obsah_state_path }}"
    dest: "{{ backup_dir_full }}/foremanctl-state.tar.gz"

When control == target, this captures everything. When they differ, this captures whatever is at obsah_state_path on the target (which may be empty or stale), while the actual secret files live on the control node.

3. Where do certificate bundles belong?

certificate-bundle generates a tarball on the server at /var/lib/foremanctl/certs/bundles/<hostname>.tar.gz. The user must manually transfer it to the proxy (or to the control node for deploy-proxy --certificate-bundle). This manual transfer does not scale to multiple proxies managed from a single control node.

4. The .installed flag is host-unaware

post_install_done_flag: "{{ obsah_state_path }}/.installed" is a single flag written on the control node (via delegate_to: localhost in post_install). If the control node manages both a server and multiple proxies, there is no way to distinguish which target has been installed. Deploying a proxy after a server overwrites the flag, and vice versa.

5. Direct coupling to parameters.yaml format

The smoker playbook (development/playbooks/smoker/smoker.yaml) loads parameters.yaml directly as a vars_file, bypassing obsah. Any restructuring of parameters.yaml will break this pattern.

6. forge and foremanctl share state

Both wrappers set OBSAH_STATE=${OBSAH_BASE}/.var/lib/foremanctl, so development workflows via forge and production workflows via foremanctl read and write the same parameters.yaml and secret files. Running foremanctl deploy followed by forge deploy-dev on the same machine will cross-contaminate persisted parameters.

Design Goals

• The control node can be separate from the target node
• A single control node can manage a server and multiple proxies
• State for different targets does not interfere
• The design borrows from Ansible concepts where possible
• Backward compatibility with the current single-host RPM-install model (control == target)

Possible Approaches

Approach A: Per-host state directories

Borrow from Ansible's fact cache pattern. Instead of a flat obsah_state_path, organize state by inventory hostname, with each host directory containing its own parameters, secrets, and flags:

.var/lib/foremanctl/
├── server.example.com/
│   ├── parameters.yaml
│   ├── foreman-db-password
│   ├── foreman-oauth-consumer-key
│   ├── certificates-ca-password
│   ├── ...
│   └── .installed
└── proxy1.example.com/
    ├── parameters.yaml
    ├── ...
    └── .installed

All secrets belong to the host they were generated for. The CA password lives in the server's directory because the server owns the CA; proxy deployments that need the CA password (to validate bundles) receive it via the certificate bundle tarball, not via a shared file.

obsah gains a concept of "target identity" and scopes all state by it. The obsah_state_path passed to Ansible points to the host-specific directory (e.g. .var/lib/foremanctl/server.example.com/), so secret file lookups and flags are automatically scoped without changing role code.

Host identity resolution

The critical design question is how obsah determines which host directory to use. Options:

1. Derive from inventory + limit: obsah resolves the target host from the Ansible inventory after applying any --limit. This is implicit and matches how Ansible works, but requires obsah to parse inventory files or shell out to ansible-inventory.
2. Explicit --target argument: the wrapper passes a target identifier. foremanctl would set this based on the command (deploy -> server host, deploy-proxy -> proxy host). Simple and explicit, but adds a new concept to the CLI.
3. Host group as fallback: when no specific host is resolvable (e.g. localhost inventory with a quadlet group), use the group name as the directory key. This handles the single-host and development cases where the inventory hostname is not meaningful.

The chosen mechanism must handle the forge development case, where the inventory is localhost-based and the target is always the quadlet host group. In this case the directory key would likely be the VM hostname or the group name.

Boundaries: obsah owns per-host parameters.yaml read/write, sets obsah_state_path to <hostname>/, resolves the host directory, and falls back to the current flat layout when no host scoping is configured. foremanctl manages inventory and provides host identity to obsah. The CLI experience is unchanged — "pass flags and they stick."

Tradeoff: most transparent to users. Requires obsah changes to add host-scoping to persist_path() and state_path().

Approach B: Parameters file as inventory

Restructure parameters.yaml into a host-keyed structure with all -> group -> host variable precedence:

# parameters.yaml
all:
  certificates_source: default
server:
  flavor: katello
  tuning: large
hosts:
  server.example.com:
    group: server
  proxy1.example.com:
    group: proxy
    foreman_name: server.example.com

obsah would resolve the target host, merge variables in precedence order, and persist back to the appropriate scope.

Gap: this only addresses parameter persistence. The 13+ secret files from ansible.builtin.password lookups still land in a flat obsah_state_path. Per-host directory isolation (Approach A) is still needed for secrets, making this approach additive rather than alternative.

Reimplementing Ansible's variable precedence in obsah is also nontrivial — every edge case needs answering (does --tuning persist to host scope or group scope?). If parameter layering is desired, actual Ansible group_vars/ and host_vars/ directories would leverage existing semantics instead of re-implementing them.

Things to Consider

What belongs on the control node vs target node?

Artifact	Current Location	Should Live On
parameters.yaml	Control node	Control node (per-host)
DB/OAuth password files	Control node (via lookups)	Control node (per-host)
CA private key + certs	Target node (/var/lib/foremanctl/certs)	Server target only
Certificate bundles	Generated on server target	Server target, transferred to proxy
.installed flag	Control node (via delegate_to: localhost)	Control node (per-host)
Backup archives	Target node	Target node
Ansible log	Control node	Control node

Ansible concepts to borrow

• host_vars / group_vars: natural place for per-host parameter persistence
• Fact caching: Ansible can cache facts per host; a similar pattern could work for foremanctl state
• Delegate_to / run_once: already used in migrate playbook for control-node-local operations
• Ansible Vault: could encrypt secrets at rest instead of plain-text files

forge and development workflows

In development, forge targets the quadlet host group (a Vagrant VM) via a localhost-based inventory. Per-host state directories work without special handling — the directory key would be the VM hostname or group name (e.g. .var/lib/foremanctl/quadlet.example.com/).

If forge and foremanctl continue sharing the same OBSAH_STATE base path, they must use different host directory keys to avoid cross-contamination. Alternatively, forge could use a separate base (e.g. .var/lib/forge).

Certificate bundle transfer

With per-host state directories, certificate bundles can flow automatically:

1. foremanctl certificate-bundle --hostname proxy1.example.com generates the bundle on the server
2. foremanctl copies the tarball to <base>/<proxy1.example.com>/certificate-bundle.tar.gz on the control node
3. foremanctl deploy-proxy reads the bundle from the proxy's state directory automatically

Restore and disaster recovery

The backup role currently archives the entire obsah_state_path as a single tarball. With per-host directories, decisions are needed:

• Does a backup of the server include only <server>/ state, or all host directories on the control node?
• If the control node dies, restoring requires all host directories — the backup must either include them all or the user must have separate backups per host
• The restore procedure must place state back into the correct per-host directory, not a flat path

persist: false parameters

Parameters marked persist: false (e.g. certificates_bundle) continue to mean "do not write to any parameters.yaml" — the scoping is orthogonal. This should be explicitly documented.

Migration path

Any redesign must handle upgrading from the current flat state directory. A concrete migration would:

1. Detect the flat layout (existence of parameters.yaml and secret files directly in obsah_state_path with no host subdirectories)
2. Determine the current target hostname (from inventory, or prompt the user)
3. Create <hostname>/ under the state base path
4. Move secret files and .installed into the host directory
5. Move parameters.yaml into the host directory

When the flat parameters.yaml contains merged state from both deploy and deploy-proxy runs, clean separation may not be possible automatically. The migration should handle the common single-host case (move as-is) and warn when ambiguous state is detected, leaving manual resolution to the user.

The migrate command already demonstrates this pattern (migrating from foreman-installer answers to foremanctl parameters) and can serve as a template.

Impact on obsah

obsah's persist_path() currently returns a single path. It would need to either:

• Accept a hostname/scope parameter to return a host-specific path
• Support a callback/plugin model where the wrapper controls persistence behavior

A phased approach may be useful: foremanctl could handle host directory creation and secret file scoping in its own wrapper logic first, then add native per-host parameter persistence to obsah.

[ekohl]

Bit of a brain dump.

When looking at #633 I was leaning to option A and wondered if we should have a foremanctl add-target $hostname pattern. Then a more generic foremanctl deploy --target $target. In #527 I played with aligning the deploy and deploy-proxy playbooks so ideally speaking there is just a single deploy playbook that is generic. However, it will have some challenges in showing parameters that are irrelevant.

Perhaps we should aim for a sort of deploy role, as a replacement for https://github.com/theforeman/foreman-operations-collection. Then in the deploy commands we can have a bit more flexibility in which parameters we expose.

[ehelms]

One idea that was proposed in other conversations was to have dedicated wrapper scripts for now. I took that idea and tried to make a single wrapper script that acts like multiple based upon the target: #693

I also still owe updates to this issue to include the various multi-host workflows we envision wanting.

[ehelms]

I have updated the description with user stories that describe the various control->target node separation scenarios I could think of.

[ekohl]

US1: Deploy a server and add proxies over time A sysadmin deploys a Foreman server from their workstation, then days or weeks later adds 2-5 content proxies one at a time. Each deploy-proxy invocation must not inherit or overwrite the server's persisted parameters, and vice versa. Ongoing operations (upgrades, config changes, feature enablement, certificate renewal, backup) target one host at a time without affecting other hosts' state.

When deploying (content) proxies in Satellite we already register them to Foreman and pull content from it. We have tools like REX and Ansible available. I'm worried foremanctl grows into a tool that provides the same capabilities as Foreman.

Turning it the other way: what is needed to deploy a (content) proxy from Foreman without foremanctl?

US2: Certificate bundle flow from a single control node When adding a proxy, the sysadmin generates a certificate bundle -- either on the server or directly on the control node -- and deploy-proxy picks it up automatically from the control node's state without manual file transfer between machines.

I'd love to get away from the certificate bundle flow. Instead, use services so that manual renewal isn't needed. Tools like certbot (using the ACME protocol) prove that it's possible to create a secure workflow. FreeIPA also implements this as a service (https://deepwiki.com/freeipa/freeipa/6.1-certificate-authority), so does Vault (https://developer.hashicorp.com/vault/tutorials/pki/pki-engine). Candlepin and Puppet being their own CA really complicates things, but ideally you'd create an intermediate CA structure.

Let's invest into that rather than building foremanctl into a more complicated tool than it needs to.

US3: Rebuild a failed server or proxy When a server dies, the sysadmin restores it from backup on the control node and reconnects existing proxies. When a proxy dies, the sysadmin redeploys just that proxy from the control node using its stored state (secrets, parameters, certificates) without affecting the server or other proxies.

If you can deploy and manage a machine from Foreman (storing its secrets and parameters), you'd be able to leverage all the tools we already provide for this in normal workflows. When there's a service to automatically provide fresh certificates, do you really need to store those in a backup? Remember: best state is no state.

US4: Manage identical but independent Foreman servers A sysadmin maintains multiple independent Foreman servers (e.g., dev/staging/prod) with the same features, tuning, and configuration but different hostnames and separate data. They may define the config once and apply it to each server, or deploy one and replicate its settings to others -- all from a single control node or from separate control nodes sharing a configuration template.

This could be interesting, but is it important enough? By using a central foremanctl you also don't allow doing rolling upgrades. Unless foremanctl becomes multi-version aware but I'd love to avoid that.

US5: Development environment alongside production A developer uses forge for a dev environment and foremanctl for a test deployment on the same machine. Running one must not contaminate the other's persisted parameters or secrets. This includes the case where the developer runs a dev Foreman server via forge and then deploys a proxy to connect to it.

Here I'd advocate for making it easier to use other OBSAH_STATE directories to separate them.

US6: Multiple development environments A developer runs multiple forge environments simultaneously (e.g., two VMs with different configurations, or a dev server and a dev proxy). Each environment's state must be isolated.

This feels very similar to the case above and easier OBSAH_STATE directory management sounds useful.

[jeremylenz]

When deploying (content) proxies in Satellite we already register them to Foreman and pull content from it. We have tools like REX and Ansible available. I'm worried foremanctl grows into a tool that provides the same capabilities as Foreman.

Turning it the other way: what is needed to deploy a (content) proxy from Foreman without foremanctl?

Sure, but I like the rule - "foremanctl is for configuring your Foreman base infra, REX is for configuring your registered hosts." We previously relied on self-REX for things like cloud connector and it was a mistake. I think it still makes sense to have foremanctl manage the smart proxy.

I'd love to get away from the certificate bundle flow. Instead, use services so that manual renewal isn't needed. Tools like certbot (using the ACME protocol) prove that it's possible to create a secure workflow. FreeIPA also implements this as a service (https://deepwiki.com/freeipa/freeipa/6.1-certificate-authority), so does Vault (https://developer.hashicorp.com/vault/tutorials/pki/pki-engine). Candlepin and Puppet being their own CA really complicates things, but ideally you'd create an intermediate CA structure.

This seems like a lot of work that we don't have time for right now. We could always do it later.

[ekohl]

This seems like a lot of work that we don't have time for right now. We could always do it later.

We also don't have a lot of time for multi-host control. I'd rather spend time on the long term goal than on this short term thing.