add quadlet recurring job runner and timers for core rake tasks

[archanaserver]

• define templated Quadlet container [email protected] via containers.podman.podman_container (state: quadlet)
• run rake tasks inside the foreman image with same secrets/mounts as foreman.service
• so far added per-instance timers for:
  • reports:daily
  • db:sessions:clear
  • reports:expire
  • audits:expire

Refs: #61 and #252

[archanaserver]

so now the @.container is now defined with containers.podman.podman_container (state: quadlet), and only the .timer units are templated. Implemented a generic foreman-recurring@ runner with per-job timers (reports:daily, db:sessions:clear, reports:expire, audits:expire). Also left ldap:refresh_usergroups out for now as a candidate for the in-app scheduling path (foreman-tasks / sidekiq-cron) to better support HA. In the follow up PRs I'll add weekly/monthly and katello/foreman-tasks timers.

[archanaserver]

@evgeni so moved back to the explicit item.rake approach and added the remaining recurring tasks from the cron. Because of the : vs container-name restriction, and I ended up using a small explicit mapping (rake field) here instead of %I. So the instance names avoid : since Podman doesn’t support it, and we still pass the correct rake task cleanly.

Also opened two follow-ups PR for katello and foreman_tasks #298 and #299 which actually needs code refactor once this one is merged because to make a generic [email protected]

Question: One thing i was unsure about is for testing, should we keep the exact same behavior as the old cron jobs, or is a journalctl-based flow acceptable as the new baseline for recurring tasks? (not sure if i have seen this conversation anywhere, or i might have missed it)

[evgeni]

Question: One thing i was unsure about is for testing, should we keep the exact same behavior as the old cron jobs, or is a journalctl-based flow acceptable as the new baseline for recurring tasks? (not sure if i have seen this conversation anywhere, or i might have missed it)

You mean logging? Yeah I think journal logging is fine.

[ekohl]

I'd like to take a step back and ask if there's a better and more native way in Foreman to do this.

Instead of having a huge sprawl of tasks, we should ask ourselves if there's a better way. In foreman-tasks we have recurring logic and perhaps we can find a way to bring that to core Foreman.

Perhaps a short term hack is to create rake tasks in core:

• foreman-rake cron:hourly
• foreman-rake cron:daily
• foreman-rake cron:weekly
• foreman-rake cron:monthly

Then individual other tasks can add themselves as prerequisites. Plugins can do the same so you wouldn't need to think about individual cronjobs for plugins. The upside is that you only load the whole of Foreman once (which makes it faster if you perform multiple tasks), but the downside is that if one crashes the later tasks may not run.

A slightly more advanced approach is to build a registry similar to what upgrade:run does. If there is recurring logic present then perhaps that could automatically take it over.

[evgeni]

I like the idea with foreman-rake cron:<when>, it would also (accidentally) solve the naming problem, as you could have foreman-cron@<when> timers/services, thus avoiding the : in the name and being able to use real instances again.

[archanaserver]

@evgeni @ekohl So based on the SAT-36361 discussion for now I’ve opened a Foreman core theforeman/foreman#10784 that adds generic cron:hourly/daily/weekly/monthly rake tasks (Foreman::Cron) as shared entrypoints for recurring jobs. And follow up in foremanctl and switch these per-job timers to just 4 generic cron:* timers here #332.

For this PR I’d still like to get the recurring infra + timers in as a first iteration, and then do the switch to cron:* in a separate, to keep the PR smaller.

[evgeni]

nitpicks and some minor questions

[evgeni]

Here is the final diff I applied locally when testing things:

diff --git src/roles/foreman/tasks/main.yaml src/roles/foreman/tasks/main.yaml
index 291edc9..9d277d4 100644
--- src/roles/foreman/tasks/main.yaml
+++ src/roles/foreman/tasks/main.yaml
@@ -161,7 +161,7 @@
     - worker-hosts-queue
 
 - name: Define templated Quadlet for recurring Foreman rake tasks
-  when: foreman_recurring_tasks_enabled | default(true)
+  when: foreman_recurring_tasks_enabled
   loop: "{{ foreman_recurring_tasks }}"
   loop_control:
     label: "{{ item.instance }}"
@@ -173,9 +173,7 @@
     sdnotify: false
     network: host
     hostname: "{{ ansible_facts['fqdn'] }}"
-    user: foreman
-    working_dir: /usr/share/foreman
-    command: "bash -lc 'foreman-rake {{ item.rake }}'"
+    command: "foreman-rake {{ item.rake }}"
     volume:
       - 'foreman-data-run:/var/run/foreman:z'
     secrets:
@@ -188,18 +186,12 @@
       - 'foreman-client-cert,type=mount,target=/etc/foreman/client_cert.pem'
       - 'foreman-client-key,type=mount,target=/etc/foreman/client_key.pem'
     quadlet_options:
-      - |
-        [Install]
-        WantedBy=default.target foreman.target
       - |
         [Unit]
         PartOf=foreman.target
-        Requires=foreman.service
-        After=foreman.service
         StartLimitIntervalSec=0
       - |
         [Service]
-        ExecStartPre=/usr/bin/flock -n /run/foreman-recurring-%i.lock -c /usr/bin/true
         TimeoutStartSec=30m
         TimeoutStopSec=2m
         KillMode=mixed
@@ -261,7 +252,7 @@
   register: foreman_status
 
 - name: Enable & start recurring timers
-  when: foreman_recurring_tasks_enabled | default(true)
+  when: foreman_recurring_tasks_enabled
   ansible.builtin.systemd:
     name: "foreman-recurring@{{ item.instance }}.timer"
     enabled: true
diff --git src/roles/foreman/templates/[email protected] src/roles/foreman/templates/[email protected]
index b22dcbe..7d8a3e2 100644
--- src/roles/foreman/templates/[email protected]
+++ src/roles/foreman/templates/[email protected]
@@ -9,4 +9,6 @@ RandomizedDelaySec=120
 AccuracySec=1min
 
 [Install]
-WantedBy=timers.target
+WantedBy=timers.target foreman.target
+[Unit]
+PartOf=foreman.target

works perfectly fine! thanks a ton for working on this!

[archanaserver]

yeah seems like everything is working perfectly \o/

[evgeni]

dev-ack!

[shubhamsg199]

ACK, Tested and works as expected.