Backup and restore container gateway postgres DB #905

ianballou · 2024-07-23T20:27:08Z

Adds backup and restore support for the container gateway DB. Only adds support for the postgres version of the database for now.

One notable change was that the DB commands need to be run as foreman-proxy since no password is saved for the database. The container gateway uses unix socket auth for the connection, so only the owning user can access the DB (as far as I know).

As such, I've added support for running generic commands as a different user.

To test, try backup and restore on both a normal Foreman/Katello install and on a smart proxy with the container gateway installed.

ToDos:

Figure out why stderr is ending up in the container gateway database dump
Write tests

ianballou · 2024-07-24T14:11:18Z

I looked into the issue where stderr makes it into the dump file. It seems that this was always the case since the final command looks like sudo -u foreman-proxy -- pg_dump -Fc database > dump_file 2>&1. That would redirect stderr into stdout, which is being redirected into dump_file.

I'm guessing this wasn't an issue before because pg_dump was always getting run as root. In my case, foreman-proxy had no permissions to change directories into the place where I started the command, which caused an error which then made it literally into the dump file.

ianballou · 2024-07-24T21:19:12Z

I also ended up adding lines to change the backup ownership to foreman-proxy:foreman-proxy in order for the foreman-proxy user to have access. I noticed we deal with this issue slightly differently for restore -- there's a simple error telling the user to change the backup permissions. This is a bit different for backup though, because the user isn't the one creating the inner backup directory. I'm curious if anyone can see any issues coming out of this.

wbclark

Hey @ianballou !

I've shared some thoughts from my initial reading of this.

Regarding the proposal for some more abstraction here now that we've added a fourth(!) DB, it looks like it could also make sense for the definitions for online DB backup procedures and DB restore procedures?

I am curious to know what you think about it, and perhaps @evgeni would have some insight as well about whether this has been discussed before, and whether it's worth doing now

wbclark · 2024-08-06T15:52:55Z

definitions/procedures/restore/extract_files.rb

+      feature(:foreman_database) || feature(:candlepin_database) ||
+        feature(:pulpcore_database) || feature(:container_gateway_database)


Suggested change

feature(:foreman_database) || feature(:candlepin_database) ||

feature(:pulpcore_database) || feature(:container_gateway_database)

%i[

foreman_database

candlepin_database

pulpcore_database

container_gateway_database

].any? { |db| feature(db) }

[nit] this one sparks joy

wbclark · 2024-08-06T16:20:52Z

definitions/checks/container_gateway/db_up.rb

@@ -0,0 +1,29 @@
+module Checks
+  module ContainerGateway
+    class DBUp < ForemanMaintain::Check


[idea] It looks like we'll have nearly identical code at definitions/checks/{foreman,candlepin,pulpcore,container_gateway}/db_up.rb... what do you think about abstracting these into a common subclass of ForemanMaintain::Check?

It sounds like a good idea to me, the files appear to be very similar.

ianballou · 2024-08-08T16:39:22Z

I originally thought this would be required for Katello 4.14 since I assume the change to postgres would introduce a regression in backup & restore. It looks like it does not, in fact it appears that offline backup is backing up the container gateway DB for free somehow. Once the installer is run, the container_gateway DB is correctly created. So, users restoring at least can still just do a smart proxy sync to restore their content.

As such, I don't think we need to rush this for the upcoming Foreman release in case there are other concerns with the implementation that pop up. I'm personally a bit busy with getting other things in shape for the release, but please let me know if anyone thinks there's a new regression that I'm missing.

ianballou · 2024-08-09T12:43:47Z

@wbclark I've implemented your ideas

evgeni · 2024-08-13T18:56:39Z

It looks like it does not, in fact it appears that offline backup is backing up the container gateway DB for free somehow.

It will not anymore (since we merged #893), so caution! :)

definitions/features/container_gateway_database.rb

ianballou · 2024-10-23T13:50:37Z

Since this has slipped a little on our team priorities, I'm going to call this a PoC and get things updated once the strategy is determined to be sound.

evgeni · 2024-11-06T07:43:08Z

This needs a rebase, as we re-did quite some parts of base_database.rb, sorry.
Is my assumption correct, we do not support external DB for proxies at all right now?

ianballou · 2024-11-06T19:00:58Z

* Is my assumption correct, we do not support external DB for proxies at all right now?

I'll preface this by saying I'm not sure if we support external DBs for proxies, but, if we do ...

We did not test external container gateway external DBs. If I remember correctly, we have all of the connection information ready to be modified to an external host, so it should work to be remote.

If my PR here does not respect external databases but should, I'll fix it.

evgeni

I think this can be simplified, see inline comments :)

evgeni · 2024-11-07T12:57:00Z

lib/foreman_maintain/concerns/base_database.rb

+          base_cmd = if config['connection_string']
+                       'pg_restore'
+                     else
+                       base_command(config, 'pg_restore')
+                     end
+          dump_cmd = base_cmd +
                     ' --no-privileges --clean --disable-triggers -n public ' \
                     "-d #{config['database']} #{file}"
+          if config['user']
+            execute!("chown -R #{config['user']}:#{config['user']} #{File.dirname(file)}")
+          end
          execute!(dump_cmd, :hidden_patterns => [config['password']],
-            :valid_exit_statuses => [0, 1])
+            :valid_exit_statuses => [0, 1], :user => config['user'])


If the DB is remote, we don't need to change to config[user] (=foreman-proxy), as the DB communication happens over the network and doesn't require a specific user to initiate it.

evgeni · 2024-11-07T13:08:36Z

definitions/features/container_gateway_database.rb

+    connection_string = config[:db_connection_string]
+    if connection_string
+      uri = URI.parse(connection_string)
+      @configuration['connection_string'] = connection_string
+      @configuration['user'] = 'foreman-proxy'
+      @configuration['database'] = uri.path[1..]
+    end


So looking at theforeman/puppet-foreman_proxy#835, the connection_string is something like postgres:///container_gateway (for a local setup) or postgres://configurableuser:[email protected]:5432/container_gateway (for a remote DB, supported or not).

Now, the other code in this PR depends on the user definition whether it switches to that (shell) user or not.
That switching is unnecessary for the remote-DB case!

Suggested change

connection_string = config[:db_connection_string]

if connection_string

uri = URI.parse(connection_string)

@configuration['connection_string'] = connection_string

@configuration['user'] = 'foreman-proxy'

@configuration['database'] = uri.path[1..]

end

connection_string = config[:db_connection_string]

if connection_string

uri = URI.parse(connection_string)

@configuration['connection_string'] = connection_string

@configuration['user'] = 'foreman-proxy' unless uri.host

@configuration['database'] = uri.path[1..]

end

definitions/procedures/repositories/index_katello_repositories_container_metadata.rb

evgeni · 2024-11-07T13:12:40Z

lib/foreman_maintain/concerns/base_database.rb

      def dump_db(file, config = configuration)
-        execute!(dump_command(config) + " > #{file}", :hidden_patterns => [config['password']])
+        if config['user']
+          execute!("chown -R #{config['user']}:#{config['user']} #{File.dirname(file)}")
+        end
+        execute!(dump_command(config) + " -f #{file}",
+          :hidden_patterns => [config['password']], :user => config['user'])
      end


Right now, this relies on the DB credentials we parse from the config and in the super special case of gateway, the shell user of the proxy.

For restores, we have this "if local, use postgres user, else the parsed config as root", if we use the same logic for dumping, we reduce the special-casing for gateway (as we either use postgres, which has access to all databases anyway, or a remote connection that doesn't require foreman-proxy user).

When I wrote this I didn't think I could use the postgres user for some reason. Using postgres would make the code here a whole lot simpler 👍

ianballou · 2025-01-14T23:24:00Z

@evgeni I've rebased this and changed the PR to use the postgres user for "local" database commands.

I need to get the tests passing again.

I've tested this on a container gateway enabled smart proxy with online backup & restore.

ianballou · 2025-07-28T20:44:34Z

The latest code ensures all local DB connections use unix sockets with a simple connection string (like postgres:///foreman). It's been tested on a capsule and I'm currently testing on a satellite system (so far so good).

Edit: the satellite system test worked.

adamruzicka

Haven't tested, but looks reasonable at a glance

adamruzicka · 2025-07-30T10:45:26Z

definitions/features/container_gateway_database.rb

+    if connection_string
+      uri = URI.parse(connection_string)
+      @configuration['connection_string'] = connection_string
+      @configuration['database'] = uri.path[1..]


The path[1..] is there "just" to strip the leading forward slash?

Yep, lemme change it to delete_prefix('/') to make it more clear.

adamruzicka · 2025-07-30T10:49:24Z

lib/foreman_maintain/concerns/base_database.rb

-        execute!(dump_command, :env => base_env)
+        if local?
+          dump_command = "pg_dump --format=c #{configuration['connection_string']}"
+          execute!("chown -R postgres:postgres #{File.dirname(file)}")


Why wasn't this needed previously?

Previously we ran this as root, but since now I'm using the postgres user, we need to ensure the dump directory is reachable by postgres. Alternatively we could just throw an error saying to ensure the directory is reachable by the postgres user if making the change for the user is overstepping.

adamruzicka · 2025-07-30T10:52:10Z

lib/foreman_maintain/concerns/base_database.rb

+          execute?(
+            "runuser - postgres -c 'psql -d #{configuration['database']} -c \"SELECT 1 as ping\"'"
+          )


Could the query be passed on stdin same as on L76-78?

I think it could, let me try.

ianballou · 2025-08-01T18:08:20Z

@adamruzicka I think I've addressed your comments.

Allow executing commands as a different user Use --file and --format with pg_dump Refactor DBUps into subclasses Fix rubocop errors Use postgres user for container gateway backup/restore Remove user config for container gateway backup

ianballou force-pushed the backup-container-gateway branch from 384b86f to 5814c0e Compare July 23, 2024 21:30

ianballou force-pushed the backup-container-gateway branch 4 times, most recently from 214ff13 to 4818b98 Compare July 24, 2024 21:15

ianballou force-pushed the backup-container-gateway branch 2 times, most recently from 3ae9e8c to 9f3a763 Compare July 24, 2024 21:47

ianballou marked this pull request as ready for review July 24, 2024 21:48

ianballou force-pushed the backup-container-gateway branch 2 times, most recently from 0762bac to 1a0161a Compare July 25, 2024 13:53

wbclark reviewed Aug 6, 2024

View reviewed changes

ianballou force-pushed the backup-container-gateway branch from 1a0161a to fb56093 Compare August 9, 2024 12:41

ianballou requested a review from wbclark August 9, 2024 12:43

evgeni reviewed Aug 13, 2024

View reviewed changes

definitions/features/container_gateway_database.rb Outdated Show resolved Hide resolved

evgeni requested changes Nov 7, 2024

View reviewed changes

pr-processor bot added the Waiting on contributor label Nov 7, 2024

ianballou force-pushed the backup-container-gateway branch from 0c1dfc0 to cc3b128 Compare November 7, 2024 18:05

ianballou force-pushed the backup-container-gateway branch 5 times, most recently from 4158dc1 to 58141a0 Compare January 14, 2025 22:39

ianballou force-pushed the backup-container-gateway branch 3 times, most recently from 876b819 to 1954032 Compare January 14, 2025 23:21

ianballou requested a review from evgeni January 14, 2025 23:22

ianballou force-pushed the backup-container-gateway branch 4 times, most recently from e5f68d3 to 6375b67 Compare July 28, 2025 20:26

ianballou force-pushed the backup-container-gateway branch 4 times, most recently from 3ee90d9 to 26050cf Compare July 28, 2025 21:00

adamruzicka reviewed Jul 30, 2025

View reviewed changes

ianballou force-pushed the backup-container-gateway branch 5 times, most recently from 7600871 to 4da99fe Compare August 1, 2025 18:03

ianballou added 3 commits September 5, 2025 15:23

Backup container gateway DB

1883bf3

Allow executing commands as a different user Use --file and --format with pg_dump Refactor DBUps into subclasses Fix rubocop errors Use postgres user for container gateway backup/restore Remove user config for container gateway backup

Fix local postgres user relate tests

3e91187

Use postgres for local connections in all DBs

3347a9e

ianballou force-pushed the backup-container-gateway branch from 9438828 to 3347a9e Compare September 5, 2025 19:24

ianballou requested a review from adamruzicka September 5, 2025 19:24

		feature(:foreman_database) \|\| feature(:candlepin_database) \|\|
		feature(:pulpcore_database) \|\| feature(:container_gateway_database)

Backup and restore container gateway postgres DB #905

Are you sure you want to change the base?

Backup and restore container gateway postgres DB #905

Uh oh!

Conversation

ianballou commented Jul 23, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ianballou commented Jul 24, 2024

Uh oh!

ianballou commented Jul 24, 2024

Uh oh!

wbclark left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

wbclark Aug 6, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ianballou commented Aug 8, 2024

Uh oh!

ianballou commented Aug 9, 2024

Uh oh!

evgeni commented Aug 13, 2024

Uh oh!

Uh oh!

ianballou commented Oct 23, 2024

Uh oh!

evgeni commented Nov 6, 2024

Uh oh!

ianballou commented Nov 6, 2024

Uh oh!

evgeni left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ianballou commented Jan 14, 2025

Uh oh!

ianballou commented Jul 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

adamruzicka left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ianballou commented Aug 1, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

ianballou commented Jul 23, 2024 •

edited

Loading

wbclark Aug 6, 2024 •

edited

Loading

ianballou commented Jul 28, 2025 •

edited

Loading