Podman Machine Date gets behind after sleep

[podollb]

This is fixed in Podman v4.5.0. You also need to re-create your podman machine with podman 4.5.0 in order for the fix to take effect. Feel free to re-open this issue if the time is sitll not synced after upgrading your version.

Originally posted by @ashley-cui in #18552

This still happens to me regularly (but seems to have just started after one of the last few upgrades):

~ podman version
Client:        Podman Engine
Version:       5.6.1
API Version:   5.6.1
Go Version:    go1.25.1
Git Commit:    1e2b2315150b2ffa0971596fb5da8cd83f3ce0e1
Built:         Thu Sep  4 15:46:52 2025
Build Origin:  pkginstaller
OS/Arch:       darwin/arm64

Server:       Podman Engine
Version:      5.2.4
API Version:  5.2.4
Go Version:   go1.22.7
Built:        Sun Oct  6 19:00:00 2024
OS/Arch:      linux/arm64
~

[ashley-cui]

Hi! Could you provide some more information, maybe a reproducer, or some more detail on how long it takes the machine to desync? Also, it looks like your server is much older than the client, does this still happen with a new podman machine?

[Honny1]

Could you please use these commands to check if the NTP service has stopped working?

podman machine ssh systemctl status chronyd

podman machine ssh timedatectl status

[github-actions]

A reviewer has determined we need more information to understand the reported issue. A comment on what is missing should be provided. Be certain you:

• provide an exact reproducer where possible
• verify you have provided all relevant information - minimum is podman info
• answer any follow up questions

If no response to the needs-info is provided in 30 days, this issue may be closed by our stale bot.

For more information on reporting issues on this repository, consult our issue guide.

[Luap99]

somewhat related #26185

[podollb]

Hi! Could you provide some more information, maybe a reproducer, or some more detail on how long it takes the machine to desync? Also, it looks like your server is much older than the client, does this still happen with a new podman machine?

It seems like it's off whenever my machine goes to sleep and it starts back up. I usually don't notice it until my machine has been off for a day or two, and then it's obvious (due to failing tests in my codebase due to the date being off in psql that is running in a docker container). If I restart or recreate my podman machine, it will temporarily fix it, but it comes back.

[podollb]

~ podman machine ssh systemctl status chronyd
● chronyd.service - NTP client/server
     Loaded: loaded (/usr/lib/systemd/system/chronyd.service; enabled; preset: enabled)
    Drop-In: /usr/lib/systemd/system/service.d
             └─10-timeout-abort.conf
             /usr/lib/systemd/system/chronyd.service.d
             └─platform-chrony.conf
     Active: active (running) since Tue 2025-10-14 11:38:41 CDT; 1 day 3h ago
       Docs: man:chronyd(8)
             man:chrony.conf(5)
   Main PID: 999 (chronyd)
      Tasks: 1 (limit: 4158)
     Memory: 2.1M (peak: 3.3M)
        CPU: 191ms
     CGroup: /system.slice/chronyd.service
             └─999 /usr/sbin/chronyd -F 2

Oct 14 11:38:41 localhost systemd[1]: Started chronyd.service - NTP client/server.
Oct 14 11:41:05 localhost.localdomain chronyd[999]: Can't synchronise: no selectable sources
Oct 14 12:32:37 localhost.localdomain chronyd[999]: Source 172.234.37.140 replaced with 66.118.229.14 (2.fedora.pool.ntp.org)
Oct 14 12:49:56 localhost.localdomain chronyd[999]: Source 23.142.248.8 replaced with 15.204.87.223 (2.fedora.pool.ntp.org)
Oct 14 13:41:40 localhost.localdomain chronyd[999]: Source 96.231.54.40 replaced with 67.217.246.204 (2.fedora.pool.ntp.org)
Oct 14 14:10:20 localhost.localdomain chronyd[999]: Selected source 15.204.87.223 (2.fedora.pool.ntp.org)
Oct 14 14:10:20 localhost.localdomain chronyd[999]: System clock wrong by 83047.577379 seconds
Oct 15 13:14:28 localhost.localdomain chronyd[999]: System clock was stepped by 83047.577378 seconds
Oct 15 13:14:28 localhost.localdomain chronyd[999]: System clock TAI offset set to 37 seconds
Oct 15 13:19:54 localhost.localdomain chronyd[999]: Selected source 144.202.41.38 (2.fedora.pool.ntp.org)

And:

~ podman machine ssh timedatectl status
               Local time: Wed 2025-10-15 19:48:12 UTC
           Universal time: Wed 2025-10-15 19:48:12 UTC
                 RTC time: Wed 2025-10-15 19:48:13
                Time zone: n/a (UTC, +0000)
System clock synchronized: yes
              NTP service: active
          RTC in local TZ: no

[podollb]

I deleted/recreated my podman machine. Hopefully that helps...

~ podman version
Client:        Podman Engine
Version:       5.6.2
API Version:   5.6.2
Go Version:    go1.25.1
Built:         Tue Sep 30 09:50:46 2025
Build Origin:  brew
OS/Arch:       darwin/arm64

Server:       Podman Engine
Version:      5.6.2
API Version:  5.6.2
Go Version:   go1.24.7
Git Commit:   9dd5e1ed33830612bc200d7a13db00af6ab865a4
Built:        Mon Sep 29 19:00:00 2025
OS/Arch:      linux/arm64
~

[podollb]

Still no luck...problem came back again today. As you see, I issue the date command on my host machine (today is Oct. 16th) and my podman VM thinks it's Oct. 15

~ date
Thu Oct 16 10:18:06 CDT 2025


~ podman machine ssh timedatectl status
               Local time: Wed 2025-10-15 17:20:35 CDT
           Universal time: Wed 2025-10-15 22:20:35 UTC
                 RTC time: Wed 2025-10-15 22:20:35
                Time zone: America/Chicago (CDT, -0500)
System clock synchronized: yes
              NTP service: active
          RTC in local TZ: no


~ podman machine ssh systemctl status chronyd
● chronyd.service - NTP client/server
     Loaded: loaded (/usr/lib/systemd/system/chronyd.service; enabled; preset: enabled)
    Drop-In: /usr/lib/systemd/system/service.d
             └─10-timeout-abort.conf
             /usr/lib/systemd/system/chronyd.service.d
             └─platform-chrony.conf
     Active: active (running) since Wed 2025-10-15 15:01:31 CDT; 2h 19min ago
 Invocation: d420b5ede1ea46cfa287e1898c77eabc
       Docs: man:chronyd(8)
             man:chrony.conf(5)
   Main PID: 943 (chronyd)
      Tasks: 1 (limit: 9301)
     Memory: 1.9M (peak: 3.1M)
        CPU: 96ms
     CGroup: /system.slice/chronyd.service
             └─943 /usr/sbin/chronyd -n -F 2

Oct 15 16:17:14 localhost.localdomain chronyd[943]: Can't synchronise: no selectable sources (0 unreachable sources)
Oct 15 16:20:23 localhost.localdomain chronyd[943]: Root distance of 204.2.134.173 (2.fedora.pool.ntp.org) exceeds maxdistance of 3.000 seconds
Oct 15 16:20:25 localhost.localdomain chronyd[943]: Root distance of 204.93.207.12 (2.fedora.pool.ntp.org) exceeds maxdistance of 3.000 seconds
Oct 15 16:21:30 localhost.localdomain chronyd[943]: Jitter of 204.93.207.12 (2.fedora.pool.ntp.org) exceeds maxjitter of 1.000 seconds
Oct 15 16:21:31 localhost.localdomain chronyd[943]: Root distance of 45.79.82.45 (2.fedora.pool.ntp.org) exceeds maxdistance of 3.000 seconds
Oct 15 16:21:31 localhost.localdomain chronyd[943]: Root distance of 172.235.60.8 (2.fedora.pool.ntp.org) exceeds maxdistance of 3.000 seconds
Oct 15 16:22:35 localhost.localdomain chronyd[943]: Jitter of 45.79.82.45 (2.fedora.pool.ntp.org) exceeds maxjitter of 1.000 seconds
Oct 15 16:22:36 localhost.localdomain chronyd[943]: Jitter of 172.235.60.8 (2.fedora.pool.ntp.org) exceeds maxjitter of 1.000 seconds
Oct 15 16:23:37 localhost.localdomain chronyd[943]: Jitter of 204.2.134.173 (2.fedora.pool.ntp.org) exceeds maxjitter of 1.000 seconds
Oct 15 17:03:27 localhost.localdomain chronyd[943]: Source 45.79.82.45 replaced with 139.94.144.123 (2.fedora.pool.ntp.org)


~ podman version
Client:        Podman Engine
Version:       5.6.2
API Version:   5.6.2
Go Version:    go1.25.1
Built:         Tue Sep 30 09:50:46 2025
Build Origin:  brew
OS/Arch:       darwin/arm64

Server:       Podman Engine
Version:      5.6.2
API Version:  5.6.2
Go Version:   go1.24.7
Git Commit:   9dd5e1ed33830612bc200d7a13db00af6ab865a4
Built:        Mon Sep 29 19:00:00 2025
OS/Arch:      linux/arm64

Any other ideas?

If I restart my podman VM I can get it back to normal...

~ podman machine stop && podman machine start

Machine "podman-machine-default" stopped successfully
Starting machine "podman-machine-default"

This machine is currently configured in rootless mode. If your containers
require root permissions (e.g. ports < 1024), or if you run into compatibility
issues with non-podman clients, you can switch using the following command:

	podman machine set --rootful

API forwarding listening on: /var/run/docker.sock
Docker API clients default to this address. You do not need to set DOCKER_HOST.

Machine "podman-machine-default" started successfully
~
~ podman machine ssh timedatectl status
               Local time: Thu 2025-10-16 10:22:05 CDT
           Universal time: Thu 2025-10-16 15:22:05 UTC
                 RTC time: Thu 2025-10-16 15:22:05
                Time zone: America/Chicago (CDT, -0500)
System clock synchronized: no
              NTP service: active
          RTC in local TZ: no
~

[Honny1]

From the logs, it looks like chronyd is having trouble reaching the time server. Was your host machine asleep, disconnected from the internet for a while, or did it have a slow connection? @podollb

You can run this command to force a time sync without a restart: podman machine ssh sudo chronyc -a makestep.

Do applehv or libkrun have an option to 'Sync time with host'? @jakecorrenti

[Luap99]

Do applehv or libkrun have an option to 'Sync time with host'? @jakecorrenti

I think that is what @ninja-quokka was looking into as part of #26185 / containers/podman-machine-os#175, not sure if he has made progress on that since?

[ninja-quokka]

I didn't make any further progress on it. Vfkit supports timesync, krunkit doesn't.

Next steps where to research the other hypervisors we use to see if there was a common way we could implement this.

The crux of the issue is we should be using the host as the time source, not an external NTP server.

Due to the daemonless nature of Podman I could only image this working with support from the hypervisors.