Skip to content

Conversation

stepanblyschak
Copy link
Contributor

What I did

I implemented a fix to stop timers upon reload/load_minigraph. If "config reload" or "config load_minigraph" is executed soon after system has started a timer may expire in the middle of the command causing subsequent failures.

Here's what is observed:

2025 Aug  6 18:29:35.532425 sonic INFO python[36429]: ansible-command Invoked with _uses_shell=True _raw_params=config load_minigraph --override_config -y warn=True stdin_add_newline=True strip_empty_ends=True argv=None chdir=None execut
able=None creates=None removes=None stdin=None
2025 Aug  6 18:29:35.754152 sonic NOTICE switch_trimming: 'load_minigraph' executing with command: config load_minigraph --override_config -y
2025 Aug  6 18:30:30.306602 sonic INFO systemd[1]: Stopped target sonic.target - SONiC services target..

2025 Aug  6 18:30:31.333507 sonic DEBUG systemd[1]: tacacs-config.timer: Timer elapsed.
2025 Aug  6 18:30:31.334003 sonic DEBUG systemd[1]: sonic.target: Installed new job sonic.target/start as 2793
2025 Aug  6 18:30:32.371292 sonic DEBUG systemd[1]: sonic.target changed dead -> active

How I did it

Stop timers associated with sonic.target

How to verify it

I put a breakpoint after stopping sonic.target. Soon after reboot I execute "config reload" command. I then wait for ~5-6 min, no services start. Without this fix, services would start.

Previous command output (if the output of a command-line utility has changed)

New command output (if the output of a command-line utility has changed)

Signed-off-by: Stepan Blyschak <[email protected]>
@mssonicbld
Copy link
Collaborator

/azp run

Copy link

Azure Pipelines successfully started running 1 pipeline(s).

@mssonicbld
Copy link
Collaborator

/azp run

Copy link

Azure Pipelines successfully started running 1 pipeline(s).

@dgsudharsan
Copy link
Collaborator

@stepanblyschak Lets have the label until Azure/sonic-utilities.msft#226 is merged. @r12f This is an issue we found in 202412 testing. Please help to prioritize

@r12f
Copy link

r12f commented Aug 16, 2025

@stepanblyschak Lets have the label until Azure/sonic-utilities.msft#226 is merged. @r12f This is an issue we found in 202412 testing. Please help to prioritize

thanks Sud! merged in 202412 already.


# Get the list of dependencies for sonic.target to fetch timer units
for service in _get_sonic_services():
if service.endswith('.timer'):
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

service

Just curios, these services are already dependencies of sonic.target. Why original code does not stop the them?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here's one of the timers:

# /lib/systemd/system/hostcfgd.timer
[Unit]
Description=Delays hostcfgd daemon until SONiC has started
PartOf=hostcfgd.service

[Timer]
OnUnitActiveSec=0 sec
OnBootSec=1min 30 sec
Unit=hostcfgd.service

[Install]
WantedBy=timers.target sonic.target

The relation between sonic.target and the timer is only WantedBy, meaning start of sonic.target starts the timer but does not stop it

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants