feat: agent resource sync API

[fregataa]

resolves #2142 https://github.com/lablup/giftbox/issues/262

Agents's sync-and-get-kernels() API

The API that synchronizes agent's kernels to kernel information specified by API parameters (preparing_kernels, pulling_kernels, running_kernels, terminating_kernels). It assumes that the kernel information given by the parameter is the "truth".
If any of kernel information mismatch between kernel_registry and running_kernels(or terminating_kernels), agent injects termination event to terminate the kernel.

sync-and-get-kernels() API returns actual { running, terminating, terminated } kernels (which is not used for now). actual_terminated_kernels contains terminated kernels specified as running_kernels by API parameter.

How to use

1. Call POST /session/_/sync-agent-resource manager API
2. Set a config to trigger resource-sync. it should be any of [after-scheduling, before-kernel-creation and on-creation-failure].

• on-creation-failure: Set by default. Call resource sync when kernel creation failed by InsufficientResource exception
• after-scheduling: Call resource sync right after scheduling on a scaling group
• before-kernel-creation: Call resource sync before calling create-kernels agent API

# manager.toml

[manager]
agent-resource-sync-trigger = ["after-scheduling"]
# Default is ["on-creation-failure"]
# agent-resource-sync-trigger = ["after-scheduling", "on-creation-failure"]

---

Note

on-creation-failure option cannot not handle ExceptionGroup including multiple InsufficientResource exceptions, which is raised by creation failure of multi kernel session. It covers only creation failure of single kernel session.
This will be resolved after merge lablup/callosum#30

Checklist: (if applicable)

• Milestone metadata specifying the target backport version
• Mention to the original issue
• API server-client counterparts (e.g., manager API -> client SDK)
• Documentation
  • Contents in the docs directory
  • docstrings in public interfaces and type annotations

---

📚 Documentation preview 📚: https://sorna--2180.org.readthedocs.build/en/2180/

---

📚 Documentation preview 📚: https://sorna-ko--2180.org.readthedocs.build/ko/2180/

[graphite-app]

Your org has enabled the Graphite merge queue for merging into main

Add the label “flow:merge-queue” to the PR and Graphite will automatically add it to the merge queue when it’s ready to merge. Or use the label “flow:hotfix” to add to the merge queue as a hot fix.

You must have a Graphite account and log in to Graphite in order to use the merge queue. Sign up using this link.

[fregataa]

Warning

This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
Learn more

• enhance: produce DoPrepareEvent after finishing scheduling in all scaling groups #2181
• feat: agent resource sync API #2180 👈
• feat: schedule function returns list of kernel-agent binding #2530
• feat: Impl Resource sync interface in agent side #2529
• main

This stack of pull requests is managed by Graphite. Learn more about stacking.

Join @fregataa and the rest of your teammates on Graphite

[fregataa]

Closed this PR because we will implement kernel heartbeat and revamp agent heartbeat, which leads to resolve this issue