Skip to content

refactor: extract telemetry span lifecycle helpers - #6788

Open
deepujain wants to merge 1 commit into
NVIDIA:mainfrom
deepujain:refactor/6631-extract-otel-spans
Open

deepujain wants to merge 1 commit into
NVIDIA:mainfrom
deepujain:refactor/6631-extract-otel-spans

Conversation

@deepujain

@deepujain deepujain commented Aug 24, 2026 •

Copy link
Copy Markdown
  • I, the PR author, have personally reviewed every line of this PR.

What does this PR do?

Large training modules are harder to review and maintain when telemetry lifecycle code is mixed into the training loop. This change moves OpenTelemetry span state and lifecycle helpers from megatron/training/training.py into megatron/training/telemetry_spans.py, while leaving the training call sites unchanged.

The extracted module owns startup and checkpoint-interval spans, shutdown state, and one-time atexit and SIGTERM hook installation. Startup timestamps are passed explicitly to avoid a circular dependency on training.py.

CPU-only tests cover startup closure, interval rerooting, one-time hook installation, hard SIGTERM teardown, and graceful SIGTERM flushing.

Issue tracking

Fixes #6631

Contribution process

Pre-checks

  • I have added relevant unit tests
  • I have added relevant functional tests
  • I have added proper typing to my code Typing guidelines
  • I have added relevant documentation
  • I have run the autoformatter.sh on my PR

Validation

  • Five focused lifecycle regressions passed in the earlier isolated CPU harness; the extracted module and tests are byte-identical on the repaired head.
  • python3 -m py_compile passed for all changed Python files on the repaired head.
  • git diff --check passed.
  • Independent exact-head review found no blocking lifecycle, compatibility, or conflict-resolution defects.

The repaired commit is rebased on current main and signed. Full package test collection on the local macOS host is blocked by the repository's unavailable Triton/GPU dependencies, so hosted GPU validation remains a maintainer-run gate.

@copy-pr-bot

copy-pr-bot Bot commented Aug 24, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@deepujain
deepujain force-pushed the refactor/6631-extract-otel-spans branch 2 times, most recently from 876532a to 561dc17 Compare September 3, 2026 02:17
@deepujain
deepujain force-pushed the refactor/6631-extract-otel-spans branch from 561dc17 to 6fbfa2e Compare September 24, 2026 21:50
Signed-off-by: dejain <deepujain@gmail.com>

Signed-off-by: Deepak Jain <deepujain@gmail.com>
@deepujain
deepujain force-pushed the refactor/6631-extract-otel-spans branch from 6fbfa2e to 29a7ea9 Compare September 29, 2026 23:26
@deepujain
deepujain marked this pull request as ready for review September 29, 2026 23:55
@deepujain
deepujain requested a review from a team as a code owner September 29, 2026 23:55
@svcnvidia-nemo-ci
svcnvidia-nemo-ci requested a review from a team September 29, 2026 23:55
@svcnvidia-nemo-ci svcnvidia-nemo-ci added the waiting-on-maintainers Waiting on maintainers to respond label Sep 30, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

community-request waiting-on-maintainers Waiting on maintainers to respond

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Extract the OpenTelemetry span-lifecycle helpers out of megatron/training/training.py

2 participants