Skip to content

Conversation

@llfl
Copy link
Contributor

@llfl llfl commented Dec 5, 2025

Motivation

The following error may occur when executing self_check_during_idle() in scheduler_runtime_checker_mixin.py:

[2025-12-04 14:56:05 PP0 TP1] Scheduler hit an exception: Traceback (most recent call last):
  File "/sources/sglang_debug/python/sglang/srt/managers/scheduler.py", line 2678, in run_scheduler_process
    scheduler.event_loop_pp()
  File "/venv_sglang_sources_sglang_debug_py312/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/sources/sglang_debug/python/sglang/srt/managers/scheduler_pp_mixin.py", line 135, in event_loop_pp
    self.self_check_during_idle()
  File "/sources/sglang_debug/python/sglang/srt/managers/scheduler_runtime_checker_mixin.py", line 300, in self_check_during_idle
    self.check_memory()
  File "/sources/sglang_debug/python/sglang/srt/managers/scheduler_runtime_checker_mixin.py", line 221, in check_memory
    self._check_req_pool()
  File "/sources/sglang_debug/python/sglang/srt/managers/scheduler_runtime_checker_mixin.py", line 197, in _check_req_pool
    raise_error_or_warn(
  File "/sources/sglang_debug/python/sglang/srt/utils/common.py", line 3788, in raise_error_or_warn
    if strict:
       ^^^^^^
  File "/sources/sglang_debug/python/sglang/srt/environ.py", line 78, in __bool__
    raise RuntimeError(
RuntimeError: Please use `envs.YOUR_FLAG.get()` instead of `envs.YOUR_FLAG`

Although this error occurs within an exception context, the Environ Error prevents further exception details from being printed, making it impossible to obtain the correct exception information.

Modifications

Replace all instances of envs.YOUR_FLAG with envs.YOUR_FLAG.get().

Result

[2025-12-04 17:02:09 PP0 TP0] Scheduler hit an exception: Traceback (most recent call last):
  File "/sources/sglang_debug/python/sglang/srt/managers/scheduler.py", line 2678, in run_scheduler_process
    scheduler.event_loop_pp()
  File "/venv_sglang_sources_sglang_debug_py312/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/sources/sglang_debug/python/sglang/srt/managers/scheduler_pp_mixin.py", line 135, in event_loop_pp
    self.self_check_during_idle()
  File "/sources/sglang_debug/python/sglang/srt/managers/scheduler_runtime_checker_mixin.py", line 300, in self_check_during_idle
    self.check_memory()
  File "/sources/sglang_debug/python/sglang/srt/managers/scheduler_runtime_checker_mixin.py", line 221, in check_memory
    self._check_req_pool()
  File "/sources/sglang_debug/python/sglang/srt/managers/scheduler_runtime_checker_mixin.py", line 197, in _check_req_pool
    raise_error_or_warn(
  File "/sources/sglang_debug/python/sglang/srt/utils/common.py", line 3789, in raise_error_or_warn
    raise ValueError(message)
ValueError: req_to_token_pool memory leak detected!available_size=2050, total_size=2048

The correct exception information has been printed.

Signed-off-by: Kun(llfl) <[email protected]>
@gemini-code-assist
Copy link
Contributor

Warning

You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again!

Copy link
Collaborator

@ShangmingCai ShangmingCai left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. A clean fix.

@ShangmingCai
Copy link
Collaborator

/tag-and-rerun-ci

@github-actions github-actions bot added the run-ci label Dec 5, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants