Refined Kubernetes scheduler implementation

[senseipri]

Description

Adds a Kubernetes-backed scheduler implementation for AReaL using StatefulSet-based worker orchestration.

This PR:

• implements KubernetesScheduler
• reuses existing HTTP guard APIs
• adds Kubernetes Python client integration
• adds pod health diagnostics and rollback handling
• mirrors local/Slurm scheduler semantics
• adds Kubernetes documentation (EN + ZH)
• adds unit and optional integration tests

Related Issue

#Native Kubernetes (K8S) scheduler [CC]

Type of Change

• 🐛 Bug fix
• ✨ New feature
• 💥 Breaking change
• 📝 Documentation update
• ♻️ Refactoring
• ⚡ Performance improvement
• ✅ Test coverage improvement

Checklist

• I have read the Contributing Guide
• Pre-commit hooks pass (pre-commit run --all-files)
• Relevant tests pass; new tests added for new functionality
• Documentation updated (if applicable; built with ./docs/build_all.sh)
• Branch is up to date with main
• Self-reviewed via /review-pr command
• This PR was created by a coding agent via /create-pr
• This PR is a breaking change

Breaking Change Details (if applicable):

N/A

Additional Context

Design highlights:

• Uses StatefulSets for stable worker identity and ordinal mapping
• Uses Kubernetes Python client instead of kubectl subprocesses
• Includes scoped pod selectors, diagnostics, and fork rollback semantics
• Explicitly rejects unsupported Slurm-specific scheduling semantics

---

Need help? Check the Contributing Guide or ask in
GitHub Discussions!

[gemini-code-assist]

Code Review

This pull request introduces the KubernetesScheduler, enabling AReaL to deploy and manage worker groups on Kubernetes using StatefulSets and headless Services. The implementation includes integration with existing trainers, comprehensive documentation in English and Chinese, and new unit and integration tests. The review feedback focuses on improving the robustness of the scheduler by using more specific exception handling for Kubernetes and network requests, and simplifying the code by removing redundant attribute fallbacks.

[gemini-code-assist]

The except Exception: is too broad. It's better to catch requests.RequestException to handle network-related errors specifically, without masking other potential issues.

Suggested change

	except Exception:
	except requests.RequestException:

[gemini-code-assist]

The exception handling for requests errors can be simplified. requests.exceptions.Timeout and requests.exceptions.ConnectionError are both subclasses of requests.exceptions.RequestException. You can catch the base class to cover all request-related errors.

Suggested change

	except requests.exceptions.Timeout as e:
	last_error = f"Timeout: {e}"
	except requests.exceptions.ConnectionError as e:
	self._check_pods_health(health_role)
	last_error = f"Connection error: {e}"
	except requests.exceptions.RequestException as e:
	self._check_pods_health(health_role)
	last_error = f"Request error: {e}"

[garrett4wade]

Review Findings

1. Unused health_role in create_engine (MEDIUM)

In areal/infra/scheduler/kubernetes.py, create_engine assigns health_role = self._colocated_roles.get(wi.role, wi.role) but never uses it. Both call_engine and async_call_engine use health_role with self._check_pods_health(health_role) during retry loops, so this looks like a copy-paste leftover where the health check was forgotten.

Suggestion: Either remove the dead assignment, or add a self._check_pods_health(health_role) call before the HTTP request for consistency with call_engine.

2. Redundant exit code extraction in _check_pods_health (MEDIUM)

The exit code is extracted twice with different defaults:

exit_code = int(_obj_get(terminated, "exit_code", _obj_get(terminated, "exitCode", 0)))
if terminated and exit_code != 0:
    exit_code = int(_obj_get(terminated, "exit_code", _obj_get(terminated, "exitCode", -1)))
    raise WorkerFailedError(...)

The second extraction is dead code — if the first returned non-zero, the second returns the identical value. If the first returned 0 (field missing), the branch is skipped. Additionally, a terminated container with no exitCode field silently defaults to 0 (success), which could mask edge cases.

Suggestion: Simplify to a single extraction, and consider defaulting to -1 for terminated containers with missing exit codes:

if terminated:
    exit_code = int(_obj_get(terminated, "exit_code", _obj_get(terminated, "exitCode", -1)))
    if exit_code != 0:
        raise WorkerFailedError(...)

3. PR title typo (LOW)

"schedular" → "scheduler"

4. Residual risk: No RBAC guidance in docs

The docs mention service account permissions but don't provide a sample ClusterRole/RoleBinding manifest. This is the most common setup friction for K8s integrations — users will need to figure out the exact verbs and resources themselves. Consider adding a minimal RBAC example covering Services, StatefulSets, Pods, pod logs, and pod events.

[senseipri]

Thanks for the detailed review and suggestions, I will address the requested fixes and push the updates. Really appreciate the feedback on both the Kubernetes integration and the scheduler semantics consistency.

[senseipri]

@garrett4wade I guess I have resolved the issues in the native kubernetes PR. Can you please see if it is merge ready?

[sitabulaixizawaluduo]

Thanks for contribute, please run "pre-commit" before submit and rebase your branch.