[Feature] Free-Threaded Python (3.14t / nogil) Support for SGLang

[liusy58]

Motivation

CPython 3.13 introduced an experimental free-threaded build (PEP 703) that removes the Global Interpreter Lock (GIL). CPython 3.14t is the first release where this build is stable enough for production use, and the ecosystem of packages with free-threading wheels is growing rapidly.

vLLM has demonstrated (vllm-project/vllm#28762) that a large-scale CUDA-accelerated serving framework can run under a free-threaded interpreter. SGLang shares a similar dependency footprint and architecture. Enabling free-threaded support would unlock true CPU-level parallelism in the Python layer — benefiting the tokenizer manager, scheduler, HTTP server, disaggregation controller, and other asyncio + thread-pool components that today serialize behind the GIL.

This RFC tracks the dependency readiness, identifies the internal code changes needed, and proposes a phased plan to make uv pip install sglang work out of the box in a clean Python 3.14t environment on Linux x86-64 (CPU and CUDA).

Background

Term	Meaning
3.14t	The free-threaded (no-GIL) CPython 3.14 build, identified by the t suffix in the ABI tag (cp314t).
cp314	The regular (with-GIL) CPython 3.14 build.
Stable ABI	A subset of the CPython C API that is ABI-compatible across Python versions (PEP 384). Extensions built against the stable ABI only need one wheel per platform, not one per Python minor version.
Py_GIL_DISABLED	The compile-time macro set in free-threaded builds. C extensions must check this to use thread-safe reference counting and data structures.

Why 3.14t and not 3.13t?

• CPython 3.14t itself is significantly more stable than 3.13t.
• Key packages (cffi, aiohttp, etc.) support 3.14t but will not back-port to 3.13t.
• PyTorch 2.10.0 (January 2026) will ship full cp314t wheels; PyTorch 2.9.0 already has "preview" wheels.

Prerequisites

Free-threaded Python support depends on SGLang first supporting regular Python 3.14 (with-GIL). This RFC assumes that prerequisite is met or being tracked separately.

Dependency Audit

The following tables categorize every SGLang dependency with compiled (C/C++/Rust/CUDA) code by its current free-threading readiness. Pure-Python packages are omitted — they work automatically.

Core Dependencies (from pyproject.toml dependencies)

Package	Current Version	cp314t Wheels on PyPI	Builds from Source	Tracking Issue	Notes
torch	2.9.1	Preview (2.9.0); full in 2.10.0	Yes	—	Core dependency. 2.10.0 (Jan 2026) is the target.
torchaudio	2.9.1	Follows torch	Yes	—
torchvision	latest	Follows torch	Yes	—
torchao	0.9.0	Follows torch	TBD	—
sglang-kernel	0.4.1	❌ No	❌ Needs work	—	SGLang-owned. Uses stable ABI (cp310). Needs Py_GIL_DISABLED audit. See below.
flash-attn-4	≥4.0.0b4	❌ No	TBD	—	CUDA extension.
flashinfer_python	0.6.7.post2	❌ No	Resolved	flashinfer-ai/flashinfer#1687	Uses stable ABI; fix was to not use limited API with free-threaded Python.
flashinfer_cubin	0.6.7.post2	❌ No	TBD	—	Binary CUDA kernels, may need rebuild only.
cuda-python	12.9	TBD	TBD	—	NVIDIA-maintained.
xgrammar	0.1.32	✅ Yes (≥0.1.31)	✅ Yes	xgrammar#500	Full support.
llguidance	≥0.7.11	✅ Yes (≥1.6.0)	✅ Yes	llguidance#256	Full support.
msgspec	latest	✅ Yes (≥0.20.0)	✅ Yes	—	Full support.
outlines	0.1.11	TBD (outlines-core)	✅ Yes	outlines-core#248	Depends on outlines-core.
sentencepiece	latest	TBD	TBD	—	C++ extension.
tiktoken	latest	TBD	TBD	—	Rust extension.
orjson	latest	✅ Yes	✅ Yes	—	Rust extension, typically early adopter.
pyzmq	≥25.1.2	✅ Yes	✅ Yes	—
numpy	latest	✅ Yes	✅ Yes	—
scipy	latest	✅ Yes	✅ Yes	—
aiohttp	latest	✅ Yes	✅ Yes	—	Supports 3.14t.
pybase64	latest	TBD	TBD	—	C extension.
setproctitle	latest	TBD	TBD	—	C extension.
nvidia-ml-py	latest	Pure Python	N/A	—
pillow	latest	✅ Yes	✅ Yes	—
pydantic	latest	✅ Yes	✅ Yes	—	Rust-compiled core.
uvloop	latest	TBD	TBD	—	C extension (libuv). Critical for asyncio perf.
soundfile	0.13.1	TBD	TBD	—	CFFI-based.
compressed-tensors	latest	TBD	TBD	—
quack-kernels	≥0.3.0	❌ No	TBD	—	CUDA extension.
kernels	latest	TBD	TBD	—
smg-grpc-servicer	≥0.5.0	TBD	TBD	—	gRPC bindings.

Optional Dependencies (diffusion, ray, tracing)

Package	cp314t Status	Notes
ray	❌ No 3.14 support at all	Made optional in vLLM. SGLang already has it as optional ([ray]). Not blocking.
opencv-python-headless	❌ No	Diffusion extra only. Tracking: opencv/opencv#27933, build fix PR: opencv-python#1051.
xformers	✅ Yes (≥0.0.35)	Resolved by removing CPython C API dependency.

SGLang-Owned Compiled Components

Component	Language	Build System	Free-Threading Status
sgl-kernel	C++17 / CUDA	scikit-build-core + CMake	❌ Uses stable ABI (cp310). Needs Py_GIL_DISABLED audit and potentially per-version builds.
cpp_radix_tree (radix cache)	C++	pybind11 (torch extension)	❌ Needs thread-safety audit.
ngram_corpus_ffi	C++	pybind11 (torch extension)	❌ Needs thread-safety audit.
hf3fs_utils	C++	pybind11 (torch extension)	⚠️ Uses py::gil_scoped_release. Needs Py_GIL_DISABLED review.
sgl-model-gateway (router)	Rust (PyO3)	maturin	✅ PyO3 has free-threading support.
multimodal_gen CUDA kernels	C++ / CUDA	setuptools	❌ Needs audit.

Internal Code Audit

GIL-Dependent Patterns

The following patterns in SGLang's Python code rely on the GIL for correctness and must be reviewed:

1. Explicit GIL assumptions. staging_handler.py documents: "CPython GIL guarantees ordering" for flag visibility between threads. Under free-threading, this needs an explicit memory barrier or threading.Event.

2. py::gil_scoped_release in C++ extensions. hf3fs_utils.cpp uses py::gil_scoped_release to release the GIL during memcpy. Under free-threading, pybind11's gil_scoped_release is a no-op, but the surrounding data structures must be thread-safe without the GIL.

3. Shared mutable state across threads. The following areas use thread pools or daemon threads that share mutable Python objects:

   • TokenizerManager: ThreadPoolExecutor for tokenization with shared request queues
   • CacheController: background threads for cache management
   • KVEvents: daemon threads for ZMQ event distribution
   • weight_utils.py: multi-threaded safetensors loading with shared iterators

4. Module-level mutable globals. Several modules use module-level dicts/lists as registries (model registry, format registry, etc.). Under free-threading, concurrent imports or first-access initialization can race.

5. asyncio + ThreadPoolExecutor interaction. run_coroutine_threadsafe and call_soon_threadsafe are used throughout the disaggregation layer. These are asyncio-safe but the callbacks they schedule may access shared state without locks.

Thread-Safety Classification

Risk Level	Pattern	Example Locations	Mitigation
High	GIL-ordering assumptions	staging_handler.py:192	Replace with threading.Event or atomics
High	Shared mutable containers across threads	weight_utils.py (iterator sharing)	Add threading.Lock or use queue.Queue
Medium	Module-level mutable registries	Model/format registries	Use threading.Lock for lazy init
Medium	concurrent.futures with shared closures	async_dynamic_batch_tokenizer.py	Audit closure captures
Low	py::gil_scoped_release in C++	hf3fs_utils.cpp	Verify data is not shared with Python threads
Low	Pure asyncio code (single-threaded event loop)	HTTP server, tokenizer manager main loop	Safe by design (single-threaded)

Proposed Plan

Phase 0: Python 3.14 (with-GIL) Support

Goal: pip install sglang works on regular Python 3.14.

Phase 1: Dependency Readiness Tracking

Goal: All dependencies are installable in a 3.14t environment.

Phase 2: sgl-kernel Free-Threading Support

Goal: sglang-kernel builds and passes tests under 3.14t.

Phase 3: Internal C++ Extension Audit

Goal: All SGLang-owned C++ extensions are thread-safe without the GIL.

Phase 4: Python-Level Thread-Safety Fixes

Goal: SGLang's Python code does not rely on GIL for correctness.

Phase 5: CI and Release

Goal: Free-threaded builds are tested and released.

Phase 6: Performance Validation and Optimization

Goal: Quantify the benefit of free-threading for SGLang workloads.

References

• vLLM free-threaded Python tracking issue (vllm-project/vllm#18937)
• PEP 703 – Making the Global Interpreter Lock Optional in CPython
• CPython 3.14 free-threaded howto
• PyTorch free-threading support

Related resources

No response

[chz34]

I once tried build such package and I think orjson doesn't support free-threaded python either. cp314 wheel is provied on Pypi but not cp314t.

And get explicit error orjson v3.11.8 does not support free-threaded Python while trying build from source.

It can be built with an optional env, but I think it's not stable.

ORJSON_BUILD_FREETHREADED=1 maturin build

[rgommers]

Hi, great to see this issue, thanks for working on this @liusy58. A few quick notes on your table:

• sentencepiece, uvloop, tiktoken, pybase64 and setproctitle all have support and 314t wheels on PyPI
• flashinfer-cubin has only compiled CUDA kernels, and the minor amount of pure Python code it contains is thread-safe, it will work as is
• Some packages that still need work and their tracking issues:
  • cuda-python: EPIC: Support free-threading build NVIDIA/cuda-python#350
  • soundfile: Support the Python 3.14t free-threaded build bastibe/python-soundfile#467
  • torchao: TorchAO Missing 3.13T (free-threading) Wheels pytorch/ao#3243

[liusy58]

@rgommers Thank you! Would you be interested in supporting SGLang no-GIL? If so, I can add you as an assignee.

[rgommers]

Hi @liusy58, I don't have a ton of bandwidth, however I (together with others on my team) can offer input on dependencies, work on porting & thread-safety of some of the widely used dependencies that still need work (torchao and soundfile first I think), as well as PR review wherever that makes sense (our team has been working on free-threading for about two years now). We won't be able to take on auditing/porting SGLang itself or SGLang subprojects though.

[liusy58]

@rgommers Could you please provide some guidance on how to obtain the sentencepiece, uvloop, tiktoken, pybase64, and setproctitle packages that have pre-built (314t) wheels available on PyPI?

[rgommers]

Sure. My team maintains a tracker at https://py-free-threading.github.io/tracking/, it contains the status and (if available) first release that had full support with wheels on PyPI. I'd look there first - that says sentencepiece 0.2.1, tiktoken 0.12.0, and setproctitle 1.3.6. That tracker is manually maintained, so it won't contain everything. So for other packages, there's a bit more digging - I usually start by checking the latest release on PyPI for cp314t wheels, e.g. for uvloop: https://pypi.org/project/uvloop/#files. Based on release history, it's usually easy to check what the first version with such wheels was, in this case uvloop 0.22.1. For completeness, if I need to make sure there are no major open issues, I usually check for the relevant PR or tracking issue (e.g., MagicStack/uvloop#693). For pybase64 it's 1.4.2, and mayeut/pybase64#722.

[rgommers]

• soundfile 0.14.0 is compatible, it both builds from source and the py3-none-* wheels on PyPI work with free-threaded interpreters.
• The next torchao release (0.18.0) will build from source.
• kernels will not support free-threading before Python 3.15 (see huggingface/kernels#494).
• flash-attn-4 is pure Python and relies on CuTe DSL, which has full support for 3.14t including wheels from version 4.5.2, so that should all work now (table entries above should be Pure Python and N/A like for nvidia-ml-py.

It would be useful to update the table if you can @liusy58, to make it easier to see what's left.

[liusy58]

@rgommers I have an experimental PR up for Python 3.14 support: #25924

[github-actions]

This issue has been automatically closed due to inactivity. Please feel free to reopen it if needed.

[rgommers]

Are you able to reopen this issue @liusy58?