Make the specializing interpreter thread-safe in `--disable-gil` builds #115999

swtaarrs · 2024-02-27T19:08:29Z

Feature or enhancement

Proposal:

In free-threaded builds, the specializing adaptive interpreter needs to be made thread-safe. We should start with a small PR to simply disable it in free-threaded builds, which will be correct but will incur a performance penalty. Then we can work out how to properly support specialization in a free-threaded build.

These two commits from Sam's nogil-3.12 branch can serve as inspiration:

There are two primary concerns to balance while implementing this functionality on main:

Runtime overhead: There should be no performance impact on normal builds, and minimal performance impact on single-threaded code running in free-threaded builds.
Reducing code duplication/divergence: We should come up with a design that is minimally disruptive to ongoing work on the specializing interpreter. It should be easy for other devs to keep the free-threaded build working without having to know too much about it.

Has this already been discussed elsewhere?

I have already discussed this feature proposal on Discourse

Links to previous discussion of this feature:

Specialization Families

Linked PRs

The text was updated successfully, but these errors were encountered:

brandtbucher · 2024-02-27T19:36:18Z

(subscribing myself)

…aded builds (#116013) For now, disable all specialization when the GIL might be disabled.

swtaarrs · 2024-03-01T17:12:51Z

This is now a performance (rather than correctness) issue for free-threaded builds, so I'm going to focus on more time-sensitive issues for a while.

…e-threaded builds (python#116013) For now, disable all specialization when the GIL might be disabled.

corona10 · 2024-04-20T06:30:07Z

@swtaarrs Out of curiosity, is there any progress or plan for this issue?

Fidget-Spinner · 2024-04-24T17:22:28Z

@corona10 I'm planning to work on this after I get the deferred reference stack in. However, there are no concrete plans as of now. I'm really happy for you or anyone else to propose a design for the specializing interpreter with free-threaded safety!

corona10 · 2024-04-25T01:23:43Z

@Fidget-Spinner cc @swtaarrs
Nice. I was also thinking about how to make it thread-safe in a seamless way since I agree with @swtaarrs.
But there is no good idea yet to solve the issue right now since I am not in a full-time position for this task :)
So it will be happy to see you have a good plan.
(I am curious that we can make them per-thread mechanism...)

By the way, in the short term, can we enable the specializer to be used only for the main thread if we can not solve the issue before 3.13 is released?
We can easily track the performance degradation between the default build because most of pyperformance benchmark are based on a single thread :)

Fidget-Spinner · 2024-04-26T13:09:59Z

@corona10 for 3.13, I think generally we're focusing on scalability across multicore rather than single-threaded perf for 3.13. It's a bit too near to feature freeze for me to feel safe re-enabling specialization at this point. There are a lot of unsolved problems still even with specialization only on the main thread. Consider the following:

Two threads sharing the same code object, A and B. A is main thread.
Thread B is in LOAD_ATTR_METHOD_WITH_VALUES's action (after guards, it is in the middle of loading from a method)
Thread A is in LOAD_ATTR_METHOD_WITH_VALUES's guard, but then deopts, meaning the method reference is now most likely dead/invalid.
Thread B loads from LOAD_ATTR_METHOD_WITH_VALUE's method, it is now holding a dangling pointer.
Thread B pushes dangling pointer to the stack. Everything crashes.

I'm reading a few papers to get some inspiration and also looking at how CRuby and other runtimes deal with this. Will post back when I have an actual plan.

Stop the world when invalidating function versions The tier1 interpreter specializes `CALL` instructions based on the values of certain function attributes (e.g. `__code__`, `__defaults__`). The tier1 interpreter uses function versions to verify that the attributes of a function during execution of a specialization match those seen during specialization. A function's version is initialized in `MAKE_FUNCTION` and is invalidated when any of the critical function attributes are changed. The tier1 interpreter stores the function version in the inline cache during specialization. A guard is used by the specialized instruction to verify that the version of the function on the operand stack matches the cached version (and therefore has all of the expected attributes). It is assumed that once the guard passes, all attributes will remain unchanged while executing the rest of the specialized instruction. Stopping the world when invalidating function versions ensures that all critical function attributes will remain unchanged after the function version guard passes in free-threaded builds. It's important to note that this is only true if the remainder of the specialized instruction does not enter and exit a stop-the-world point. We will stop the world the first time any of the following function attributes are mutated: - defaults - vectorcall - kwdefaults - closure - code This should happen rarely and only happens once per function, so the performance impact on majority of code should be minimal. Additionally, refactor the API for manipulating function versions to more clearly match the stated semantics.

…ython#124997) Stop the world when invalidating function versions The tier1 interpreter specializes `CALL` instructions based on the values of certain function attributes (e.g. `__code__`, `__defaults__`). The tier1 interpreter uses function versions to verify that the attributes of a function during execution of a specialization match those seen during specialization. A function's version is initialized in `MAKE_FUNCTION` and is invalidated when any of the critical function attributes are changed. The tier1 interpreter stores the function version in the inline cache during specialization. A guard is used by the specialized instruction to verify that the version of the function on the operand stack matches the cached version (and therefore has all of the expected attributes). It is assumed that once the guard passes, all attributes will remain unchanged while executing the rest of the specialized instruction. Stopping the world when invalidating function versions ensures that all critical function attributes will remain unchanged after the function version guard passes in free-threaded builds. It's important to note that this is only true if the remainder of the specialized instruction does not enter and exit a stop-the-world point. We will stop the world the first time any of the following function attributes are mutated: - defaults - vectorcall - kwdefaults - closure - code This should happen rarely and only happens once per function, so the performance impact on majority of code should be minimal. Additionally, refactor the API for manipulating function versions to more clearly match the stated semantics.

…thon#126607) Enable specialization of LOAD_GLOBAL in free-threaded builds. Thread-safety of specialization in free-threaded builds is provided by the following: A critical section is held on both the globals and builtins objects during specialization. This ensures we get an atomic view of both builtins and globals during specialization. Generation of new keys versions is made atomic in free-threaded builds. Existing helpers are used to atomically modify the opcode. Thread-safety of specialized instructions in free-threaded builds is provided by the following: Relaxed atomics are used when loading and storing dict keys versions. This avoids potential data races as the dict keys versions are read without holding the dictionary's per-object lock in version guards. Dicts keys objects are passed from keys version guards to the downstream uops. This ensures that we are loading from the correct offset in the keys object. Once a unicode key has been stored in a keys object for a combined dictionary in free-threaded builds, the offset that it is stored in will never be reused for a different key. Once the version guard passes, we know that we are reading from the correct offset. The dictionary read fast-path is used to read values from the dictionary once we know the correct offset.

…thongh-126616)

…E` (python#126600) Add free-threaded specialization for `UNPACK_SEQUENCE` opcode. `UNPACK_SEQUENCE_TUPLE/UNPACK_SEQUENCE_TWO_TUPLE` are already thread safe since tuples are immutable. `UNPACK_SEQUENCE_LIST` is not thread safe because of nature of lists (there is nothing preventing another thread from adding items to or removing them the list while the instruction is executing). To achieve thread safety we add a critical section to the implementation of `UNPACK_SEQUENCE_LIST`, especially around the parts where we check the size of the list and push items onto the stack. --------- Co-authored-by: Matt Page <[email protected]> Co-authored-by: mpage <[email protected]>

Record success in `specialize` This matches the existing behavior where we increment the success stat for the generic opcode each time we successfully specialize an instruction.

…ython#127169) The specialization only depends on the type, so no special thread-safety considerations there. STORE_SUBSCR_LIST_INT needs to lock the list before modifying it. `_PyDict_SetItem_Take2` already internally locks the dictionary using a critical section.

…BSCR (pythongh-127227)

…ongh-127514)

…pythongh-127128) Use existing helpers to atomically modify the bytecode. Add unit tests to ensure specializing is happening as expected. Add test_specialize.py that can be used with ThreadSanitizer to detect data races. Fix thread safety issue with cell_set_contents().

…h-127426) No additional thread safety changes are required. Note that sending to a generator that is shared between threads is currently not safe in the free-threaded build.

…-threaded builds (python#127123) The CALL family of instructions were mostly thread-safe already and only required a small number of changes, which are documented below. A few changes were needed to make CALL_ALLOC_AND_ENTER_INIT thread-safe: Added _PyType_LookupRefAndVersion, which returns the type version corresponding to the returned ref. Added _PyType_CacheInitForSpecialization, which takes an init method and the corresponding type version and only populates the specialization cache if the current type version matches the supplied version. This prevents potentially caching a stale value in free-threaded builds if we race with an update to __init__. Only cache __init__ functions that are deferred in free-threaded builds. This ensures that the reference to __init__ that is stored in the specialization cache is valid if the type version guard in _CHECK_AND_ALLOCATE_OBJECT passes. Fix a bug in _CREATE_INIT_FRAME where the frame is pushed to the stack on failure. A few other miscellaneous changes were also needed: Use {LOCK,UNLOCK}_OBJECT in LIST_APPEND. This ensures that the list's per-object lock is held while we are appending to it. Add missing co_tlbc for _Py_InitCleanup. Stop/start the world around setting the eval frame hook. This allows us to read interp->eval_frame non-atomically and preserves the behavior of _CHECK_PEP_523 documented below.

… free-threaded builds (#128164) Finish specialization for LOAD_ATTR in the free-threaded build by adding support for class and instance receivers.

Enable free-threaded specialization of LOAD_CONST.

Fix a few thread-safety bugs to enable test_opcache when run with TSAN: * Use relaxed atomics when clearing `ht->_spec_cache.getitem` (pythongh-115999) * Add temporary suppression for type slot modifications (pythongh-127266) * Use atomic load when reading `*dictptr`

Fix a few thread-safety bugs to enable test_opcache when run with TSAN: * Use relaxed atomics when clearing `ht->_spec_cache.getitem` (gh-115999) * Add temporary suppression for type slot modifications (gh-127266) * Use atomic load when reading `*dictptr`

Make tuple iteration more thread-safe, and actually test concurrent iteration of tuple, range and list. (This is prep work for enabling specialization of FOR_ITER in free-threaded builds.) The basic premise is: Iterating over a shared iterable (list, tuple or range) should be safe, not involve data races, and behave like iteration normally does. Using a shared iterator should not crash or involve data races, and should only produce items regular iteration would produce. It is not guaranteed to produce all items, or produce each item only once. (This is not the case for range iteration even after this PR.) Providing stronger guarantees is possible for some of these iterators, but it's not always straight-forward and can significantly hamper the common case. Since iterators in general aren't shared between threads, and it's simply impossible to concurrently use many iterators (like generators), better to make sharing iterators without explicit synchronization clearly wrong. Specific issues fixed in order to make the tests pass: - List iteration could occasionally fail an assertion when a shared list was shrunk and an item past the new end was retrieved concurrently. There's still some unsafety when deleting/inserting multiple items through for example slice assignment, which uses memmove/memcpy. - Tuple iteration could occasionally crash when the iterator's reference to the tuple was cleared on exhaustion. Like with list iteration, in free-threaded builds we can't safely and efficiently clear the iterator's reference to the iterable (doing it safely would mean extra, slow refcount operations), so just keep the iterable reference around.

Fix a few thread-safety bugs to enable test_opcache when run with TSAN: * Use relaxed atomics when clearing `ht->_spec_cache.getitem` (pythongh-115999) * Add temporary suppression for type slot modifications (pythongh-127266) * Use atomic load when reading `*dictptr` (cherry picked from commit f151d27) Co-authored-by: Sam Gross <[email protected]>

Fix a few thread-safety bugs to enable test_opcache when run with TSAN: * Use relaxed atomics when clearing `ht->_spec_cache.getitem` (gh-115999) * Add temporary suppression for type slot modifications (gh-127266) * Use atomic load when reading `*dictptr` (cherry picked from commit f151d27)

Add free-threaded versions of existing specialization for FOR_ITER (list, tuples, fast range iterators and generators), without significantly affecting their thread-safety. (Iterating over shared lists/tuples/ranges should be fine like before. Reusing iterators between threads is not fine, like before. Sharing generators between threads is a recipe for significant crashes, like before.)

corona10 · 2025-03-15T13:39:34Z

@mpage @colesbury @Yhg1s Now we can close the issue right since #128798 is merged?

colesbury · 2025-03-15T13:40:51Z

Yes

corona10 · 2025-03-15T13:41:08Z

And maybe worth to update What's new for it?

…n#128798) Add free-threaded versions of existing specialization for FOR_ITER (list, tuples, fast range iterators and generators), without significantly affecting their thread-safety. (Iterating over shared lists/tuples/ranges should be fine like before. Reusing iterators between threads is not fine, like before. Sharing generators between threads is a recipe for significant crashes, like before.)

swtaarrs added type-feature A feature request or enhancement topic-free-threading labels Feb 27, 2024

swtaarrs self-assigned this Feb 27, 2024

swtaarrs mentioned this issue Feb 27, 2024

PEP 703 -- Making the Global Interpreter Lock Optional in CPython #108219

Open

swtaarrs mentioned this issue Feb 27, 2024

gh-115999: Disable the specializing adaptive interpreter in free-threaded builds #116013

Merged

colesbury pushed a commit that referenced this issue Mar 1, 2024

gh-115999: Disable the specializing adaptive interpreter in free-thre…

339c8e1

…aded builds (#116013) For now, disable all specialization when the GIL might be disabled.

swtaarrs removed their assignment Mar 1, 2024

woodruffw pushed a commit to woodruffw-forks/cpython that referenced this issue Mar 4, 2024

pythongh-115999: Disable the specializing adaptive interpreter in fre…

fad3958

…e-threaded builds (python#116013) For now, disable all specialization when the GIL might be disabled.

adorilson pushed a commit to adorilson/cpython that referenced this issue Mar 25, 2024

pythongh-115999: Disable the specializing adaptive interpreter in fre…

96d4fbd

…e-threaded builds (python#116013) For now, disable all specialization when the GIL might be disabled.

diegorusso pushed a commit to diegorusso/cpython that referenced this issue Apr 17, 2024

pythongh-115999: Disable the specializing adaptive interpreter in fre…

d31dae8

…e-threaded builds (python#116013) For now, disable all specialization when the GIL might be disabled.

mpage self-assigned this Aug 8, 2024

bedevere-app bot mentioned this issue Sep 10, 2024

gh-115999: Implement thread-local bytecode and enable specialization for BINARY_OP #123926

Merged

mpage added a commit to mpage/cpython that referenced this issue Sep 13, 2024

Merge branch 'main' into pythongh-115999-thread-local-bytecode

d34adeb

mpage added a commit to mpage/cpython that referenced this issue Sep 17, 2024

Merge branch 'main' into pythongh-115999-thread-local-bytecode

c2d8693

mpage added a commit to mpage/cpython that referenced this issue Sep 25, 2024

Merge branch 'main' into pythongh-115999-thread-local-bytecode

aa330b1

mpage added a commit to mpage/cpython that referenced this issue Sep 26, 2024

Merge branch 'main' into pythongh-115999-thread-local-bytecode

7dfd1ca

mpage added a commit to mpage/cpython that referenced this issue Sep 28, 2024

Merge branch 'main' into pythongh-115999-thread-local-bytecode

dd144d0

mpage added a commit to mpage/cpython that referenced this issue Sep 30, 2024

Merge branch 'main' into pythongh-115999-thread-local-bytecode

b6380de

This was referenced Oct 3, 2024

gh-115999: Refactor LOAD_GLOBAL specializations to avoid reloading {globals, builtins} keys #124953

Merged

gh-115999: Stop the world when invalidating function versions #124997

Merged

mpage added a commit to mpage/cpython that referenced this issue Oct 5, 2024

Merge branch 'main' into pythongh-115999-thread-local-bytecode

adb59ef

mpage added a commit to mpage/cpython that referenced this issue Oct 7, 2024

Merge branch 'main' into pythongh-115999-refactor-load-global

e68f7e0

ebonnal pushed a commit to ebonnal/cpython that referenced this issue Jan 12, 2025

pythongh-115999: Add free-threaded specialization for TO_BOOL (py…

0da364f

…thongh-126616)

ebonnal pushed a commit to ebonnal/cpython that referenced this issue Jan 12, 2025

pythongh-115999: Add partial free-thread specialization for BINARY_SU…

89cd78d

…BSCR (pythongh-127227)

ebonnal pushed a commit to ebonnal/cpython that referenced this issue Jan 12, 2025

pythongh-115999: Use light-weight lock for UNPACK_SEQUENCE_LIST (pyth…

eb5179f

…ongh-127514)

bedevere-app bot mentioned this issue Jan 13, 2025

gh-115999: Add free-threaded specialization for FOR_ITER #128798

Merged

mpage added a commit to mpage/cpython that referenced this issue Jan 14, 2025

Merge branch 'main' into pythongh-115999-load-attr-instance-merged

8c70ea4

mpage added a commit that referenced this issue Jan 14, 2025

gh-115999: Specialize LOAD_ATTR for instance and class receivers in…

b5ee025

… free-threaded builds (#128164) Finish specialization for LOAD_ATTR in the free-threaded build by adding support for class and instance receivers.

bedevere-app bot mentioned this issue Jan 27, 2025

gh-115999: Enable free-threaded specialization of LOAD_CONST #129365

Merged

Yhg1s added a commit that referenced this issue Jan 29, 2025

gh-115999: Enable free-threaded specialization of LOAD_CONST (#129365)

5c930a2

Enable free-threaded specialization of LOAD_CONST.

colesbury mentioned this issue Feb 7, 2025

gh-117657: Enable test_opcache under TSAN #129831

Merged

colesbury mentioned this issue Feb 26, 2025

[3.13] gh-117657: Enable test_opcache under TSAN (GH-129831) #130597

Merged

bedevere-app bot mentioned this issue Mar 15, 2025

gh-115999: Update whats news for free-threaded 3.14 #131285

Open

picnixz added the interpreter-core (Objects, Python, Grammar, and Parser dirs) label Mar 15, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Make the specializing interpreter thread-safe in `--disable-gil` builds #115999

Make the specializing interpreter thread-safe in `--disable-gil` builds #115999

swtaarrs commented Feb 27, 2024 •

edited by bedevere-app bot

Loading

Tasks

brandtbucher commented Feb 27, 2024

swtaarrs commented Mar 1, 2024

corona10 commented Apr 20, 2024

Fidget-Spinner commented Apr 24, 2024

corona10 commented Apr 25, 2024

Fidget-Spinner commented Apr 26, 2024

corona10 commented Mar 15, 2025

colesbury commented Mar 15, 2025

corona10 commented Mar 15, 2025

Make the specializing interpreter thread-safe in --disable-gil builds #115999

Make the specializing interpreter thread-safe in --disable-gil builds #115999

Comments

swtaarrs commented Feb 27, 2024 • edited by bedevere-app bot Loading

Feature or enhancement

Proposal:

Has this already been discussed elsewhere?

Links to previous discussion of this feature:

Specialization Families

Tasks

Linked PRs

brandtbucher commented Feb 27, 2024

swtaarrs commented Mar 1, 2024

corona10 commented Apr 20, 2024

Fidget-Spinner commented Apr 24, 2024

corona10 commented Apr 25, 2024

Fidget-Spinner commented Apr 26, 2024

corona10 commented Mar 15, 2025

colesbury commented Mar 15, 2025

corona10 commented Mar 15, 2025

Make the specializing interpreter thread-safe in `--disable-gil` builds #115999

Make the specializing interpreter thread-safe in `--disable-gil` builds #115999

swtaarrs commented Feb 27, 2024 •

edited by bedevere-app bot

Loading