Manage CPython thread states automatically across Julia tasks

AGENTS.md

@@ -5,7 +5,7 @@
 - **Python tests**:
   - Copy `pysrc/juliacall/juliapkg-dev.json` to `pysrc/juliacall/juliapkg.json` before running (do **not** commit this copy).
   - Execute with `uv run pytest -s --nbval ./pytest` (add `--cov=pysrc` when coverage is needed).
-  - Sometimes `juliapkg` requires Julia 1.10–1.11; `juliaup` already provides 1.11.7 in this environment.
+- Sometimes `juliapkg` requires Julia 1.10–1.11; `juliaup` already provides 1.11.7 in this environment.
 
 The majority of tests live in the Julia package; Python tests cover functionality that cannot be exercised from Julia (e.g., JuliaCall-specific behavior). Run both suites—typically Julia first—in whichever order makes sense.
 

docs/src/juliacall.md

@@ -157,13 +157,11 @@ caveats.
 
 Most importantly, you can only call Python code while Python's
 [Global Interpreter Lock (GIL)](https://docs.python.org/3/glossary.html#term-global-interpreter-lock)
-is locked by the current thread. You can use JuliaCall from any Python thread, and the GIL
-will be locked whenever any JuliaCall function is used. However, to leverage the benefits
-of multi-threading, you can unlock the GIL while executing any Julia code that does not
-interact with Python.
+JuliaCall borrows the Python thread state which entered Julia and automatically detaches it
+while arbitrary Julia code runs. Nested Python interaction from that Julia code temporarily
+reattaches the same state, and the borrowed state is restored before returning to Python.
 
-The simplest way to do this is using the `_jl_call_nogil` method on Julia functions to
-call the function with the GIL unlocked.
+The historical `_jl_call_nogil` spelling remains available as a compatibility alias:
 
 ```python
 from concurrent.futures import ThreadPoolExecutor, wait
@@ -173,16 +171,9 @@ fs = [pool.submit(jl.Libc.systemsleep._jl_call_nogil, 5) for _ in range(4)]
 wait(fs)
 ```
 
-In the above example, we call `Libc.systemsleep(5)` on four threads. Because we
-called it with `_jl_call_nogil`, the GIL was unlocked, allowing the threads to run in
-parallel, taking about 5 seconds in total.
-
-If we did not use `_jl_call_nogil` (i.e. if we did `pool.submit(jl.Libc.systemsleep, 5)`)
-then the above code will take 20 seconds because the sleeps run one after another.
-
-It is very important that any function called with `_jl_call_nogil` does not interact
-with Python at all unless it re-locks the GIL first, such as by using
-[PythonCall.GIL.@lock](@ref).
+Ordinary calls provide the same automatic resource management, so
+`pool.submit(jl.Libc.systemsleep, 5)` is preferred. PythonCall operations nested inside
+Julia callbacks are safe without explicit region or lock calls.
 
 You can also use [multi-threading from Julia](@ref jl-multi-threading).
 

docs/src/pythoncall-reference.md

@@ -222,8 +222,15 @@ Py(x::MyType) = x.py
 
 ## Multi-threading
 
-These functions are not exported. They support multi-threading of Python and/or Julia.
-See also [`juliacall.AnyValue._jl_call_nogil`](@ref julia-wrappers).
+PythonCall manages Python thread state automatically. These exported macros are optional
+performance and concurrency hints; users normally do not need them for correctness.
+
+```@docs
+@pyregion
+@pyregionbreak
+```
+
+The older `PythonCall.GIL` names remain as compatibility aliases.
 
 ```@docs
 PythonCall.GIL.lock

docs/src/pythoncall.md

@@ -471,42 +471,35 @@ See [Installing Python packages](@ref python-deps).
 
     Multi-threading support is experimental and can change without notice.
 
-From v0.9.22, PythonCall supports multi-threading in Julia and/or Python, with some
-caveats.
-
-Most importantly, you can only call Python code while Python's
-[Global Interpreter Lock (GIL)](https://docs.python.org/3/glossary.html#term-global-interpreter-lock)
-is locked by the current thread. Ordinarily, the GIL is locked by the main thread in Julia,
-so if you want to run Python code on any other thread, you must unlock the GIL from the
-main thread and then re-lock it while running any Python code on other threads.
-
-This is made possible by the macros [`PythonCall.GIL.@unlock`](@ref) and
-[`PythonCall.GIL.@lock`](@ref) or the functions [`PythonCall.GIL.unlock`](@ref) and
-[`PythonCall.GIL.lock`](@ref) with this pattern:
+PythonCall APIs automatically establish the Python thread state they need, so ordinary
+operations can be called from any Julia task or thread without explicit locking. The
+optional [`@pyregion`](@ref) macro amortizes those transitions across straight-line,
+Python-heavy work. Use [`@pyregionbreak`](@ref) around Julia-heavy code which deliberately
+yields, waits, or blocks cooperatively:
 
 ```julia
-PythonCall.GIL.@unlock Threads.@threads for i in 1:4
-  PythonCall.GIL.@lock pyimport("time").sleep(5)
+Threads.@threads for i in 1:4
+  @pyregion pyimport("time").sleep(5)
 end
 ```
 
 In the above example, we call `time.sleep(5)` four times in parallel. If Julia was
 started with at least four threads (`julia -t4`) then the above code will take about
 5 seconds.
 
-Both `@unlock` and `@lock` are important. If the GIL were not unlocked, then a deadlock
-would occur when attempting to lock the already-locked GIL from the threads. If the GIL
-were not re-locked, then Python would crash when interacting with it.
+Both region macros nest arbitrarily, and neither is required for correctness. A nested
+PythonCall operation inside `@pyregionbreak` temporarily re-enters Python automatically.
+On a GIL-enabled Python, attaching a state can block that Julia worker while CPython
+arbitrates access; free-threaded Python uses the same state-management machinery.
 
 With multiple Julia threads you need exactly one interactive thread, see the [FAQ](@ref faq-multi-threading).
 
 You can also use [multi-threading from Python](@ref py-multi-threading).
 
 ### Caveat: Garbage collection
 
-If Julia's GC collects any Python objects from a thread where the GIL is not currently
-locked, then those Python objects will not immediately be deleted. Instead they will be
-queued to be deleted in a later GC pass.
+If Julia's GC collects Python objects while no Python thread state is already attached,
+those objects are queued rather than making a finalizer block while attaching a state.
 
 If you find you have many Python objects not being deleted, you can call
-[`PythonCall.GC.gc()`](@ref) or `GC.gc()` while the GIL is locked to clear the queue.
+[`PythonCall.GC.gc()`](@ref) or `GC.gc()` to clear the queue.

pytest/test_all.py

@@ -105,6 +105,8 @@ def test_julia_gc():
             end
         end
         GC.gc()
+        @test !isempty(PythonCall.GC.QUEUE.items)
+        PythonCall.GC.gc()
         @test isempty(PythonCall.GC.QUEUE.items)
         """
     )

src/API/exports.jl

@@ -3,6 +3,8 @@ export @py
 export @pyconst
 export @pyeval
 export @pyexec
+export @pyregion
+export @pyregionbreak
 export ispy
 export Py
 export pyabs

src/API/macros.jl

@@ -2,6 +2,8 @@
 macro pyconst end
 macro pyeval end
 macro pyexec end
+macro pyregion end
+macro pyregionbreak end
 
 # Convert
 macro pyconvert end

src/C/context.jl

@@ -21,6 +21,11 @@ A handle to a loaded instance of libpython, its interpreter, function pointers,
 end
 
 const CTX = Context()
+const FINALIZE_HOOK = Ref{Function}(() -> begin
+    if Py_FinalizeEx() == -1
+        @warn "Py_FinalizeEx() error"
+    end
+end)
 
 function _atpyexit()
     if CTX.is_initialized && !CTX.is_preinitialized
@@ -282,9 +287,7 @@ function init_context()
             Py_InitializeEx(0)
             atexit() do
                 CTX.is_initialized = false
-                if Py_FinalizeEx() == -1
-                    @warn "Py_FinalizeEx() error"
-                end
+                FINALIZE_HOOK[]()
             end
         end
         CTX.is_initialized = true

src/C/pointers.jl

@@ -19,6 +19,8 @@ const CAPI_FUNC_SIGS = Dict{Symbol,Pair{Tuple,Type}}(
     # GIL & THREADS
     :PyEval_SaveThread => () => Ptr{Cvoid},
     :PyEval_RestoreThread => (Ptr{Cvoid},) => Cvoid,
+    :PyThreadState_New => (Ptr{Cvoid},) => Ptr{Cvoid},
+    :PyThreadState_GetInterpreter => (Ptr{Cvoid},) => Ptr{Cvoid},
     :PyGILState_Ensure => () => PyGILState_STATE,
     :PyGILState_Release => (PyGILState_STATE,) => Cvoid,
     :PyGILState_GetThisThreadState => () => Ptr{Cvoid},
@@ -278,6 +280,7 @@ const CAPI_OBJECTS = Set([
     $([:($name::PyPtr = C_NULL) for name in CAPI_EXCEPTIONS]...)
     $([:($name::PyPtr = C_NULL) for name in CAPI_OBJECTS]...)
     PyOS_InputHookPtr::Ptr{Ptr{Cvoid}} = C_NULL
+    PyThreadState_GetUnchecked::Ptr{Cvoid} = C_NULL
 end
 
 const POINTERS = CAPIPointers()
@@ -295,8 +298,14 @@ const POINTERS = CAPIPointers()
     )
     $([:(p.$name = dlsym(lib, $(QuoteNode(name)))) for name in CAPI_OBJECTS]...)
     p.PyOS_InputHookPtr = dlsym(CTX.lib_ptr, :PyOS_InputHook)
+    p.PyThreadState_GetUnchecked = let q = dlsym_e(lib, :PyThreadState_GetUnchecked)
+        q == C_NULL ? dlsym(lib, :_PyThreadState_UncheckedGet) : q
+    end
 end
 
+PyThreadState_GetUnchecked() =
+    ccall(POINTERS.PyThreadState_GetUnchecked, Ptr{Cvoid}, ())
+
 for (name, (argtypes, rettype)) in CAPI_FUNC_SIGS
     args = [Symbol("x", i) for (i, _) in enumerate(argtypes)]
     @eval $name($(args...)) = ccall(POINTERS.$name, $rettype, ($(argtypes...),), $(args...))

src/Convert/pyconvert.jl

@@ -369,13 +369,15 @@ On failure, evaluates to `onfail`, which defaults to `return pyconvert_unconvert
 """
 macro pyconvert(T, x, onfail = :(return $pyconvert_unconverted()))
     quote
-        T = $(esc(T))
-        x = $(esc(x))
-        ans = pytryconvert(T, x)
-        if pyconvert_isunconverted(ans)
-            $(esc(onfail))
-        else
-            pyconvert_result(T, ans)
+        @pyregion begin
+            T = $(esc(T))
+            x = $(esc(x))
+            ans = pytryconvert(T, x)
+            if pyconvert_isunconverted(ans)
+                $(esc(onfail))
+            else
+                pyconvert_result(T, ans)
+            end
         end
     end
 end
@@ -387,10 +389,19 @@ Convert the Python object `x` to a `T`.
 
 If `d` is specified, it is returned on failure instead of throwing an error.
 """
-pyconvert(::Type{T}, x) where {T} = @autopy x @pyconvert T x_ error(
-    "cannot convert this Python '$(pytype(x_).__name__)' to a Julia '$T'",
-)
-pyconvert(::Type{T}, x, d) where {T} = @autopy x @pyconvert T x_ d
+function pyconvert(::Type{T}, x) where {T}
+    @pyregion begin
+        @autopy x @pyconvert T x_ error(
+            "cannot convert this Python '$(pytype(x_).__name__)' to a Julia '$T'",
+        )
+    end
+end
+
+function pyconvert(::Type{T}, x, d) where {T}
+    @pyregion begin
+        @autopy x @pyconvert T x_ d
+    end
+end
 
 """
     pyconvertarg(T, x, name)
@@ -399,12 +410,16 @@ Convert the Python object `x` to a `T`.
 
 On failure, throws a Python `TypeError` saying that the argument `name` could not be converted.
 """
-pyconvertarg(::Type{T}, x, name) where {T} = @autopy x @pyconvert T x_ begin
-    errset(
-        pybuiltins.TypeError,
-        "Cannot convert argument '$name' to a Julia '$T', got a '$(pytype(x_).__name__)'",
-    )
-    pythrow()
+function pyconvertarg(::Type{T}, x, name) where {T}
+    @pyregion begin
+        @autopy x @pyconvert T x_ begin
+            errset(
+                pybuiltins.TypeError,
+                "Cannot convert argument '$name' to a Julia '$T', got a '$(pytype(x_).__name__)'",
+            )
+            pythrow()
+        end
+    end
 end
 
 function init_pyconvert()

src/Core/Core.jl

@@ -32,6 +32,8 @@ using Markdown: Markdown
 
 import ..PythonCall:
     @pyconst,
+    @pyregion,
+    @pyregionbreak,
     @pyeval,
     @pyexec,
     ispy,

src/Core/Py.jl

@@ -1,5 +1,5 @@
-incref(x::C.PyPtr) = (C.Py_IncRef(x); x)
-decref(x::C.PyPtr) = (C.Py_DecRef(x); x)
+incref(x::C.PyPtr) = @pyregion (C.Py_IncRef(x); x)
+decref(x::C.PyPtr) = @pyregion (C.Py_DecRef(x); x)
 
 """
     ispy(x)
@@ -85,10 +85,12 @@ Use this to eagerly free a Python object, rather than waiting for Julia's GC to
 it at some indeterminate point in the future.
 """
 function unsafe_pydel(x::Py)
-    ptr = getptr(x)
-    if ptr != C.PyNULL
-        C.Py_DecRef(ptr)
-        setptr!(x, C.PyNULL)
+    @pyregion begin
+        ptr = getptr(x)
+        if ptr != C.PyNULL
+            C.Py_DecRef(ptr)
+            setptr!(x, C.PyNULL)
+        end
     end
     return
 end
@@ -99,12 +101,14 @@ macro autopy(args...)
     body = args[end]
     # ans = gensym("ans")
     esc(quote
+        @pyregion begin
         # $([:($t = $ispy($v) ? $v : $Py($v)) for (t, v) in zip(ts, vs)]...)
         # $ans = $body
         # $([:($ispy($v) || $unsafe_pydel($t)) for (t, v) in zip(ts, vs)]...)
         # $ans
         $([:($t = $Py($v)) for (t, v) in zip(ts, vs)]...)
         $body
+        end
     end)
 end
 
@@ -291,15 +295,7 @@ function _propertynames(x::Py, private::Bool)
     return Symbol[Symbol(pystr_asstring(word)) for word in words]
 end
 
-function Base.propertynames(x::Py, private::Bool = false)
-    if C.PyGILState_Check() == 1
-        _propertynames(x, private)
-    else
-        C.on_main_thread() do
-            _propertynames(x, private)
-        end::Vector{Symbol}
-    end
-end
+Base.propertynames(x::Py, private::Bool = false) = @pyregion _propertynames(x, private)
 
 Base.Bool(x::Py) = pytruth(x)