Manage CPython thread states automatically across Julia tasks

[cjdoris]

Motivation

• Replace the old assumption that Julia permanently holds Python's GIL with a task-safe thread-state model so PythonCall APIs work correctly from any Julia task/thread.
• Provide user-facing, nestable hints (@pyregion/@pyregionbreak) that amortize attach/detach transitions without exposing GIL details.
• Preserve compatibility with existing callback paths (JuliaCall/JlWrap) and keep finalizers non-blocking.

Description

• Added a new internal module Region that implements per-OS-thread persistent PythonCall-owned PyThreadState + cooperative semaphore, and per-task dynamic TaskState with stickiness and token-based nested enter/exit semantics; exported macros @pyregion and @pyregionbreak use this machinery. (src/Region/Region.jl, src/API/macros.jl, src/API/exports.jl)
• Hooked startup/finalization so standalone PythonCall detaches after init and finalization re-attaches a known interpreter state before calling Py_FinalizeEx(). (src/C/context.jl, src/Region/Region.jl, src/PythonCall.jl)
• Replaced the old PyGILState_* usage internally for region semantics and exposed macros; provided a compatibility layer for the existing PythonCall.GIL API by routing it through the new region enter/exit functions. (src/GIL/GIL.jl, src/C/pointers.jl)
• Applied @pyregion / @pyregionbreak throughout the core C-API call sites to ensure correct attach/detach behavior while keeping regions tight; updated @py lowering so generated code has one outer @pyregion and each @jl splice becomes a @pyregionbreak. (bulk edits under src/Core/*, src/PyMacro/PyMacro.jl, src/JlWrap/*, src/GC/GC.jl, src/Core/builtins.jl)
• Revised finalizer/decref logic to avoid attaching a Python state in finalizers; finalizers enqueue pointers if no attached state exists and PythonCall.GC.gc() can be used to drain the queue inside a region. (src/GC/GC.jl, test/GC.jl)
• Adjusted low-level C pointer handling to expose a safe unchecked thread-state getter (PyThreadState_GetUnchecked / _PyThreadState_UncheckedGet) and PyThreadState_New usage for persistent per-thread states. (src/C/pointers.jl, src/C/C.jl)
• Added task/region tests that exercise nested region/break combinations, detach-yield-resume, multi-threaded interleaving, and Python->Julia callbacks. (test/Region.jl)
• Bumped the package Julia requirement to 1.12 to rely on OncePerThread / OncePerTask primitives and updated julia/juliapkg metadata where appropriate.

Testing

• Ran the Julia test suite with multiple threads: JULIA_NUM_THREADS=4 julia --project -e 'using Pkg; Pkg.test()' — all tests passed (4,095 tests passed).
• Ran coverage-enabled tests: JULIA_NUM_THREADS=4 julia --project -e 'using Pkg; Pkg.test(coverage=true)' — suite passed (4,095 tests passed); uncovered lines are limited to defensive rollback/impossible-token branches and the Python-originated root-break path that requires JuliaCall-specific scenarios.
• Performed a threaded smoke test invoking PythonCall from Threads.@spawn to verify detached initialization and cross-thread Python calls — succeeded (simple runtime checks demonstrated PyThreadState_GetUnchecked() is NULL outside regions and that @pyregion amortizes transitions).
• Attempted the Python-side tests (uv run pytest -s --nbval ./pytest) but they failed to run in this environment due to an external constraint: the environment's Python 3.14 OpenSSL/juliapkg interaction restricts Julia to ≤1.11 and therefore conflicts with the new Julia ≥1.12 requirement; the Python tests are otherwise intended to run in a matching environment and were not failing due to region logic.
• Documentation build attempted with docs/make.jl; Documenter ran but could not infer repository remote source links because this checkout lacks a valid origin remote (non-test blocking).

If you want, I can (1) open follow-ups to address remaining minor uncovered branches and defensive assertions, (2) add CI matrix notes to run the Python tests under a Python/Julia combination that satisfies OpenSSL/juliapkg constraints, or (3) narrow/expand any @pyregion placements based on preferred granularity.

---

Codex Task