Dear Maintainer,

Since upgrading python3.13 from 3.13.5-2+deb13u2 to 3.13.5-2+deb13u3, an
application test suite that uses SQLAlchemy's asyncio ORM (and therefore
greenlet) segfaults reliably. The interpreter dies inside the cyclic GC and
inside pymalloc, at random places, in a different test on every run.

The same code, the same wheels, and the same greenlet version are stable on
upstream CPython 3.13.5 -- the identical upstream version Debian ships -- so
the regression appears to come from the Debian patch set in deb13u3 rather
than from CPython 3.13.5 itself.

Evidence
--------

Each row is a full run of the same test suite (~6850 tests, pytest-xdist -n 30):

  Debian 3.13.5-2+deb13u3                    5-10 crashed workers per run
  Debian 3.13.5-2+deb13u3, gc.disable()      clean, 0 crashes
  upstream CPython 3.13.5 (identical ver.)   clean, 0 crashes
  upstream CPython 3.13.12                   clean, 0 crashes

Disabling the cyclic garbage collector makes the crashes disappear entirely,
which is what first pointed at the GC.

Backtraces (symbols via debuginfod.debian.net)
----------------------------------------------

Crash 1 -- GC linked list is corrupt. Note the object pointer 0x7dcd499d914f
is not even correctly aligned, so it is not a live object:

  #4 _PyGCHead_SET_PREV (gc=..., prev=...) at ../Include/internal/pycore_gc.h:191
  #5 _PyObject_GC_UNTRACK (op=0x7dcd499d914f) at ../Include/internal/pycore_object.h:423
  #6 listiter_dealloc (self=0x7dcd499d914f) at ../Objects/listobject.c:3950
  #7 _PyEval_EvalFrameDefault (...) at ../Python/generated_cases.c.h:2888

Crash 2 and 3 -- pymalloc's free list is corrupt:

  #4 pymalloc_alloc (state=..., nbytes=40) at ../Objects/obmalloc.c:2129
  #5 _PyObject_Malloc (ctx=..., nbytes=40) at ../Objects/obmalloc.c:2149
  #6 PyObject_Malloc (size=40) at ../Objects/obmalloc.c:1316

Other runs died in _PyObject_GC_New and PyObject_GetIter. In every core, the
crashing Python code was running inside a greenlet-switched stack:

  #7 greenlet::UserGreenlet::inner_bootstrap (...) at src/greenlet/greenlet_refs.hpp:245
  #8 greenlet::UserGreenlet::g_initialstub (...) at src/greenlet/TUserGreenlet.cpp:317
  #9 greenlet::UserGreenlet::g_switch (...) at src/greenlet/TUserGreenlet.cpp:185

Suspected cause
---------------

The deb13u3 changelog includes:

  - Avoid garbage collecting objects too early when sharing __dict__
    (Closes: #1108039)

That is the only GC/object-lifetime change in the u2 -> u3 delta, and the
symptoms are object-lifetime symptoms, so it seems the most likely candidate.
I have not bisected the patch set to confirm this, and I may be wrong about
which patch is responsible -- the load-bearing facts are the four rows in the
table above.

What I ruled out
----------------

- Not application code: the crashes are unchanged with our latest commit
  reverted, and were absent before the upgrade.
- Not a stale/ABI-mismatched greenlet: rebuilding greenlet 3.5.3 from source
  against the current headers changes nothing. 3.5.3 is the current release.
- Not memory pressure: 61 GB free, no OOM killer activity.

Reproducing it
--------------

I could not reduce this to a small script, which is worth stating plainly.
Neither a long-running server (30k requests, 64-way concurrency, 9 minutes)
nor synthetic greenlet + GC + SQLAlchemy loops (30 parallel processes,
fresh engine and event loop per iteration) crash. It needs the full suite.

The application is open source, so the recipe is:

  git clone https://github.com/maziggy/bambuddy
  cd bambuddy
  python3.13 -m venv venv
  ./venv/bin/pip install -r requirements.txt -r requirements-dev.txt
  cd backend && ../venv/bin/python -m pytest tests/ -n 30 -q

On deb13u3 this produces several "worker crashed" failures within ~75
seconds; the specific tests differ every run. Adding a sitecustomize.py with
gc.disable() on PYTHONPATH makes it pass.

Relevant versions: greenlet 3.5.3, SQLAlchemy 2.0.51, aiosqlite, pytest-xdist.

System
------

Debian 13 (trixie), amd64, kernel 7.0.12-1-pve
python3.13         3.13.5-2+deb13u3
libpython3.13      3.13.5-2+deb13u3
python3-greenlet   (from PyPI wheel in venv, 3.5.3)

Thanks for maintaining python3.13.

Hi Martin (2026.07.13_06:58:31_+0000)

Urgh, sorry about that.

Can you test this against upstream >= 3.13.13? That includes the same
patch.

This change is also in >= 3.14.4 and >= 3.15.0a7.

If it's working in these versions, but not with our backported patch,
then we're clearly missing a related patch, and we must find it.

If it's not working with the latest upstream releases, then there's an
upstream bug here that we should take upstream.

Stefano

Hi Stefano,

Tested as you asked, and then bisected it. Short answer: upstream >= 3.13.13 is fine, and you are missing a related patch. I think I have found it.

You are missing gh-130555, commit d567f451e157: "[3.13] gh-130555: Fix use-after-free in dict.clear() with embedded values (gh-145268) (#145430)".

deb13u3 carries debian/patches/traverse-managed-dicts.patch, which is gh-130327, commit 702d08578394 ("Always traverse managed dictionaries, even when inline values are available"). Upstream shipped both of those in 3.13.13. You took the first and not the second.

They are not independent. gh-130555 is a prerequisite for gh-130327:

- gh-130327 makes the GC always traverse a split table's values, including embedded/inline ones. It removes the !mp->ma_values->embedded guard in dict_traverse.
- Without gh-130555, clear_lock_held() still clears an embedded-values dict with Py_CLEAR(oldvalues->values[i]) in a loop, and only sets oldvalues->size = 0 afterwards. Py_CLEAR decrefs, which can run arbitrary code and re-enter the GC while the value slots still hold dangling pointers and size is still non-zero.
- So the now-unconditional traverse walks freed pointers. That is exactly what we see: the cores die in _PyObject_GC_UNTRACK / _PyGCHead_SET_PREV with a misaligned object pointer, and in pymalloc_alloc.
- gh-130555 introduces clear_embedded_values(), which NULLs every slot and sets size = 0 first, then decrefs from a local array. A re-entrant traverse can then never observe a dangling slot.

So applying gh-130327 alone does not merely fail to fix something - it introduces the use-after-free. deb13u2 does not have this crash; deb13u3 does.

The bisection. Each row is three full runs of the same test suite under pytest-xdist -n 30, all interpreters built with gcc 14.2.0 and an identical ./configure, all with greenlet 3.5.3 and SQLAlchemy 2.0.51:

  Debian 3.13.5-2+deb13u3                              		18, 22 crashed workers
  upstream 3.13.12 (has neither patch)                 		0, 0, 0
  upstream 3.13.14 (has both)                          		0, 0, 0
  upstream 3.13.5 + gh-130327 only (= your backport)  16, 10, 14
  upstream 3.13.5 + gh-130327 + gh-130555              	0, 0, 0

The last two rows are the useful ones: same tarball, same compiler, same flags, one commit apart.

Worth noting for severity: this is silent heap corruption in a stable release, not just a crash. We only caught it because a 30-way parallel test suite turns it into reliable segfaults. A single-process workload will more often corrupt quietly than die.

Happy to test a deb13u4 before you upload, if that is useful - I can turn a build around the same day.

Thanks for the quick reply.

Martin

I can confirm this regression in an environment with no greenlet and no
asyncio, which may help narrow the triage: plain threading.Thread workloads
crash the same way, so greenlet stack switching is not an essential
ingredient.

Environment
-----------
- Debian 13 (trixie) on Raspberry Pi 5, aarch64, kernel 6.18.34+rpt-rpi-2712
  (16K pages), python3.13 / libpython3.13 3.13.5-2+deb13u3
  ("3.13.5 (main, Jun 13 2026, 14:18:01) [GCC 14.2.0]")
- Workload: weewxd (WeeWX 5.4.0 weather daemon) in a venv on the system
  interpreter; long-running, ~10-12 threads (serial I/O, HTTP posters,
  report generation), C extensions: sqlite3 (stdlib), numpy 2.4.1,
  Pillow 12.1.0, Cheetah3, pyephem.
- WeeWX calls an explicit gc.collect() from its main thread every 3 hours
  (all other threads running normally; no gc.freeze/disable).

Evidence
--------
Seven identical hosts ran this workload for months on 3.13.5-2+deb13u2 with
zero crashes. unattended-upgrades installed deb13u3 on 2026-07-11 06:47 PDT.
First SIGSEGV 21 hours later; four total within the next 48 hours, on four
different hosts.

Every crash occurred 12-300 ms after the 3-hourly explicit gc.collect()
completed (the daemon logs each collection; the log line immediately
precedes the fatal signal every time):

  host A  gc log 11:10:04.626  ->  SIGSEGV 11:10:04.638  (+12 ms)
  host B  gc log 14:10:04.397  ->  SIGSEGV 14:10:04.540  (+143 ms)
  host C  gc log 05:10:04.534  ->  SIGSEGV 05:10:04.548  (+14 ms)
  host B' gc log 04:00:03.186  ->  SIGSEGV ~04:00:03.5   (+~300 ms, prior day)

Most collections survive; the observed failure rate is roughly 3-4% per
collection with this thread mix.

The crashing thread differs every time and is always in an ordinary
allocation; faulting addresses are garbage low pointers, consistent with
corrupted GC linked lists / pymalloc freelists as in the original report:

Core 1 (crash while stdlib sqlite3 builds a row tuple; Python-level: a
report thread inside sqlite3 Cursor.execute):
  #3  _PyObject_GC_NewVar ()
  #4  PyTuple_Pack ()
  #5  _sqlite3.cpython-313-aarch64-linux-gnu.so
  si_addr = 0x2e9152

Core 2 (crash in pure-Python config parsing, no C extension in the frame;
Python-level: configobj._load building strings):
  #3  PyUnicode_New ()
  #4  (python3.13 + 0x167958)
  si_addr = 0x26f3ca

A third crash (no core retained by coredumpctl, but faulthandler traceback
captured) died in the main thread inside a numpy allocation
(PyType_GenericAlloc called from _multiarray_umath) while two report
threads were in sqlite3 fetches.

Full cores (~500 MB uncompressed each) and complete
faulthandler/gdb thread dumps are preserved and available on request --
happy to run further gdb commands against them or test candidate packages.
I have not tested upstream 3.13.13/3.14.x on these hosts, but the identical
workload was stable for months on 3.13.5-2+deb13u2 and crashes started the
morning deb13u3 was installed, on multiple machines independently.

Since the fleet runs unattended, I have also seen that the crash reproduces
with nothing more exotic than: threads doing sqlite3 reads + a main-thread
gc.collect(). If a minimal reproducer would help, that combination is where
I would start.

John Kline

Since then my seven weewxd instances have completed 145 of the explicit
3-hourly gc.collect() passes that previously triggered the crashes,
with zero segfaults.  On deb13u3 the same fleet crashed on roughly 3-4%
of these passes (4 crashes over ~2 days), so approximately five crashes
would have been expected in those 145 passes had we stayed on deb13u3.
This further confirms the regression is confined to deb13u3.

Is a deb13u4 adding gh-130555 (d567f451e157) planned?  I'm happy to
test a candidate package on this fleet before upload.  I'd also like
to stop holding at deb13u2, since that means running without deb13u3's
security fixes.

John Kline

Hi John (2026.07.16_14:14:34_+0000)
https://deb.debusine.debian.net/debian/r-stefanor-python/

Stefano

Thanks for the quick turnaround, Stefano.

I have the test build (3.13.5-2+deb13u4 from your debusine workspace)
installed on seven of my nine trixie/arm64 machines as of this morning,
including five of the seven weewxd instances that exercised the crash.
Each of those five has now completed its first post-install
gc.collect() pass cleanly.  The remaining two weewxd hosts go tonight;
I'll follow up in a couple of days with the accumulated pass count
across the fleet (for reference: deb13u3 crashed 4 times in ~130
passes; deb13u2 and, so far, deb13u4 have zero).

John Kline

We believe that the bug you reported is fixed in the latest version of
python3.13, which is due to be installed in the Debian FTP archive.

A summary of the changes between this version and the previous one is
attached.

Thank you for reporting the bug, which will now be closed.  If you
have further comments please address them to 1141977@bugs.debian.org,
and the maintainer will reopen the bug report if appropriate.

Debian distribution maintenance software
pp.
Stefano Rivera <stefanor@debian.org> (supplier of updated python3.13 package)

(This message was generated automatically at their request; if you
believe that there is a problem with it please contact the archive
administrators by mailing ftpmaster@ftp-master.debian.org)
Format: 1.8
Date: Wed, 15 Jul 2026 17:25:40 -0300
Source: python3.13
Architecture: source
Version: 3.13.5-2+deb13u4
Distribution: trixie
Urgency: medium
Maintainer: Matthias Klose <doko@debian.org>
Changed-By: Stefano Rivera <stefanor@debian.org>
Closes: 1141977
Changes:
 python3.13 (3.13.5-2+deb13u4) trixie; urgency=medium
 .
   * Patch: Fix use-after-free in dict.clear() with embedded values.
     Resolves a regression in 3.13.5-2+deb13u3. (Closes: #1141977)
Checksums-Sha1:
 5863ea53817c7c5ab83761620c153302324f76f4 3721 python3.13_3.13.5-2+deb13u4.dsc
 b9d0c777043edb21da90bb954d2b8fe46463f07c 296316 python3.13_3.13.5-2+deb13u4.debian.tar.xz
 4ee8f510af471698a982784b09cd615a58544d8a 9970 python3.13_3.13.5-2+deb13u4_source.buildinfo
Checksums-Sha256:
 72134c9f6705d0d87a53cf48c5a997f088b6b675ad5d87f57fa6e9f338393519 3721 python3.13_3.13.5-2+deb13u4.dsc
 b5cc42821fb6a6f91b7a5ac1da5b313bb251288ca03f03d683b4c6ca453ece11 296316 python3.13_3.13.5-2+deb13u4.debian.tar.xz
 6aa424d1b145d624d09ddc0ffbc29d9b72d07a9f9926f71bcb1d2aa4607e5c6f 9970 python3.13_3.13.5-2+deb13u4_source.buildinfo
Files:
 63ef34ed3b893a4e9eff74b891196e36 3721 python optional python3.13_3.13.5-2+deb13u4.dsc
 ca31cc2e9e3aada703a8e0f69fa18e00 296316 python optional python3.13_3.13.5-2+deb13u4.debian.tar.xz
 a75409c13d5cc3a4d3860beb5b4d52f0 9970 python optional python3.13_3.13.5-2+deb13u4_source.buildinfo
-----BEGIN PGP SIGNATURE-----

iIoEARYKADIWIQTumtb5BSD6EfafSCRHew2wJjpU2AUCaloYExQcc3RlZmFub3JA
ZGViaWFuLm9yZwAKCRBHew2wJjpU2K+zAPsHUo+P1MjOZ7eJi2UpsMyzC+qnuDDl
0c62f+Un6xpGXQD+M0Lh89GoWfkdaOwqIgjaTo1QnIatgpLgOkEB0OqUhgc=
=ph37
-----END PGP SIGNATURE-----

Following up with final results from the fleet test of the 3.13.5-2+deb13u4
test build, as promised.

The test build has now run for about 21 hours (installed midday 2026-07-16,
all hosts rebooted) on 7 production weewxd instances across 7 Raspberry Pi 5
machines (arm64, trixie).  Each instance runs an explicit gc.collect() every
3 hours, so the fleet has accumulated:

  49 consecutive clean GC passes, 0 crashes, 0 core dumps.

Largest collection per host (objects collected in a single pass):

  bambi5t        60886
  judygirldog5t  58153
  cosmo5t        49817
  ella5t         44104
  mrpojangles5t  40008
  judy5t         36825
  charlemagne5t  28475

For comparison, under deb13u3 this same fleet crashed 4 times in roughly 130
passes (~3% per pass), and every fatal pass had collected at least ~22.9k
objects.  On deb13u4, every one of the 7 hosts has now completed at least one
pass larger than the smallest collection that crashed deb13u3, including
several far beyond the deb13u3 range, all clean.

As far as this fleet can tell, deb13u4 fully resolves the issue.  Thanks for
the quick turnaround on the fix, and glad to see it accepted into
proposed-updates.

John Kline