(Observed on the Ubuntu 26.04 sync of this package, which is unmodified
from Debian; the defect is in the Debian patch series and applies to the
Debian package equally.)
SUMMARY
=======
debian/patches/2016_opus_plc.patch (out-of-tree Opus/PLC support) adds
RTP-seqno-based lost-frame tracking to ast_translate() in
main/translate.c. Its "late packet" handling cannot cope with a mid-call
RTP sequence-number reset (SSRC/source change by a carrier SBC, a routine
event): after such a reset it silently discards EVERY subsequent inbound
frame on that translation path — for 30 seconds up to ~11 minutes — then
"self-heals" and repeats on the next source change.
The failure is completely silent (no log output at any verbosity/debug
level), which makes it look like a network/NAT problem and is, we
believe, a widely-misdiagnosed cause of "intermittent one-way audio that
fixes itself" reports on Debian/Ubuntu Asterisk installations.
The defect exists dormant in older releases too (e.g. asterisk 16.x
packages carried the same patch), but only manifests since upstream
Asterisk (>= 18-era) began stamping received frames with the wire RTP
sequence number (res_rtp_asterisk.c: "rtp->f.seqno = seqno;"), which
arms the patch's gate. This explains why upgrading from asterisk 16 to
20/22 packages surfaces the bug with unchanged configuration.
ROOT CAUSE
==========
From 2016_opus_plc.patch (as applied to main/translate.c, ast_translate()):
if ((path->f.seqno <= rtp_seqno_max_value) && (path->f.seqno !=
f->seqno)) {
if (f->seqno < path->f.seqno) { /* seqno overrun situation */
frames_missing = rtp_seqno_max_value + f->seqno -
path->f.seqno - 1;
} else {
frames_missing = f->seqno - path->f.seqno - 1;
}
/* Out-of-order packet - more precise: late packet */
if ((rtp_seqno_max_value + 1) / 2 < frames_missing) {
if (consume) {
ast_frfree(f);
}
...
return NULL; /* <-- silent discard */
}
...
Two independent failure triggers:
1. Mid-call source change: the carrier's SBC re-anchors media (new SSRC,
new random seqno). If the new seqno lands "behind" the stored
path->f.seqno, frames_missing computes > 32768 and the frame is
discarded as "late". Crucially, path->f.seqno is only updated in
framein() — a code path the discard never reaches — so the state
NEVER advances and every subsequent packet is judged "late" as well.
The lock only clears when the live seqno (incrementing at 50/s for
20 ms audio) numerically passes the stale stored value:
gap/50 seconds = 30 s ... ~11 min of total inbound audio loss.
2. Fresh translation paths start with path->f.seqno == 0 (the pvt is
calloc'd; the patch's own comment assumes an init value of 0x10000
that does not exist on all alloc paths). Any new path whose first
frame carries seqno > 32768 (~50% probability) enters the same
eternal-late state from the very first packet.
Both were observed and instrumented in production-like conditions
(details below).
IMPACT / OBSERVED BEHAVIOUR
===========================
- Intermittent one-way (or both-way, when both legs are hit) audio in
bridged/conference calls, lasting 30 s to 11 min, self-healing,
correlating with the carrier SBC's in-dialog re-INVITEs/media
handovers. Roughly 1 in 3 conference calls affected with our carrier.
- tcpdump shows inbound RTP arriving with speech content and 0 loss;
Asterisk transmits payload of pure alaw digital silence (0xd5) to the
peer(s). rx counters keep incrementing; "rtp set debug" shows packets
received; nothing at all is logged.
- Affects any call path that involves a translation (e.g. alaw <->
slin for ConfBridge/MixMonitor/etc.), i.e. nearly everything.
DIAGNOSIS EVIDENCE (summary)
============================
We instrumented every silent frame-drop site in the read path of a
production-identical staging system (probes in res_rtp_asterisk.c,
chan_pjsip.c, main/channel.c, main/rtp_engine.c, main/translate.c).
During live faults, only the translate.c late-packet discard fired — at
exactly the inbound packet rate (50/s), thousands of consecutive frames,
on precisely the channel whose party became inaudible, while all other
layers delivered valid voice frames.
REPRODUCTION
============
1. Debian/Ubuntu asterisk 20.x/22.x package, any PJSIP trunk, alaw,
a call path involving transcoding (ConfBridge is convenient).
2. Mid-call, change the RTP source seqno space (a carrier SBC media
handover does this; in a lab, re-anchor the sender or switch SSRC
with a discontinuous sequence number).
3. If the new seqno is below the previous stream's, all inbound audio
on that leg goes silent; "rtp set debug" keeps showing received
packets; nothing is logged. Audio returns by itself once the seqno
passes the stale stored value.
PROPOSED FIX (tested)
=====================
Treat a sustained run of "late" verdicts as a stream reset and resync,
instead of discarding forever. Genuinely late single packets (real
reordering) are still dropped, preserving the patch's PLC intent. A
reset costs 3 frames (60 ms) instead of minutes of silence.
Deployed and validated on the affected system: faults eliminated across
test batches; each real source-switch now logs exactly one resync event
and is inaudible to callers.
--- a/include/asterisk/translate.h
+++ b/include/asterisk/translate.h
@@ (struct ast_trans_pvt, after 'struct ast_frame f;')
struct ast_frame f; /*!< used in frameout. */
+ int seqno_late_streak; /*!< consecutive late-packet verdicts;
>=3 => stream reset, resync */
--- a/main/translate.c
+++ b/main/translate.c
@@ ast_translate(), the late-packet branch added by 2016_opus_plc.patch
/* Out-of-order packet - more precise: late packet */
if ((rtp_seqno_max_value + 1) / 2 < frames_missing) {
- if (consume) {
- ast_frfree(f);
- }
- /*
- * Do not pass late packets to any transcoding module,
because that
- * confuses the state of any library (packets
inter-depend). With
- * the next packet, this one is going to be treated as lost
packet.
- */
- return NULL;
+ /*
+ * A sustained run of "late" verdicts is not reordering —
it is a
+ * stream reset (SSRC/source change, e.g. by a carrier
SBC). The
+ * previous behaviour discarded EVERY frame until seqno
wraparound
+ * (30 s – 11 min of one-way silence, completely unlogged).
+ * Resync after 3 consecutive late verdicts; genuinely late
+ * single packets are still dropped as before.
+ */
+ if (++path->seqno_late_streak >= 3) {
+ ast_debug(1, "Translation path seqno resync after %d
late frames (f->seqno=%d path->f.seqno=%d)\n",
+ path->seqno_late_streak, f->seqno, (int)path->f.seqno);
+ path->seqno_late_streak = 0;
+ frames_missing = 0; /* stream reset: translate
normally */
+ } else {
+ if (consume) {
+ ast_frfree(f);
+ }
+ /*
+ * Do not pass late packets to any transcoding module,
because that
+ * confuses the state of any library (packets
inter-depend). With
+ * the next packet, this one is going to be treated as
lost packet.
+ */
+ return NULL;
+ }
+ } else {
+ path->seqno_late_streak = 0;
}
Notes for the maintainer:
- struct ast_trans_pvt is calloc'd, so the new field needs no explicit
init. (The threshold could be a #define; 3 was chosen to still drop
up to two genuinely-reordered stragglers.)
- Independent of the resync, the fresh-path case (trigger 2) suggests
path->f.seqno should be initialised to 0x10000 ("no seqno seen yet")
on every allocation path, as the patch's own comment assumes.
- An alternative, more conservative remedy is to drop the late-packet
discard entirely for non-buffering translators, but the resync above
is minimal and preserves existing behaviour for real reordering.
PRIOR REPORTS OF THE SAME DEFECT (this report provides their root cause)
========================================================================
- Ubuntu bug #2053070 "opus PLC patches cause audio loss" (2024-02-13,
still open/triaged): concluded the patch is "broken and should be
removed" but without a code-level cause or reproducible test case.
https://bugs.launchpad.net/ubuntu/+source/asterisk/+bug/2053070
- Upstream asterisk/asterisk#507 "MixMonitor doesn't record after Blind
Transfer" (2023-12, Debian 12/Asterisk 18): same defect surfacing as
one-way recordings after a transfer (a transfer churns translation
paths, hitting the fresh-path trigger). Reporter confirmed on
2024-02-13: dropping 2015_opus.patch + 2016_opus_plc.patch fixed it.
https://github.com/asterisk/asterisk/issues/507
- Debian bug #1060275 "Codec translation notice" (2024-01): the log-spam
facet ("N lost frame(s)" NOTICEs on plain GSM/G.722/alaw translations
with opus not even loaded). The response was to comment the NOTICE
out ("disabled by default to not flood log in production"), which made
the remaining audio-loss defect completely silent and much harder to
diagnose — as our investigation experienced first-hand.
Unlike the above, this report identifies the exact code defect, both
triggers, and provides a minimal tested fix that keeps the PLC feature
(rather than removing the patches wholesale).
ADDITIONAL CONTEXT
==================
- Not related to upstream GitHub issue #2059 / commit f0c1c1a7b
("res_pjsip_sdp_rtp: Fix intermittent call drop on reINVITE sdp
change") — we backported that fix first and the fault persisted; the
RX-silent defect reported here is specific to 2016_opus_plc.patch.
- Environment: Ubuntu 26.04 (package synced unmodified from Debian),
Asterisk 22.5.2, PJSIP trunk (TCP) to SpeakUp NL, alaw only,
plain RTP (no ICE/SRTP/WebRTC/TURN).