Dear Maintainer,

I don't know in which version it happened, but the header files no
longer define the PangoFcFontClass type (and PangoFcFontMapClass),
which makes accessing the documented public members inside
and subclassing impossible. According to the docs (e.g.
https://developer.gnome.org/pango/stable/PangoFcFont.html), to implement a
new fc-backend requires subclassing both PangoFcFontMap and PangoFcFont,
which is no longer possible in 1.46, breaking all third-party renderers
(e.g. ours, which uses pango to implement opengl rendering in games).

I've dug a bit deeper. Unfortunately, pango no longer has a ChangeLog, so
it's not clear to me when this was changed or why, but this seems to be
an intentional change, i.e. between pango 1.42 and pango 1.46 a bunch of
public members and structs required to subclass pango have been moved into
private header files.

I've opened an issue at https://gitlab.gnome.org/GNOME/pango/-/issues/513,
as massively removing parts of the documented api and disallowing
extensibility can't be good.

If this breakage is indeed intended, maybe debian could simply ship the
missing header files (pango/*-private.h)? According to the docs, they are
required to implement new backends).

Control: merge 974139 974140
Control: retitle 974139 libpango1.0-dev: PangoFcFont, PangoFcFontMap no longer subclassable
Control: forwarded 974139 https://gitlab.gnome.org/GNOME/pango/-/issues/513
Control: tags 974139 = upstream wontfix

Sorry, I don't think it's appropriate for Debian to be unilaterally
changing Pango's API. If upstream makes these APIs public again, we can
pick that up; but if upstream confirms that they're intentionally private,
making them public again in Debian is just going to cause us more trouble
in future.

    smcv

Actually, upstream has changed both API and ABI unilaterally - the removed
definitions are part of the public ABI, so at some point, debian would
have to bump the .soname.

Upstream has now confirmed that it was intentional - apparently, they
don't want people to be able to extend pango themselves anymore, so in the
long term I have to switch to something else anyway.

Now, since these definitions are part of the public ABI, I guess it is
safe to ship these header files myself until actual breakage occurs, but
that's really a bit insane.

The rationale behind debian shipping these headers would be that the
deifntions inside are ABI parameters, so cannot change without a .soname
change, at which point ABI compatibility doesn't matter anymore.

(Pango has broken their ABI in the past without bumping the .soname, so
this is definitely something you should watch out for).

Anyway, thanks for your feedback!

Actually, looking at 1.47 in experimental, it seems the ABI has been
changed without bumping the soname (the size of the PangoFcFontClass
struct has changed, which is part of the publicv ABI), which means
existing binaries will suffer memory corruption.

So debian would at least have to bump the soname to soemthing like 1.1.

This has happened before - pango upstream does not seem to care much about
ABI compatibility, but it's trivial to avoid in debian by bumping the
soname, so old binaries don't cause bad things to happen to users.

Distribution-specific SONAME bumps, without coordination with upstreams,
cause incompatibilities for years to come (see: libcurl) and I would
strongly prefer to avoid them. The upstream developer of a library "owns"
the namespace of SONAME version numbers; if we bump the SONAME version,
then we cannot complain when the upstream later uses the same version
number for a different ABI, leaving us unable to ever re-converge.

It is also the upstream developer who decides which parts of a library are
or aren't considered to be part of the public API/ABI. Yes, reclassifying
public API/ABI as private breaks derived projects that were relying
on it, and it's unfortunate that the Pango maintainers were unaware
that derived projects were relying on this part of the API; and yes, we
*could* make a Debian-specific fork of Pango, and link our GTK/etc. to
a Debian-specific Pango branch instead of the upstream Pango; but that
doesn't seem a sustainable thing to do.

I'm sorry that this has broken your use-case, but I don't think taking
an adversarial tone is going to help you to achieve the result you are
aiming for. People who perceive that they are being attacked will tend to
become increasingly defensive and unwilling to change their position,
which is presumably the opposite of what you want.

    smcv

I agree it is an unfortunate situation that upstream does this, but
a) it's a debian policy violation and b) it introduces unchecked out
of bounds accesses, which are always potential security bugs. This is
especially troublesome as the mere display of text from untrusted sources
can trigger this.

If you think that this bug report is a bit muddled I can open a new bug
only about the policy vioaltion and memory corruption issue, so there is
no confusion about header files or other incompatiiblities, which are a
separate issue.

True, but in this case, the developer has decided that the relevant parts
ARE part of the public ABI. A developer cannot undo this, just as a
developer cannot make 1=2, no matter how much she or he insist on it.

(which the pango developers don't do btw., it's not disputed that the
relevant parts are part of the public ABI, all that the pango developers
have said is that they don't have the manpower to bump the soname when on
breaking changes).

Not only that, they also introduce random memory corruption and potential
security bugs.

Please note that the problem is not reclassifying the public ABI as
private, the problem is breaking the ABI without bumping the soname
introducing serious actual bugs.

Reclassifying thew ABI as private would not have caused this, neither
would breaking the ABI and bumping the soname cause this kind of issue.

The pango maintainers obviously were aware of that, because it's
documented in the pango documentation multiple timesa and in the nissue I
reported they admitted they were aware of it, but don't care (apparently
because it is just too much work, a fair point).

Bumping the soname is not forking pango, but required by debian policy.

Note that bumping thew soname is all that's required to fix this issue, even
if the fix is not to my liking, but keep in mind this escalated from "some
ABI/API is no longer accessible" to "

Can you point out where I have taken "adversial tone"? What I reported is
a policy violation and apart from that a serious issue (unchecked memory
accesses that can be triggered simply by displaying text).

I suppose the bets way to proceed for you is to not feel attacked - I am
not aware of attacking you, and if you wrongly got the impression I did,
rest assured that this was not the intent of my words, I respect you as a
fellow human and so on.

Since this is cleared up, I hope we can go back to the actual etchnical
issues here?

So, the core issue there is less that PangoFcFontClass,
PangoFcFontMapClass are no longer available for subclassing, but more
that they *were* visible in the past, and therefore code outside Pango
(like yours) might already be relying on them to continue to have the
same struct layout that they had in buster. Is that correct?

Are you aware of code in Debian that subclasses them, or is this only a
concern for non-Debian code making use of the fact that subclassing is/was
allowed? (ABI breaks are ABI breaks either way, but it matters from the
release-management point of view.)

It might be possible to revert the changes to the layout of those classes
to how they were in buster, mark them as private/deprecated/something,
and if necessary move the new functionality that upstream subsequently
added into an intermediate subclass that is then used by their in-tree
subclasses. The good news is that, because the affected versions treat
those classes as private and non-subclassable, this approach shouldn't
break anything that has been compiled against the affected versions and
expects to see the ABI that is currently in bullseye.

This sort of thing seems to be a recurring problem in libraries that have
two APIs: one facing towards "normal" users of the library (in Pango's
case this means things like a typical GTK app), and one facing towards
projects that extend the library itself (like Pango engines and
renderers), with API/ABI breaks in the typical app-facing API surface
treated as more important than API/ABI breaks in the extension API.

I am not a primary maintainer of the pango1.0 package in Debian, but the
listed maintainers have too many responsibilities and not enough time
(and so do I, but here I am answering bugmail anyway). They're welcome
to take over, but will probably say the same things I'm saying.

I am unlikely to be able to keep compatibility with the old ABI in
a long-term-sustainable way without spending a significant amount of
time doing careful and patient work to persuade upstream that this is
both necessary and viable. We do not have authority over the upstream
maintainer, who is more overworked than I am, and attempting to force
him to take action is unlikely to be effective. I don't really have
the bandwidth to do this *either*, but *someone* needs to respond to
GNOME-team bug reports, so it might as well be another thing on my
growing list of responsibilities.

If I do this, I would ask you not to jump in on upstream issues/MRs
and insist that it happen sooner: antagonizing the overworked upstream
developer is not going to make this happen quicker, and might make it not
happen at all. In particular, if we propose making Debian-specific ABI
changes or setting a Debian-specific SONAME, I expect that to be perceived
as a threat, which would harm our ability to work constructively with
that upstream in future.

Bumping the SONAME makes us incompatible with upstream, potentially on a
long-term basis (see libcurl, libpcre3) and should not be done lightly.
In the past, Debian has been a lot more enthusiastic about bumping SONAMEs
on a distro-specific basis, but the results of doing so keep coming back
to haunt us, and I think the project (and in particular the release team)
is coming round to a consensus that it's usually a bad idea, and should
be a last resort at best.

Pragmatically, I suspect a SONAME transition might also cause us more
crashes in the short term than the affected ABIs do: while the transition
is incomplete, apps will end up loading both the "old" and "new" Pango
into the same address space, which seems likely to be bad news. Versioned
symbols wouldn't help here because the GType system is a flat global
namespace.

    smcv

Yes. And in addition, upstream has changed their members and increased
their sizes, so existing binaries might or might not work, and might or
might not cause out of bounds accesses.
most libpango users don't use this functionality (at leats not in this
way), but it is the only way to get your own renderer as far as I can see.

There are two issues: subclassing embeds (by necessity) the structure inside
the client binary. That is probably quite rare, as most programs just use one
of the existing backends. The size change will affect that.

The other issue is changed structure members. This is probably quite a bit
more common (but again, I have no way to find out), because a lot more code
might want to access the previously public members. This will typically not
(immediately) cause out of bounds accesses, but more likely in crashes, but
every case

I also only looked at two of the many classes in pango, namely
pangofc-font and pangofc-fontmap, but the changes are larger and ongoing,
and a few minor changes already have caused issues with existing programs
such as lilypad and have been undone, and I initially hoped that pango
upstream might undo the other breaking changes, but it seems clear to me
that the changes overall are extensive and they are not willing to undo
more than tiny parts of it (although I got very frustrated because I just
couldn't get a clear answer).

So I don't know what else changed in detail, and even less what the impact
is, and even less than that how to find affected programs.

I also don't know if this ABI breakage is ongoing - in the past, pango
develoeprs have broken the AIB before without changing the soname (I
opened a bug report for this in debian in 2009 and the then debian
maintainer got upstream to partially change it), but the conflicting claim
by pango upstream is that essentially nbody is working on pango, but on
the other hand, there is recent widespread breakage introduced by changign
a lot of the API, and and unknown amount of the ABI.

So while I don't know the extent of actual ABI breakage overall, most of the
changes so far seem to have *prepared* libpango for more extensive ABI
breakage.

As a data point, I created a copy of the class structure with extra
padding at the end to aovid out of bounds issues and recompiled one of my
opengl clients with it. It did seem to run with 1.45, but crashed with
1.47, so my idea with just adding the header files back wouldn't work
anyway, as pango not only changed the struct size, it also changed the
meaning of some struct members in incompatible ways.

So debian would have to bump the soname on, possibly, every release. And/or
would have to assess the amount fo breakage and the danger introduced on
every release.

Mabye this explains some of my frustration and the wording I used -
upstream is aware of this, but clearly doesn't care and the result puts
distributions such as debian in a horrible spot. And again, it all makes
very little sense, because of the claim that pango is essentially in
maintenance-only mode, there are no developers left, so who does all this
breakage and why.

That's of course only my opinion based on two of the pango classes that
directly affect me. Since this will force me to dump pango and implement
my own replacement for pango (probably based on raqm, but it's still a lot
of work), these changes in pango will, medium-term, not affect me at all
anymore.

Yes, making part of the API private but keeping the ABI would have been
what pango clearly should have done, followed by a 2.0 release which then
removes all the deprecated stuff. Upstream has said they won't do this, of
course.

Yes, but I don't see how this amounts to anything less then forking pango.

For example, some of the crashes I observed are due to pango not
maintaining the freetype face pointer members anymore, causing null
pointer accesses. To fix this, you'd essentially have to roll back to an
older version of pango, as trhe changes seem to be extensive.

Possibly. From my own (maybe limited) experience, I only ever had this
kind of issue with pango, and I use hundreds of libraries in my many
public and proprietary projects. But maybe I am just lucky because I don't
use so amyn gnoem libraries and possibly this is more of a gnome thing.

My experience is that on incompatible ABI changes, soname gets bumped or
the change is considered a bug.

Well, it totally makes sense to me. I'm quite unhappy to have to implement
my own pango library now. I wouldn't be pleased if it was just API
breakage and would require adjustments (such as switching from freetype
to harfbuzz), but the fact that the problem is two levels higher makes it
so frustrating: not only is not not an API change, they simply retracted
the API completely, disallowing people to use pango at all, and on top
of that, existing binaries now crash and might even have security bugs
that might be able to be triggered by outside (e.g. when my game client
displays chat messages and allocates and changes pangofc-fonts... And I
wouldn't even know where to start anylzing that).

I would conjecture that you probably won't even be able to keep
compatibility for short term - the changes in 1.47 seem to be quite a bit
larger than just hiding struct members and enlarging the structs.

I only found out the extent of these changes yesterday.

upstream claims no maintainer exists, actually.

I will not communicate with upstream anymore, I wasn't lying when I told
tham I have to disengange.

It's clear to me that these changes will stay, and I have to rip out pango
out of all my programs. I have to do that with speed, as existing binaries
will start, and debian is not the only target, so even if debian could
magically fix the ABI, it won't help me.

I think what pango upstream does is completely irresponsible.

It's not clear to me where this incompatibility would come from - existing
code would compile and run just fine. What wouldn't work would be existing
binaries from other distributions, but if the ABI is incompatible, thats not
actually more incompatible than not bumping the soname, it is safer though.

Granted, I am not a distribution maintainer, and don't envy you, so maybe
I am overlooking something in the big picture. But header files and
library names wouldn't change by a soname bump, i.e. there is typically no
reliance on the soname in program sources.

There are of course exceptions when programs dynamically load libraries,
which happens a lot with, say, libGL, maybe with libcurl, but probably not
with pango.

Well, I could have told the release team that it is a bad idea all
along. The question is always what is "badder", bumping the soname or
not. When an upstream project builds shared libraries it is always the
best solution if they bump it on ABI breaking changes. Not doing so
causes issues for every downstream, including but not limited to all OS
distributions.
normal part of their work, so debian surely must be able to deal with
this. Existing binaries will continue to work as opposed to having unknown
bugs.

Anyway, the TL;DR is (yeah, I put it at the end :) that the changes seem
to be extensive, and not limited to a single pango release. Just restoring
the members/size doesn't seem to be as feasible as I originally thought,
as there have been *actual* semantic changes behind the scenes. And I only
looked at two of the classes.

PS: Thanks for caring - I am relying on the overhelmingly good work of
Debian maintainers ever since I switched to debian some 20 years ago.