Package: libpango1.0-udeb
Version: 1.44.7-3
Severity: grave
Tags: d-i
Justification: renders package unusable
X-Debbugs-Cc: debian-boot@lists.debian.org

Hi,

In the “better late than never” department… I've completed a debugging session
that shows that upgrading libpango1.0-udeb from 1.42.4-8 to 1.44.7-3 hangs the
installer in various situations.

Interestingly, it doesn't *always* do that.

 - I know for sure this upgrade is responsible for the rescue mode
   breakage *under some locales* tracked at:
https://bugs.debian.org/987377

   e.g. getting a shell into the system to be rescued works fine in
   French or in German, but not in English (which hasn't any crazy fonts
   or glyphs compared to the two other ones).


 - It seems to a kind user performing many tests that this could be
   linked to the actual hardware configuration:
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=987377#54

   (See the “For short […]” conclusion, it's priceless.)

   I'll be trying to get more results from Étienne.


 - I'm *not yet* certain this also triggers direct failures for Netinst
   images for several languages, tracked at:
https://bugs.debian.org/987449

   but I have a rather strong suspicion, as mentioned in:
https://bugs.debian.org/987377#59

   I'll try to confirm this in the next few days.


Chris Hofstaedtler provided some perf data, which might help figure out
what's happening for someone who's knowledgeable about pango internals:
https://bugs.debian.org/987449#15

Please keep debian-boot@lists.debian.org in the loop when replying, and
thanks for your help!


Cheers,

Cyril Brulebois <kibi@debian.org> (2021-04-26):
…

It's been confirmed, so I've added a “block” between both reports.

I've also confirmed an earlier version, uploaded to experimental
(1.44.6-1) also triggers the issue, hence the recent “found”.

I'm now down to investigating upstream releases that weren't packaged
into Debian. For that, I'm using #987449 with Sinhala (since that's
almost instantaneous, as opposed to #987377 which needs going through
various menus).

This time around, testing 1.43.0 with debian/ carried over from
debian/1.42.4-8, adjusted for docs (some files are missing) and symbols
(one new symbol), the problem cannot be triggered in an obvious manner.

Moving to 1.44* tags now.


Cheers,

Cyril Brulebois <kibi@debian.org> (2021-04-29):

Alright, I think I'm hitting my limits here.

Versions 1.44, 1.44.2, and 1.44.3 all fail in a similar way:

    [39/144] cc -Ipango/libpangoft2-1.0.so.0.4400.0.p -Ipango -I../pango -I. -I.. -I/usr/include/glib-2.0 -I/usr/lib/x86_64-linux-gnu/glib-2.0/include -I/usr/include/fribidi -I/usr/include/harfbuzz -I/usr/include/freetype2 -I/usr/include/libpng16 -I/usr/include/uuid -I/usr/include/cairo -I/usr/include/pixman-1 -fdiagnostics-color=always -pipe -D_FILE_OFFSET_BITS=64 -Wall -Winvalid-pch -std=gnu99 -D_POSIX_C_SOURCE=200809L -D_POSIX_THREAD_SAFE_FUNCTIONS -D_GNU_SOURCE -g -O2 -ffile-prefix-map=/home/kibi/hack/pango1.0.git=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -Wimplicit-function-declaration -Wstrict-prototypes -Wmissing-prototypes -Wnested-externs -Wold-style-definition -Wno-int-conversion -Wno-discarded-qualifiers -fno-strict-aliasing -Wpointer-arith -Wmissing-declarations -Wformat=2 -Wformat-nonliteral -Wformat-security -Wunused -Wcast-align -Wmissing-noreturn -Wmissing-format-attribute -Wmissing-include-dirs -Wlogical-op -Wno-uninitialized -Wno-shadow -Werror=implicit -Werror=nonnull -Werror=init-self -Werror=main -Werror=missing-braces -Werror=sequence-point -Werror=return-type -Werror=trigraphs -Werror=array-bounds -Werror=write-strings -Werror=address -Werror=int-to-pointer-cast -Werror=pointer-to-int-cast -Werror=empty-body -Werror=write-strings -Wundef -Werror=redundant-decls -fvisibility=hidden '-DG_LOG_DOMAIN="Pango"' -DG_LOG_USE_STRUCTURED=1 -DPANGO_COMPILATION '-DSYSCONFDIR="/etc"' '-DLIBDIR="/usr/lib/x86_64-linux-gnu"' -DPANGO_DISABLE_DEPRECATION_WARNINGS -MD -MQ pango/libpangoft2-1.0.so.0.4400.0.p/pangofc-font.c.o -MF pango/libpangoft2-1.0.so.0.4400.0.p/pangofc-font.c.o.d -o pango/libpangoft2-1.0.so.0.4400.0.p/pangofc-font.c.o -c ../pango/pangofc-font.c
    FAILED: pango/libpangoft2-1.0.so.0.4400.0.p/pangofc-font.c.o
    cc -Ipango/libpangoft2-1.0.so.0.4400.0.p -Ipango -I../pango -I. -I.. -I/usr/include/glib-2.0 -I/usr/lib/x86_64-linux-gnu/glib-2.0/include -I/usr/include/fribidi -I/usr/include/harfbuzz -I/usr/include/freetype2 -I/usr/include/libpng16 -I/usr/include/uuid -I/usr/include/cairo -I/usr/include/pixman-1 -fdiagnostics-color=always -pipe -D_FILE_OFFSET_BITS=64 -Wall -Winvalid-pch -std=gnu99 -D_POSIX_C_SOURCE=200809L -D_POSIX_THREAD_SAFE_FUNCTIONS -D_GNU_SOURCE -g -O2 -ffile-prefix-map=/home/kibi/hack/pango1.0.git=. -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -Wimplicit-function-declaration -Wstrict-prototypes -Wmissing-prototypes -Wnested-externs -Wold-style-definition -Wno-int-conversion -Wno-discarded-qualifiers -fno-strict-aliasing -Wpointer-arith -Wmissing-declarations -Wformat=2 -Wformat-nonliteral -Wformat-security -Wunused -Wcast-align -Wmissing-noreturn -Wmissing-format-attribute -Wmissing-include-dirs -Wlogical-op -Wno-uninitialized -Wno-shadow -Werror=implicit -Werror=nonnull -Werror=init-self -Werror=main -Werror=missing-braces -Werror=sequence-point -Werror=return-type -Werror=trigraphs -Werror=array-bounds -Werror=write-strings -Werror=address -Werror=int-to-pointer-cast -Werror=pointer-to-int-cast -Werror=empty-body -Werror=write-strings -Wundef -Werror=redundant-decls -fvisibility=hidden '-DG_LOG_DOMAIN="Pango"' -DG_LOG_USE_STRUCTURED=1 -DPANGO_COMPILATION '-DSYSCONFDIR="/etc"' '-DLIBDIR="/usr/lib/x86_64-linux-gnu"' -DPANGO_DISABLE_DEPRECATION_WARNINGS -MD -MQ pango/libpangoft2-1.0.so.0.4400.0.p/pangofc-font.c.o -MF pango/libpangoft2-1.0.so.0.4400.0.p/pangofc-font.c.o.d -o pango/libpangoft2-1.0.so.0.4400.0.p/pangofc-font.c.o -c ../pango/pangofc-font.c
    ../pango/pangofc-font.c: In function ‘get_face_metrics’:
    ../pango/pangofc-font.c:371:44: error: ‘HB_OT_METRICS_UNDERLINE_SIZE’ undeclared (first use in this function); did you mean ‘HB_OT_METRICS_TAG_UNDERLINE_SIZE’?
      371 |   if (hb_ot_metrics_get_position (hb_font, HB_OT_METRICS_UNDERLINE_SIZE, &position))
          |                                            ^~~~~~~~~~~~~~~~~~~~~~~~~~~~
          |                                            HB_OT_METRICS_TAG_UNDERLINE_SIZE
    ../pango/pangofc-font.c:371:44: note: each undeclared identifier is reported only once for each function it appears in
    ../pango/pangofc-font.c:374:44: error: ‘HB_OT_METRICS_UNDERLINE_OFFSET’ undeclared (first use in this function); did you mean ‘HB_OT_METRICS_TAG_UNDERLINE_OFFSET’?
      374 |   if (hb_ot_metrics_get_position (hb_font, HB_OT_METRICS_UNDERLINE_OFFSET, &position))
          |                                            ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
          |                                            HB_OT_METRICS_TAG_UNDERLINE_OFFSET
    ../pango/pangofc-font.c:377:44: error: ‘HB_OT_METRICS_STRIKEOUT_SIZE’ undeclared (first use in this function); did you mean ‘HB_OT_METRICS_TAG_STRIKEOUT_SIZE’?
      377 |   if (hb_ot_metrics_get_position (hb_font, HB_OT_METRICS_STRIKEOUT_SIZE, &position))
          |                                            ^~~~~~~~~~~~~~~~~~~~~~~~~~~~
          |                                            HB_OT_METRICS_TAG_STRIKEOUT_SIZE
    ../pango/pangofc-font.c:380:44: error: ‘HB_OT_METRICS_STRIKEOUT_OFFSET’ undeclared (first use in this function); did you mean ‘HB_OT_METRICS_TAG_STRIKEOUT_OFFSET’?
      380 |   if (hb_ot_metrics_get_position (hb_font, HB_OT_METRICS_STRIKEOUT_OFFSET, &position))
          |                                            ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
          |                                            HB_OT_METRICS_TAG_STRIKEOUT_OFFSET

Version 1.44.4 is the first one I was able to build, using the packaging
from debian/1.44.6-1 (first 1.44.x version that was packaged and that's
also known to be buggy).

This would suggest the regression in caused by something in here:

    Overview of changes in 1.44.4
    =============================
    - Add an insert-hyphens attribute
    - Reinstate the return type of pango_fc_font_lock_face
    - Fix a problem with ellipses getting the wrong font
    - fc: Improve filtering by font format
    - Re-add PangoFcFont to public headers
    - Install PangoFc and PangoOT introspection files
    - Fix ink rectangles to have positive height
    - Fix mark positioning
    - Switch to using harfbuzz for metrics

    Overview of changes in 1.44.3
    =============================
    - Install pango-ot headers
    - Make subpixel positioning optional
    - fc: Ignore fonts with unsupported formats

    Overview of changes in 1.44.2
    =============================
    - Disable ligatures when letterspacing
    - Set design coords on hb_font_t
    - Expose more font options in pango-view
    - OS X: Make 'system-ui' font work
    - Keep deprecated pango-fc apis in headers
    - Make hex boxes work, always
    - introspection: Various build fixes
    - introspection: Add PangoPT, PangoFT2 namespaces
    - layout: Make the new line-spacing opt-in

    Overview of changes in 1.44.1
    =============================
    - Fix a crash with allow_break attributes
    - Fix Emoji spacing
    - Fix up includes and pkg-config requires
    - Correct some cases for hyphen insertion

    Overview of changes in 1.44.0
    =============================
    - Use harfbuzz for shaping on all platforms
    - Stop using freetype for font loading; this
        drops support for type1 and bitmap fonts
    - Add a getter for hb_font_t
    - Make PangoCoverage a GObject
    - Add a pango_tailor_break api
    - font metrics: Add line height
    - layout: Support line spacing
    - layout: Draw hyphens for line breaks
    - Add an attribute to suppress line breaking
    - cairo: Don't render hex boxes for space
    - Add an attribute to show invisible characters
    - Stop quantizing glyph positions
    - Add tests for itemization and line breaking
    - Remove language and shape engine remnants
    - Rename meson options: gtk_doc, introspection
    - Require GLib 2.59.2
    - Require Harfbuzz 2.0

but as mentioned initially, I don't think I can go deeper in my “blind”
(I don't know anything about pango or rendering in general) bisection of
pango1.0.

At first, the upstream/fix-deadlocks* branches sounded promising,
especially with such a commit message:

    Fix hangs that people have observed

but those are regression fixes in the 1.48.x series, and a quick look
around in previous versions doesn't seem to show similar initialization
routines, so someone else is likely happening in 1.44.x…

Help/pointers appreciated.


Cheers,

* Cyril Brulebois <kibi@debian.org> [210429 18:03]:
     ../pango/pangofc-font.c:371:44: error: ‘HB_OT_METRICS_UNDERLINE_SIZE’ undeclared (first use in this function); did you mean ‘HB_OT_METRICS_TAG_UNDERLINE_SIZE’?
[..]

Just passing by, but I can build 1.44.0 with Debians 1.44.6 debian/
directory, when adding this patch from upstream:
https://gitlab.gnome.org/GNOME/pango/-/commit/d835004502c801a8a16cc436a38900e548ecde52.patch

HTH,
Chris

Cyril Brulebois, le jeu. 29 avril 2021 18:54:56 +0200, a ecrit:

commit d835004502c801a8a16cc436a38900e548ecde52
Author: Ebrahim Byagowi <ebrahim@gnu.org>
Date:   Sat Aug 10 14:05:40 2019 +0430

    Use latest version of metrics naming in pangofc-font

diff --git a/pango/pangofc-font.c b/pango/pangofc-font.c
index 98e77288..21644b57 100644
--- a/pango/pangofc-font.c
+++ b/pango/pangofc-font.c
@@ -365,16 +365,16 @@ get_face_metrics (PangoFcFont      *fcfont,
 #if HB_VERSION_ATLEAST(2,5,4)
   hb_position_t position;

-  if (hb_ot_metrics_get_position (hb_font, HB_OT_METRICS_UNDERLINE_SIZE, &position))
+  if (hb_ot_metrics_get_position (hb_font, HB_OT_METRICS_TAG_UNDERLINE_SIZE, &position))
     metrics->underline_thickness = position;

-  if (hb_ot_metrics_get_position (hb_font, HB_OT_METRICS_UNDERLINE_OFFSET, &position))
+  if (hb_ot_metrics_get_position (hb_font, HB_OT_METRICS_TAG_UNDERLINE_OFFSET, &position))
     metrics->underline_position = position;

-  if (hb_ot_metrics_get_position (hb_font, HB_OT_METRICS_STRIKEOUT_SIZE, &position))
+  if (hb_ot_metrics_get_position (hb_font, HB_OT_METRICS_TAG_STRIKEOUT_SIZE, &position))
     metrics->strikethrough_thickness = position;

-  if (hb_ot_metrics_get_position (hb_font, HB_OT_METRICS_STRIKEOUT_OFFSET, &position))
+  if (hb_ot_metrics_get_position (hb_font, HB_OT_METRICS_TAG_STRIKEOUT_OFFSET, &position))
     metrics->strikethrough_position = position;
 #endif
 }

which probably can thus be used as it is.

Samuel

Chris Hofstaedtler <zeha@debian.org> (2021-04-29):

Yes, the earlier stable releases from the 1.44.x series looked like
build-side stabilization and I suppose I could end up cherry-picking one
or more patches to get a bit further.

Anyway, looking at the diffstat between 1.43.0 (last known good) and
1.44.4 (first known bad), we have this:
 179 files changed, 19157 insertions(+), 7045 deletions(-)

Between 1.43.0 and 1.44 (if it were an earlier known bad):
 165 files changed, 11006 insertions(+), 6766 deletions(-)

which is a huge diff still.

From your earlier report on #987449[1], something around
pango_default_break seemed to be happening. (In passing, I'd love to
learn how you generated it so that I can try my luck with #987377.)

 1. https://bugs.debian.org/987449#15

and that helps a lot dig into that huge diff, which touched the file
holding that function, adding e.g. this hunk:

+           /* Dependent Vowels for Indic language */
+           if (_pango_is_Virama (prev_wc) ||
+               _pango_is_Vowel_Dependent (prev_wc))
+             attrs[i].backspace_deletes_character = TRUE;

Not knowing anything about it:
https://en.wikipedia.org/wiki/Indic_languages

redirects to:
https://en.wikipedia.org/wiki/Indo-Aryan_languages

which lists Sinhala, Marathi, but neither Persian nor Kannada.

There are a lot of other changes in that file anyway…


In the end, I might try and bisect deeper, if I can get stuff to build
without too much hassle…


Cheers,

* Cyril Brulebois <kibi@debian.org> [210430 09:47]:

Indeed :-(
- I created a Debian bullseye install with a kernel version matching
  the installer, and also installed linux-perf(-5.10)
- From the running, already stuck installer, I mounted the existing
  install, did the whole mount --bind stuff for sys, proc, dev.
  I cannot remember if one needs to actually chroot.
- And then its mostly `perf top` and/or `perf record` + `perf
  report`. The former gives you an immediate view, while the latter
  writes into a file which you can look at later.

Having more debug symbols would have been useful, but I did not
manage to set that up.

Chris

(hangs when selecting Sinhala), but so much changed between 1.43.0 and
1.44.0 that bisecting between them might not be a very good route.

I've also been able to attach a debugger to debconf. My preliminary finding
is: we enter gtk_container_idle_sizer() in GTK 2 and never exit, because
every time we go into gtk_container_check_resize(), we end up in
gtk_text_layout_emit_changed() which emits a signal that eventually calls
_gtk_container_queue_resize(), so gtk_container_idle_sizer() has more work
to do and we loop indefinitely.

I assume this isn't meant to happen? But please don't mistake me for an
expert on GTK 2, or Pango, or Harfbuzz.

Steps go something like this (is there a better way?):

Host preparation:
- Build interesting libraries (pango1.0, harfbuzz, glib2.0, gtk+2.0) with
  DEB_BUILD_OPTIONS="noopt nostrip nocheck"; we'll need these later

VM preparation:
- Create a new VM booting from virtio disk /dev/vda
- Have an up-to-date bullseye installation on /dev/vda
  (I copied an existing bullseye GNOME installation and upgraded it)
- Make sure a user can log in via ssh and can sudo to root
- Power off the VM
- Attach a CD-ROM device with firmware-bullseye-DI-rc1-amd64-netinst.iso
- Make it boot from CD-ROM
- Boot it
- Start graphical installer
- Proceed with installation, in English, until we get to "Configure the
  network"
- Hit Cancel and Go Back until we get to the menu of installation steps
- Send Ctrl+Alt+F2
- modprobe ext4
- mount /dev/vda1 /mnt
- mount --rbind /dev /mnt/dev
- mount --rbind /proc /mnt/proc
- mount --rbind /sys /mnt/sys
- mkdir /mnt/d-i
- mount --bind / /mnt/d-i
- chroot /mnt bash
  - /etc/init.d/ssh start
  - exit
- ssh in to the chroot'd ssh server; this is our debugging environment
- On the graphical console, send Ctrl+Alt+F5 to go to X11

Debugging loop:
- Use rsync/ssh/cp to get the unstripped libraries into /d-i in the ssh
  session (the root directory of the d-i environment)
- Ctrl+Alt+F2, use udpkg -i to install them in the d-i environment
- In the ssh session, use pstree -p to look at the process hierarchy,
  then kill debconf and everything below it
- Hit Cancel and Go Back until we get to the menu of installation steps
- "Choose language"
- Select Sinhala
- In the ssh session:
    sudo gdb -ex 'set pagination off' \
    -ex 'set sysroot /d-i' \
    -ex 'set logging file /log' \
    -ex 'set logging overwrite off' \
    -ex 'set logging on' \
    /d-i/usr/bin/debconf `pgrep debconf`
- Debug interactively until you get a better idea

Some sample backtraces are attached.

My next step is going to be to try to hack Harfbuzz to disable the Indic
shaper (which is what seems to be in use here) and see whether that stops
the infinite loop. That's unlikely to be an acceptable solution, but it'll
at least tell us whether the Indic shaper is what's triggering this.

    smcv

...
that seems to be enough to make installation proceed.

Again, this is probably not an acceptable solution: Harfbuzz has shapers
for complex scripts for a reason, and I suspect someone who can speak
the relevant language would find the text either ugly or unreadable
when using the default shaper. However, I hope this does at least point
someone who knows more about the mechanics of text rendering and/or GTK
relayout in the right direction...

I think the reason this regressed between Pango 1.43.0 and 1.44.0 might
just be that Pango 1.44.0 uses Harfbuzz to implement more of its own
functionality.

    smcv

Hi Simon,

And thanks a lot for debugging/digging/sharing!

Simon McVittie <smcv@debian.org> (2021-05-03):
locally by deploying a hack harfbuzz udeb with your patch on top of
exactly the components used for D-I Bullseye RC 1), but that doesn't
explain the failure for Persian.

Of course, one can also apply the same kind of hack to other scripts
that share the same dedicated Arabic shaper, via:

    -       return &_hb_ot_complex_shaper_arabic;
    +       return &_hb_ot_complex_shaper_default;

While the problem was obvious for certain languages (first screen after
language selection, #987449), and could exist for any languages using a
script that comes with a dedicated shaper, that still doesn't explain
the regression with English in rescue mode (#987377). I fear this issue
could actually show up with any languages (e.g. all those sharing the
Latin script), depending on $some_conditions.

I think I'll add a GTK-level patch to easily spot whether that last
issue is similar (infinite loop with signals all around the place), and
maybe try to pinpoint when the problem started to happen in the pango1.0
Git repository.

There's also a problem with Swedish during regular install (not rescue),
that I haven't tried to reproduce yet:
https://lists.debian.org/debian-boot/2021/05/msg00018.html

This would seem to confirm one doesn't need a “complex” script to get
the issue, Latin is enough…

Despite all the above, once we have a better grasp of what's happening,
would it seem reasonable to add a hack in whatever would make sense
(pango1.0, harfbuzz, and/or gtk+2.0), but only in the udeb build, so
that we dodge the issue for Bullseye without impacting installed
systems/deb packages? If git bisecting yields a prospective (set of)
patch(es) of course…

ISTR you proposed then implemented changes to alleviate one of the
blockers for GTK3 in d-i (vte vs. libstdc++ if memory serves), but I
haven't managed to look at it during the bullseye cycle; we definitely
should look into gtk3-ifying (perhaps gtk4-ifying, time flies) d-i at
some point, but that's very likely bookworm material at this stage…

Looking at the “user-level” changes in the upstream documentation, that
is indeed what seems very plausible to my inexperienced eyes.


Cheers,

*if* we can find a suitable hack.

pango1.0 just has one build shared between the udeb and the full .deb,
so if the hack goes there, a prerequisite would be to build it twice
(fairly straightforward, just annoying). gtk+2.0 and harfbuzz already
build twice, so it would be easy to add -DDEBIAN_INSTALLER to the udeb
build, or similar.

The problem is finding the right hack. I don't think bisecting Pango is
necessarily going to get you very far: if you arrive at the version/commit
where Pango switched from doing text layout internally to using Harfbuzz
for everything, then that'll be a dead end. Conditionally reverting the
switch from internal layout engines to Harfbuzz doesn't seem likely to
yield a reasonable patch - the internal layout engines were deleted about
2 years ago and the rest of the pango codebase has moved on since then.

I've added some printf debugging to GTK2 (attached) and it looks as
though the problem is that the layout code flaps between two different
pixel sizes for the same "line" of text (GTK calls it a "line" but
it's really more like a paragraph - it will be line-wrapped to fit the
available width). With the test-case of going back to language selection
and choosing Sinhala, which I've been using because it's an easy one
to reproduce:

May 13 10:24:18 debconf: cdebconf_gtk (process:4590): Gtk - DEBUG: IA__gtk_text_layout_validate_yrange: GtkTextLayout 0x560b7b142110: validating near anchor 0x7f28cd080e20 from 0px to 576px
May 13 10:24:18 debconf: cdebconf_gtk (process:4590): Gtk - DEBUG: IA__gtk_text_layout_validate_yrange: anchor 0x7f28cd080e20 is in line 0
May 13 10:24:18 debconf: cdebconf_gtk (process:4590): Gtk - DEBUG: IA__gtk_text_layout_validate_yrange: validating a subsequent line 0
May 13 10:24:18 debconf: cdebconf_gtk (process:4590): Gtk - DEBUG: IA__gtk_text_layout_validate_yrange: line content: තේරූ භාෂාව සඳහා ස්ථාපකයේ පරිවර්තනය අසම්පූර්ණයි.
May 13 10:24:18 debconf: cdebconf_gtk (process:4590): Gtk - DEBUG: IA__gtk_text_layout_validate_yrange: was 72x102px
May 13 10:24:18 debconf: cdebconf_gtk (process:4590): Gtk - DEBUG: IA__gtk_text_layout_validate_yrange: now 73x119px
May 13 10:24:18 debconf: cdebconf_gtk (process:4590): Gtk - DEBUG: IA__gtk_text_layout_validate_yrange: validating a subsequent line 1
May 13 10:24:18 debconf: cdebconf_gtk (process:4590): Gtk - DEBUG: IA__gtk_text_layout_validate_yrange: line content:
May 13 10:24:18 debconf: cdebconf_gtk (process:4590): Gtk - DEBUG: IA__gtk_text_layout_validate_yrange: was 0x16px
May 13 10:24:18 debconf: cdebconf_gtk (process:4590): Gtk - DEBUG: IA__gtk_text_layout_validate_yrange: now 0x16px
May 13 10:24:18 debconf: cdebconf_gtk (process:4590): Gtk - DEBUG: IA__gtk_text_layout_validate_yrange: validating a subsequent line 2
May 13 10:24:18 debconf: cdebconf_gtk (process:4590): Gtk - DEBUG: IA__gtk_text_layout_validate_yrange: line content: මෙහි තේරුම සමහර සංවාද ඉංග්<200d>රීසියෙන් පෙනීමේ සැලකිය යුතු ඉඩක් ඇති බවයි.
May 13 10:24:18 debconf: cdebconf_gtk (process:4590): Gtk - DEBUG: IA__gtk_text_layout_validate_yrange: was 66x153px
May 13 10:24:18 debconf: cdebconf_gtk (process:4590): Gtk - DEBUG: IA__gtk_text_layout_validate_yrange: now 66x153px
May 13 10:24:18 debconf: cdebconf_gtk (process:4590): Gtk - DEBUG: IA__gtk_text_layout_validate_yrange: validating a subsequent line 3
May 13 10:24:18 debconf: cdebconf_gtk (process:4590): Gtk - DEBUG: IA__gtk_text_layout_validate_yrange: line content:
May 13 10:24:18 debconf: cdebconf_gtk (process:4590): Gtk - DEBUG: IA__gtk_text_layout_validate_yrange: was 0x16px
May 13 10:24:18 debconf: cdebconf_gtk (process:4590): Gtk - DEBUG: IA__gtk_text_layout_validate_yrange: now 0x16px
May 13 10:24:18 debconf: cdebconf_gtk (process:4590): Gtk - DEBUG: IA__gtk_text_layout_validate_yrange: validating a subsequent line 4
May 13 10:24:18 debconf: cdebconf_gtk (process:4590): Gtk - DEBUG: IA__gtk_text_layout_validate_yrange: line content: ඔබට විකල්ප භාෂාව පිළිබඳ හොඳ අවබෝධයක් නොමැති නම්, එක්කෝ වෙනස් භාෂාවක් තෝරා ගැනීම හෝ ස්ථාපනය නැවැත්වීමට නිර්දේශ කරයි.
May 13 10:24:18 debconf: cdebconf_gtk (process:4590): Gtk - DEBUG: IA__gtk_text_layout_validate_yrange: was 72x289px
May 13 10:24:18 debconf: cdebconf_gtk (process:4590): Gtk - DEBUG: IA__gtk_text_layout_validate_yrange: now 70x323px
May 13 10:24:18 debconf: cdebconf_gtk (process:4590): Gtk - DEBUG: IA__gtk_text_layout_validate_yrange: Revalidated invalid lines from 0 to 4
May 13 10:24:18 debconf: cdebconf_gtk (process:4590): Gtk - DEBUG: IA__gtk_text_layout_validate_yrange: Top of first line y=0
May 13 10:24:18 debconf: cdebconf_gtk (process:4590): Gtk - DEBUG: IA__gtk_text_layout_validate_yrange: First line 0, y=0
May 13 10:24:18 debconf: cdebconf_gtk (process:4590): Gtk - DEBUG: IA__gtk_text_layout_validate_yrange: Last line 4, y=627
May 13 10:24:18 debconf: cdebconf_gtk (process:4590): Gtk - DEBUG: IA__gtk_text_layout_validate_yrange: Delta height 51
...
May 13 10:24:18 debconf: cdebconf_gtk (process:4590): Gtk - DEBUG: IA__gtk_text_layout_validate_yrange: GtkTextLayout 0x560b7b142110: validating near anchor 0x7f28cd080e20 from 0px to 627px
May 13 10:24:18 debconf: cdebconf_gtk (process:4590): Gtk - DEBUG: IA__gtk_text_layout_validate_yrange: anchor 0x7f28cd080e20 is in line 0
May 13 10:24:18 debconf: cdebconf_gtk (process:4590): Gtk - DEBUG: IA__gtk_text_layout_validate_yrange: validating a subsequent line 0
May 13 10:24:18 debconf: cdebconf_gtk (process:4590): Gtk - DEBUG: IA__gtk_text_layout_validate_yrange: line content: තේරූ භාෂාව සඳහා ස්ථාපකයේ පරිවර්තනය අසම්පූර්ණයි.
May 13 10:24:18 debconf: cdebconf_gtk (process:4590): Gtk - DEBUG: IA__gtk_text_layout_validate_yrange: was 73x119px
May 13 10:24:18 debconf: cdebconf_gtk (process:4590): Gtk - DEBUG: IA__gtk_text_layout_validate_yrange: now 72x102px
May 13 10:24:18 debconf: cdebconf_gtk (process:4590): Gtk - DEBUG: IA__gtk_text_layout_validate_yrange: validating a subsequent line 1
May 13 10:24:18 debconf: cdebconf_gtk (process:4590): Gtk - DEBUG: IA__gtk_text_layout_validate_yrange: line content:
May 13 10:24:18 debconf: cdebconf_gtk (process:4590): Gtk - DEBUG: IA__gtk_text_layout_validate_yrange: was 0x16px
May 13 10:24:18 debconf: cdebconf_gtk (process:4590): Gtk - DEBUG: IA__gtk_text_layout_validate_yrange: now 0x16px
May 13 10:24:18 debconf: cdebconf_gtk (process:4590): Gtk - DEBUG: IA__gtk_text_layout_validate_yrange: validating a subsequent line 2
May 13 10:24:18 debconf: cdebconf_gtk (process:4590): Gtk - DEBUG: IA__gtk_text_layout_validate_yrange: line content: මෙහි තේරුම සමහර සංවාද ඉංග්<200d>රීසියෙන් පෙනීමේ සැලකිය යුතු ඉඩක් ඇති බවයි.
May 13 10:24:18 debconf: cdebconf_gtk (process:4590): Gtk - DEBUG: IA__gtk_text_layout_validate_yrange: was 66x153px
May 13 10:24:18 debconf: cdebconf_gtk (process:4590): Gtk - DEBUG: IA__gtk_text_layout_validate_yrange: now 66x153px
May 13 10:24:18 debconf: cdebconf_gtk (process:4590): Gtk - DEBUG: IA__gtk_text_layout_validate_yrange: validating a subsequent line 3
May 13 10:24:18 debconf: cdebconf_gtk (process:4590): Gtk - DEBUG: IA__gtk_text_layout_validate_yrange: line content:
May 13 10:24:18 debconf: cdebconf_gtk (process:4590): Gtk - DEBUG: IA__gtk_text_layout_validate_yrange: was 0x16px
May 13 10:24:18 debconf: cdebconf_gtk (process:4590): Gtk - DEBUG: IA__gtk_text_layout_validate_yrange: now 0x16px
May 13 10:24:18 debconf: cdebconf_gtk (process:4590): Gtk - DEBUG: IA__gtk_text_layout_validate_yrange: validating a subsequent line 4
May 13 10:24:18 debconf: cdebconf_gtk (process:4590): Gtk - DEBUG: IA__gtk_text_layout_validate_yrange: line content: ඔබට විකල්ප භාෂාව පිළිබඳ හොඳ අවබෝධයක් නොමැති නම්, එක්කෝ වෙනස් භාෂාවක් තෝරා ගැනීම හෝ ස්ථාපනය නැවැත්වීමට නිර්දේශ කරයි.
May 13 10:24:18 debconf: cdebconf_gtk (process:4590): Gtk - DEBUG: IA__gtk_text_layout_validate_yrange: was 70x323px
May 13 10:24:18 debconf: cdebconf_gtk (process:4590): Gtk - DEBUG: IA__gtk_text_layout_validate_yrange: now 72x289px
May 13 10:24:18 debconf: cdebconf_gtk (process:4590): Gtk - DEBUG: IA__gtk_text_layout_validate_yrange: Revalidated invalid lines from 0 to 4
May 13 10:24:18 debconf: cdebconf_gtk (process:4590): Gtk - DEBUG: IA__gtk_text_layout_validate_yrange: Top of first line y=0
May 13 10:24:18 debconf: cdebconf_gtk (process:4590): Gtk - DEBUG: IA__gtk_text_layout_validate_yrange: First line 0, y=0
May 13 10:24:18 debconf: cdebconf_gtk (process:4590): Gtk - DEBUG: IA__gtk_text_layout_validate_yrange: Last line 4, y=576
May 13 10:24:18 debconf: cdebconf_gtk (process:4590): Gtk - DEBUG: IA__gtk_text_layout_validate_yrange: Delta height -51

Obviously I can't read Sinhala, but
"තේරූ භාෂාව සඳහා ස්ථාපකයේ පරිවර්තනය අසම්පූර්ණයි."
presumably ought to be one size or the other, if everything else (e.g.
font) is kept constant: it shouldn't flap between 73x119px and 72x102px
on consecutive layout runs. I think the next step is to trace into how GTK
is arriving at those sizes, and where the size flapping is taking place
(probably in GTK, Pango or Harfbuzz).

The given sizes also seem weird: the width is a lot less than I would have
expected, as though the text is getting wrapped one character per line or
something like that... perhaps if we managed to escape from this infinite
loop, GTK would allocate more width for the GtkTextView and the text would
get re-wrapped at a sensible width?

Another possible route to look at might be to force the GtkTextView to
fill the whole width of the window, so that a narrow layout is never
considered?

The steps to use GTK 3 in d-i would be:

- convert cdebconf-gtk-udeb from GTK 2 to GTK 3
- convert cdebconf-gtk-entropy from GTK 2 to GTK 3
- convert cdebconf-gtk-terminal from GTK 2 to GTK 3 and, simultaneously,
  from libvte9-udeb to libvte-2.91-0-udeb

The big blocker for libvte-2.91-0-udeb used to be that it's written in
C++, but I've switched it to be built with -static-libstdc++ so that we
don't need a libstdc++6-udeb (its public API/ABI is GLib-flavoured C,
only the internals are C++). The result is that libvte-2.91-0-udeb isn't
much larger than libvte9-udeb.

matchbox-keyboard-udeb also uses GTK 2 (#967624), but if it's a separate
GUI program then it doesn't *necessarily* have to move to GTK 3 at the
same time that cdebconf-gtk does. I'm also not sure whether it's actively
used for anything?

    smcv

Hi Simon,

Simon McVittie <smcv@debian.org> (2021-05-13):

Yeah, that's what I had in mind.

I fear this might be the case, yes…

I haven't looked into that at all (at least yet)…

Based on the (perceived) unlikeliness that we might find a fix for GTK 2
(yeah, inertia… but it had been working so neatly for so long that
moving away from it just hadn't happened…), I've checked what would
happen with GTK 3 in cdebconf and cdebconf-gtk-terminal (I had forgotten
about cdebconf-gtk-entropy until writing this reply).

The installer seems to be working somewhat. I'm seeing strange things
regarding layout, regarding widget expansion (basically we have some
wasted vertical space). I'm also seeing a different behaviour regarding
the expose (GTK 2) vs. draw (GTK 3) event handling, meaning the banner
doesn't get repainted automatically; trying to do that on my own results
on it being painted over and over again (that happens with a
pango_cairo_show_layout), meaning “Mode de récupération” (fr) gets
rendered on top of “Rescue mode” (en) after selecting French. I wouldn't
call any of those two issues a blocker for 11.0 *if we were to go the
“Switch to GTK 3” route*.

I'm also seeing a warning when spawning a shell, that comes from
vte2.91; again, not a blocker. But exiting the shell leads to a crash,
and the installer gets restarted. The syslog has this:

    May 17 15:28:46 debconf: cdebconf_gtk (process:257): GLib - DEBUG: setenv()/putenv() are not thread-safe and should not be used after threads are created
    May 17 15:28:46 debconf: cdebconf_gtk (process:257): VTE - WARNING: (../src/vtepty.cc:670):bool _vte_pty_spawn_sync(VtePty*, const char*, const char* const*, const char* const*, GSpawnFlags, GSpawnChildSetupFunc, gpointer, GDestroyNotify, GPid*, int, GCancellable*, GError**): runtime check failed: ((spawn_flags & forbidden_spawn_flags()) == 0)
    May 17 15:31:14 kernel: [  218.924148] event_listener[263]: segfault at 0 ip 00007f2ecb006500 sp 00007f2ecaf12a98 error 4 in plugin-terminal.so[7f2ecb006000+2000]

I'm assuming the DEBUG/WARNING parts aren't too scary, and that the
segfault upon exit might be unrelated. I would call that a blocker for a
new release candidate of the installer since that would leave /target
mounted, possibly with other filesystems, and one couldn't re-enter the
shell.

I'll check what needs to happen with cdebconf-gtk-entropy, and whether
that might have a link with the segfault…

It seems safe to ignore at the moment. (ISTR that was an optional
component that could be used on some devices, rather than something we
rely on in most cases, but I could be wrong.)


Cheers,

Cyril Brulebois <kibi@debian.org> (2021-05-17):

(No change.)

Additionally, even with all 3 cdebconf-gtk-* packages converted, we
still get libgtk2.0-0-udeb pulled into a netboot-gtk build, because this
package pulls it:

    build/pkg-lists/gtk-common:libgail18-udeb

Adding debian-accessibility@ to the loop for awareness, and for input
since it's been a while since I last looked into accessibility details…
Would we need some prospective libgail-3-0-udeb to replace it in a GTK 3
world?


And for those not following #debian-boot, I'm finding myself between a
rock and a hard place, as both options (trying to work around the
rendering-related hangs versus switching to GTK 3 at the last moment)
are very far from ideal… I'm not sure what we'll end up doing, and I'm
happy to get some “votes” pro or against any of those options, and to
hear about other options entirely.


Cheers,

Cyril Brulebois, le lun. 17 mai 2021 18:12:35 +0200, a ecrit:

gail is only needed for gtk2, gtk3 includes the accessibility stack
by itself.

Samuel

My instinct is that it's far, far too late to be moving to GTK 3 this
cycle, and I'd prefer to get a suitable hack into GTK 2 if we can
find one. We can make it #ifdef DEBIAN_INSTALLER to avoid disrupting
the installed system.

I think d-i will need to move to GTK 3 early in the Debian 12 cycle,
but hard freeze is not the time to be fast-forwarding through 10 years
of GUI library development.

I've continued to look into GTK2/Pango with some rather extensive printf
debugging. I have other responsibilities and I'm trying to learn how
GTK/Pango text layout works from first principles (I'm in the GNOME
team but not really a GUI developer), so it's slow going, but I have
some ideas for things that might be able to break the loop. It's not
clear to me yet whether this is a GTK 2 bug, or a Pango bug, or what.
GNOME team members who actually know what they're doing are welcome
to step in any time.

Upstream are going to be incredibly reluctant to help us with GTK 2,
given that it has been deprecated in favour of GTK 3 for a decade, and
is officially EOL now that GTK 4 has stable releases. Pango does seem
like it's doing the wrong thing here, but perhaps GTK 2 or cdebconf is
holding it wrong.

Getting
https://salsa.debian.org/installer-team/cdebconf/-/merge_requests/4
into cdebconf would make this easier, although we can revert it for the
sake of a smaller diff once we think we have a solution if the release
team are grumpy about it.

Philip Hands wanted to get this running under automated testing, but my
current GTK 2 and Pango patches are spamming the syslog at 3 million lines
a minute when they get into the loop, so we're going to have logistical
difficulties in saving logs.

It looks as though the problem is that the size GTK chooses for a
GtkTextView (a debconf "note" or similar) is flapping between two
values. GTK asks Pango "if you wrap lines at width x, how much space
will you need?", then uses the result to re-run the layout algorithm,
which changes the width available, which causes the layout algorithm to
be re-run and so on. Under normal circumstances, this runs for a few
iterations and then stabilizes at one size, terminating the loop, but
when I select Sinhala from the language menu, the width flaps between
71 and 72 pixels, with each re-layout resulting in the other width.

I think this might be related to the fact that when the layout is
calculated at the narrower width, Pango decides that there isn't space
to put the "." at the end of a paragraph on the same line as the last
word, and so wraps it to the next line, which is fine; except that
you'd expect the space required for the last word to get a bit smaller
when the "." is not included, but actually Pango tells GTK that it will
need 1 pixel *more* for "අසම්පූර්ණයි" than for "අසම්පූර්ණයි.", resulting in it
flapping between the narrower and wider width. I don't know why that
happens - perhaps the Indic shaper is drawing a character at the end of
a line more elaborately, or something?

However, either 71 or 72 pixels seems a ridiculous width to be trying to
squish multiple sentences into, so arguably it's a bug in GTK 2's layout
algorithm that it is even considering that as a size. The GTK 2 layout
algorithm does not necessarily make much sense, and the fact that it
can feed back into itself is probably a GTK 2 design flaw. During the
GTK 3 cycle it was redone to be in terms of "height for width" (if I
give you this width, how much height will you need? the mode cdebconf
will end up using) and "width for height" (the opposite, rarely used) -
so hopefully GTK 3 avoids this failure mode. However, it also seems wrong
that telling Pango that less width is available can result in it saying
it needs *more* minimum width, and maybe preventing this from happening
would get GTK 2 to do the right thing.

My next thing to get a try when I get a chance will be to make GTK refuse
to obey Pango when GTK asks Pango to stick to a width limit and Pango goes
outside that limit, with a g_warning(). This might result in the text
being clipped at the right margin (or left margin in Hebrew/Arabic), or
even having its "ink rectangle" overlap with the next widget to the right
(or left); but for d-i, which always uses the full screen width and in
practice has a generous amount of space for its text, we might get away
with it? In a very brief test that seemed to resolve the Sinhala thing,
but I haven't tried the other paths that lead to infinite loops. Please
could someone who has tried the other scenarios provide a walkthrough?

If you want to replicate my super-verbose printf debugging:

* https://salsa.debian.org/installer-team/cdebconf/-/merge_requests/4
* apply https://people.debian.org/~smcv/bug987587/20210516/gtk+2.0/ to
  gtk+2.0 (the patch to d/rules applies directly, the patch to GTK
  applies via d/patches)
* apply https://people.debian.org/~smcv/bug987587/20210516/pango/ to
  pango1.0
* repeat test steps that I previously sent to this bug
* look at the syslog and despair
* you will want to truncate the syslog before repeating debugging,
  and send some strategic SIGSTOP to the debconf process when not actively
  letting it run, because otherwise the initramfs will run out of RAM

I'll send a separate reply to -boot about the possibility of moving to
GTK 3.

    smcv

Hi Simon!

Simon McVittie <smcv@debian.org> (2021-05-17):

If you're happy with further assisting us with getting away with GTK 2
one more time, that's very fine with me! Huge thanks for all your work
and help so far!

From my little “feasibility survey” over the last few days, there seem
to be a number of obvious issues to solve indeed!

I'm very happy to see this patch merged, and possibly released into
bullseye. The most obvious side effect I can anticipate is possibly
bigger /var/log/installer/syslog files in the installed systems, but
users are expected to compress them anyway before attaching them in
installation reports (BTS and/or lists size limits).

Would you like that to be released into unstable right away? I'm happy
to deal with it if that helps testing stuff.

Without looking into the code, one might try and keep track of results
that have been seen, and if N/N+1 is detected, maybe just assume this
should be N+1 and move on with other computations? But anyway, further
down your mail you seemed to have ideas already.

Maybe the Indic shaper makes the problem more obvious but I definitely
saw a similar hang (even if I hadn't patched the installer to generate
log lines to make extra sure) with plain English when starting in rescue
mode.
 - buttons at the bottom;
 - checkboxes to toggle password/passphrase visibility (but they might
   be on their own line actually);
 - anything that can be accompanied by a scrollbar.

All in all, we should have plenty of space most of (if not all) the
time. And to be honest, if we end up having rendering glitches but a
non-hanging/crashing installer for Buster, I can live with it.

Walkthrough for the English vs. rescue mode, using QEMU:

 - Get an sda.img image to “rescue” (see below).

 - Build a netboot/gtk/mini.iso from git master (the TRANSSTATUS is
   important for languages that aren't English, as that file makes it
   possible to distinguish which translations are usable):
     make -C build/ rebuild_netboot-gtk USE_UDEBS_FROM=bullseye TRANSSTATUS=translation-status

 - Run build/dest/netboot/gtk/mini.iso against it with:
     kvm -m 1G -cdrom build/dest/netboot/gtk/mini.iso -hda sda.img -boot d

 - Accept all default values.

 - Once the disk has been detected and scanned, a number of options
   are proposed, pick the first one (executing a shell in the context of
   the installed system) → hang.

To speed things up, you can fetch an image to rescue along with my last
mini.iso build here:a
https://mraw.org/~kibi/smcv/

Note: The hang doesn't happen if one picks French or German at the
beginning.

Thanks for the details, I'll give it a shot!


Cheers,

It wasn't obvious to me where we'd keep track of this, or how correct
it would be to do that - cache invalidation is reputed to be one of the
hardest problems in computer science, and this would be one more thing
that needs to be invalidated on at least those occasions when the layout
has legitimately changed (but without invalidating it too often because
that would destroy the workaround).

Having reproduced the English/rescue issue and got further with the
Sinhala/install issue with the GTK MR referenced below, I think it can
also happen that the layout flaps by an amount greater than 1 pixel
(I think the most I've seen is 4px), so a special case for n/n+1 isn't
going to be enough.

One of the reasons this took me a while is that I got distracted by the
difference I was seeing being exactly 1px, which I thought might be to
do with GTK adding 1px of extra width to make sure there's space for a
cursor - but after tracing through it, it seemed like GTK is always
adding/removing that width correctly.

The other thing I wanted to try was to make cdebconf create the GtkTextView
in an empty state, and then populate it with text a little later (perhaps
after layout but before drawing, or perhaps drawing one frame without text
and then adding the text in the next frame, I'm not actually sure). And
that also works:
https://salsa.debian.org/installer-team/cdebconf/-/merge_requests/6

Following the rule of thumb that bad interactions between two components
should often be fixed on *both* sides, I'd be tempted to clone this bug,
reassign to both gtk+2.0 and cdebconf, and apply both changes.

As for Pango, I'm afraid figuring out whether it is doing something wrong
here is beyond my expertise. If I can characterize the maybe-bug in a clear
enough way I can try asking upstream - but as I said before, upstream will
be very reluctant to touch this as soon as I mention GTK 2, which has been
on life-support for a decade and is now officially dead.

    smcv

Simon McVittie <smcv@debian.org> (2021-05-19):

Looking at the traces I saved when installing in Swedish with just the
GTK patch, I'm seeing a bunch of 1px to 4px differences, but it can even
reach 8px!

    […] we asked Pango to wrap text for width 186px but it now wants 194px. Clamping result to 186px!

That looks good to me, I'll do so in a moment, and deal with (re)trying
to understand everything going on with your patch before merging and
uploading.

As mentioned before, GTK 2 has been working just fine for us… until it
no longer did. It's *very*hugely*appreciated* that you've helped us that
much this time around. We'll do the work next time, and I don't think
it's important enough to try and dive into Pango some more at this
stage, unless someone experiences similar issues with a hrm more recent
and supported toolkit version…

Thanks again, you're a lifesaver!


Cheers,

Control: clone 987587 -2 -3
Control: reassign -2 libgtk2.0-0-udeb 2.24.33-1
Control: retitle -2 libgtk2.0-0-udeb: should avoid GtkTextView relayout loop if width from Pango oscillates
Control: tags -2 + patch pending
Control: reassign -3 cdebconf-gtk-udeb 0.257
Control: retitle -3 cdebconf-gtk-udeb: should give GTK a chance to do layout before adding lots of text
Control: tags -3 + patch
Control: retitle 987587 libpango1.0-udeb: asking for less width can result in more width being requested
Control: severity 987587 normal
Control: tags 987587 + help

As discussed with kibi on the merge requests, let's do this.

Clone -2 is for GTK, and is resolved by
<https://salsa.debian.org/gnome-team/gtk2/-/merge_requests/2>. I'll
handle this.

Clone -3 is for cdebconf, and is resolved by
<https://salsa.debian.org/installer-team/cdebconf/-/merge_requests/6>.
Please don't merge until the new bug number is available.

I'll leave #987587 assigned to Pango, but I'm de-escalating it to non-RC
since it isn't at all obvious to me whether it's a bug, and resolving the
clones -2 and -3 will fix the release-critical issue.

    smcv

...

Nod, absolutely. Thanks very much for your efforts here Simon!