I've received a spam where the "From:" contains the U+3164 HANGUL FILLER
character. This corrupts the display when Mutt is running in GNU Screen.

To reproduce:

  mutt -F muttrc -f mbox

with the attached muttrc and mbox files.

See attached video.webm (screen recording) for what I obtained.

This issue is not reproducible with upstream's GNU Screen 5.0.0,
and not with tmux either.

alignment issue.

Note that a consequence is that one can delete a wrong mail message
because the cursor is badly positioned (this happened when I tried
to get a minimal testcase).

With the default Mutt settings, it seems that one just gets an
alignment issue with the mbox file I provided.

* Vincent Lefevre <vincent@vinc17.net> [251224 14:43]:

Can you check if this was fixed upstream, IOW this works with the
current upstream GNU screen version?

Chris

I said in my bug report:

This issue is not reproducible with upstream's GNU Screen 5.0.0,
and not with tmux either.

And Debian's screen version is based on the head of upstream's
screen-v4 branch, if this is what you meant: this is 4.9.1 +
3 commits (unrelated to this bug) fixing the 3 CVEs, and Debian
has these fixes in 4.9.1-3.

Hi Vincent,

Can you please test with screen 5.0.1-1 from unstable?

Cheers,
Peter Dey

Hi,

I confirm that this is now working correctly, while it was still
buggy with screen 4.9.1-3.1, just before the upgrade.

BTW, I had mentioned an alignment issue in

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1123944#10

With GNOME Terminal, it still occurs, but also outside GNU Screen,
so that one cannot conclude.

But with xterm 409 and Mutt's default settings, e.g.

  mutt -F /dev/null -f mbox

it occurs only inside GNU Screen (but everything is fine with the
muttrc I had provided with xterm, and with or without GNU Screen).

The issue can be seen under bash or zsh with

  printf "x\u3164%*sz\rabc\n" $((COLUMNS-2)) y

For instance, with a 20-column xterm, without GNU Screen (I've added
digits to show columns; they are not part of the output):

12345678901234567890
abc               yz

Ditto with GNU Screen (screen 5.0.1-1):

12345678901234567890
x                 ya
bc

Hi Vincent,
"abc", followed by 15 spaces or non-visible characters, followed by "yz"
at the end of the column?

Using the printf snippet you've provided, I have been unable to
reproduce your expected output, even without introducing screen into the
mix:

xterm 390-1ubuntu3        / bash 5.2.21-2ubuntu4
rxvt-unicode 9.31-3build2 / bash 5.2.21-2ubuntu4

Both of these print:
x□
abc

(I have replaced the non-printable dashed square with an actual unicode
square)

Windows Terminal 1.24.10921.0 / OpenSSH_for_Windows_9.5p2 / bash
5.2.15-2+b7
Putty 0.83                                                / bash
5.2.15-2+b7

Both of these print:
x
abc

For all of the four tested scenarios, the characters "yz" were not
visible.

Cheers,
Peter

of a dashed square, there is a double-width space that is copy-pasted
as U+3164 HANGUL FILLER. This is also what I obtain with GNOME Terminal
under Debian/unstable.

But under Debian/unstable, I get

abc               yz

(with "z" being on the last column) with both

  xterm 409-1
  rxvt-unicode 9.31-3+b2

(whether I use my current config or the default config).

Then I checked consistency with wcwidth:

The wcwidth of U+3164 HANGUL FILLER is 2 under Debian 12 (bookworm)
while it is 0 under Debian/unstable. This would explain the behavior.

I can see in the glibc repository:

commit 25c9c3789ecf35a5dc303261abcef7993112017d
Author: Jules Bertholet <julesbertholet@quoi.xyz>
Date:   2024-02-23 20:54:57 +0000

    localedata: Fix several issues with the set of characters considered 0-width [BZ #31370]

    = `Default_Ignorable_Code_Point`s should have width 0 =

    Unicode specifies (https://www.unicode.org/faq/unsup_char.html#3) that characters
    with the `Default_Ignorable_Code_Point` property

    > should be rendered as completely invisible (and non advancing, i.e. “zero width”),
    if not explicitly supported in rendering.

    Hence, `wcwidth()` should give them all a width of 0, with two exceptions:

    - the soft hyphen (U+00AD SOFT HYPHEN) is assigned width 1 by longstanding precedent
    - U+115F HANGUL CHOSEONG FILLER needs a carveout
      due to the unique behavior of the conjoining Korean jamo characters.
      One composed Hangul "syllable block" like 퓛
      is made up of two to three individual component characters, or "jamo".
      These are all assigned an `East_Asian_Width` of `Wide`
      by Unicode, which would normally mean they would all be assigned
      width 2 by glibc; a combination of (leading choseong jamo) +
      (medial jungseong jamo) + (trailing jongseong jamo) would then have width 2 + 2 + 2 = 6.
      However, glibc (and other wcwidth implementations) special-cases jungseong and jongseong,
      assigning them all width 0,
      to ensure that the complete block has width 2 + 0 + 0 = 2 as it should.
      U+115F is meant for use in syllable blocks
      that are intentionally missing a leading jamo;
      it must be assigned a width of 2 even though it has no visible display
      to ensure that the complete block has width 2.

    However, `wcwidth()` currently (before this patch)
    incorrectly assigns non-zero width to
    U+3164 HANGUL FILLER and U+FFA0 HALFWIDTH HANGUL FILLER;
    this commit fixes that.

    Unicode spec references:
    - Hangul:  §3.12 https://www.unicode.org/versions/Unicode15.0.0/ch03.pdf#G24646 and
      §18.6 https://www.unicode.org/versions/Unicode15.0.0/ch18.pdf#G31028
    - `Default_Ignorable_Code_Point`: §5.21 https://www.unicode.org/versions/Unicode15.0.0/ch05.pdf#G40095.

[...]

where BZ #31370 is at

https://sourceware.org/bugzilla/show_bug.cgi?id=31370

This fix was included in glibc 2.40:

https://sourceware.org/pipermail/libc-alpha/2024-July/158467.html
The GNU C Library version 2.40 is now available
[...]
  [31370] localedata: wcwidth() does not treat
    DEFAULT_IGNORABLE_CODE_POINTs as zero-width
[...]

So it seems that your Ubuntu release is too old.

I suppose that the behavior of GNOME Terminal under Debian/unstable
is a bug (perhaps it is assuming the old, buggy glibc behavior).

Now, concerning GNU Screen, I recall that I get

12345678901234567890
x                 ya
bc

with my current xterm options, but I get

12345678901234567890
x                 ab
c

with the xterm default options, which is also incorrect.
I suppose that's the reverseWrap option that makes the
difference (unset by default, but I set it in my config
as it gives a better behavior).

With a TrueType font, in both cases, there is a dashed square
around the "x".