When piping text into heirloom-mailx, it fails to specify the charset used with
the From: line if the name inherited from /etc/passwd is in UTF-8.

cat file | mail -s "some subject" user@domain.ltd

q-funk:x:1000:1000:Martin-Éric Racine,,,:/home/q-funk:/bin/bash

Setting "set sendcharsets=utf-8" only seems to affect the MIME declaration that
appears in the message headers:

User-Agent: Heirloom mailx 12.4 7/29/08
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 8bit

However, it does not solve the above issue:

Message-Id: <1252199465.516755.6377.nullmailer@iki.fi>
From: Martin-?ric Racine <q-funk@iki.fi>

Yet, we see that:

$ file /etc/passwd
/etc/passwd: UTF-8 Unicode text

Is there something I'm missing?

- -- System Information:
Debian Release: squeeze/sid
  APT prefers testing
  APT policy: (500, 'testing')
Architecture: i386 (i686)

Kernel: Linux 2.6.30-020630-generic (SMP w/1 CPU core)
Locale: LANG=fi_FI.UTF-8, LC_CTYPE=fi_FI.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash

Versions of packages heirloom-mailx depends on:
ii  base-files               5.0.0           Debian base system miscellaneous f
ii  libc6                    2.9-25          GNU C Library: Shared libraries
ii  libgssapi-krb5-2         1.7dfsg~beta3-1 MIT Kerberos runtime libraries - k
ii  libssl0.9.8              0.9.8k-4        SSL shared libraries

heirloom-mailx recommends no packages.

Versions of packages heirloom-mailx suggests:
ii  nullmailer [mail-transport-ag 1:1.04-1.1 simple relay-only mail transport a

- -- no debconf information
iEYEARECAAYFAkqjD5EACgkQeXr56x4Muc3csgCfTAG3dc6f83Va4obf7AxgCKvv
zicAn2ZsQes2yjLcRJipIsz3Hkhfywv0
=N5oH
-----END PGP SIGNATURE-----

* Martin-Éric Racine:

As far as I know, using non-ASCII characters in the GECOS field of
/etc/passwd is not specified at all. So far, I haven't found anything
Debian's main policy file, passwd(5), the adduser(8) and useradd(8)
manpages, nor the documentation of base-passwd. (If you have found more
than I, let me know.)

From an application's standpoint, I'd tend to assume the GECOS field
either to be a comma-sparated string of ASCII characters or a
comma-separated string of byte values.

Yes, I can see how useless both options are is your case. :-)

Basing mailx' interpretation of the GECOS field on the sendcharset
variable, as you suggested is probably not a good idea.

As a workaround, please  try setting your real name to a pre-encoded
string in the .mailrc.

Cheers,
-Hilko

While it is not specified, it has become a de-facto standard in Debian
and its derivatives to use UTF-8 for everything, including the real
name that appears in the GECOS field of /etc/passwd.

We cannot assume that anymore now that Debian uses UTF-8 for everything.

Why not?

Do you really expect all users on a given system to start doing that,
just because their name includes non-ascii characters? Please remember
that both Debian and Ubuntu nowadays allow non-ascii GECOS content
under the presumption that it will be in UTF-8.

Martin-Éric

* Martin-Éric Racine:

I had also thought about UTF-8 becoming the standard encoding in many
places in Debian, be it de-iure or de-facto. But I am not going to
assume that this extends to /etc/passwd.

And how should non-ASCII characters  in other kinds of user databases be
treated, such as NIS or LDAP?

If you can point me to a text passage in the policy (or any relevant
discussion on the mailing lists), I will be happy to reconsider my
opinion.

(it's sendcharsets, sorry for the typo)

sendcharsets is about the target charset.

Not at all. I just thought that this workaround might be helpful for you
until the larger issues get sorted out. Feel free to ignore my
suggestion. :-)

They have alway allowed non-ascii content in the GECOS field, but I see
no such presumption.

From the sources I have seen, existing tools for manipulating
/etc/passwd will happily accept *any* byte sequence from the terminal.
If an administrator has still set his console to iso-8859-1, that's what
is used, without conversion.

That's of course slightly more complicated.  However, as far as
/etc/passwd is concerned, testing the content with 'file' would be a
rather easy way to determine whether to use UTF-8 or something else.
Anyhow, the current approach in the global config to try iso-8859-1
and then utf-8 is broken, because it only works for non-EURO western
languages. The only correct assumption to make is utf-8 and if that
fails, then parse 'env' for whatever deprecated locale the user
currently has.

Every default installation of Debian or Ubuntu writes GECOS content in
UTF-8 based on the fullname that is given when creating the account,
because a Debian or Ubuntu system nowadays defaults to UTF-8 locales
and uses that to produce the GECOS info.

OK, what should we parse then to make a correct guess?

Well, I'd indeed hope we can sort this out.  Besides, I have a dozen
of different environment variables that already set my real name.
Can't we parse any of those? Oh, but that would probably fail too,
because it doesn't tell the encoding either, right?

It indeed accepts anything and, there days, with the locales using
UTF-8 variants be default, it really does get anything. :-)

Martin-Éric

Hello Martin-Éric,

This bug is now >10 years old, so I somewhat hope it got fixed as the
general adoption of UTF-8 increased in the last years. I tried to
reproduce the issue you described with the latest version of s-nail in
unstable (14.9.15-1), but I'm not sure I'm doing it right. The best
thing would be if you could try to reproduce with a more recent version
on s-nail and report back. Do you think you could give it a shot?

Thanks!

Paride

su 1. jouluk. 2019 klo 18.44 Paride Legovini (pl@ninthfloor.org) kirjoitti:

All my hosts currently run the traditional bsd-mailx.

Martin-Éric

Martin-Éric Racine wrote in <CAPZXPQeWka2gtdYjnKUpAgC4behjG3hx3+5ns0K9H\
oNYD-dA5Q@mail.gmail.com>:
 |su 1. jouluk. 2019 klo 18.44 Paride Legovini (pl@ninthfloor.org) kirjoitti:
 |> On Sun, 06 Sep 2009 <q-funk@iki.fi> wrote:
 |>> Package: heirloom-mailx
 |>> Version: 12.4-1.1+b1
 |>> Severity: normal
 |>>
 |>> When piping text into heirloom-mailx, it fails to specify the charset \
 |>> used with
 |>> the From: line if the name inherited from /etc/passwd is in UTF-8.
 |>>
 |>> cat file | mail -s "some subject" user@domain.ltd
 |>>
 |>> q-funk:x:1000:1000:Martin-Éric Racine,,,:/home/q-funk:/bin/bash
 |>
 |> Hello Martin-Éric,
 |>
 |> This bug is now >10 years old, so I somewhat hope it got fixed as the
 |> general adoption of UTF-8 increased in the last years. I tried to
 |> reproduce the issue you described with the latest version of s-nail in
 |> unstable (14.9.15-1), but I'm not sure I'm doing it right. The best
 |> thing would be if you could try to reproduce with a more recent version
 |> on s-nail and report back. Do you think you could give it a shot?
 |
 |All my hosts currently run the traditional bsd-mailx.

Hilko Bengen is possibly sometimes a bit difficult to get on.  If
i read the bugs' thread, he referred to sendcharsets= as the
target character set, and this is correct.  What we need to know
to get your (former) problem right is the input / source character
set.

So as long as your locale (man 7 locale) states the correct
character set (for example doing "export LC_ALL=fr_FR.utf8" in the
shell, before starting this MUA, for example), or you set the
ttycharset variable directly (it is normally derived from the
user's locale), this should just work.  (Iff conversion to
sendcharsets= is possible, iff that is necessary.)
This is true for all programs which adhere to locale(7).

(Note this MUA now supports a mime-force-sendout variable which
can be used to enforce sending out data, even if character set
conversion fails.  For unattended usage on systems which may
generate logs in whatever encoding.)