Dear Maintainer,
Since 6.14.1+ds-4, a fresh Mono installation has an empty X.509 trust store and
no supported way to populate one. Every outbound TLS connection from every Mono
application fails at handshake. The only workaround is a manual per-user symlink
(see WORKAROUND below); nothing in the archive sets it up, and the machine-wide
equivalent is deleted by debian/mono-libraries.preinst on every install or
upgrade.
I appreciate you inherited this package recently (6.14.1+ds2-3, Closes: #1133397);
the change below predates your maintainership. Analysis and suggested patches are
offered in that spirit.
REPRODUCER
$ csharp -e 'new System.Net.WebClient().DownloadString("https://debian.org")'
System.Net.WebException: Error: TrustFailure (Authentication failed, ...)
---> Mono.Btls.MonoBtlsException: Ssl error:1000007d:SSL routines:
OPENSSL_internal:CERTIFICATE_VERIFY_FAILED
at ./external/boringssl/ssl/handshake_client.c:1132
at Mono.Btls.MonoBtlsContext.ProcessHandshake ()
ROOT CAUSE
Commit af77cf49f "Drop generation of a Mono certificates store" (6.14.1+ds-4)
removed, in a single change:
- debian/mono-keystore the /etc/ca-certificates/update.d/ hook that ran
cert-sync /etc/ssl/certs/ca-certificates.crt
- debian/mono-devel.triggers activate update-ca-certificates
- Depends: ca-certificates from mono-devel
- /usr/bin/cert-sync and /usr/lib/mono/4.5/cert-sync.exe, moved from
mono-devel.install into debian/not-installed
Commit 7473d3eb8 then added to debian/mono-libraries.preinst an unconditional
rm --recursive /etc/mono/certstore
rm /usr/share/.mono (when a symlink)
so any machine-wide store created by hand is destroyed on the next mono-libraries
install or upgrade. (Per-user stores under ~/.config/.mono are not touched.)
The premise of the removal does not hold. Bug #821831 was a piuparts report about
unowned files after purge (policy 6.8/10.8). The only argument for removing the
feature outright is Jo Shields' 2016 comment that "Mono 4.8 ... should remove the
need for a separate cert store entirely". That never happened. In 6.14.1 the BTLS
provider still resolves its roots exclusively from the Mono store:
mcs/class/System/Mono.Btls/MonoBtlsProvider.cs:382-391 <http://Mono.Btls/MonoBtlsProvider.cs:382-391>
SetupDefaultCertificateStore() -> AddUserStore() + AddMachineStore()
mcs/class/System/Mono.Btls/MonoBtlsX509StoreManager.cs:71-84 <http://Mono.Btls/MonoBtlsX509StoreManager.cs:71-84>
MachineTrustedRoots -> MX.X509StoreManager.NewLocalMachinePath + "Trust"
= /usr/share/.mono/new-certs/Trust
With that directory absent, Directory.Exists() at MonoBtlsProvider.cs:404 is
false, no lookup is added, the X509_STORE is empty, and every chain fails.
There is no fallback to /etc/ssl/certs. store.SetDefaultPaths() exists at
MonoBtlsProvider.cs:385 but is inside "#if MONODROID". SSL_CTX_set_default_verify_paths
is never called; MonoBtlsX509Store.LoadLocations() has zero callers; SSL_CERT_FILE
and SSL_CERT_DIR are never read.
WHY certmgr IS NOT A SUBSTITUTE
mono-devel still ships certmgr, but certmgr writes the *legacy* store only
(mcs/tools/security/certmgr.cs:150-152 -> X509StoreManager.LocalMachine/CurrentUser <http://X509StoreManager.LocalMachine/CurrentUser>,
newFormat=false), i.e. /usr/share/.mono/certs/Trust holding DER *.cer files
(Mono.Security.X509/X509Store.cs:138-159). BTLS reads only new-certs/Trust, via
X509_LOOKUP_hash_dir over PEM <hash>.0 files (X509Store.cs:224-239,
mono/btls/btls-x509-lookup.c:73-76). The two stores differ in both directory and
encoding, so certmgr cannot populate a store BTLS will read. cert-sync writes both
(cert-sync.cs:121-133) and is the only tool that does.
MONO_TLS_PROVIDER=legacy does not help either (confirmed empirically below):
"legacy" is a bare alias for BTLS
on Linux (Mono.Net.Security/MonoTlsProviderFactory.cs:275 <http://Mono.Net.Security/MonoTlsProviderFactory.cs:275> and :312-320); no
separate legacy provider exists in this source. Note man/mono.1:1660-1662 still
documents "legacy" as a distinct stack, which is stale and is misleading users.
EMPIRICAL VERIFICATION
Traced on a working mono 6.12.0.199+dfsg-6 install (with ca-certificates-mono
present), since a stock 6.14.1 install has no populated store to compare against.
Full strace available on request. Every cert-related syscall of a successful
handshake, in order:
openat("/usr/lib/../lib/libmono-btls-shared.so", O_RDONLY|O_CLOEXEC) = 6
newfstatat("/home/USER/.config/.mono/new-certs/Trust", ...) = -1 ENOENT
newfstatat("/usr/share/.mono/new-certs/Trust", {st_mode=S_IFDIR|0755}) = 0
newfstatat("/usr/share/.mono/new-certs/Trust/ee64a828.0", ...) = -1 ENOENT
newfstatat("/usr/share/.mono/new-certs/Trust/865fbdf9.0", {st_size=2976}) = 0
openat("/usr/share/.mono/new-certs/Trust/865fbdf9.0", O_RDONLY) = 7
Counted over the whole trace:
occurrences of /etc/ssl : 0
occurrences of ca-certificates.crt : 0
occurrences of .mono/certs/Trust : 0
So at runtime Mono never consults the system bundle, and never consults the
legacy store. `openssl x509 -subject_hash` on that file returns exactly 865fbdf9,
confirming plain OpenSSL hash-dir probing of <hash>.0, then <hash>.1, and so on.
Store sufficiency was then tested directly, masking the machine store with an
empty bind mount inside a private mount namespace (process-local; nothing on disk
was modified):
user store contents result
------------------------------------------------ -----------
nothing TrustFailure
certs/Trust/ski-*.cer only, 121 roots TrustFailure
new-certs/Trust/<hash>.0 only, 121 roots OK
both OK
new-certs/Trust as a symlink to /etc/ssl/certs OK
new-certs/Trust is necessary AND sufficient; certs/Trust is neither. The last
row shows BTLS's existing directory lookup already works unmodified against the
hash directory update-ca-certificates maintains in /etc/ssl/certs.
Confirming the tools diverge, in throwaway HOMEs:
certmgr -add -c Trust <cert> -> creates ONLY certs/Trust/ski-<SKI>.cer
(no new-certs directory is created at all)
cert-sync --user <bundle> -> "121 new root certificates were added",
creates BOTH certs/Trust/ski-*.cer
and new-certs/Trust/<hash>.0
mozroots behaves like certmgr, writing only certs/Trust — and separately, its
download URL appears to be dead, so it is not a viable fallback either.
MONO_TLS_PROVIDER was empirically ruled out as a workaround:
legacy => OK btls => OK default => OK bogusvalue => "TLS Support not available."
The bogus value failing proves the variable is honored. But diffing the
cert-related paths between the legacy and btls traces yields no difference beyond
a PID-suffixed /dev/shm name: legacy still dlopens libmono-btls-shared.so and
still reads new-certs/Trust. With only certs/Trust populated, legacy fails exactly
as btls does.
cert-sync IS STILL BUILT
mcs/tools/security/Makefile:9-10 lists cert-sync.exe in SECURITY_PROGRAMS,
alongside certmgr.exe/chktrust.exe/mozroots.exe <http://certmgr.exe/chktrust.exe/mozroots.exe>, under the same single condition;
scripts/Makefile.am:78 generates the /usr/bin/cert-sync wrapper the same way as
its shipped siblings. Both land in debian/tmp on every build. Nothing in
debian/rules or debian/patches disables them. They are suppressed solely by
debian/not-installed lines 1, 34 and 37.
Upstream's documented expectation is that the packager runs it. man/cert-sync.1:14-20:
"Its use is intended to be automated at Mono install time, by distribution
packagers, to seamlessly provide SSL support to Mono applications without further
user interaction", with the example "cert-sync /etc/ssl/certs/ca-certificates.crt".
IMPACT
Any Mono application making an HTTPS request. Two downstream projects have open
reports:
- BizHawk https://github.com/TASEmulators/BizHawk/issues/4826
- KSP-CKAN https://github.com/KSP-CKAN/CKAN/issues/4590
CKAN additionally fails to install outright, because it Depends: ca-certificates-mono
and nothing in the archive Provides it any more. Summarising the investigation in
https://github.com/KSP-CKAN/CKAN/pull/4649, the CKAN maintainers list
redistributing the archived ca-certificates-mono_6.14.1+ds-2 .deb from
snapshot.debian.org as a possible workaround.
WORKAROUND
Until this is fixed, each affected user can point their BTLS user store at the
system hash directory:
$ mkdir -p ~/.config/.mono/new-certs
$ ln -s /etc/ssl/certs ~/.config/.mono/new-certs/Trust
(Use $XDG_CONFIG_HOME instead of ~/.config if it is set.) This follows
ca-certificates updates automatically and survives package upgrades. The same
link created machine-wide under /usr/share/.mono would be removed by
debian/mono-libraries.preinst.
SUGGESTED FIXES, in increasing order of effort
1. Ship cert-sync again. Delete debian/not-installed lines 1, 34, 37 and restore
to debian/mono-devel.install:
debian/tmp/usr/bin/cert-sync
debian/tmp/usr/lib/mono/4.5/cert-sync.exe*
and to debian/mono-devel.manpages:
debian/tmp/usr/share/man/man1/cert-sync.1
This gives users a documented recovery path. It does not by itself fix a fresh
install.
2. Restore automatic population. Reinstate debian/mono-keystore (recoverable
verbatim from af77cf49f^) into /etc/ca-certificates/update.d/, the
"activate update-ca-certificates" trigger, and Depends: ca-certificates.
To avoid regressing #821831, the store must this time be owned properly or
cleaned in postrm purge rather than left unowned.
Please note this needs to hang off a package that runtime users install.
In 6.12, libmono-system4.0-cil and libmono-security4.0-cil both carried
Recommends: ca-certificates-mono (= 6.12.0.199+dfsg-6), so any default install
that included System.dll received a populated store. Both are now provided by
mono-libraries (Provides: libmono-system4.0-cil, libmono-security4.0-cil), so
mono-libraries is the natural home for the replacement.
Also required: drop or gate the "rm --recursive /etc/mono/certstore" in
debian/mono-libraries.preinst:4-10, which otherwise deletes the store on every
upgrade.
3. Make BTLS read the system store, which is what #821831 assumed would happen and
removes the need for a Mono-specific store at all. The smallest form is to move
the existing call out of the Android-only branch in
mcs/class/System/Mono.Btls/MonoBtlsProvider.cs:382-391 <http://Mono.Btls/MonoBtlsProvider.cs:382-391>, i.e. add
store.SetDefaultPaths ();
to the #else arm after AddMachineStore(store). BoringSSL's
X509_STORE_set_default_paths (external/boringssl/crypto/x509/x509_d2.c:62-80)
then adds a hash_dir lookup on /etc/ssl/certs, which update-ca-certificates
already maintains in exactly the form X509_LOOKUP_hash_dir wants (the
symlink row in EMPIRICAL VERIFICATION demonstrates this). It calls
ERR_clear_error() before returning, so the absence of /etc/ssl/cert.pem on
Debian is harmless. This is one line, needs no ABI change, and would also make
SSL_CERT_DIR/SSL_CERT_FILE work. It would need forwarding upstream to
https://gitlab.winehq.org/mono/mono .
(LoadLocations() is the more targeted alternative but throws when the file is
missing, so it would need a File.Exists guard; SetDefaultPaths is safer.)
4. Add Provides: ca-certificates-mono to whichever package ends up carrying this,
so existing downstream Depends lines such as CKAN's resolve again.
Options 1 and 4 alone unbreak CKAN's installability; 2 or 3 is needed to fix the
handshake on a fresh install.
Disclosure: the source analysis and this report were prepared with help from AI.
I have reviewed it; the failure is from my own Ubuntu 26.04 machine.
Thanks for taking on the package.