Dear Maintainer,
With OpenSSL configured in FIPS-only mode (only the "base" and "fips"
providers
active; the "default" provider disabled), "apt-get update"/"install"
against an
HTTPS repository intermittently fails with:
Err:N https://...repo...
OpenSSL error: error:0308010C:digital envelope routines::unsupported
Error reading from server - read (5: Input/output error)
Root cause
----------
During the TLS handshake, libssl performs an implicit EVP_MD_fetch() for the
legacy MD5 / MD5-SHA1 digests (used for pre-TLS-1.2 handshake signing and
the
TLS 1.0/1.1 PRF). Under a FIPS-only provider configuration those digests are
unavailable (they exist only in the default provider), so the fetch fails
and
leaves "error:0308010C ... unsupported" on the thread's OpenSSL error queue.
This is benign: the handshake completes fine. "openssl s_client" to the same
host under the identical FIPS config connects successfully with the very
same
failed MD5/MD5-SHA1 fetches (verifiable with an LD_PRELOAD trace of
EVP_MD_fetch).
The actual failure is that apt does not clear the OpenSSL error queue
before its
TLS I/O. In methods/connect.cc, TlsFd::Read() and TlsFd::Write() call
SSL_read()/SSL_write() and then HandleError() -> SSL_get_error() WITHOUT a
preceding ERR_clear_error(). When SSL_read() later returns <= 0 for a benign
reason, SSL_get_error() consults the non-empty error queue, returns
SSL_ERROR_SSL, and apt reports the stale MD5 error as a fatal read failure
(errno = EIO -> "Error reading from server").
This violates the documented precondition in SSL_get_error(3): "The current
thread's error queue must be empty before the TLS/SSL I/O operation is
attempted, [...] as the SSL_get_error() function uses the error queue
[...]."
PostgreSQL fixed the identical class of bug (stale FIPS-mode error-queue
entry
misreported later) by calling ERR_clear_error() "on the way in"; libpq
already
does this around its OpenSSL I/O.
This did not occur before Debian 13 / apt 3.0: apt 2.6 (bookworm) used
GnuTLS
for its TLS transport, which does not touch OpenSSL's providers or error
queue.
Reproduction (Debian 13)
------------------------------
Minimal, self-contained Dockerfile. The build itself fails at the final RUN
(installing Docker from an HTTPS repo) -- "docker build ." is the whole
repro:
FROM debian:13-slim
SHELL ["/bin/bash", "-o", "pipefail", "-c"]
# FIPS-only OpenSSL: install the FIPS provider, then activate base +
fips and
# disable the default provider.
RUN apt-get update --yes \
&& apt-get install --yes --no-install-recommends \
ca-certificates openssl openssl-provider-fips \
&& MODULES_DIR="$(openssl version -m | cut -d'"' -f2)" \
&& openssl fipsinstall -out /etc/ssl/fipsmodule.cnf -module
"${MODULES_DIR}/fips.so"
RUN sed -i 's|^#\s*\.include\s\+fipsmodule.cnf|.include
/etc/ssl/fipsmodule.cnf|' /etc/ssl/openssl.cnf \
&& sed -i 's/^default\s*=\s*default_sect/# default = default_sect/'
/etc/ssl/openssl.cnf \
&& sed -i 's/^#\s*fips\s*=\s*fips_sect/fips = fips_sect\nbase =
base_sect\n\n[base_sect]\nactivate = 1/' /etc/ssl/openssl.cnf
# ("openssl list -providers" now shows only base + fips.)
# Install Docker from its official HTTPS apt repo (any HTTPS repo
triggers it;
# this is just a convenient public one). This RUN fails:
# OpenSSL error: error:0308010C ... Error reading from server
# E: Package 'docker-ce' has no installation candidate
RUN apt-get install --no-install-recommends -y ca-certificates curl
gnupg \
&& install -m 0755 -d /etc/apt/keyrings \
&& curl -fsSL https://download.docker.com/linux/debian/gpg | gpg
--dearmor -o /etc/apt/keyrings/docker.gpg \
&& echo "deb [signed-by=/etc/apt/keyrings/docker.gpg]
https://download.docker.com/linux/debian trixie stable" \
> /etc/apt/sources.list.d/docker.list \
&& apt-get update \
&& apt-get install --no-install-recommends -y docker-ce docker-ce-cli
containerd.io
Build it:
docker build .
The build fails at the final RUN with the error:0308010C / "Error reading
from
server" message above. (The underlying trigger is a read returning <= 0
while
the stale error is queued, so in principle a fluke pass is possible; in
practice
fetching the Docker repo over HTTPS this way fails on essentially every
build,
matching what we see in CI. If a build does pass, rebuild with --no-cache.)
For contrast, the connection itself is fine and the failed MD5 fetch is
benign --
both of these succeed under the identical FIPS config:
# same handshake, succeeds, proving MD5 is not actually needed:
openssl s_client -connect download.docker.com:443 -servername
download.docker.com </dev/null
# and apt works if the default provider is made available:
OPENSSL_CONF=/dev/null apt-get update # (with the docker.list
source above)
System information
------------------
Debian release: 13 (trixie), amd64
Versions of relevant packages:
apt 3.0.3
libssl3t64 3.5.6-1~deb13u2 (OpenSSL; apt's TLS backend in 3.0)
openssl 3.5.6-1~deb13u2
libc6 2.41-12+deb13u3
Reproduced in a stock debian:13-slim container (see Dockerfile above).
Suggested fix
-------------
Clear the OpenSSL error queue immediately before each
SSL_read()/SSL_write() in
methods/connect.cc, mirroring libpq:
ssize_t Read(void *buf, size_t count) override {
assert(ssl);
+ ERR_clear_error();
return HandleError(SSL_read(ssl, buf, count));
}
ssize_t Write(void *buf, size_t count) override {
assert(ssl);
+ ERR_clear_error();
return HandleError(SSL_write(ssl, buf, count));
}
This makes apt robust to any benign leftover OpenSSL error, not just the
FIPS/MD5 case.
Michael Hamill
Senior Software Engineer II
michael.hamill@wellhive.com
www.wellhive.com
WELLHIVE CONFIDENTIALITY NOTICE: The contents of this email message and any attachments are intended solely for the addressee(s). Unless otherwise indicated, it contains information that is confidential, privileged and/or exempt from disclosure under applicable law. If you are not the named addressee, you are not authorized to read, print, retain, copy or disseminate this message or any part of it. If you have received this message in error, please notify the sender immediately by e-mail and delete all copies of the message.