#737799 STARTTLS fails with 8192-bit keys

#737799#5
Date:
2014-02-06 01:11:42 UTC
From:
To:
Version: 8.14.4-4.1
Severity: normal

I have this config in /etc/mail/starttls.m4, corresponding to my own
private CA and some certificates that I generated and signed:

  define(`confCACERT_PATH', `/etc/ssl/certs')dnl
  define(`confCACERT',      `/etc/ssl/public/ca.pem')dnl
  define(`confSERVER_CERT', `/etc/ssl/public/sendmail-server.pem')dnl
  define(`confSERVER_KEY',  `/etc/ssl/private/sendmail-server.key')dnl
  define(`confCLIENT_CERT', `/etc/ssl/public/sendmail-client.pem')dnl
  define(`confCLIENT_KEY',  `/etc/ssl/private/sendmail-client.key')dnl

If sendmail-server.key and sendmail-client.key are 8192 bits, sending
queued mail from my local system fails with the following error in
/var/log/mail.log:

  Feb  5 19:26:13 psy sm-mta[21534]: starting daemon (8.14.4): SMTP+queueing@00:05:00
  Feb  5 19:26:14 psy sm-msp[21555]: starting daemon (8.14.4): queueing@00:10:00
  Feb  5 19:26:15 psy sm-mta[21568]: STARTTLS=server, error: accept failed=-1, SSL_error=1, errno=0, retry=-1, relay=localhost [127.0.0.1]
  Feb  5 19:26:15 psy sm-msp-queue[21540]: STARTTLS=client, error: connect failed=0, SSL_error=1, errno=0, retry=-1
  Feb  5 19:26:15 psy sm-msp-queue[21540]: STARTTLS=client: 21540:error:14094417:SSL routines:SSL3_READ_BYTES:sslv3 alert illegal parameter:s3_pkt.c:1108:SSL alert number 47
  Feb  5 19:26:15 psy sm-mta[21568]: STARTTLS=server: 21568:error:1408E098:SSL routines:SSL3_GET_MESSAGE:excessive message size:s3_both.c:483:
  Feb  5 19:26:15 psy sm-msp-queue[21540]: ruleset=tls_server, arg1=SOFTWARE, relay=[127.0.0.1], reject=403 4.7.0 TLS handshake.
  Feb  5 19:26:15 psy sm-msp-queue[21540]: s1601pq7016584: to=redacted@redacted, ctladdr=jim (1000/1000), delay=00:24:24, xdelay=00:00:01, mailer=relay, pri=480853, relay=[127.0.0.1] [127.0.0.1], dsn=4.0.0, stat=Deferred: 403 4.7.0 TLS handshake.
  Feb  5 19:26:15 psy sm-mta[21568]: s160QEGo021568: localhost [127.0.0.1] did not issue MAIL/EXPN/VRFY/ETRN during connection to MTA-v4

On a hunch, I regenerated both keys as 4096 bit keys, signed them with
the CA, and tried again.  This worked fine, with no other changes:

  Feb  5 19:33:12 psy sm-mta[22363]: starting daemon (8.14.4): SMTP+queueing@00:05:00
  Feb  5 19:33:14 psy sm-msp[22383]: starting daemon (8.14.4): queueing@00:10:00
  Feb  5 19:33:14 psy sm-mta[22371]: STARTTLS=server, relay=localhost [127.0.0.1], version=TLSv1/SSLv3, verify=OK, cipher=DHE-RSA-AES256-SHA, bits=256/256
  Feb  5 19:33:14 psy sm-msp-queue[22369]: STARTTLS=client, relay=[127.0.0.1], version=TLSv1/SSLv3, verify=OK, cipher=DHE-RSA-AES256-SHA, bits=256/256
  Feb  5 19:33:15 psy sm-mta[22371]: s160XEib022371: from=<jim@redacted>, size=1007, class=0, nrcpts=1, msgid=<20140206000150.GA16379@redacted>, proto=ESMTP, daemon=MTA-v4, relay=localhost [127.0.0.1]
  Feb  5 19:33:15 psy sm-msp-queue[22369]: s1601pq7016584: to=redacted@redacted, ctladdr=jim (1000/1000), delay=00:31:24, xdelay=00:00:01, mailer=relay, pri=660853, relay=[127.0.0.1] [127.0.0.1], dsn=2.0.0, stat=Sent (s160XEib022371 Message accepted for delivery)

I'm certain I generated the keys and signed the certificates the exact
same way in both situations, and the Sendmail configuration did not
change.  The only difference was that it fails when the keys for the
client certs were generated with:

  openssl genrsa -out out.key 8192

and worked fine with:

  openssl genrsa -out out.key 4096

#737799#10
Date:
2014-02-06 01:31:38 UTC
From:
To:
It looks like this is actually an OpenSSL bug with >4096 bit keys:
http://rt.openssl.org/Ticket/Display.html?user=guest&pass=guest&id=319

#737799#15
Date:
2026-07-12 07:10:29 UTC
From:
To:
That link does not work anymore.
Is that issue fixed?