#747098 openssh-client: key negotiation fails for mtu>1500 and Ciphers (options) > 3

Package:
openssh-client
Source:
openssh
Description:
secure shell (SSH) client, for secure access to remote machines
Submitter:
john mcdougald
Date:
2015-08-27 19:45:06 UTC
Severity:
important
#747098#5
Date:
2014-05-05 14:32:22 UTC
From:
To:


After a recent dist-upgrade I could no longer ssh to a local server (running Wheezy).
Both client and target have Gigabit interfaces with an MTU of 9000.  In each
case the interface is a brige (br0 with only member being eth0).

The client hangs while performing key negotiation:

root@rocky:~# ssh -vvv dusty
OpenSSH_6.6.1, OpenSSL 1.0.1g 7 Apr 2014
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: Applying options for *
debug3: ciphers ok: [3des-cbc,aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc]
debug2: ssh_connect: needpriv 0
debug1: Connecting to dusty [10.115.10.75] port 22.
debug1: Connection established.
debug1: permanently_set_uid: 0/0
debug3: Incorrect RSA1 identifier
debug3: Could not load "/root/.ssh/id_rsa" as a RSA1 public key
debug1: identity file /root/.ssh/id_rsa type 1
debug1: identity file /root/.ssh/id_rsa-cert type -1
debug1: identity file /root/.ssh/id_dsa type -1
debug1: identity file /root/.ssh/id_dsa-cert type -1
debug1: identity file /root/.ssh/id_ecdsa type -1
debug1: identity file /root/.ssh/id_ecdsa-cert type -1
debug1: identity file /root/.ssh/id_ed25519 type -1
debug1: identity file /root/.ssh/id_ed25519-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_6.6.1p1 Debian-5
debug1: Remote protocol version 2.0, remote software version OpenSSH_6.0p1 Debian-4+deb7u1
debug1: match: OpenSSH_6.0p1 Debian-4+deb7u1 pat OpenSSH* compat 0x04000000
debug2: fd 3 setting O_NONBLOCK
debug3: load_hostkeys: loading entries for host "dusty" from file "/root/.ssh/known_hosts"
debug3: load_hostkeys: found key type ECDSA in file /root/.ssh/known_hosts:33
debug3: load_hostkeys: loaded 1 keys
debug3: order_hostkeyalgs: prefer hostkeyalgs: ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug2: kex_parse_kexinit: curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
debug2: kex_parse_kexinit: ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com,ssh-dss-cert-v01@openssh.com,ssh-rsa-cert-v00@openssh.com,ssh-dss-cert-v00@openssh.com,ssh-ed25519,ssh-rsa,ssh-dss
debug2: kex_parse_kexinit: 3des-cbc,aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc
debug2: kex_parse_kexinit: 3des-cbc,aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc
debug2: kex_parse_kexinit: hmac-md5-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-ripemd160-etm@openssh.com,hmac-sha1-96-etm@openssh.com,hmac-md5-96-etm@openssh.com,hmac-md5,hmac-sha1,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: hmac-md5-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-ripemd160-etm@openssh.com,hmac-sha1-96-etm@openssh.com,hmac-md5-96-etm@openssh.com,hmac-md5,hmac-sha1,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: none,zlib@openssh.com,zlib
debug2: kex_parse_kexinit: none,zlib@openssh.com,zlib
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit: first_kex_follows 0
debug2: kex_parse_kexinit: reserved 0
debug2: kex_parse_kexinit: ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
debug2: kex_parse_kexinit: ssh-rsa,ssh-dss,ecdsa-sha2-nistp256
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-sha2-256,hmac-sha2-256-96,hmac-sha2-512,hmac-sha2-512-96,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-sha2-256,hmac-sha2-256-96,hmac-sha2-512,hmac-sha2-512-96,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: none,zlib@openssh.com
debug2: kex_parse_kexinit: none,zlib@openssh.com
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit: first_kex_follows 0
debug2: kex_parse_kexinit: reserved 0
debug2: mac_setup: setup hmac-md5
debug1: kex: server->client 3des-cbc hmac-md5 none
debug2: mac_setup: setup hmac-md5
debug1: kex: client->server 3des-cbc hmac-md5 none
debug1: sending SSH2_MSG_KEX_ECDH_INIT
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY

Changing my mtu from 9000 to 1500 fixes the problem.  Further investigation gets a little strange though...

If I leave my mtu at 9000 and specify the cipher on the ssh command line: #ssh -c aes128-ctr dusty
the client connects just fine.  I tried each cipher in order and each time the client connected successfully.

I then edited the '/etc/ssh/ssh_config' and placed a single cipher option on the "Ciphers" line, this also worked.
I then started adding cipher options back and discovered something strange: if my FIRST cipher was 3des-cbc, I could specify
a total of three ciphers on the 'Ciphers' line, adding a fourth resulted in the failed key exchange.  HOWEVER, if 3des-cbc
was not the first option, I could only specify TWO options on the 'Ciphers' line, adding the third resulted in the same
failed key exchange.  I did not try further permutations.

QUICK FIX:
1. change client machine MTU = 1500 (no changes to 'etc/ssh/ssh_config')
2. specify two or fewer cipher options inside '/etc/ssh/ssh_config'

John

#747098#10
Date:
2014-07-23 13:05:12 UTC
From:
To:
Dear Maintainer,

same here - ssh over a 10 GE interface to a wheezy machine with mtu 9000 hangs
until a tcp timeout after:

debug1: expecting SSH2_MSG_KEX_ECDH_REPLY

The attached wireshark capture on the client machine shows the
ssh Client: Key Exchange Init packet
with 2034 byte is retransmitted on the tcp level until the server machine closes the connection.

The workaround with reducing the mtu to 1500 works here, too.

Please forward this information if this bug is not in openssh itself.

BTW, bug #747096 looks like nearly a verbatim copy and could proably deleted or at least marked
as duplicate.

Thanks,
 Hermann

#747098#17
Date:
2015-08-27 19:28:59 UTC
From:
To:
I've been seeing an issue that is very similar to what is described in
747098 and 612607, where there's a hang after

  ...
  debug1: kex: server->client aes128-ctr umac-64@openssh.com zlib@openssh.com
  debug1: kex: client->server aes128-ctr umac-64@openssh.com zlib@openssh.com
  debug1: sending SSH2_MSG_KEX_ECDH_INIT
  debug1: expecting SSH2_MSG_KEX_ECDH_REPLY

This only happens when connecting through an OpenConnect tunnel; ssh
works fine when I'm not using VPN.

Specifying a cipher didn't help, but lowering the MTU for the VPN tunnel
from 1500 to 1300 fixed things.

# ip li set mtu 1300 dev vpn0

mike