ss/config file not working but ssh works

I am trying to login to final_host from localhost via an intermediate hop, i.e:

localhost -> hop -> final_host 

This works:

localhost:~$ ssh -t user@hop "ssh user@final_host" 

This also works:

localhost:~$ ssh user@hop hop:~$ ssh user@final_host finalhost:~$ 

But this doesn't:

localhost:~$ ssh -J user@hop_ip "ssh user@final_host_ip" -vvv OpenSSH_7.6p1, LibreSSL 2.6.2 debug1: Reading configuration data /etc/ssh/ssh_config debug1: /etc/ssh/ssh_config line 48: Applying options for * debug1: Setting implicit ProxyCommand from ProxyJump: ssh -l user -vvv -W '[%h]:%p' hop_ip debug1: Executing proxy command: exec ssh -l user -vvv -W '[final_host_ip]:22' hop_ip debug1: key_load_public: No such file or directory debug1: identity file /Users/user/.ssh/id_rsa type -1 debug1: key_load_public: No such file or directory debug1: identity file /Users/user/.ssh/id_rsa-cert type -1 debug1: key_load_public: No such file or directory debug1: identity file /Users/user/.ssh/id_dsa type -1 debug1: key_load_public: No such file or directory debug1: identity file /Users/user/.ssh/id_dsa-cert type -1 debug1: key_load_public: No such file or directory debug1: identity file /Users/user/.ssh/id_ecdsa type -1 debug1: key_load_public: No such file or directory debug1: identity file /Users/user/.ssh/id_ecdsa-cert type -1 debug1: key_load_public: No such file or directory debug1: identity file /Users/user/.ssh/id_ed25519 type -1 debug1: key_load_public: No such file or directory debug1: identity file /Users/user/.ssh/id_ed25519-cert type -1 debug1: Local version string SSH-2.0-OpenSSH_7.6 debug1: permanently_drop_suid: 501 OpenSSH_7.6p1, LibreSSL 2.6.2 debug1: Reading configuration data /etc/ssh/ssh_config debug1: /etc/ssh/ssh_config line 48: Applying options for * debug2: ssh_connect_direct: needpriv 0 debug1: Connecting to hop_ip port 22. debug1: Connection established. debug1: key_load_public: No such file or directory debug1: identity file /Users/user/.ssh/id_rsa type -1 debug1: key_load_public: No such file or directory debug1: identity file /Users/user/.ssh/id_rsa-cert type -1 debug1: key_load_public: No such file or directory debug1: identity file /Users/user/.ssh/id_dsa type -1 debug1: key_load_public: No such file or directory debug1: identity file /Users/user/.ssh/id_dsa-cert type -1 debug1: key_load_public: No such file or directory debug1: identity file /Users/user/.ssh/id_ecdsa type -1 debug1: key_load_public: No such file or directory debug1: identity file /Users/user/.ssh/id_ecdsa-cert type -1 debug1: key_load_public: No such file or directory debug1: identity file /Users/user/.ssh/id_ed25519 type -1 debug1: key_load_public: No such file or directory debug1: identity file /Users/user/.ssh/id_ed25519-cert type -1 debug1: Local version string SSH-2.0-OpenSSH_7.6 debug1: Remote protocol version 2.0, remote software version OpenSSH_7.2p2 Ubuntu-4ubuntu2.4 debug1: match: OpenSSH_7.2p2 Ubuntu-4ubuntu2.4 pat OpenSSH* compat 0x04000000 debug3: fd 5 is O_NONBLOCK debug1: Authenticating to hop_ip:22 as 'user' debug3: hostkeys_foreach: reading file "/Users/user/.ssh/known_hosts" debug3: record_hostkey: found key type ED25519 in file /Users/user/.ssh/known_hosts:1 debug3: load_hostkeys: loaded 1 keys from hop_ip debug3: order_hostkeyalgs: prefer hostkeyalgs: ,ssh-ed25519 debug3: send packet: type 20 debug1: SSH2_MSG_KEXINIT sent debug3: receive packet: type 20 debug1: SSH2_MSG_KEXINIT received debug2: local client KEXINIT proposal debug2: KEX algorithms: curve25519-sha256,,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha256,diffie-hellman-group14-sha1,ext-info-c debug2: host key algorithms: ,ssh-ed25519,,,,,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,rsa-sha2-512,rsa-sha2-256,ssh-rsa debug2: ciphers ctos: ,aes128-ctr,aes192-ctr,aes256-ctr,, debug2: ciphers stoc: ,aes128-ctr,aes192-ctr,aes256-ctr,, debug2: MACs ctos: ,,,,,,,hmac-sha2-256,hmac-sha2-512,hmac-sha1 debug2: MACs stoc: ,,,,,,,hmac-sha2-256,hmac-sha2-512,hmac-sha1 debug2: compression ctos: none,,zlib debug2: compression stoc: none,,zlib debug2: languages ctos: debug2: languages stoc: debug2: first_kex_follows 0 debug2: reserved 0 debug2: peer server KEXINIT proposal debug2: KEX algorithms: ,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha1 debug2: host key algorithms: ssh-rsa,rsa-sha2-512,rsa-sha2-256,ssh-ed25519 debug2: ciphers ctos: ,aes128-ctr,aes192-ctr,aes256-ctr,, debug2: ciphers stoc: ,aes128-ctr,aes192-ctr,aes256-ctr,, debug2: MACs ctos: ,,,,,,,hmac-sha2-256,hmac-sha2-512,hmac-sha1 debug2: MACs stoc: ,,,,,,,hmac-sha2-256,hmac-sha2-512,hmac-sha1 debug2: compression ctos: none, debug2: compression stoc: none, debug2: languages ctos: debug2: languages stoc: debug2: first_kex_follows 0 debug2: reserved 0 debug1: kex: algorithm: debug1: kex: host key algorithm: ssh-ed25519 debug1: kex: server->client cipher: MAC: <implicit> compression: none debug1: kex: client->server cipher: MAC: <implicit> compression: none debug3: send packet: type 30 debug1: expecting SSH2_MSG_KEX_ECDH_REPLY debug3: receive packet: type 31 debug1: Server host key: ssh-ed25519 SHA256:c18UMgj7nokTZJHMGnbsOgxDHlIZc2r184efDHtoTLE debug3: hostkeys_foreach: reading file "/Users/user/.ssh/known_hosts" debug3: record_hostkey: found key type ED25519 in file /Users/user/.ssh/known_hosts:1 debug3: load_hostkeys: loaded 1 keys from hop_ip debug1: Host 'hop_ip' is known and matches the ED25519 host key. debug1: Found key in /Users/user/.ssh/known_hosts:1 debug3: send packet: type 21 debug2: set_newkeys: mode 1 debug1: rekey after 134217728 blocks debug1: SSH2_MSG_NEWKEYS sent debug1: expecting SSH2_MSG_NEWKEYS debug3: receive packet: type 21 debug1: SSH2_MSG_NEWKEYS received debug2: set_newkeys: mode 0 debug1: rekey after 134217728 blocks debug2: key: server_key_folder/.ssh/id_rsa (0x7f953cc00d70), agent debug2: key: /Users/user/.ssh/id_rsa (0x0) debug2: key: /Users/user/.ssh/id_dsa (0x0) debug2: key: /Users/user/.ssh/id_ecdsa (0x0) debug2: key: /Users/user/.ssh/id_ed25519 (0x0) debug3: send packet: type 5 debug3: receive packet: type 7 debug1: SSH2_MSG_EXT_INFO received debug1: kex_input_ext_info: server-sig-algs=<rsa-sha2-256,rsa-sha2-512> debug3: receive packet: type 6 debug2: service_accept: ssh-userauth debug1: SSH2_MSG_SERVICE_ACCEPT received debug3: send packet: type 50 debug3: receive packet: type 51 debug1: Authentications that can continue: publickey,password debug3: start over, passed a different list publickey,password debug3: preferred publickey,keyboard-interactive,password debug3: authmethod_lookup publickey debug3: remaining preferred: keyboard-interactive,password debug3: authmethod_is_enabled publickey debug1: Next authentication method: publickey debug1: Offering public key: RSA SHA256:IhOfM2s2i/vFgY/Mj962CoNez631HDIMDRjxFvDhOEI server_key_folder/.ssh/id_rsa debug3: send_pubkey_test debug3: send packet: type 50 debug2: we sent a publickey packet, wait for reply debug3: receive packet: type 60 debug1: Server accepts key: pkalg rsa-sha2-512 blen 279 debug2: input_userauth_pk_ok: fp SHA256:IhOfM2s2i/vFgY/Mj963CoNez631HDIMDRvxFvDhOEI debug3: sign_and_send_pubkey: RSA SHA256:IhOfM2s2i/vFgY/Mj963CoNez631HDIMDRvxFvDhOEI debug3: send packet: type 50 debug3: receive packet: type 52 debug1: Authentication succeeded (publickey). Authenticated to hop_ip ([hop_ip]:22). debug3: ssh_init_stdio_forwarding: final_host_ip:22 debug1: channel_connect_stdio_fwd final_host_ip:22 debug1: channel 0: new [stdio-forward] debug2: fd 7 setting O_NONBLOCK debug2: fd 8 setting O_NONBLOCK debug1: getpeername failed: Bad file descriptor debug3: send packet: type 90 debug2: fd 5 setting TCP_NODELAY debug3: ssh_packet_set_tos: set IP_TOS 0x10 debug1: Requesting debug3: send packet: type 80 debug1: Entering interactive session. debug1: pledge: network debug3: receive packet: type 80 debug1: client_input_global_request: rtype want_reply 0 debug3: receive packet: type 91 debug2: channel_input_open_confirmation: channel 0: callback start debug2: channel_input_open_confirmation: channel 0: callback done debug2: channel 0: open confirm rwindow 2097152 rmax 32768 debug1: ssh_exchange_identification: \033]1337;PushKeyLabels=fish_%self\033]1337;SetKeyLabel=F2=GitSSH-2.0-OpenSSH_7.2p2 Ubuntu-4ubuntu2.4 debug1: ssh_exchange_identification: debug1: ssh_exchange_identification: \024n��3�\030d\206\206�\235A6 debug1: ssh_exchange_identification: ,aes128-ctr,aes192-ctr,aes256-ctr,, debug1: ssh_exchange_identification: sh.com,,,,,,hmac-sha2-256,hmac-sha2-512,hmac-sha1 debug3: receive packet: type 98 debug1: client_input_channel_req: channel 0 rtype reply 1 debug3: send packet: type 100 debug3: receive packet: type 96 debug2: channel 0: rcvd eof debug2: channel 0: output open -> drain debug2: channel 0: obuf empty debug2: channel 0: close_write debug2: channel 0: output drain -> closed debug1: channel 0: FORCE input drain debug2: channel 0: ibuf empty debug2: channel 0: send eof debug3: send packet: type 96 debug2: channel 0: input drain -> closed debug3: receive packet: type 97 debug2: channel 0: rcvd close debug3: channel 0: will not send data after close debug2: channel 0: almost dead debug2: channel 0: gc: notify user debug1: stdio forwarding: done ssh_exchange_identification: Connection closed by remote host 

It gets stuck for a while before closing the connection. Using a config file with ProxyJump, ProxyCommand -W or ProxyCommand + nc also doesn't work and gets stuck at the same point. Additionally this stopped working at some point, I was able to use ProxyCommand in the past and it suddenly stopped working one day without apparent config change from my side. What could be going on here and what can I do to debug further?

EDIT: I should point out that other users of the same system are able to login without issue via config file so the problem is most likely local.

1 Answer

Authentication to final_host succeeds, but you get this error: getpeername failed: Bad file descriptor.

The ssh daemon on the final_host and hop should connect file descriptors for you to use (stdin, stdout, stderr) on your localhost. This fails.

The function mentioned should point you in the right direction. You are using an IP address that might not have a reverse dns entry, nor a line in /etc/hosts.