SSH Agent Forwarding not working even when using `ssh -A`

First of all, I've checked all the following

• Agent Forwarding not working
• Extra configuration required for ssh-agent forwarding?
• SSH Agent Forwarding Not Working

but none of them helped. So, here is my question:

I can ssh from A to B, or from A to C, but not from A to B, then B to C. When connecting from A to either B or C, I always use ssh -A to force SSH Agent Forwarding.
But why I'm still not able to connect A -> B -> C without being asked for passphrase?

UPDATE: nearly three years later, the same problem still haunts me, but now I've narrowed down the problem a bit:

Symptom: I can ssh A -> B, or A -> C, but not A -> B -> C, or A -> C -> B.

The problem is exactly described by the topic -- SSH Agent Forwarding is not working.

From

Troubleshoot SSH issues

it says:

To list your loaded keys, enter ssh-add -l. If you don't see the SSH key you want to use...

then there is problem -- the SSH key you want to use is NOT loaded.

This is what's happening when I'm at A -> B or A -> C. I.e., right after I did ssh -A into an intermediate server. the SSH key is lost, not forwarded and not loaded.

$ ssh-add -l The agent has no identities. 

That's the reason why I cannot ssh further without the passphrase.

It does have SSH_AUTH_SOCK variable setup and several ssh-agent around though:

$ echo "$SSH_AUTH_SOCK" /tmp/ssh-RtEuLOmFDBet/agent.3722 $ ps -e | grep [s]sh-agent 3723 ? 00:00:00 ssh-agent 4613 ? 00:00:00 ssh-agent 

It does not seem to be related to my own environment as they are identical, or the /etc/ssh/sshd_config file, as I've compared the ones from the intermediate servers that are working or not working.

UPDATE END.

More info: all three machines, are configured with the standard Ubuntu ssh configuration. I.e., the AllowAgentForwarding option is not in /etc/ssh/sshd_config, though I doubt whether it should, because I saw "Since agent forwarding defaults on, it should be enough to remove any AllowAgentForwarding line from sshd_config." from Extra configuration required for ssh-agent forwarding?.

Some say ssh-add will do, but when I do it on B or C, it asks me to Enter passphrase for my .ssh/id_rsa. Some say check the SSH_AUTH_SOCK, but I do have it on B or C (either from A to B, or from A to C):

$ env | grep SSH_AUTH_SOCK SSH_AUTH_SOCK=/tmp/ssh-RTScJ5PZh9Mh/agent.2083 

Is the Agent Forwarding not working because of missing the AllowAgentForwarding option ? Then which one (A, B, or C) should I put it in? Wouldn't ssh -A be enough? Also I have the .ssh/id_rsa files on both B and C, is that the reason ssh-add ask for the passphrase for them?

EDIT:

Here is the log of -Avvv from B to C:

OpenSSH_6.2p2 Ubuntu-6ubuntu0.1, OpenSSL 1.0.1e 11 Feb 2013 debug1: Reading configuration data /home/myid/.ssh/config debug1: Reading configuration data /etc/ssh/ssh_config debug1: /etc/ssh/ssh_config line 19: Applying options for * debug2: ssh_connect: needpriv 0 debug1: Connecting to boxc. debug1: Connection established. debug3: Incorrect RSA1 identifier debug3: Could not load "/home/myid/.ssh/id_rsa" as a RSA1 public key debug1: identity file /home/myid/.ssh/id_rsa type 1 debug1: Checking blacklist file /usr/share/ssh/blacklist.RSA-1024 debug1: Checking blacklist file /etc/ssh/blacklist.RSA-1024 debug1: identity file /home/myid/.ssh/id_rsa-cert type -1 debug3: Incorrect RSA1 identifier debug3: Could not load "/home/myid/.ssh/id_dsa" as a RSA1 public key debug1: identity file /home/myid/.ssh/id_dsa type 2 debug1: Checking blacklist file /usr/share/ssh/blacklist.DSA-1024 debug1: Checking blacklist file /etc/ssh/blacklist.DSA-1024 debug1: identity file /home/myid/.ssh/id_dsa-cert type -1 debug1: identity file /home/myid/.ssh/id_ecdsa type -1 debug1: identity file /home/myid/.ssh/id_ecdsa-cert type -1 debug1: Enabling compatibility mode for protocol 2.0 debug1: Local version string SSH-2.0-OpenSSH_6.2p2 Ubuntu-6ubuntu0.1 debug1: Remote protocol version 2.0, remote software version OpenSSH_6.2p2 Ubuntu-6 debug1: match: OpenSSH_6.2p2 Ubuntu-6 pat OpenSSH* debug2: fd 3 setting O_NONBLOCK debug3: put_host_port: boxc debug3: load_hostkeys: loading entries for host "boxc" from file "/home/myid/.ssh/known_hosts" debug3: load_hostkeys: found key type RSA in file /home/myid/.ssh/known_hosts:15 debug3: load_hostkeys: loaded 1 keys debug3: order_hostkeyalgs: prefer hostkeyalgs: ,,ssh-rsa debug1: SSH2_MSG_KEXINIT sent debug1: SSH2_MSG_KEXINIT received debug2: kex_parse_kexinit: ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1 debug2: kex_parse_kexinit: ,,ssh-rsa,,,,,,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-dss debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,,,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour, debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,,,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour, debug2: kex_parse_kexinit: ,,,,,,,,,hmac-md5,hmac-sha1,,,hmac-sha2-256,hmac-sha2-512,hmac-ripemd160,,hmac-sha1-96,hmac-md5-96 debug2: kex_parse_kexinit: ,,,,,,,,,hmac-md5,hmac-sha1,,,hmac-sha2-256,hmac-sha2-512,hmac-ripemd160,,hmac-sha1-96,hmac-md5-96 debug2: kex_parse_kexinit: ,zlib,none debug2: kex_parse_kexinit: ,zlib,none debug2: kex_parse_kexinit: debug2: kex_parse_kexinit: debug2: kex_parse_kexinit: first_kex_follows 0 debug2: kex_parse_kexinit: reserved 0 debug2: kex_parse_kexinit: ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1 debug2: kex_parse_kexinit: ssh-rsa,ssh-dss debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,,,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour, debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,,,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour, debug2: kex_parse_kexinit: ,,,,,,,,,hmac-md5,hmac-sha1,,,hmac-sha2-256,hmac-sha2-512,hmac-ripemd160,,hmac-sha1-96,hmac-md5-96 debug2: kex_parse_kexinit: ,,,,,,,,,hmac-md5,hmac-sha1,,,hmac-sha2-256,hmac-sha2-512,hmac-ripemd160,,hmac-sha1-96,hmac-md5-96 debug2: kex_parse_kexinit: none, debug2: kex_parse_kexinit: none, debug2: kex_parse_kexinit: debug2: kex_parse_kexinit: debug2: kex_parse_kexinit: first_kex_follows 0 debug2: kex_parse_kexinit: reserved 0 debug2: mac_setup: found debug1: kex: server->client aes128-ctr debug2: mac_setup: found debug1: kex: client->server aes128-ctr debug1: sending SSH2_MSG_KEX_ECDH_INIT debug1: expecting SSH2_MSG_KEX_ECDH_REPLY debug1: Server host key: RSA ed:26:20:93:4c:88:ef:17:70:e3:d4:7a:42:4c:8e:69 debug3: put_host_port: [192.168.2.122]:21 debug3: put_host_port: boxc debug3: load_hostkeys: loading entries for host "boxc" from file "/home/myid/.ssh/known_hosts" debug3: load_hostkeys: found key type RSA in file /home/myid/.ssh/known_hosts:15 debug3: load_hostkeys: loaded 1 keys debug3: load_hostkeys: loading entries from file "/home/myid/.ssh/known_hosts" debug3: load_hostkeys: found key type RSA in file /home/myid/.ssh/known_hosts:16 debug3: load_hostkeys: loaded 1 keys debug1: Host 'boxc' is known and matches the RSA host key. debug1: Found key in /home/myid/.ssh/known_hosts:15 debug1: ssh_rsa_verify: signature correct debug2: kex_derive_keys debug2: set_newkeys: mode 1 debug1: SSH2_MSG_NEWKEYS sent debug1: expecting SSH2_MSG_NEWKEYS debug2: set_newkeys: mode 0 debug1: SSH2_MSG_NEWKEYS received debug1: Roaming not allowed by server debug1: SSH2_MSG_SERVICE_REQUEST sent debug2: service_accept: ssh-userauth debug1: SSH2_MSG_SERVICE_ACCEPT received debug2: key: /home/myid/.ssh/id_rsa (0x7f7e....e760), debug2: key: /home/myid/.ssh/id_dsa (0x7f7e....e7a0), debug2: key: /home/myid/.ssh/id_ecdsa ((nil)), debug1: Authentications that can continue: publickey debug3: start over, passed a different list publickey debug3: preferred gssapi-keyex,gssapi-with-mic,publickey,keyboard-interactive,password debug3: authmethod_lookup publickey debug3: remaining preferred: keyboard-interactive,password debug3: authmethod_is_enabled publickey debug1: Next authentication method: publickey debug1: Offering RSA public key: /home/myid/.ssh/id_rsa debug3: send_pubkey_test debug2: we sent a publickey packet, wait for reply debug1: Server accepts key: pkalg ssh-rsa blen 149 debug2: input_userauth_pk_ok: fp 22:32:...:1d:e3 debug3: sign_and_send_pubkey: RSA 22:32:...:1d:e3 debug1: key_parse_private_pem: PEM_read_PrivateKey failed debug1: read PEM private key done: type <unknown> Enter passphrase for key '/home/myid/.ssh/id_rsa': 

I've compared with my good session (A->C), and found all are none different, other than the last 3 lines, which starts with "key_parse_private_pem: PEM_read_PrivateKey failed". The good session instead has:

debug1: Enabling compression at level 6. debug1: Authentication succeeded (publickey). Authenticated to boxc 

Everything else are the same.

Again, my environment:

$ apt-cache policy openssh-server openssh-server: Installed: 1:6.2p2-6ubuntu0.1 Candidate: 1:6.2p2-6ubuntu0.4 Version table: 1:6.2p2-6ubuntu0.4 0 500 saucy-updates/main amd64 Packages 1:6.2p2-6ubuntu0.3 0 500 saucy-security/main amd64 Packages % sshd -v sshd: illegal option -- v OpenSSH_6.2p2 Ubuntu-6ubuntu0.1, OpenSSL 1.0.1e 11 Feb 2013 

Thanks

2 Answers

I've been trying to troubleshoot this problem for nearly seven years, and finally it get solved -- I launch keychain in my ~/.profile, which starts its own 'ssh-agent', even on machine B & C. This is the source of the problem, because keychain's ssh-agent overshadow the sshd provided one.

Removing it (keychain) from my ~/.profile solved the problem.

Update, another possibility, ssh-agent etc. usually get started as part of starting up the GUI on the local system. e.g., in another case, the call's hidden in /etc/X11/xdm/sys.xsession!

I confirm my SSH Agent Forwarding is working by doing, in MachineA,

ssh -t MachineB ssh MachineC 

while ssh MachineB then, within it ssh MachineC was failing.

I'll start it (ssh-agent from keychain etc) manually only from machine A from now on.

Yet another cause: If the target host's fingerprint doesn't match with your ~/.ssh/known_hosts, SSH automatically disables Agent Forwarding.

The solution is:

$ ssh -A -o UserKnownHostsFile=/dev/null my-target-host 

After SSH into the remote/intermediate host, do this to verify that the Agent Forwarding is in effect:

remote-host$ echo $SSH_AUTH_SOCK /tmp/ssh-AyHL6WclXl/agent.107234 <== Means Agent Forwarding is in effect.