Is it working?

[mxxcon]

How can I tell if this is properly working, and if not, how to troubleshoot this?
I'm using https://github.com/babun/babun
I have Pageant running. I installed your ssh-pageant into /usr/bin/, I added provided snippet into both my .bashrc and .zshrc, restarted my shell. However, when I try to ssh to a remote host i'm still prompted for a password.
If I use plain putty or other 'regular' terminal app, Pageant picks up key authentication..

[cuviper]

If your rc files are starting the agent properly, you'll get an environment variable for the socket. Try running "echo $SSH_AUTH_SOCK" to see if you have anything. You can also try "ssh-add -l" to list the keys it can find, and if it can talk to the agent at all.

In fact, the easiest way to test operation directly is "ssh-pageant ssh-add -l", which will start an agent just to run that command. Try that to make sure the basics are working, before we worry about your rc files.

I don't know anything about babun, but in a quick search it doesn't look like they tweak ssh at all, so it's probably ok there.

[mxxcon]

echo $SSH_AUTH_SOCK gives just an empty line.
ssh-add -lgives Could not open a connection to your authentication agent.
ssh-pageant ssh-add -l returns back to the prompt with no output.

[cuviper]

OK, the first two results indicate that the environment isn't being set in your rc files. What exactly did you add?

The last one seems to be working, just that your running Pageant doesn't have any keys in it at the moment. Otherwise you would have seen a connection error like the standalone ssh-add -l. Try adding some keys and then see if you can list them. You can even use ssh-pageant ssh-add FILE to load openssh-style key files into Pageant.

[mxxcon]

I do have my key loaded in Pageant...
At the buttom of both ~/.zshrc and ~/.bashrc I added

# ssh-pageant
eval $(/usr/bin/ssh-pageant -r -a "/tmp/.ssh-pageant-$USERNAME")

Right now i'm in zsh, but I tried switching to bash with the same results.

[cuviper]

What happens if you run that eval line directly in the shell? Or maybe try the very simple eval $(ssh-pageant), which will use a random agent path.

And actually, I just found that with an empty Pageant, ssh-add will say, "The agent has no identities." Also if Pageant isn't running, ssh-pageant will report SSH_AGENT_FAILURE, and ssh-add still just says no identities. If there's a problem with the socket, or SSH_AUTH_SOCK isn't set at all, then you get "Could not open a connection to your authentication agent." as you saw.

So I'm really not sure what's going on when you get nothing from ssh-pageant ssh-add -l. :/

For comparison, does charade work for you?

[mxxcon]

eval $(ssh-pageant) thinks for a moment and then back to prompt with no output.
I'll give charade a try.

[cuviper]

How about "ssh-pageant ; echo $?" ?

I have a hunch that maybe babun is a 32-bit cygwin environment, and maybe you downloaded 64-bit ssh-pageant? Or vice-versa. But I just found that running the wrong/mixed versions will fail silently with exit code 127, as echo $? will show.

If that's the case, go download the other bit version. :) Unfortunately, there's nothing I could do in ssh-pageant to diagnose this, because this would be a failure to run the program at all. I can try to clarify this in the README though.

[mxxcon]

I downloaded both versions

{ /bin }  » uname -a
CYGWIN_NT-6.3-WOW64 WNYCLT800017 1.7.29(0.272/5/3) 2014-04-07 13:44 i686 Cygwin
{ /bin }  » ssh-pageant32 ; echo $?
127
{ /bin }  » ssh-pageant64 ; echo $?
127

😞

[cuviper]

OK, that "i686" means this is 32-bit Cygwin. Also WOW64 is the confusing name for the environment of 32-bit applications running on 64-bit Windows.

Yet they both fail? Ugh. I just double-checked that I didn't mess up the 32-bit download either.

How about ldd /usr/bin/ssh-pageant.exe? I don't use anything but cygwin1.dll and standard windows libraries, but it's worth checking. And if nothing says "not found", then it could still be some missing symbol, especially if babun is not a fully current version of cygwin.

Actually, ssh-pageant 1.4 just started using some additional Cygwin APIs, like cygwin_conv_path. If missing symbols are your issue, then 1.3 might be better for you.

[cuviper]

OK, I downloaded babun and looked at dependency walker of ssh-pageant against that version of cygwin1.dll, and it's missing symbol __cxa_atexit. That's not something I use directly, but it's made for running destructors of global data. It might even be something I link to which indirectly causes that.

The prebuilt ssh-pageant 1.3 did not link to that symbol, so it may actually work for you. However, when I try a fresh build of 1.3, I do get that symbol now, so it must be something changed in newer Cygwin. By that logic, you might also be ok compiling your own copy of 1.4 in babun's older Cygwin.

[mxxcon]

After compiling it from source when I relaunch babun I get

ssh-pageant pid 5028

and in the process list I see /usr/bin/ssh-pageant

echo $SSH_AUTH_SOCK still gives an empty line.
ssh-add -l still gives Could not open a connection to your authentication agent.
ssh-pageant ssh-add -l shows signature of my ssh key.
If I try to ssh anywhere I'm still prompted for the password.

[cuviper]

It's a good sign that the process started! But if SSH_AUTH_SOCK isn't set, and ssh-add -l doesn't work, I don't expect any other ssh operations will use it. At least ssh-pageant ssh-add -l shows that the agent is working in principle.

So now it's just a question of getting that environment variable set. When you don't say -c or -s, it looks at $SHELL to see if that looks like a csh-style shell, and if even that's missing then it assumes sh-style. I would expect you to see error messages from the eval if this was wrong, but maybe not. You can try running the command without eval to see what it's doing -- if you see setenv, that's csh-style. Both bash and zsh are sh-style shells.

Or you can cheat and set SSH_AUTH_SOCK yourself, since you're using a fixed -a path anyway. But I would like to know what's going wrong here. :)

BTW, thank you for sticking with this so long! I will definitely have to collect some of these troubleshooting steps into the README.

[mxxcon]

And I really appreciate you responding so quickly to help me troubleshoot this. 👍

hmm, it does seem like SSH_AUTH_SOCK is not being set. After killing all remaining processes and restarting babun, running export or env shows this variable is still not set.
if I manually run

{ ~ }  » /bin/ssh-pageant -r -a "/tmp/.ssh-pageant-$USERNAME"
SSH_AUTH_SOCK='/tmp/.ssh-pageant-e030508'; export SSH_AUTH_SOCK;

and then env, it still does not show this var.

[cuviper]

To be clear, you're manually running the assignment and export too? Just running ssh-pageant won't do, as it can't change the environment of the shell. (That's why it's usually an eval line.)

[mxxcon]

I added

eval $(/usr/bin/ssh-pageant -r -a "/tmp/.ssh-pageant-$USERNAME")

to my ~/.profile, exited babun, killed any remaining ssh-pageant processes, restarted babun, saw ssh-pageant pid 9272, ran env|sort and SSH_AUTH_SOCK is not there.

[cuviper]

Since you see the pid line, the eval must be running, as that echo is one of the commands from ssh-pageant. I can only guess that something else is overriding it later. Maybe put your own line like echo "ssh-pageant sock $SSH_AUTH_SOCK" right after the eval? I see that babun comes with keychain, which manages ssh-agent processes, and there are also some oh-my-zsh scripts for ssh-agent and gpg-agent (which can also do ssh). I wouldn't expect any of these to run by default and clobber SSH_AUTH_SOCK, but who knows. Otherwise, I'm out of ideas. :/

[cuviper]

@mxxcon, have you discovered anything else?

[mxxcon]

I'm sorry, i've been busy last couple of days. I'll poke around with this some more.

[mxxcon]

Looks like this is something zsh-specific. When I switched my shell to bash it started working.

I'll try to troubleshoot this a bit more, otherwise will post in babun's repo to help me figure out what's going on.

[Ginja]

Going to piggy back off this issue. I'm also using babun, and experiencing a similar issue. However, I have $SSH_AUTH_SOCK set, I see the process is started, and PuTTY's pageant is running with my private key loaded:

# env | sort
SHELL=/bin/bash
SSH_AUTH_SOCK=/tmp/.ssh-pageant-[username]
SSH_PAGEANT_PID=5948
SVN_SSH=C:\Users\[username]\code\PuTTY\plink.exe

# ssh-add -l
Could not open a connection to your authentication agent.

# ssh-pageant ssh-add -l
Could not open a connection to your authentication agent.

# ssh-pageant -v
ssh-pageant 1.4
Copyright (C) 2009-2014  Josh Stone
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Edit: Not sure if it's suppose to, but the socket referenced in SSH_AUTH_SOCK never gets created (i.e. ls /tmp//tmp/.ssh-pageant-[username] returns file not found).

Edit2: This is what's in my .bashrc file

eval $(/usr/bin/ssh-pageant -r -a "/tmp/.ssh-pageant-$USERNAME")

[cuviper]

@Ginja It's probably just a typo, but you have a repeat in your ls /tmp//tmp/....

It's troubling that even ssh-pageant ssh-add -l doesn't work, as this removes a lot of the variability. Could your problem be something like issue #13? What version of Pageant do you have?

A couple other useful commands: stat $SSH_AUTH_SOCK ; ps -fW | grep -i pageant

[Ginja]

@cuviper yes it was a typo, sorry.

I'm using Pageant version 0.63, and here's the output of the command you suggested:

$ stat $SSH_AUTH_SOCK ; ps -fW | grep -i pageant
stat: cannot stat `/tmp/.ssh-pageant-[username]': No such file or directory
       0    5240       0 ?        15:00:58 C:\Users\[username]\code\PuTTY\pageant.exe
       0    8960       0 ?        19:13:13 C:\Users\[username]\.babun\cygwin\bin\ssh-pageant.eked,xe

Edit: I'm able to successfully use ssh-pageant if I use plink, but I don't think that's a good thing (merely based on what you mentioned in #13)

ssh-pageant plink.exe hostname

In case it matters, I am using a .ssh/config file that I use on non-Windows servers, which points to the non-pageant key, but I've tried commenting it out, and replacing the Identity File value with the .ppk key. But still no luck.

[cuviper]

Well, plink will talk to pageant directly, so ssh-pageant isn't doing anything there.

Are you mixing runtimes? That ps output makes me think so. The first column UID is 0, which is what I expect for "external" processes like pageant, but for ssh-pageant it should be able identify the UID by name. Similarly, the paths are both full Windows paths, but for native I expect to see a cygwin path like /bin/ssh-pageant.

So, it's not necessarily a problem for your ssh-pageant to use a different runtime than the other ssh programs, as long as the socket implementation is still compatible. You'll have to try and see. But I think the first problem here is that they have different mount namespaces. That is, /tmp/ for ssh-pageant is not the same as /tmp/ for ssh-add et al, so they can't even find each other.

Either rebuild ssh-pageant in the same runtime, or try a common path for the socket. As of 1.4, it should work fine to use a Windows path for this, so perhaps:
ssh-pageant -a "$LOCALAPPDATA/ssh-pageant-sock" ssh-add -l

[Ginja]

Thanks for your help, @cuviper.

I was able to resolve the issue by compiling the latest ssh-pageant from source. Previously I was downloading the latest binary, and moving it over to /usr/bin.

$ ssh-pageant.exe ssh-add -l
2048 [hash] imported-openssh-key (RSA)

$ stat $SSH_AUTH_SOCK ; ps -fW | grep -i pageant
  File: `/tmp/.ssh-pageant-[username]'
  Size: 0               Blocks: 1          IO Block: 65536  socket
Device: 1e0078h/1966200d        Inode: 2533274790429416  Links: 1
Access: (0600/srw-------)  Uid: (35556/[username])   Gid: ([gid]/Domain Users)
Access: 2015-02-13 09:22:40.557201900 -0800
Modify: 2015-02-13 09:22:40.558201900 -0800
Change: 2015-02-13 09:22:40.558201900 -0800
 Birth: 2015-02-13 09:22:40.557201900 -0800
       0    1316       0 ?        09:16:58 C:\Users\[username]\code\PuTTY\pageant.exe
[username].sh    5532       1 ?        09:22:40 /usr/bin/ssh-pageant

[cuviper]

Great! I'm still curious, do you have multiple cygwins installed and in your path? The downloaded copy of ssh-pageant must have been getting its runtime from somewhere...

Also, you gave me the idea for #28, which will be useful to know if ssh-pageant can talk to pageant at all. And I still need to capture essential steps from this whole discussion into documentation, which is why I'm leaving this open.

[Ginja]

I do have multiple cygwins in my path. And I really should look at uninstalling it because I prefer Babun's installation.

#28 sounds like a good idea.

[avail]

Hello. Sorry for bumping such an old issue, but, it doesn't seem to work for me. The ENV variables are set, ssh-add -l outputs my keys, but when I try sshing into my server, I can't. I am 100% certain that is the correct key, as I can run putty and it logs in properly.

[cuviper]

Are you sure the environment is set? The first three lines of your screenshot show just the output of running ssh-pageant, but you have to actually run those commands to modify your environment. The easiest way to do that is like this for bash: eval $(ssh-pageant)

When you run ssh-pageant ssh-add -l, that's starting a new temporary agent just for the duration of the ssh-add -l command. That's useful to see if Pageant communication is working, but doesn't tell you if the environment is set in your shell. Try running ssh-add -l alone to test the environment.

[avail]

I have tried that as well, sorry for not mentioning (I have executed all of the commands manually) - I don't use bash, so I don't have eval - still doesn't work though.

Here's an image showing me executing said lines, and trying ssh-add -l

[cuviper]

What shell is it?

Error connecting to agent: No such file or directory

I would first check whether .ssh-pageant actually exists at that path. If it does, then perhaps your shell just has different quoting rules that you'll have to adapt in setting SSH_AUTH_SOCK. At least we know the raw environment value should be ok, since ssh-pageant ssh-add -l worked.

[avail]

it's just a regular command prompt (cmd.exe). The file is present, yes. Modifying the quotation marks to doublequotes, and all the slashes to either backslashes or regular ones doesn't give any different result.

[cuviper]

Ah, cmd is odd in that it doesn't parse quotes at all, so they end up in the literal environment value.

Try a bare value, maybe even set SSH_AUTH_SOCK=%LOCALAPPDATA%/.ssh-pageant

[avail]

Oh, that one worked. Wow.
Thanks for your help. Now to make myself some tool equivalent to eval

[cuviper]

It looks like you're using a fixed -a SOCKET path. If so, I would just hardcode that set too. The pid environment is only needed if you want ssh-pageant -k to work.

[avail]

So I've noticed. Yeah, I'll do just that.

[kenchilada]

Cool project. Can't get it to work using prebuilt cygwin64 1.4. Pageant is definitely running and has my keys added and works elsewhere in Windows.

CYGWIN_NT-10.0 DTxxx 2.6.1(0.305/5/3) 2016-12-16 11:55 x86_64 Cygwin

SOCKDIR=$(cygpath ${LOCALAPPDATA})
if [ -d ${SOCKDIR} ]; then
        eval $(/usr/bin/ssh-pageant -r -a "${SOCKDIR}/ssh-pageant-$USERNAME.sock")
fi

This results in SSH_AUTH_SOCK being set correctly on the environment and...

$ ls -n ${SSH_AUTH_SOCK}
srw-------+ 1 1049818 1049089 0 Feb 13 16:53 /cygdrive/c/Users/Ken/AppData/Local/ssh-pageant-ken.sock

However....

$ ssh-add -l
error fetching identities for protocol 2: agent refused operation
The agent has no identities.

And...

$ ssh-pageant ssh-add -l
The agent has no identities.

Any ideas? I feel like perhaps it cannot communicate with Pageant but I am not smart enough to go further.

[cuviper]

@kenchilada Are your ssh-pageant and ssh-add running in the exact same Cygwin environments? What version of Pageant are you running?

I really should get around to #28...

[kenchilada]

I think I was trying something stupid. I've been connecting to my machine through sshd with putty.exe, and thought this would all work everywhere because ssh-pageant is using a socket. I suppose what I was looking for was really just agent forwarding in putty.exe.

If I use regular Cygwin mintty like a normal person, the ssh-pageant ssh-agent -l works just fine.

I'm curious why it didn't work like I thought? Is it simply because sshd.exe is running as a different user and cannot access the socket?

[cuviper]

Ah yeah, when running via sshd you can just use normal agent forwarding. Normally, ssh-pageant doesn't access any socket (except for the -r reuse test) - it creates a socket for others that want that unixy interface. It accesses Pageant through Windows messages, and you're probably in a different security context under sshd than the Pageant running on the desktop.