Make SSH use `gpg-agent`

2018-04-01

This document covers how to "trick" SSH commands into using gpg-agent instead of ssh-agent, which makes it possible to hold your SSH secret keys on a YubiKey.

Pre-requisites

The two obvious dependencies are an SSH client, and gnupg. One or both of these are usually installed on most Linux and macOS machines.

Most Linux distros come with openssh already installed, however some distros may split the client and server bits into separate packages. Some distros may install gnupg as well - if not, you should be able to use yum, apt-get, or a similar command, to install the necessary packages. Search

CentOS, Fedora, RedHat, etc.

yum install openssh-clients gnupg2 gnupg2-smime

Debian, Ubuntu, etc.

I'm using Xubuntu 18.04 on a few workstations at home. The commands I use to configure SSH to use gpg-agent on these machines are...

sudo apt install scdaemon gpg-agent

mkdir -p ~/.gnupg
echo use-agent >> ~/.gnupg/gpg.conf
echo enable-ssh-support >> ~/.gnupg/gpg-agent.conf

xfconf-query -c xfce4-session -p /compat/LaunchGNOME -n -t bool -s false
xfconf-query -c xfce4-session -p /startup/ssh-agent/enabled -n -t bool -s false

If you're curious, this document is my checklist for setting up Xubuntu. Unfortulately Keybase doesn't render Markdown to HTML like my web server does, but Markdown is pretty easy to read on its own.

Other Linux distros

I don't have the exact commands for every other distro out there. For gnupg you should search for packages with names like gnupg, gpg2, or maybe just gpg.

macOS - GPGTools

Note: I don't use GPGTools anymore, but I'm leaving this info here. See "macOS - Homebrew" below for more information.

For macOS, the openssh client is installed as a basic part of the OS, however gnupg is not. There are two ways to install the gnupg tools:

Visit https://gpgtools.org/ and install the current version of GPG Suite.

Not only will this give you the gpg command line tools, but it also includes a System Preferences widget to control some aspects of how gpg and gpg-agent work, along with a Mail.app plugin to support signing and encrypting email.
brew install --cask gpg-suite will install the same package, using Homebrew.

Note: you can also use brew install --cask gpg-suite-no-mail if you don't need the Mail.app plugin.

Note that both methods end up installing the same software, I just find it easier to use the command line, so I use Homebrew on my macOS machines.

Also note that the Mail.app plugin is not free. It's not horribly expensive, and it's not a "subscription" (it's a one-time purchase for each "major version" of the GPG Suite package), however they only allow five "activations", and the "Paddle" framework wants to connect to api.paddle.com on a regular basis.

I don't like the whole "limited number of activations" thing, and I hate any kind of system which contstantly "phones home" like like this, so ... while I do believe in supporting the authors of the software I use, I figure the donation I sent them a few back covers my use of the command line tools and the Preferences widget, and I use Thunderbird with Enigmail instead of their Mail.app plugin.

macOS - Homebrew

I was working on another page today (2022-01-22) and noticed that the machine (a MacBook Air M1) appeared to have three different versions of gpg installed, from a combination of "MacGPG2", "GPGTools", and Homebrew. In the interest of "cleaning up", I decided to remove all but one - and the Homebrew version is what I decided to keep, since it's a dependency of a few other Homebrew packages I use, and because it's quicker and easier to install. (I'm familiar enough with gpg and key management that I don't really need the key management GUI and System Preferences widget.)

After downloading and running the GPGTools Uninstaller (direct download link) I discovered that the "MacGPG2" version was also gone, and the Homebrew version was the only thing left on the machine. (Apparently "GPG Suite", "GPG Tools", and "MacGPG2" are all the same thing.) I ran into some issues after removing GPGTools ... long story short, GPG uses a program called pinentry to ask the user for a PIN code when a "card" requires one. The pinentry program from GPGTools was the only one on the machine, so the error was because gpg-agent wanted to ask for a PIN but had no way to do so.

The fix was to install a "pinentry" program using Homebrew. Running "brew search pinentry" command showed that there's a "pinentry-mac" package, and "brew info pinentry-mac" confirmed that it is what it sounds like, and after installing it, I'm able to ssh just like I did before removing GPGTools.

TL;DR This command will install the necessary packages from Homebrew.

brew install gnupg pinentry-mac

I also had to configure gpg-agent. Details are below, but here's the short version:

mkdir -p $HOME/.gnupg
cat > $HOME/.gnupg/gpg-agent.conf <<EOF
enable-ssh-support
pinentry-program    /opt/homebrew/bin/pinentry-mac
EOF
gpg-connect-agent killagent /bye
gpg-connect-agent /bye

After restarting gpg-agent, everything is working again.

Setup - Linux

To make the current shell use gpg-agent (and therefore the YubiKey) instead of the normal ssh-agent ...

Manual process

Make sure the GPG_TTY variable is set.
```
export GPG_TTY=$(tty)
```

Make sure that the SSH_AUTH_SOCK variable points to the S.gpg-agent.ssh socket.

unset SSH_AGENT_PID
export SSH_AUTH_SOCK="$( gpgconf --list-dirs agent-ssh-socket )"

Any commands executed in this shell will use gpg-agent as the SSH agent.

Automatic process (shell, per-user)

To make sure that your shell always sets the GPG_TTY and SSH_AUTH_SOCK variables correctly, add the following to your .bash_profile (or the appropriate file, if your login shell is not bash)

########################################
# Set things up for using gpg-agent

export GPG_TTY=$(tty)

function use-gpg-agent-for-ssh {
    SOCK="$( gpgconf --list-dirs agent-ssh-socket )"
    if [[ -n "${SOCK:-}" ]]
    then
        unset SSH_AGENT_PID
        export SSH_AUTH_SOCK="$SOCK"
    fi
}

use-gpg-agent-for-ssh

Note that this creates a function to "do the work", and then calls that function. This way if you decide you don't want this all the time, you can comment out just the function call (the last line), and then you can type use-gpg-agent-for-ssh in any shell to easily "activate" the change within that shell.

Once you have added this, every new interactive shell will use the changes. A quick way to test it is to open a new terminal window, which will contain a new shell. Once you have verified that it's working, you can either close the shell you're working in and open a new window, or you can run "source ~/.bash_profile" to read the updated profile into the current shell.

Note that setting the variables in this way will only affect shells and any processes started from those shells. In particular, it will NOT affect processes started by something other than your shell, such as cron jobs.

Automatic process (all users)

The process is the same as the "shell, per-user" process above, except that instead of editing your ~/.bash_profile file...

You will edit /etc/profile, so that all users will use it.
On some systems (such as CentOS 6 or 7, and probably 8 although I haven't tried it yet) you may be able to create an /etc/profile.d/use-gpg-agent-for-ssh.sh file instead.

If your system has multiple users, and some of them may wants to use the normal ssh-agent, you may want to not include calling the function (i.e. the final use-gpg-agent-for-ssh line) in what you add to the system-wide profile. In this case, users who do want to use gpg-agent by default can add a user-gpg-agent-for-ssh line to their ~/.bash_profile, and anybody on the system can manually type that command to use gpg-agent within that shell.

Setup - macOS

In macOS, LaunchAgents are configurations which starts a process or runs a command automatically. macOS comes with a LaunchAgent which does the following, every time a user logs in:

Creates a UNIX socket with a dynamic name, and sets things up so that ssh-agent is automatically started, listening on on that socket, the first time a process connects to the socket. (If multiple users are logged in, each user will have their own socket and their own ssh-agent process.)
Exports an SSH_AUTH_SOCK environment variable whose value is the path to that dynamically generated socket.

We need to change things around so that the SSH_AUTH_SOCK variable points to the name of a socket where gpg-agent is listening.

My first thought was to change the value of the SSH_AUTH_SOCK variable itself, and I did figure out how to do this automatically when the user logs in, by disabling the built-in LaunchAgent which runs ssh-agent. However...

OS X 10.11 "El Capitan" added a security feature called System Integrity Protection (or "SIP"). This made things more difficult, in that you had to disable SIP (which requires rebooting into "recovery mode") before you could disable the LaunchAgent, and then reboot to re-enable SIP again afterward (because SIP itself is actually a good idea, I just don't think that the automatic ssh-agent startup should have been included within its scope.)
macOS 10.15 "Catalina" added another feature where the root filesystem is mounted "read only", which added another set of hoops that had to be jumped through.
macOS 11.0 "Big Sur" took it a step further by digitally signing the contents of the root filesystem. I haven't actually tried it, but it sounds like if you were to delete or change the LaunchAgent file, the signatures won't match and the OS would refuse to boot at all.

While I was hunting for information about how to disable this LaunchAgent in Catalina, I found this article which explained a different way to solve the problem. Instead of disabling the macOS LaunchAgent, we can add our own LaunchAgent which runs after theirs, which replaces the UNIX socket created by the built-in LaunchAgent, with a symbolic link to the UNIX socket where gpg-agent is listening for SSH agent requrests. By doing this, any client which uses the $SSH_AUTH_SOCK value to connect to an SSH agent, still uses the randomly generated filename which was pointing to ssh-agent, however now points to to gpg-agent, and that's what the SSH client ends up talking to.

The only part of this I'm not clear about is how to ensure that our LaunchAgent runs after Apple's LaunchAgent. It's probably something as simple as "launchd processes the system LaunchAgents before any user LaunchAgents", but I haven't seen any official documentation which says that, so ... while I've never seen it happen, I'm not totally convinced that the two LaunchAgents wont accidentally run in the wrong order at some point.

Quick version

Install the GPG software, using one of the following methods:
- Install GPG Tools, with or without the GPG Mail support. (I haven't used "GPG Mail" since they started charging for it, and I don't use "GPG Tools" at all anymore.)
- Homebrew.
```
brew install gnupg pinentry-mac
```
  Quick configuration:
```
mkdir -p $HOME/.gnupg
cat > $HOME/.gnupg/gpg-agent.conf <<EOF
enable-ssh-support
pinentry-program    /opt/homebrew/bin/pinentry-mac
EOF
gpg-connect-agent killagent /bye
gpg-connect-agent /bye
```

Install the two LaunchAgent files.

cd ~/Library/LaunchAgents
curl -O https://jms1.net/yubikey/net.jms1.gpg-agent.plist
curl -O https://jms1.net/yubikey/net.jms1.gpg-agent-symlink.plist

Either log out and log back in, or reboot the machine.
When you log back in, verify that the SSH_AUTH_SOCK environment variable points to a temp file which is a symlink to your $HOME/.gnupg/S.gpg-agent.ssh file (or technically a named pipe).
```
% ls -l $SSH_AUTH_SOCK
lrwxr-xr-x  1 jms1  wheel  34 Dec  6 10:44 /private/tmp/com.apple.launchd.gR4WHD21R5/Listeners -> /Users/jms1/.gnupg/S.gpg-agent.ssh
```
If you already have a YubiKey with an SSH key loaded, verify that you're able to see the key.

With the YubiKey NOT inserted:
```
% ssh-add -l
The agent has no identities.
```
With the YubiKey inserted:
```
% ssh-add -l
4096 SHA256:xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx cardno:nnnnnnnnnnnn (RSA)
```
In both commands. "-l" is a lowercase "L", not the digit "one".

Also note that the SSH public key doesn't have the same "comment" that you might normally have after a key. Remember that the YubiKey only really stores the secret key, from which the public key is derived. It doesn't store any kind of metadata about the key. The comment it shows is the serial number of the YubiKey.

Download links for the LaunchAgent files

net.jms1.gpg-agent.plist - starts gpg-agent
net.jms1.gpg-agent-symlink.plist - replaces the UNIX socket with a symlink

Details

Configure `gpg-agent`

To configure gpg-agent to support SSH, add this line to $HOME/.gnupg/gpg-agent.conf:

enable-ssh-support

To configure gpg-agent to find its "pinentry" program...

Find the full path to the pinentry program. I did this by typing "pinent" and then hitting TAB, which showed the following output:
```
pinentry         pinentry-curses  pinentry-mac     pinentry-tty
```
From these, it seemed obvious to me that "pinentry-mac" was the one I wanted, so I found the full path to that...
```
$ which -a pinentry-mac
/opt/homebrew/bin/pinentry-mac
```
Once you have the path, add this line to $HOME/.gnupg/gpg-agent.conf:
```
pinentry-program    /opt/homebrew/bin/pinentry-mac
```

If you changed the gpg-agent.conf file for any reason, you should restart the running gpg-agent process:

gpg-connect-agent killagent /bye
gpg-connect-agent /bye

Make `gpg-agent` start automatically

Create $HOME/Library/LaunchAgents/net.jms1.gpg-agent.plist with the following contents: (adjust the path to gpg-connect-agent as needed)

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN"
  "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
  <dict>
    <key>Label</key>
    <string>net.jms1.gpg-agent</string>
    <key>RunAtLoad</key>
    <true/>
    <key>KeepAlive</key>
    <false/>
    <key>ProgramArguments</key>
    <array>
      <string>/usr/local/MacGPG2/bin/gpg-connect-agent</string>
      <string>/bye</string>
    </array>
  </dict>
</plist>

Tell launchd to use it.

launchctl load net.jms1.gpg-agent.plist

Replace the socket with a symlink

Create $HOME/Library/LaunchAgents/net.jms1.gpg-agent-symlink.plist with the following contents: (adjust the path to the socket file as needed)

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/ProperyList-1.0/dtd">
<plist version="1.0">
  <dict>
    <key>Label</key>
    <string>net.jms1.gpg-agent-symlink</string>
    <key>ProgramArguments</key>
    <array>
      <string>/bin/sh</string>
      <string>-c</string>
      <string>/bin/ln -sf $HOME/.gnupg/S.gpg-agent.ssh $SSH_AUTH_SOCK</string>
    </array>
    <key>RunAtLoad</key>
    <true/>
  </dict>
</plist>

Tell launchd to use it.

launchctl load net.jms1.gpg-agent-symlink.plist

Restart

You will need to either reboot, or log out and log back in, in order to activate these changes.

Make sure it worked

After rebooting or logging back in, make sure it worked.

Make sure the variable exists, pointing to a random name.
```
$ env | grep SSH
SSH_AUTH_SOCK=/private/tmp/com.apple.launchd.CaehyEWKPw/Listeners
```
The "CaehyEWKPw" portion of the name will be different every time you log into the machine. This is normal.

Make sure that name is a symlink, pointing to the gpg-agent SSH socket.

$ ls -l $SSH_AUTH_SOCK
lrwxr-xr-x  1 jms1  wheel  34  Feb 18 00:55 /private/tmp/com.apple.launchd.CaehyEWKPw/Listeners -> /Users/jms1/.gnupg/S.gpg-agent.ssh

Note: the command has a "lowercase L" option.

Make sure the agent is reachable.
```
$ gpg-connect-agent -v /bye
gpg-connect-agent: closing connection to agent
```
You should just see the message shown above.
Make sure the YubiKey is connected.

Make sure gpg is able to talk to your YubiKey.

$ gpg --card-status
Reader ...........: Yubico YubiKey OTP FIDO CCID
Application ID ...: D276000124010304xxxxxx
...

Make sure the agent is able to talk to the YubiKey.

$ ssh-add -l
4096 SHA256:l7CsDA23ENutkRsZ5jhlqJfl2syaiJfHni7b95e8dQ4 cardno:0006xxxxxxxx (RSA)

Usage

If you've gone through the setup process above, and the SSH_AUTH_SOCK variable points to the S.gpg-agent.ssh socket, you don't really need to do anything differently - just use ssh, scp, sftp, or whatever, the same way you already do. As long as your SSH client works with an agent, and your YubiKey is physically plugged into the computer, it should all "just work".

If you haven't gone through the steps above ... do so.

`authorized_keys`

To get the public key line needed for authorized_keys files...

Insert the YubiKey and wait a few seconds.

Run "ssh-add -L".

$ ssh-add -L
ssh-rsa AAAAB3NzaC1yc...9toFRmxejrbw== cardno:0006xxxxxxxx

The "cardno:xxxxx" at the end of the line is a comment. When using the value in an authorized_keys file I normally replace this with something more useful than the serial number...

$ cat .ssh/authorized_keys
ssh-rsa AAAAB3NzaC1yc...9toFRmxejrbw== jms1@jms1.net 2019-03-21 hardware token

Notes

The gpg-agent automatically "contains" the Authentication Keys stored on the YubiKeys (or other OpenPGP cards) present on the system. When gpg-agent receives an authentication request, it passes it along to the YubiKey, which does the work of signing the request without sending the secret key anywhere.

Other keys can be added to the agent using ssh-add. When you do this, a copy of the secret key will be written to a file in the ~/.gnupg/private-keys-v1.d/ directory, named after the "key grip" (another kind of fingerprint, which includes the options rather than just the public key).

However, there are a few things to be aware of.

These files are stored separately, and may be encrypted using a different passphrase than the SSH secret key file.

When you add a key, you will be prompted first for the existing passphrase (to read the secret key), and then for a new passphrase (to encrypt the secret key in this new file).
Later, when you're prompted for a passphrase in order to use the key, you will need to enter the "new passphrase" rather than the original one.

The "ssh-add -d" (or -D) command will not remove these keys.

gpg-agent adds the key grips (similar to a Key ID) to a file called "~/.gnupg/sshcontrol".
Removing the key grip from this file makes the key no loger appear in the "ssh-add -l" output, and no longer be available for SSH authentication.
Removing the ~/.gnupg/sshcontrol file itself will make ALL keys no longer appear in the "ssh-add -l" output, or be available for SSH authentication. (This does not include keys stored on YubiKeys or other cards.)
Editing or removing this file will not remove the files under the ~/.gnupg/private-keys-v1.d/ directory. You will need to remove those files by hand.

Changelog

2024-06-19 jms1

moved page to new jms1.info site, updated header

2022-01-22 jms1

Updated with more info about "GPG from Homebrew" and less about GPGTools, since I don't use GPGTools anymore.
Also added more details about what a "pinentry" program does.

2021-01-07 jms1

changed "brew cask install" to "brew install --cask"

2020-12-20 jms1

Moved to jms1.info, moved this Changelog to the end of the file
Added macOS versions where security changes were added
Other minor formatting updates

2020-12-06 jms1

Verified that the macOS setup process described below (i.e. installing the two LaunchAgent files and log out/in) DOES work with macOS 11.0 "Big Sur", on both Intel and Apple Silicon processors.

This is a LOT easier to set up, so this is what I'm doing with my own machines as I upgrade them to Catalina. I've updated this page with information about that process.

2020-02-23 jms1

A few weeks ago I updated this page with information about how to set this up on Catalina. While I was thinking about it I happened across this article which accomplishes the same overall goal, but instead of disabling the macOS com.openssh.ssh-agent LaunchAgent, it creates a symlink with whatever name $SSH_AUTH_SOCK contains, pointing to $HOME/.gnupg/S.gpg-agent.ssh.

2019-09-01 jms1

last version updated on Keybase

Older jms1

I wasn't keeping any kind of changelog before this, so I can't really include more details here, other than the fact that the very first version of this page was written some time in 2018.

Generated 2024-09-05 04:05:29 +0000

initial-21-g1f47195 2024-09-05 04:05:12 +0000

jms1.info