Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

[jms1]
Me, 2023-11-14

jms1.info

This site is a collection of random documents that I've written over the past 20+ years. Many of them are technical, but some will be related to non-technical things that I happen to be interested in. I'm trying to keep them organized by topic, as you can see in the index to the left.

One of the reasons I'm putting them here is so that I know where to find them. I've found that I can't hold every little detail about everything in my brain anymore, so if I write things down, I'll be able to go back and refresh my own memory when I need to.

Another reason is that people have asked me about a lot of these topics, so having them on a web page lets me give a quick answer and include a link where they can read more information, rather than having to spend a bunch of time answering the same questions over and over again.

And if other people find the information on this site useful, then ... I'm glad I could help.

Notes

Some pages may not be complete.

The first part of how I add pages to this site is to just copy an existing Markdown file. A lot of the original files I'm copying here will contain very basic bare-bones information, enough for me to understand (usually), but possibly not useful to others. (Part of why I'm adding them here is to add the more human-friendly explanations, both for others, and for myself a year from now when I've forgotten all this.)

When I do this, my intention is to go back and add more detailed information to those pages, but I don't always remember to do it - especially if I'm in a hurry.

If you find one of these "bare bones" pages and need more information, feel free to let me know.

Some of the links on the left may not exist.

These are pages I'm planning to write, but haven't done so yet.

There's not a whole lot of content here.

This is starting to not be the case anymore, but as I mentioned above, I've been writing down and keeping random notes for over 20 years. I started the current version of this site in 2024-06, it's going to take a while for me to find, organize, prioritize, and write the pages here. Plus I'm doing this in my spare time, I have a full-time job which keeps me pretty busy and, most days, makes me not want to even look at a computer when work is done.

My Other Sites

These are some other public web sites that I've written over the years.

  • jms1.net is my original "home page". It started off containing "everything", but over time I found that certain topics were taking enough space to justify moving them to their own sites.

  • qmail.jms1.net is my web site about Qmail. When I started using qmail I took several other patches and combined them into a single patch, and I put the patch on the jms1.net site. People started using my combined patch and asking questions about it, so I moved all of the qmail-related content to its own web site.

    I was making my living as an independent contractor at the time, and unfortunately, since returning to the world of full-time employment, I haven't had time to maintain the site (or the combined patch). If you're building a mail server I still recommend qmail, but I don't really recommend my own combined patch anymore, because it is so far out of date.

  • kg4zow.us is my site about Amateur Radio (aka "ham radio"). I'm not as active on the air as I used to be, partly because I moved to a smaller city which doesn't have as many ham radio operators around, and partly because I don't have the time.

  • remarkable.jms1.info is a site I started for information about the reMarkable tablets. It's a Linux-based e-ink tablet with a textured screen that I find really nice to write on. One of the things I really like about it is that it allows SSH-as-root out of the box. I've been writing my own programs to work with them without using reMarkable's cloud service.

  • jekyll.jms1.info was my first attempt at writing this site. As the name suggests, I was using Jekyll to manage the content, but in 2024-06 I decided to re-do the entire site using mdbook, since I'm a lot more familiar with it (I use it to maintain half a dozen internal web sites for $DAYJOB), and because it's a lot easier to install, configure, and update on a new workstation. Plus, it makes more sense to me to organize the pages by subject rather than by date.

Contact

The best way to reach me is to email jms1@jms1.net. If your message bounces (my server does some pretty aggressive spam-blocking), change net to me - that will also reach me, but it works through Apple's servers. Note that I don't always check personal email every day, so if you email me and I don't respond, please be patient.

If you need an immediate response for some reason, or if you've found a problem with the information on this site, you can use Keybase chat. My username there is jms1.

About this site

I'm writing this site as a collection of Markdown files, converting them to static HTML files using mdbook with some customizations, tracking the changes using git, and hosting the finished site using Keybase Sites.

Clockwork Pi uConsole

The Clockwork Pi uConsole is a handheld Linux computer, made by Clockwork Pi.

Hardware

The uConsole is sold as a kit, which includes ...

  • a metal frame, that the parts mount into
  • a 5-inch 1280x720 display
  • a keyboard with a directional pad and "gaming" buttons (A/B/X/Y/Select/Start)
  • a trackball with L/R mouse buttons
  • a pair of small internal speakers
  • a set of circuit boards to connect the various components together
  • metal front and back covers
  • an external wifi antenna, meant to be attached to the outside of the case

No soldering is required. All parts are secured to the frame using M4 hex-head screws, and the kit comes with a 2.5mm hex driver (aka "allen wrench"), which is the only tool needed to assemble the unit.

The main board has a single USB-A port (USB 2.0), a USB-C port (for power input only), a Micro-HDMI port, and a 3.5mm headphone jack. There is room inside the unit for an expansion board, which may have its own external connectors on the other side of the unit.

The kit can can be ordered with one of several "core" boards containing an ARM- or RISC-based CPU board, or with no "core" (for people who may already have a Raspberry Pi Compute Module 4/5). I ordered mine with a Raspberry Pi Compute Module 4 core.

It can also be ordered with a 4G Cellular/GPS expansion board, with its own antenna meant to be attached to the outside of the case.

Open Source

The design is all open-source. The uConsole repo on Github contains schematics for the circuit boards, source code for the keyboard firmware, and information about how they built the kernel with the correct modules for the uConsole hardware.

Their web site sorta says that the design files for the frame and front/back covers should be there as well, but I don't see them in this repo (or in any of their repos) as of 2025-12-31.

Batteries

Batteries are NOT included with the kit. This is due to safety regulations around shipping lithium batteries.

The uConsole is powered by a pair of 18650 rechargable Li-ion batteries, with charging circuitry included on the main board. The battery holder within the unit can hold either flat-top or button-top batteries.

⇒ The Batteries page has more details, including notes about which batteries to order and which ones to avoid.

Power Supply

A power supply is NOT included with the kit. This is probably because different parts of the world use different power connectors and voltages. It uses a standard 5V USB-C connector.

Most USB-C chargers for mobile phones or tablets should be usable with the uConsole, however you need to be careful when using "high power" adapters designed for laptops.

⇒ The Power Supply page has more details.

Cases

Coming soon.

Below is a quick list of what I plan to include on the page when I write it.

  • USA Gear case
    • has very little room for accessories (designed for a camera display?)
    • semi-rigid, not solid
  • Pelican 1200
    • has more room
    • SOLID
    • other colours available (black, tan, orange, yellow)
    • more expensive
    • has a lifetime warranty

Power

Indicators

The uConsole comes with a translucent white rubber button for power. The button is translucent because there are two LEDs on either side of the button itself, which can be seen through the button:

  • A green LED, which indicates that the uConsole's power is on.

  • A yellow LED, which indicates that the uConsole is charging the internal batteries.

Power Supply

The uConsole requires a 5V 2A power supply with a USB-C connector. The power supply can be used to charge the batteries, and can also be used to power the unit when batteries are not installed.

A power supply is NOT included with the kit. Most USB-C chargers for mobile phones or tablets should be usable with the uConsole, however ...

The uConsole does not perform USB-PD power negotiation. It expects the USB-C port to receive 5V power, and can draw up to 3A. The highest I've seen it draw is 1.8A, or 5W). Chargers which only supply 5V, or which correctly implement the USB-PD power negotiation process, should work with the uConsole.

What this means is, any USB power supply which only supplies 5V, or any newer power supply which can supply more but correctly implements USB-PD, should be usable with the uConsole. Be aware that some power supplies may not negotiate USB-PD correctly and can damage the uConsole, as this forum post illustrates.

  • This power supply (I think from an old iPad) only outputs 5V, up to 2.4 A. My uConsole works just fine with it.

  • This power supply (from a newer iPad Pro) is the one I carry in the box with my uConsole. It can output 5V or 9V, up to 3A. It is marked as supporting USB-PD.

  • This power supply came with a Raspberry Pi 5 kit. It isn't explicitly marked as supporting USB-PD, however the list of voltage levels is identical to the list of voltage levels supported by USB-PD. A quick test shows that it doesn't try to supply more than 5V to the uConsole, and the uConsole was able to charge the battery while plugged into it.

Batteries

The uConsole is powered by a pair of 18650 rechargable Li-ion batteries. The battery holder within the unit can hold either flat-top or button-top batteries. The main board has the necessary circuitry to charge the batteries, so you don't have to remove them in order to recharge them.

The batteries are connected in parallel, which means that it will run with only a single battery installed. It also means that if you need to replace the batteries, you could theoretically swap them out one at a time while the system is running, however this is not recommended.

Batteries are NOT included with the kit. This is due to safety regulations around shipping lithium batteries.

You probably won't find these batteries in a normal retail store. I've seen them at a dedicated battery store, and apparently most "vape" shops sell them as well (apparently vape pens use them?) They can also be ordered from suppliers like Amazon or AliExpress, however you need to be careful not to order "fake" batteries which don't have the capacity they claim.

⚠️ Battery Capacity

The physical and electrical characteristics of Li-ion batteries limit the capacity of a 18650 cell to about 3600 mAh, with the "normal" capacity being 2500-3000 mAh. If you see batteries advertised with higher capacities (such as 5000 or 9900 mAh), be aware that the seller is probably lying about the capacity, and their claims should not be trusted.

I admit, I fell for this. In 2025-01 I bought some batteries which were advertised as having 5000 mAh capacity, for a different device (an Arpeggio mini-synth, if you're curious). The batteries do work, however it was the first time I had ever used 18650 cells so I didn't realize that only two hours of run-time wasn't normal.

I have since ordered a set of higher quality batteries (links below), with a charger that has a capacity tester function. The new batteries all test between 2890 and 2960 mAh, while the old ones that I tested only had 1380 and 1400 mAh. I have thrown the old batteries out.

Calibration

When the batteries in the uConsole are changed, you may need to calibrate the power controller so it knows the maximum capacity of the new batteries and can calculate the right percentage to show in the on-screen battery indicator.

This is covered on the Battery Calibration page.

Batteries

  • Amazon - Authentic Samsung30Q, 3.7V Flat Top Real 3000mAh 18650 Battery Rechargeable 30Q (4 Pack)

    While reading about peoples' experiences with "fake" batteries, I found several web sites which recommended buying only genuine Panasonic or Samsung batteries. These are advertised as "Authentic Samsung", so I'm hoping they will last longer than two hours.

    These arrived on 2026-01-06. The word "Samsung" doesn't appear anywhere on the batteries or packaging, but so far they seem to be a lot better than the previous "fake" batteries I was using.

    The first thing I did was put them on a charger. Unlike the previous "fake" batteries I was using (which went from empty to "full" in less than two hours), I had to charge them for over four hours before the charger considered them "full", and the uConsole immediately recognized them as 100% full, rather than showing a seemingly random number between 25% and 70%.

Charger

Power Testers

These are little gadgets that plug in between the charging cable and the uConsole, to show how much power the uConsole is drawing.

Other

Related links (mostly Clockwork Pi Forum posts) I used while writing this page, not in any particular order.

Battery Calibration

When you put a new set of batteries into the uConsole, the power controller needs to "learn" the capacity of the new batteries. This process is known as "calibration". Until you calibrate the power controller, it won't know the real capacity of your batteries, and may report 65% when they're empty, or (as mine were when I started writing this page) 34% when they've been charging overnight and are presumably full.

I've seen several web pages, mostly on the Clockwork Pi forums, which tell how to do this. Every set of directions is either a copy of the same three sentences, or a reference to those three sentences.

It seems to me that some details are missing, so I'm writing this page to try and provide directions which are (1) more complete, and (2) more detailed. A big part of this is because I'm currently trying to do the calibration process and seeing some really weird results.

My Experience So Far

A year ago I bought a set of 18650 batteries with a stated capacity of 5000 mAh, to use with a different device. I didn't notice anything wrong with them at the time, but I also had never used 18650's before so I didn't really have a good idea of what to expect.

Using the batteries I had ordered a year ago ...

  • I left the uConsole plugged in and running overnight, with a tester between the cable and the uConsole to show how much power was being transferred. When I started, the power cable was supplying 9W, presumably to run the system and charge the batteries.

  • In the morning it showed the battery level at 68%, and the power cable was supplying 4W.

  • I ran the command to start the calibration process, unplugged the power cable, and ran a command to "exercise" the CPU (computing SHA256 checksums of 1GB blocks of data from /dev/urandom).

  • About 2h30m later, the system died.

  • I plugged the power cable back in to charge the batteries. The tester on the power cable showed that it was supplying 9W.

  • After about two hours this dropped from 7W to 3W, then over the next half hour it gradually drifted down to zero. This tells me that the power controller stopped charging the batteries, presumably because the batteries were full.

  • When I powered the system back on, the power controller reported the batteries were only 43% full ... and with the power cable connected and supplying 4W, the reported battery level drifted DOWN to 25% over half an hour, and is now sitting at a steady 25%.

  • I then unplugged the cable and let the uConsole run from just the batteries. It died in about 1h45m.

Fake Batteries

While researching this page, I came across several pages which explained that, with the battery chemistries being used today, 18650 batteries cannot hold more than about 3600 mAh each. This means that the batteries I was using, did not have the capacity they claimed to have. Two of these pages also recommended buying genuine Panasonic or Samsung batteries.

Back on Amazon, I found a listing for "Authentic Samsung" batteries. I took the chance and ordered these, along with a charger with a capacity tester.

The capacity tester showed all four of the new batteries having 2890-2960 mAh each, and two of the older batteries I had been using, having 1380 and 1400 mAh (I stopped testing and threw all six of them out, not worth wasting the time to prove that junk is junk).

I've been using the new batteries for about a week, they last over four hours in the uConsole.

Procedure

To calibrate the power controller for a new set of batteries ...

  • If you have a power tester (see below), plug it in between the power cable and the uConsole so you can monitor how much power the is being consumed, rather than having to rely only on the incorrect information from the uConsole's charging circuitry. (After all, the whole point of calibrating the power controller is to be able to trust the information it provides.)

  • If the uConsole is configured with any "auto shutdown" or "battery saver" featires, disable them. We're going to be deliberately running the batteries down to empty, which means we want the machine to not shut down by itself.

  • Plug in the charging cable and wait until the batteries are full.

    If you're using a power tester, make sure it's showing how much power (watts) the uConsole is consuming. If it's showing volts and amps, you can multiply the values to get watts (i.e. 1.8A at 5V => 9W).

    You can watch the state of the battery using upower, which is normally included with the Raspberry Pi images.

    • Run upower -e to list all power-related devices. The devices will have names that look like /org/freedesktop/Upower/devices/..., the internal battery controller will have a name containing apx20x_battery.

    • Run upower -i DEVICE to see the properties of that device.

    BAT=$( upower -e | grep 'apx20x_battery' )
watch -n5 "upower -i $BAT | egrep '(state|percentage):'"

    You can leave the watch command running in a terminal window while performing the steps below, this will run the command over and over every five seconds so you can watch the numbers as they change.

    When the batteries were full, or at least when I thought the batteries were full, I noticed the following:

    • In the watch ... upower -i output, the state value started changing between charging and discharging.
    • On the power meter, the usage dropped from 9W to 2W.

  • Tell the power controller to start the calibration process. In a terminal window ...

    echo 1 | sudo tee /sys/class/power_supply/axp20x-battery/calibrate

  • Unplug the power cable, so the uConsole is running on batteries.

  • Let the unit run until the battery dies. This can take a while.

    If you'd rather not wait so long, you can do things on the system to increase it's power consumption. On my uConsole, I ran the following command in a terminal window to keep the CPU busy:

    while true ; do dd status=none bs=1G count=1 if=/dev/urandom | sha256sum ; done | cat -n

    This command makes the kernel generate 1 GB of random numbers (reading from /dev/urandom generates random bytes), calculates the SHA256 checksum of those numbers, and prints the result, over and over until the system dies. (The cat -n at the end puts a count in front of each checksum, in case you're curious how many GB of random data have been processed.)

  • When the uConsole dies, plug the power cable in, but do not power it on.

  • Wait until the batteries are full. If you're using a power tester on the wire, you'll see the power consumption drop to zero when this happens. If not, all I can suggest is to let the unit charge overnight.

  • When you power the unit on, it should have an accurate picture of the batteries' true capacity, and the percentages should be accurate.

Software

Coming soon.

Below is a quick list of what I plan to include on the page when I write it.

git

This section contains pages dealing with the git source code control system.

Move a Commit to a New Branch

Many companies or teams have a policy that all work needs to be done on a "feature branch", which is then merged into a "primary branch" using a pull request. This allows people other than yourself to review your work before it gets merged into the main code.

I'm not perfect, sometimes I forget to create a new branch first, and accidentally create commits directly on the primary branch. Usually I realize this before pushing anything, which means I can fix it on the local machine first.

Quick Explanation

It helps if you remember that a "branch" is just a name, pointing to a commit.

What we're going to do is this:

  • Create the new branch, pointing to the last of the new commits.

  • Move the main branch to point to what it was pointing at before we started creating commits.

Starting Condition

In this examples below, we're going to assume that the recent commits in the repo look like this:

$ git tree1 -a
* 67f8356 (HEAD -> main) 2024-06-20 jms1(G) fix typo
* 8a837d6 2024-06-20 jms1(G) start new feature
*   1d3158c (origin/main) 2024-06-13 jms1(G) Merge branch 'prev-feat'
|\
| * d60b020 (origin/prev-feat) 2024-06-12 jms1(G) previous feature
|/
* 3accd26 2024-05-29 jms1(G) Merge branch 'old-feat'

ℹ️ git tree1

This is one of my standard git aliases. It actually means ...

git log --date-order --decorate --graph --no-show-signature '--pretty=tformat:%C(auto)%h%d %C(brightcyan)%as %C(brightgreen)%al(%G?)%C(reset) %s'

... although you won't see the colours in the copy/pasted text on this page.

In this case, I created two commits, 8a837d6 then 67f8356, then realized I should have created a feature branch for it first.

Create a New Branch

The first part of fixing this is to create a new branch, pointing to what should be the HEAD of that new branch. The current HEAD is already pointing to that commit, so if we just create the new branch here, we'll be good.

$ git branch new-feat

Looking at the repo after this, you can see that the new "new-feat" branch exists and is pointing to the correct commit.

$ git tree1 -a
* 67f8356 (HEAD -> main, new-feat) 2024-06-20 jms1(G) fix typo
* 8a837d6 2024-06-20 jms1(G) start new feature
*   1d3158c (origin/main) 2024-06-13 jms1(G) Merge branch 'prev-feat'
|\
| * d60b020 (origin/prev-feat) 2024-06-12 jms1(G) previous feature
|/
* 3accd26 2024-05-29 jms1(G) Merge branch 'old-feat'

Move the main Branch

This will move the main branch to point to the commit that it was pointing to before we started working on the new branch.

Identify the commit where the branch should point

First, identify the commit that it should be pointing to.

In this example, the main branch should be pointing to commit 1d3158c. You can refer to the commit using its hash, or using any other branch or tag name which points to that commit. In many cases, origin/main will be usable.

Check out the branch you're moving

Next, make sure the branch you're moving (in this case, main) is checked out.

$ git checkout main

In this example, it wasn't really necessary because we were already on the main branch. However, I've trained myself to always do this, just to be on the safe side.

Move the branch

The git reset command changes what the current branch points to.

$ git reset --hard 1d3158c

At this point the repo will look like this:

$ git tree1 -a 67f8356
* 67f8356 (new-feat) 2024-06-20 jms1(G) fix typo
* 8a837d6 2024-06-20 jms1(G) start new feature
*   1d3158c (HEAD -> main, origin/main) 2024-06-13 jms1(G) Merge branch 'prev-feat'
|\
| * d60b020 (origin/prev-feat) 2024-06-12 jms1(G) previous feature
|/
* 3accd26 2024-05-29 jms1(G) Merge branch 'old-feat'

As you can see ...

  • The main branch now points to the commit that it would have pointed to if we had created the new branch before creating any commits.

  • The new new-feat branch points to the most recent commit in the work you've already done.

Keep Working

At this point, the problem is fixed. You can continue working as if you had created the branch before starting, including pushing the new branch to a remote and creating a pull request.

If you had already pushed

If you had already pushed the "bad" state to another remote (such as a "git server"), you'll need to "force push" the corrected branch to that same remote.

git push -f main

If you force-push a branch in a repo that other people use, you need to let those other people know as soon as possible that you've done so. If they had fetched changes to their workstations during the time that the remote's main was pointing to the wrong commit, their workstation's idea of origin/main (and possibly their local main) will still be pointing to the wrong commit.

In this case, they will need to update their local repo from the primary server, and possibly update their local main branch to point to the updated branch on the remote (aka origin/main), as soon as possible. Ideally they should do this before creating any new branches, or adding any commits to the branch you force-pushed. If they don't, it can cause ... problems.

  • If they have any un-committed changes,

  • Fetch changes from the remote server.

    git fetch origin

  • Force-update their local main branch to point to origin/main.

    git checkout main
git reset --hard origin/main

Multiple Remote URLs

A git "remote" is a logical construct representing another copy of the same repository, which may be updated at different times than the local repo (i.e. the directory on your workstation, with the .git/ directory in it). Each remote has one or more URLs, pointing to the other copy of that repo.

When you first clone a git repo to your machine, it automatically creates a "remote" called origin, pointing to the URL that you cloned the repo from. In most cases this is enough, especially in a typical "git server" scenario, where one machine acts as a central location that everybody else working with the repo, treats as authoritative.

This allows you to use commands like git fetch and git pull to download updates from the same source, and git push to upload changes back to that source.

In some cases, you may want a repo (on your workstation) to have multiple "remotes". There are several possible reasons you may need/want to do this.

There are a few ways to handle this.

Multiple Remotes

Repos (such as the directories on your workstation) can have multiple "remotes". I've done this in the past as a way to migrate repos from one service to another (i.e. from Bitbucket to Github), and occasionally need to do this in cases where the remotes get out of sync with each other.

As an example ...

$ git remote -v
foks    foks://(redacted)/jms1.info (fetch)
foks    foks://(redacted)/jms1.info (push)
github  git@github.com:kg4zow/jms1.info (fetch)
github  git@github.com:kg4zow/jms1.info (push)

With this kind of configuration, you can pull or push commits from either remote, as needed.

For example, to update Github's copy of the repo with any new commits that FOKS knows about, I could do something like this:

git fetch -p foks
git push --all github
git push --tags github

Setting this up

Assuming you've cloned a repo from a remote called origin ...

  • Create an empty repo on the remote server. The mechanics of this will vary based on the remote server. Using Github as an example, you could use their web interface or their command line utility.

    gh repo create --private USERNAME/REPONAME

    ⚠️ Create an EMPTY repo.

    If the repo-creation process offers you any options to set up any README, LICENSE, or other files, do not use them. These options work by starting an empty repo and adding a commit which adds those files. Pushing an existing repo into a non-empty repo like this is ... not impossible, but it's not simple (you would have to "merge" the two).

  • On your local machine, use git remote add to create a new remote.

    git remote add github git@github.com:USERNAME/REPONAME

  • Make sure your local repo is up to date with the current "primary" copy of the repo.

    git fetch -p

  • Many services (like Github) will set a repo's "primary" branch to the first branch pushed into the repo. In order to avoid issues in the future, your first push should contain only the branch that you consider to be "primary".

    First make sure your local directory is up to date with that branch.

    git checkout main
git pull

  • Push the primary branch. Because this is the first branch you're pushing, Github will set it as the primary branch in its copy of the repo (i.e. the branch that gets checked out if somebody clones the repo from Github without specifying a branch name).

    git push github main

  • Push all other branches, and any tags.

    git push --all github
git push --tags github

At this point, the new github remote contains every commit, branch, and tag that the .git/ directory on your workstation knows about.

Going forward, you can manage which commits get pulled or pushed to which remotes by specifying the remote names as part of your git commands. For example, if you push new commits to Github and need to copy them back to the original server, you might do something like this:

git fetch -p github
git checkout main
git merge github/main
git push origin main

Multiple URLS in the Same Remote

Another option is to add multiple "push" URLs to the same remote.

Every git remote has a "fetch" URL, used by operations that read from the remote. Remotes can also have "push" URLs, separate from the "fetch" URL.

Operations that write to the remote will use the "push" URLs if there are any, otherwise they will use the "fetch" URL.

If a remote has multiple push URLs, every time commits are pushed to that remote, they are sent to all of those URLs, one after the other. A single git push command can send updates to multiple remote locations.

I do this with most of my personal repos, in order to have "backup copies" of my repos on other servers, in case their "primary servers" are not available.

Setting this up

As an example, this is how I set up the repo holding the jms1.info site's source code. (These are the actual commands I typed on my workstation when I added Github as a second URL for this repo on 2025-10-17.)

  • Create the Github repo.

    gh repo create --public kg4zow/jms1.info

  • Clone the existing FOKS repo to a new directory.

    cd ~/git/
git clone foks://(redacted)/jms1.info jms1.info

  • Add the same URL as the first push URL.

    When a remote is created, git remote add adds the "fetch" URL. The git push command uses this only when the remote doesn't have any specific "push" URLs.

    If a remote has one or more "push" URLs, the git push command will not use the fetch URL. This means if you want pushes to go to the same server you pull from, you need to add a copy of the fetch URL, as a push URL.

    git remote set-url --add --push foks://(redacted)/jms1.info

    ℹ️ If you don't do this, you'll end up with a situation where "fetch" operations will pull data from one server, while "push" operations will send data to a different server. This would be unusual, but there are occasionally cases where this can be useful.

  • Add the Github URL as a new push URL.

    cd ~/git/jms1.info/
git remote set-url --add --push origin git@github.com:kg4zow/jms1.info

  • Push the contents to Github.

    Note that the first git push is only pushing the main branch. This is so that Github will make main the primary branch in its copy of the repo.

    git push main
git push --all
git push --tags

    You may notice that these commands don't have a "remote" name. This is because the repo (the directory on my workstation) only has one remote, called origin.

    Also note that these git push commands did try to push commits to FOKS. However, nothing was actually pushed because the FOKS server already had the commits and tags I was pushing.

From this point forward, whenever I make changes and push commits, the git push command pushes those changes to both locations, one after the other.

Note that the git push has a -v option which makes it show "Pushing to [URL]" for each URL it pushes to. This can be useful if you're troubleshooting an issue with not being able to push to one or more remote services.

The git-fix-remote-order script

Download

When I first started adding multiple push URLs to remotes, I sometimes ran into problems when adding URLs or changing which URL was the primary "fetch" URL. These were usually caused by my own typos, but sometimes I would forget about a URL, forget to add the fetch URL as a push URL, or add the URLs in the wrong order (making the wrong URL the "fetch" URL).

To clean this up, I found myself having to run the same set of commands every time:

  • git remote -v to see what the current remotes and URLs were
  • git remote remove to remove the current remotes
  • git remote create to create the remote with just the fetch URL
  • git remote set-url --add to add the fetch URL as a push URL, then to add the additional push URLs

And in order to avoid making the same mistakes again, I would type out the commands in a text editor, then double-check them before running anything, and THEN running them (by copy/pasting them from the text editor, to avoid typos).

This got to be rather tedious, so I wrote a script called git-fix-remote-order to take care of this. It uses a config file containing ...

  • Sections for each repo I use on my workstations
  • The URLs which belong in the origin remote, in the correct order, i.e. the first URL is the "fetch" URL and the others are all "push" URLs.
  • Alternate remote names for each URL. (This is explained below.)

As an example, this is what some of the blocks in my config file look like:

# my "cheatsheets", https://github.com/cheat/cheat
=cheatsheets
foks        foks://(redacted)/cheatsheets
internal    git@git.internal:jms1/cheatsheets
github      git@github.com:kg4zow/jms1-cheatsheets

# source for jms1.info site
=jms1.info
foks        foks://(redacted)/jms1.info
internal    git@git.internal:jms1/jms1.info
github      git@github.com:kg4zow/jms1.info

# source for kg4zow.us site
=kg4zow.us
internal    git@git.internal:jms1/kg4zow.us
foks        foks://(redacted)/kg4zow.us

# source for qmail.jms1.net site
=qmail.jms1.net
internal    git@git.internal:jms1/qmail.jms1.net

Using the script

  • Make sure the script is installed somewhere in your PATH.

    I normally have the $HOME/bin/ directory in my PATH, so I copy it there. (Others may use $HOME/.local/bin/ for the same purpose, do whatever works on your system.)

  • cd into the working directory of the repo whose remote URLs need to be updated.

  • Run the script with no options.

    This will examine the directory and the current remotes, and show you what it thinks needs to be done. The cyan-coloured lines are the actual git commands involved. (Commands starting with + are actually executed, while commands with - are just being shown but not executed.)

  • Assuming everything looks right, run the same command but add the -g option.

    This does the same thing, but it acutally runs the commands.

The last command it runs will be git remote -v, which shows you the state of the remotes after the commands were executed. Note that if you run the script without the -g option, this output will be the same as when it ran git remote -v at the beginning, since nothing will have been changed.

ℹ️ Alternate Command

You can also type git fix-remote-order instead of git-fix-remote-order.

This is because if the git command sees a sub-command that it doesn't recognize (like fix-remote-order), it searches your PATH for an executable called git-{subcommand}, and if it finds one, runs it instead.

This mechanism can be used to "create your own git commands", by writing scripts (or compiling executables) and storing them in your PATH with names starting with git-.

Splitting Remotes and Alternate Remote Names

The script works by removing all existing remotes, then adding just the remote(s) which need to exist, with the correct URLs. Normally this means it creates a single remote called origin with the URLs in the order showin the config file.

There are cases where you may need each URL to "be" its own remote. I've needed this as part of the cleanup process when commits get pushed to some remote servers but not others.

The script has a "split" mode where, instead of creating a single origin remote with all of the URLs, it creates a separate remote for each URL. The alternate remote names in the config file are used as the remote names when "splitting" a repo's remotes.

Splitting the repo's URLs into separate remotes can be useful when troubleshooting a repo where one of the remote URLs is "out of sync" with the others. The basic process looks like this:

  • Run git-fix-remote-order -sg to split the URLs into separate remotes.

  • Use whatever commands are needed to fix things by hand.

    I can't really offer more detail than this, because there are too many possible ways that repos can be "out of sync" with each other. However, the following commands might be useful ...

    • git fetch with a remote name
    • git pull with a remote name
    • git push with a remote name, and possibly with other options (including specific local:remote branch names)
    • git log1 --all
      • log1 is one of my aliases. The output will be a single line for each commit, and will include indicators showing where each remote's idea of each branch is pointing.
    • git tree1 --all
    • git commit
    • git merge
    • git reset

  • Once everything is back to its expected state (i.e. all remotes' copies of the branches are pointing to the same commits), run git-fix-remote-order -g to rejoin the remotes.

    If you see a yellow warning saying WARNING: cannot re-link branches, this means that whatever branches were originally linked to upstream branches, are no longer linked. (This kind of link is what makes git push work correctly without any branch or repo names.)

    You can re-create the link using the git branch -u command. For example, to link the local main branch to the origin/main upstream branch ...

    git branch -u origin main

My Configuration

I've been using git for over ten years. Over that time I've found a collection of configuration options that seem to work well for me. I normally configure all of my workstations with these options.

These options includes a collection of "aliases" which really make my life easier.

Configuration Options

Identity

These options set the default name, email, and PGP key used for commits. The values of the options are different on personal and work machines.

  • Personal machines

    git config --global user.name "John Simpson"
git config --global user.email "jms1@jms1.net"
git config --global user.signingkey "0xE3F7F5F76640299C5507FBAA49B9FD3BB4422EBB"

  • For $DAYJOB machines

    git config --global user.name "John Simpson"
git config --global user.email "jms1@domain.xyz"
git config --global user.signingkey "0x1234C0FFEEC0FFEEC0FFEEC0FFEEC0FFEEC0FFEE"

Note that I also have aliases for cases where I might need to sign a commit using my personal "identity", on a $DAYJOB workstation. Because my PGP and SSH keys are stored on Yubikeys, I can just plug the "other" Yubikey into the machine and use the correct alias.

These aliases are documented below.

For all machines

I use these configuration options on every machine.

git config --global core.editor "nano"
git config --global core.excludesfile "$HOME/.gitignore_global"
git config --global credential.helper "cache --timeout=300"
git config --global init.defaultBranch "main"
git config --global clone.defaultBranch main
git config --global log.showSignature true
git config --global push.default "simple"
git config --global pull.rebase false
git config --global gpg.ssh.allowedSignersFile "$HOME/.config/git/allowed_signers"

All of my commits and tags are signed. This is a requirement at $DAYJOB, and a good idea in general.

git config --global commit.gpgsign true
git config --global tag.gpgSign true

Commit message template

This sets up a text file which is used as a template when git uses a text editor to create or edit a commit message.

In particular, I generally use the 50/72 format format when writing commit messages. As you can see below, having the ^ marks at 50 and 72 characters makes it easier for me to stay within the limits.

  • To configure the file:

    git config --global commit.template "$HOME/.stCommitMsg"

  • The contents of the file:

    $ cat ~/.stCommitMsg


#                                             50 ^                  72 ^
# First line: start with ticket number(s), limit to 50 characters
# BLANK LINE
# Additional lines: limit to 72 characters
$ git config --global commit.template "$HOME/.stCommitMsg"

Note that empty lines and lines starting with # are not included in the actual commit message.

Aliases

Aliases allow you to "make up your own git commands". For example, if you were to do this ...

git config --global alias.showfiles "show --name-only"

... then git showfiles would be the same as git show --name-only.

My Usual Aliases

These are the aliases I've built up over the years. Some of these I use dozens of times every day.

git config --global alias.log1 "log --oneline --no-show-signature --abbrev=8 '--pretty=tformat:%C(auto)%h%d %C(brightcyan)%as %C(brightgreen)%al(%G?)%C(reset) %s'"
git config --global alias.tree "log --graph --decorate"
git config --global alias.tree1 "log --date-order --decorate --graph --no-show-signature '--pretty=tformat:%C(auto)%h%d %C(brightcyan)%as %C(brightgreen)%al(%G?)%C(reset) %s'"
git config --global alias.tagdates "log --tags --simplify-by-decoration --pretty=\"format:%ai %d\" --no-show-signature"
git config --global alias.taghashes "log --tags --simplify-by-decoration --pretty=\"format:%H %d\" --no-show-signature"
git config --global alias.id "describe --always --tags --long --abbrev=8 --dirty"
git config --global alias.top "rev-parse --show-toplevel"

Changes for older git versions

Some colours and tags were added between git 2.16.5 and 2.37.0.

  • %as (commit date YYYY-MM-DD) -> %ad with --date=short option
  • %al (author email local part) -> %an (author name)

For older versions without these newer colour codes, I use these aliases instead.

git config --global alias.log1 "log --oneline --no-show-signature --abbrev=8 --date=short '--pretty=tformat:%C(auto)%h%d %C(cyan)%ad %C(green)%an(%G?)%C(reset) %s'"
git config --global alias.tree1 "log --date-order --decorate --graph --no-show-signature --date=short '--pretty=tformat:%C(auto)%h%d %C(cyan)%ad %C(green)%an(%G?)%C(reset) %s'"

Sign commits using specific keys

These aliases allow me to sign commits using my personal PGP key on the work machine, or vice-versa, by physically plugging the correct Yubikey into the machine. By themselves they won't be very useful to anybody else, but they could be useful as examples if you need to use different keys for different repos.

git config --global alias.commitp "commit --gpg-sign=E3F7F5F76640299C5507FBAA49B9FD3BB4422EBB --author='John Simpson <jms1@jms1.net>'"
git config --global alias.commitw "commit --gpg-sign=1234C0FFEEC0FFEEC0FFEEC0FFEEC0FFEEC0FFEE --author='John Simpson <jms1@domain.xyz>'"

Notes

Random notes relating to git

Commits signed with SSH keys

There used to be a brief explanation here, I've moved this to its own page.

⇒ Signing commits with SSH keys

Configuration scope

The git config command operates on different files, depending on which options you give it.

OptionFileScope
--local (or none)REPO_ROOT/.git/configthe current repo
--global$HOME/.gitconfigthe current user
--system/usr/local/etc/gitconfigall users on the system
--worktreeWORKTREE_ROOT/.git/config.worktree
or REPO_ROOT/.git/config		the current "worktree"
or --local if no worktree is active
--file ___specifieddepends on the file

Changelog

2025-04-30 jms1

  • Fixed a typo
  • Removed info about git commits signed by SSH keys, since there's now a dedicated page for that

2024-06-20 jms1

  • Created this page (from pre-existing notes)

Signing commits with SSH keys

Newer versions of git support the ability to sign commits using an SSH key.

Specifically ...

  • Git v2.34 or higher
  • OpenSSH 8.0 or higher

🛑 Don't Do This

I personally think that using SSH keys to sign commits is a bad idea.

The main reason I say this is because SSH keys are not linked to specific identities. An SSH key pair is JUST a pair of numbers - there's nothing about an SSH key which says "I belong to John Simpson". If I don't tell you which SSH key I actually use, you have no way to know whether a given key belongs to me or not.

Git allows users to set their own name and email address when creating commits. This means that anybody can create a commit which claims to be made by any name and email address. Somebody could use your name and email address on a commit, and sign the commit using an SSH key that they created on their own. It wouldn't be the correct key (i.e. the key that you normally use), but it would be signed, and the commit would have your name and address as the author ... so if somebody was looking at the commit but not verifying the signature, there's a good chance this would be enough to fool them.

If a command like git show is looking at a commit which was signed using an SSH key, all it can really tell for sure is which SSH key signed it. It doesn't know who that key actually belongs to, unless you tell it which keys belong to which people. (I explain how to do this below.)

This is why you need to verify the signatures on commits - to be sure that the person who the commit says is the author, actually is the author.

Verifying commits signed by SSH keys involves some extra configuration which isn't required for commits signed by PGP keys.

Another reason for not using SSH keys to sign commits is that, if the secret key is compromised, there is no way to revoke the key. If your secret key escapes, YOU have to hunt down every person who has a copy of your public key and tell them to stop using it.

Again, I personally don't use SSH keys to sign commits, but I do feel like people should be allowed to make their own decisions. As long as you understand the risks, if you want to do it, more power to you.

Overview

There are two basic operations involved with signed commits, no matter what key type is used: signing commits, and verifying signatures.

Git needs to be configured for both of these.

Signing Commits

There are a few things to configure in order to make git sign your commits.

Sign one commit

Creating a signed commit is done by adding the -S (uppercase) or --gpg-sign option to your git commit command line, like so:

git commit -S -m 'commit message'

Note that only that one commit will be signed.

Tell git to sign commits by default

You can also configure git to sign commits by default, using this command:

git config --global commit.gpgsign true

After doing this, every git commit command will create a signed commit, without your having to include the -S option in the command.

ℹ️ Create an un-signed commit

If you do this and need to occasionally not sign a commit, you can add the --no-gpg-sign option to the git commit command, and that one commit will be created without a signature.

Documentation for the commit.gpgsign option

Tell git to use an SSH key when signing commits

Git normally uses a PGP key to sign commits. Setting this config option tells it to use an SSH key instead.

git config --global gpg.format ssh

Documentation for the gpg.format option

Tell git which SSH key to use

We can do this in one of two ways:

  • Use the filename to the public key file:

    git config --global user.signingKey $HOME/.ssh/id_rsa.pub

  • Include the public key itself in git's configuration:

    git config --global user.signingKey 'key::ssh-ed25519 AAAA... comment'

Documentation for the user.signingKey option

Verifying Commits

Before git will be able to verify commits, it first needs to be told which SSH keys belong to which people. We do this by creating an "allowed signers" file, which is a simple text file containing email addresses and SSH public keys.

Create an Allowed Signers file

The file can be created wherever you like, as long as it's readable by the git command itself. The example below will use $HOME/.git_allowed_signers as the filename, but you can use whatever filename you like.

Each line in the file should contain ...

  • One or more email addresses. If multiple email addresses use the same key, the addresses should be separated by commas.
  • An SSH public key, optionally including the same coments you normally see after the key itself.

For example ...

jms1@example.com ssh-ed25519 AAAAC3NzxxxxxxxxxxxxxxxxxxxxxxxxxRK4m 2025-04-29 fake key

Documentation for the Allowed Signers file format is in the ssh-keygen(1) man page. If your system isn't set up to view man pages, this web page has a copy, however OpenSSH 8.0 was the first version to use an Allowed Signers file and I'm not sure what version of OpenSSH this web page was taken from.

ℹ️ Trust the man pages

You should always trust the documentation in your system's man pages over anything you find online. This is because the man pages are installed with the software itself, so they're guaranteed to be the same versions.

This is especially true of programs like OpenSSH, where new options are being added with every new version.

Tell git where to find the Allowed Signers file

This is just another git config command.

git config --global gpg.ssh.allowedSignersFile $HOME/.git_allowed_signers

Documentation for the gpg.ssh.allowedSignersFile option

Tell git to always show signatures

This is optional. I do it as part of the git configuration on all of my machines.

This will make every git log or git show command use the --show-signature option by default, so you won't have to remember to include it in every command.

git config --global log.showSignature true

Messages

You may run into the following messages when working with SSH-signed commits.

Good "git" signature

Good "git" signature for user@example.com with RSA key SHA256:SHA256:oSAqmDr7/LKI+STyBrT79IcFmbyt3h1P1niXdg0I+94

✅ This is good. It means that the commit was signed by an SSH key which is in your Allowed Signers file.

This message is actually telling you three things:

  • Good "git" signature tells you that the commit's contents and message haven't been modified since the signature was created.

  • for user@example.com tells you whose SSH key signed the message.

  • with RSA key ... gives you the fingerprint of the key which signed the commit.

No principal matched.

Good "git" signature with RSA key SHA256:oSAqmDr7/LKI+STyBrT79IcFmbyt3h1P1niXdg0I+94
No principal matched.

❓ This means that the commit was signed by an SSH key that isn't recognized (i.e. isn't in your Allowed Signers file).

This mesage is actually telling you three things:

  • Good "git" signature tells you that the commit's contents and message haven't been modified since the signature was created.

  • with RSA key ... gives you the fingerprint of the key which signed the commit.

  • No principal matched means that it can't tell you who signed the commit.

If you're able to contact the commit's author and get a copy of their SSH public key, you can get its fingerprint using ssh-keygen -l, like so:

$ ssh-keygen -l -f their_id_rsa.pub
4096 SHA256:oSAqmDr7/LKI+STyBrT79IcFmbyt3h1P1niXdg0I+94 user@example.com (RSA)

As you can see, the fingerprint printed by this command matches the fingerprint shown by the git show or git log command, so you can be sure that this commit was signed by that person.

gpg.ssh.allowedSignersFile needs to be configured and exist

❓ If you work with repos containing commits signed by SSH keys, and git is showing signatures, you may see this message when looking at commits that were signed by SSH keys:

error: gpg.ssh.allowedSignersFile needs to be configured and exist for ssh signature verification

The message means exactly what it says: either you haven't configured the gpg.ssh.allowedSignersFile option, or the file that it points to doesn't exist.

If you don't want to see this message, there are a few options:

  • Use the --no-show-signature option in your command.

    git show --no-show-signature a1b2c3d4

    This option will override the log.showSignature config option.

  • You can configure git to use an Allowed Signers file, but leave the file empty.

    > $HOME/.git_allowed_signers
git config --global gpg.ssh.allowedSignersFile $HOME/.git_allowed_signers

    Note that the first command starts with >. This will create the file if it doesn't already exist, and empty the file if it does.

    ℹ️ Shell Output Redirect Operator

    This is the same > operator that you would use to send the output of a command to a text file. In this case we're not running a command, so we're sending nothing to the file.

  • You can configure git to use /dev/null as the Allowed Signers file.

    git config --global gpg.ssh.allowedSignersFile /dev/null

    This is just like setting up an Allowed Signers file and leaving it empty, except that you don't have to create the file at all. The /dev/null file always exists on macOS and Linux systems, and if anything (including git) tries to read data from it, it will act like any other empty file.

Changelog

2025-04-30 jms1

  • Created page

Keybase

Keybase is a system which provides end-to-end encrypted services, including ...

  • Text chat, between groups of specific people, and within "teams".
  • Cloud file storage, with the files accessible to yourself, a specific set of people, or a team.
  • Git repositories, accessible to yourself or to a team.
  • Encrypting, decrypting, signing, and verifying encrypted messages which can be sent by some other method (such as email, or a "corporate approved and monitored" chat system).

Keybase also provides a way to prove your identity on some other service, and to allow others to find you based on those identities. These services include Github, Reddit, and Hackernews. You can also "prove" that you own specific DNS domains and web sites.

You can also attach PGP keys to your Keybase account. This was actually Keybase's original application, as a way to find other peoples' PGP keys when all you knew them as, was a username on a site like Reddit.

For example, if you only know somebody as "kg4zow on Github", if a Keybase user has proven that they own that Github user (hint: I did), you can use Keybase to chat or share files with them, secure in the knowledge that the person you are communicating with on Keybase is the same person as "kg4zow on Github".

KBFS: Cloud File Storage

KBFS, or Keybase Filesystem, is a cloud file storage system. Keybase provides 250 GB of encrypted cloud storage for each user, as well as 100 GB for each team. This storage can be accessed by any Keybase client which is registered as a device on a user's account.

Each user also has a public directory available, whose contents can be viewed by any other Keybase user. For example, if you're logged into Keybase you can look in /keybase/public/jms1/ to see the files that I'm sharing with the world.

FUSE and Redirector

For Linux, macOS, and ms-windows systems, Keybase provides a way to "mount" KBFS so it appears as part of the machine's filesystem. The details are different for each operating system, but Linux and macOS both use a FUSE (Filesystem in USErspace) module to translate "file accesses" to the appropriate API calls needed to upload and download encrypted blocks from Keybase's cloud servers.

It's possible for multiple people to be logged into a computer at the same time, so Keybase needs to ensure that different users on the same machine can't see each others' Keybase files. The mechanics of how this happens are different for each operating sytem.

I don't want to go into a lot of technical detail, so the short version is this:

  • Each user on a computer has their own "view" of KBFS, mounted in a different directory.

  • KBFS uses a thing called a "redirector", which redirects file accesses to the user-specific mount directory for whatever user is accessing it.

    • On Linux, the redirector is mounted as /keybase.

    • On macOS, the redirector is mounted as /Volumes/Keybase. Some systems may also have /keybase as a symbolic link pointing to /Volumes/Keybase.

The idea is, all users on the system can use paths starting with /keybase/, and they will see their own "version" of KBFS, containing the files that they have access to.

Because of this, the normal way to write the names of files stored in KBFS is using paths starting with /keybase/.

ms-windows

You will note that I didn't mention ms-windows at all. This is because I haven't used ms-windows since the days of "windows 7", and I don't remember the details of how KBFS works on windows.

I have a vague memory of there being a third-party program which needs to be installed - a quick web search tells me that what I'm thinking of is probably Dokan. I don't remember if this is distributed with the Keybase installer, or if you have to download and install it yourself.

KBFS Directories

KBFS has three high-level categories of directories: public, private, and team. Under these categories, folders "exist" whose name tell who have access to them.

Public

  • /keybase/public/alice/ is readable by anybody, but only writable by Alice.

  • /keybase/public/alice,bob/ is readable by anybody, but only Alice and Bob are able to write to it. (This is not something you see a whole lot, but it works if you have a need for it.)

Private

  • /keybase/private/alice/ is only accessible by Alice (or technically, by devices on Alice's account).

  • /keybase/private/alice,bob/ is accessible to both Alice and Bob.

  • /keybase/private/alice,bob#charlie,david/ is accessible to Alice, Bob, Charlie, and David.

    • Alice and Bob (before the #) are able to read and write files.
    • Charlie and David (after the #) are able to read the files but not write them.

    As you can see, it's possible to create private folders where different people have different access. However, once that folder exists, the list of who has what access can never change. If you need to remove somebody's access, or change them from read-only to read-write, your only option is to create an entirely new folder whose name is the new list of who has what access, and move the files from one to the other. The old one will still "exist", it'll just be empty.

    Keybase added "Teams" as a way to deal with this problem. Users can be added or removed from a team, or have their roles changed, without needing to change any team or directory names.

Team

  • /keybase/team/xyzzy/ is accessible by Keybase users who are members of the xyzzy team. Each user's role within the team controls what access they have to the files in the team's folder.

  • /keybase/team/xyzzy.dev/ is accessible by Keybase users who are members of the xyzzy.dev team. This is a "sub-team" of the xyzzy team. (Sub-teams are explained below.)

ℹ️ The user and team names shown above are all examples. I don't know if there are users or teams with those names.

Teams are explained in more detail below.

Space

I mentioned this above, but to make it more obvious ...

  • Each user is given 250 GB of storage for free.
  • Each team is given 100 GB of storage for free.
  • There is currently no limit to the number of teams which can be created.

The one restriction is, teams cannot have the same name as a user. This means that, because I already have the username jms1, I could not also create a team called jms1.

Teams

"Teams" are groups of Keybase users. Users can be added to or removed from teams dynamically.

This is different than a "group of users" situation. A "group chat" between Alice, Bob, and Charlie will only ever contain those people. If you try to add a fourth person, it creates a new group chat between those four people. The original three-way chat will still exist, and the fourth person will never be able to access it.

  • When users are added to a team, they will have access to the team's chat history, shared files, and git repos.

  • When users are removed from a team, they will immediately no longer have access to the team's chat history, shared files, or git repos. (If they previously saved anything they will still have access to their own copies, but they won't be able to access

Roles

Users who are added to a team will be able to see the team's chat history, shared files, and git repositories, subject to their "role" within the team.

Available roles are:

  • reader = can participate in team chat rooms, has read-only access to the team's KBFS folders and git repositories.

  • writer = same as reader, but has read-write access to the team's KBFS folders and git repositories.

  • admin = same as writer, but can add or remove team members and set their roles, up to admin. Can also create or delete sub-teams "below" this team (so if somebody is an admin for the team xyzzy.dev, they could create an xyzzy.dev.ios sub-team).

  • owner = Can create or delete sub-teams anywhere below the top-level team, as well as add, remove, and set the role for any user in any sub-team.

Users who are an admin or owner of a team do not automatically have access to its sub-teams' chats or files. They do, however, have the ability to add themselves to the sub-team. This is referred to as "implied admin" permission. (Note that if they do this, the other team members will be able to see that the admin/owner is now a member of the team - there's no way they could give themselves access without it being visible.)

Team admins and owners can set a minimum role needed to write in the team's chat. This is normally set to reader, but can be set to writer or admin if there's a need to have people who can read the team chat but not be able to "speak" in it (i.e. an "announce-only" channel).

Sub-Teams

Teams can have "sub-teams". For example, the xyzzy top-level team might have sub-teams called xyzzy.dev, xyzzy.qa, and xyzzy.sales. Each sub-team has its own list of members, with their own roles for that team.

Keybase Sites

Keybase Sites provides simple web hosting for sites containing static files.

Keybase originally had a web site using the keybase.pub domain, where every user's /keybase/public/xxx/ directory could be viewed. This service was taken down in ... I want to say 2023-02?

They also have a service which can host static pages stored in almost any Keybase directory, using a custom domain name that you own. This is how I'm hosting the jms1.info site (where you're presumably reading this right now).

The documentation is a bit outdated. You can ignore anything that mentions the keybase.pub domain, but the "Custom domains" section still works exactly as described.

My Experience

I've been using keybase since 2017. I've had very few problems with it, and the problems I have had were mostly related to Apple making low-level changes to macOS, and Keybase/Zoom not using the beta versions to test the client before the new macOS is released to the public.

One thing I did find interesting ... when Apple first released computers with the "Apple Silicon" processors, I had an M1 MacBook Air. The Keybase app hadn't been updated to support it yet, and at the time nobody at Keybase had an M1 machine to try it with. One of the Keybase devs sent me what he thought should be a working client, and I was able to test it for them and send back some log files. Keybase released the first client which supported the M1 processor about a week later.

Your Keybase Account

Coming soon.

Keybase on macOS

Coming soon.

Before You Lose Access

Hopefully you're reading this page BEFORE disaster strikes.

I've seen a lot of cases where people create a Keybase account and then lose access to it, because they didn't read the documentation, or they were in a hurry and skipped over steps, or in a few cases, because they created their accounts long enough ago that some of the warnings that the current client shows, didn't exist at the time.

This page will explain a few things that you should do, while you have access to your account, so that if something goes wrong you won't lose access to everything stored in your account.

Background

Multiple Devices

🛑 MAKE SURE YOUR KEYBASE ACCOUNT HAS MULTIPLE DEVICES ON IT. 🛑

This is the most important thing on this entire page.

Keybase encrypts things (chat messages, files, git commits, etc.) using encryption keys which are specific to each device. These keys are stored on each device, and are never sent to any Keybase server.

If you lose the encryption keys for every device on your account, you will lose access to everything stored in the account. This includes ...

  • Chat history
  • Files stored in KBFS
  • Git repositories

If these things are accessible by other people or teams, those other people will still have access, but you won't.

I know I said it above, but I'll say it again.

🛑 MAKE SURE YOUR KEYBASE ACCOUNT HAS MULTIPLE DEVICES ON IT. 🛑

Adding devices to your account

Keybase has clients for Linux, macOS, ms-windows, Android, and iOS. Their web site has directions for how to download and install the software, as well as how to add the device to your existing account.

If you don't physically have a second device that you can install Keybase on, you can create a "paper key".

In fact, even if you have a dozen devices with Keybase installed, you should create a paper key.

If you don't know what devices are on your account, check the "Devices" tab in your Keybase app, or visit https://keybase.io/___/devices (substitute your username where you see ___ in the URL).

Paper Keys

A paper key is a sequence of 13 words which encode a device encryption key. This key is attached to your Keybase account like a normal device key.

They are called "paper keys" because you're supposed to physically write them down on paper, and lock the paper up someplace safe.

Obviously "safe" means that other people shouldn't be able to access it, but you should also consider physical safety. If it's locked up at home, what happens if your house catches fire, or floods, or if an earthquake destroys it?

As an example, the paper keys for my own Keybase accounts are ...

  • Written down on paper and stored in a fire safe at home. The paper itself has nothing on it but a collection of random words, so if somebody manages to break into the safe, they won't immediately know what the words are for - all they'll see is a collection of random words.
  • In a text file, stored on an encrypted USB stick, also stored in the fire safe at home.
  • On another encrypted USB stick, physically stored with a family member in a different part of the world.

This means if something happens to my house, up to and including permanent destruction, I can get the backup copies of the paperkeys from this family member. It might take a few days, but I wouldn't be permanently locked out of my accounts.

Resetting Your Account

The Keybase web site offers a way to "reset" your account. They do warn about this being a drastic action, but I don't feel like they make it "scary" enough.

🛑 Resetting your account starts a new account with the same username.

If you do this, you will permanently lose accesss to the content stored in the old account.

Even if you later find one of the old devices, it won't be able to log into your Keybase account anymore.

I explained above that if you lose the encryption keys for every device on your account, you lose access to everything stored in the account. When you reset your account, you are deleting the account entirely, and starting a new account with the same username. Other than the username, there is no connection between the old account and the new one.

You will also lose your memberships in any groups you may be part of. This also means that if your account was the only "owner" of any teams, those teams will now have no owner at all - which means they cannot be fully managed (and if there are also no users with the "admin" role, they cannot be managed at all).

The only time you should ever reset your account is if you are 110% sure that you will NEVER be able to regain access to the devices on the old account. If there is even a remote chance of regaining access to any of your old devices, I recommend starting a new account with a different username.

Lockdown Mode

It goes without saying that you should use a strong password for your Keybase account, and it should be a password that you aren't using for anything else.

BUT.

If somebody manages to get the password for your Keybase account, they could log into the web site as you and reset your account. Doing this wouldn't give them access to your stored information, but it would prevent YOU from being able to access it. (This is a form of "denial of service attack".)

🛑 There is no notification when an account is reset. If somebody manages to reset your account, you wouldn't know about it until you discover that you can't access your Keybase account anymore - and by then it would be too late to do anything about it.

This is not something that Keybase employees would be able to help you with. If your account is reset, whether you do it or an attacker does it, everything encrypted with the old account's device keys will be gone.

In the Keybase client, under Settings → Advanced, there is an "Enable account lockdown mode" setting. If this checkbox is turned on, Keybase will only allow the account to be reset or deleted from a logged-in Keybase device. If an attacker has your Keybase password and logs into the web site as you, the only things they could do would be to send invitations or change your notification settings.

Of course, if your account is in Lockdown Mode and you lose all of the devices, the account cannot be recovered or deleted. This means that you wouldn't be able to re-use the same username.

This makes it even more important that you not lose all of your devices, and that you have a paperkey.

⇒ This page has more details about Lockdown Mode.

ℹ️ All of my Keybase accounts have "Lockdown mode" turned on.

I'm okay with this, because I have paperkeys stored securely.

Checklist

  • Make sure your account has multiple devices attached to it.

  • Create a paperkey, write it down, and store it securely.

  • Check your devices every so often. (I check mine every few months.)

    • Check the list of devices. You can see this in the Keybase client, or by visiting https://keybase.io/USERNAME/devices (substitute your own Keybase username for USERNAME, obviously). Make sure that the devices you think are on the account, are actually there. Also make sure that your account doesn't have any devices which shouldn't be there.

    • For phones, tablets, or computers that you may not use every day, make sure their software is up to date (especially the Keybase client itself), and that they are able to log into the account.

  • Enable "Lockdown Mode" on your account, but ONLY AFTER making sure you have multiple devices and a paperkey.

Lost Access

If you have lost all devices which are on your keybase account, and you don't have an active paperkey on the account (or if you've lost the paperkey) ... I'm sorry, but there is no way to recover any content stored in your account.

Resetting Your Account

There is a way to "reset" your account. This works by starting a new "signature chain" (or "sigchain") for the username. Doing this replaces the old sigchain, which means that any encryption keys currently attached to the account, will no longer be attached to the account.

ℹ️ Signature Chains

Each Keybase account has a chain of digital signatures. Each signature represents a statement by an existing device, authorizing a change to the list of encryption keys attached to the account. This happens when you add or remove devices, paperkeys, PGP keys, or identity proofs.

Signature chains are public. You can see any Keybase account's signature chain by visiting https://keybase.io/USERNAME/sigchain. For example, this is the sigchain on my own Keybase account.

Resetting your account should only be considered as a final resort. If you do this, any content that was previously stored in your account will be permanently lost. This includes ...

  • Teams: You will be removed from any teams that you may be a member of.

    • If you were the only person with owner access to a team, resetting the account will leave the team without an owner. Other members who have admin access to the team will be able to add you back to the team as a member or as an admin, but there is no way for anybody to make you (or anybody else) an owner.

    • If you are the only person with owner or admin access to a team, resetting your account will leave that team without any way to manage members at all. Current members will remain members, but there won't be any way for anybody to add or remove members going forward.

    • Because of this, it's important to be sure that every team has multiple owners, even if the second owner is a "backup account" with a paperkey stored somewhere safe ... just like you should be doing with your primary account.

  • Files: You will no longer have access to any encrypted files, other than other users' public files.

    • Files under /keybase/private/USERNAME/ and /keybase/public/USERNAME/ will be lost.

    • Files under /keybase/team/ will still exist as part of that team, however your new account will not have access to them unless a team owner/admin adds you back to the team.

  • Git repositories: You will no longer have access to any encrypted git repos that your "old" account had.

    • Repos under keybase://private/USERNAME/ will be lost.

    • Repos under keybase://team/ will still exist, but your "new" account won't have access to them unless a team owner/admin adds you back to the team.

    Note that if you have an up-to-date clone of a Keybase repo on a computer somewhere, that could be used to create a new repo with the same commit history. However, computers need to be devices on your account in order to access Keybase git repos in the first place, so if you do have a cloned copy, chances are the computer involved is a device on your account, and you shouldn't need to reset the account.

  • Chat: You will no longer have access to any chat history.

    • For person-to-person (non-team) chats, the other people involved will still have access to the history, however the conversation will essentially be over. If you start a new conversation with them (or they try to chat with you) after you reset your account, it will be a new conversation.

    • For team chats, if your "new" account is added to the team again, you would regain access to the existing history.

If you reset your account and later find one of your devices or paperkeys, it will no longer be able to log into your account.

Essentially, the only time you should consider resetting your account is if you are 100% sure that you will never regain access to a device or paperkey which has access to the account. If there is any chance of finding one of these, you're better of creating a new Keybase account with a different username.

If your account is in Lockdown mode, the account cannot be reset except from an active device. And if you have an active device, you shouldn't need to reset the account in the first place.

I do this for my own Keybase accounts, but I also have paperkeys for each account which are physically written down and stored in safe places.

How to Reset Your Account

After reading everything above, if you're 100% sure you want to reset your account ...

  • Visit keybase.io
  • Use the "Login" link at the top right to log into your account. You will need the username and password.
  • Near the top right of your profile will be a "gear" button.

In the Meantime

After reading everything above, it should be obvious that you need more than just one device on your account, so you will be able to regain access to your account in case you lose all of your devices.

Assuming you still have access to your Keybase account ...

  • Read through the Before You Lose Access page.
  • If possible, make sure there are multiple devices on your account.
  • If you don't have more than one device, or if you want to be sure you have a way to regain access in case all of your devices are lost, make sure you have a paper key on your account.

Keybase Sites

Keybase Sites is a service built into Keybase which hosts web sites from KBFS directories. It can be used to host sites containing only static files - there is no provision for running any kind of server-side scripting.

I'm using it to host the jms1.info site you're reading right now.

Original write-up

https://jms1.pub/kbsites/ is the first set of information I wrote about Keybase Sites, and includes links to five different hostnames which illustrate how to host sites using five different kinds of sources:

  • /keybase/public/xxx/
  • /keybase/private/user#kbpbot/ (private, with kbpbot having read-only access)
  • /keybase/team/xxx/ (with the kbpbot user as a team member with "reader" access)
  • git repo keybase://team/xxx/repo, in master branch (with the kbpbot user as a team member with "reader" access)
  • git repo keybase://team/xxx/repo, in some other branch (with the kbpbot user as a team member with "reader" access)

jms1.info

Details about how I set up Keybase Sites for the jms1.info site. The site is hosted from a team directory with kbpbot as a team member with "reader" access.

Keybase Team

Create a Keybase team with the kbpbot user as a member with reader permissions. This can be either a top-level team, or a sub-team. (I'm using a sub-team.)

keybase team create jms1team.sites
keybase team add-member -s -u kbpbot -r reader jms1team.sites

Create a KBFS directory to hold the site's files. In my case, I'm using the same team to host multiple sites, so each site has its own directory within the team.

mkdir /keybase/team/jms1team.sites/jms1.info

Create a dummy index page so you can test that the site is working.

cat > /keybase/team/jms1team.sites/jms1.info/index.html <<EOF
<html>
<head>
<title>jms1.info</title>
</head>
<body>
<p>test 1 2 3</p>
</body>
</html>
EOF

DNS Records

I use djbdns to serve my DNS data. Below are the actual tinydns data lines from my DNS data, along with what I think are the equivalent records in BIND format.

  • For jms1.info ...

    +jms1.info:18.214.166.21:3600
'_keybase_pages.jms1.info:kbp=/keybase/team/jms1team.sites/jms1.info:3600

    jms1.info.                 3600  IN A    18.214.166.21
_keybase_pages.jms1.info.  3600  IN TXT  "kbp=/keybase/team/jms1team.sites/jms1.info"

    • The record for jms1.info. itself must be an A record, since the rules for CNAME records explicitly say that a resource record name which is a CNAME record, cannot also exist as any other kind of record. Since this record is the "root" of a domain, it must have an SOA record and one or more NS records, therefore it cannot also be a CNAME record.

      The IPv4 address, 18.214.166.21, is the IP that kbp.keybaseapi.com. points to. This hasn't changed in a few years now, however if it does change in the future, I'll need to remember to update my own DNS records to match it.

    • The TXT record for _keybase_pages.jms1.info. is what tells Keybase Sites where to find the site's content. Because it points to a team folder, the kbpbot user needs to have at least "reader" permissions in that team.

  • For www.jms1.info (which does exist as a separate site, containing a meta-redirect to https://jms1.info/ without the www. in the hostname) ...

    Cwww.jms1.info:kbp.keybaseapi.com:3600
'_keybase_pages.www.jms1.info:kbp=/keybase/team/jms1team.sites/www.jms1.info:3600

    www.jms1.info.                 3600  IN CNAME  kbp.keybaseapi.com
_keybase_pages.www.jms1.info.  3600  IN TXT    "kbp=/keybase/team/jms1team.sites/www.jms1.info"

    • The record for www.jms1.info. is a CNAME record, pointing to kbp.keybaseapi.com. This makes the www.jms1.info name resolve to the same IP address(es) that kbp.keybaseapi.com points to, without having to worry about manually updating them if kbp.keybaseapi.com ever changes.

    • The TXT record for _keybase_pages.jms1.info. is what tells Keybase Sites where to find the site's content. Because it points to a team folder, the kbpbot user needs to have at least "reader" permissions in that team.

Linux

I built my first Linux machine in 1992-11. Since then I've been using Linux on a regular basis, including almost every day since 1995-09 when I started working at my first ISP.

The pages in this section have to do with Linux, mostly system and network administration stuff.

CentOS 7 - Fix yum repos

2024-07-06

I have a fair number of scripts which run inside of containers. I do this for several reasons, including:

  • I can run things which require Linux, on my macOS workstations, using colima.

  • I can run scripts or programs which have complicated dependencies, without having to mess with installing those dependencies on every machine where I want to run the program. Everything is already installed within the container.

Many of these use the centos:7.9.2009 container (aka centos:7), because at the time I wrote them, I was using CentOS 7 on a day-to-day basis at work, both at work and personally. One of the reasons I do this at work is to build RPM packages for CentOS and RHEL systems, on my macOS machines.

CentOS 7 officially went end-of-life on 2024-06-30. At work they're paying for an extended-lifetime support contract with Red Hat, who provides us with access to a set of yum repos whose packages receive security and bug-fix updates for RHEL 7. These updated packages can only be used for RHEL 7 machines (not CentOS 7), and they can only be used for work-related machines for which the company pays a license. (It's not my money.)

We've spent the last few months replacing our CentOS 7 machines with RHEL 7 because of this. (And because somebody "higher up" heard a Red Hat employee say that convert2rhel leaves CentOS artifacts on the converted system, took that to mean it doesn't work, and ordered us not to use it, so we had to build all new VMs and migrate their programs and data by hand ... but that's a different discussion.)

vault.centos.org

CentOS has a server called vault.centos.org which contains copies of the CentOS yum repositories for retired CentOS versions, going back to CentOS 2.1.

When CentOS 7 went EOL, its packages were added to the vault as well, the mirrorlist.centos.org servers (which handled automatically redirecting yum clients to a working mirror) were powered off, and the hostname removed from DNS. And while the mirror.centos.org mirror servers are still running, they use different directory names and don't contain any RPMs.

This means that servers which are still using CentOS 7, as well as containers started from the centos:7.9.2009 container image, need to be re-configured to use vault.centos.org.

⚠️ This should be a temporary measure.

The RPM packages in vault.centos.org will NEVER be updated, even for security fixes. This might be okay for containers which are never accessible from the outside world, however servers should be upgraded or migrated to a different OS which does receive security updates.

Update repo files

The sed commands updates the original /etc/yum.repos.d/Centos-*.repo files to use the vault.centos.org servers instead.

sed -i -e '/^mirrorlist/d;/^#baseurl=/{s,^#,,;s,/mirror,/vault,;}' /etc/yum.repos.d/CentOS*.repo

I've started adding this command to scripts which run inside of containers built from the centos:7.9.2009 image. And for custom container images which use centos:7.9.2009 as the starting point for custom containers, I've updated their Dockerfiles like so:

FROM    centos:7.9.2009
RUN     sed -i -e '/^mirrorlist/d;/^#baseurl=/{s,^#,,;s,/mirror,/vault,;}' /etc/yum.repos.d/CentOS*.repo
RUN     yum -y update && yum -y install xxx yyy zzz && yum clean all
...

Changelog

2024-07-06 jms1

  • wrote this page from an Obsidian note which had just the sed one-liner

Using colima to run x86 Containers on ARM

2024-07-29

I started off using Docker Desktop to run containers on my macOS workstation, both at work and for personal projects. For a while it was actually pretty cool.

However, Docker (the company) changed. They started collecting detailed usage information from the software, they wouldn't let you use the software without logging into a "Docker account" (which lets them correlate the usage information with a specific person), and then they changed their licensing and started demanding that commercial users to pay for it - with a ridiculously high pricetag at the time ($25/mo per user? really?)

So I started looking for alternatives, and a colleague pointed me to colima. This is an open source program which combines Lima (LInux MAchines, which runs Linux VMs on macOS) and a container runtime (Docker, Podman, or containerd). I've been using this at work, and for personal projects, for a few years now.

Rosetta 2

One of my home machines is a MacBook Air with an Apple M2 processor, and now that Apple no longer sells Intel machines, it looks like any future Apple machines I buy will also use "Apple Silicon" (aka "ARM") processors as well. Being a different processor architecture, it has a totally different instruction set, and therefore cannot run x86 code by itself.

When Apple switched from PowerPC to Intel processors, they also released a program called Rosetta, which allowed PowerPC executables to run on Intel-based machines. When they released the first M1 machine, they also released Rosetta 2, which translates x86_64 code into ARM code, either "on the fly" while a process is running, or "ahead of time" the first time you run a program.

Colima

Colima on macOS works by creating Linux VMs with Docker or Podman running in them, and passing any docker or podman commands to that VM.

It can use one of two methods to create these Linux VMs:

  • QEMU is an open-source software-based virtualization framework which has been around for 20+ years. This is the default technology used by KVM, which I've been using for many years to run Linux VMs on Linux hosts.

    QEMU has the ability to create VMs running a wide range of CPU architectures, on a range of host operating systems and architectures. For the purposes of this page, this includes being able to run x86_64 (64-bit Intel) VMs on an aarch64 (64-bit ARM, aka "Apple Silicon") host.

    Colima uses QEMU by default.

  • VZ is Apple's native virtualization framework in macOS 11 and later. The framework itself is built into macOS, but Apple doesn't offer any kind of user interface to manage VMs, just an API for other programs to use. If you're interested, there are programs out there to create VMs using VZ, such as UTM and VirtualBuddy.

    Colima can use VZ under macOS 13 and later. On earlier macOS versions, colima will only use QEMU.

With colima on macOS, the container runtime is a process running within the Linux VM that colima creates. This means that the images you pull are actually stored within that VM, and the containers you run are running on that VM.

Note that colima can run multiple Linux VMs at the same time.

Creating Colima VMs

The colima start command will create a new Linux VM, if one doesn't already exist.

Each Linux VM is identified using a "profile" name. If you create a Linux VM without giving it a profile name, it will use the name default. If you have a profile called default, other colima commands will use it unless you include a --profile option in those commands.

For me, 99% of what I use containers for is to run x86_64 containers, so my default profile is an "x86_64 using VZ and Rosetta 2" VM.

The commands listed below only cover the options needed to set the virtualization runtime (QEMU or VM) and CPU architecture of the VM (aarch64 or x86_64). Other options, such as CPU count, RAM, and disk size, are not shown, but should be added to these commands if needed.

On an Intel Mac

  • Intel (x86_64) VM using QEMU

    colima start --profile qemu_x86_64 \
    --cpu-type max

    • The --cpu-type max option tells QEMU to mirror the x86 CPU capabilities of the underlying host. Without this, the virtualized CPU won't be able to run AlmaLinux/RHEL 9.

    • You can also use the --arch x86_64 option, but it isn't necessary since it will be the default on an Intel-based Mac.

  • Intel (x86_64) VM using VZ (❓ not tested yet)

    colima start --profile vz_x86_64 \
    --cpu-type max \
    --vm-type vz \

    • The --cpu-type max option tells VZ to mirror the x86 CPU capabilities of the underlying host. Without this, the virtualized CPU won't be able to run AlmaLinux/RHEL 9.

    • You can also use the --arch x86_64 option, but it isn't necessary since it will be the default on an Intel-based Mac.

  • ARM (aarch64) VM using QEMU (❓ not tested yet)

    colima start --profile qemu_aarch64 \
    --arch aarch64

On an ARM (Apple Silicon) Mac

  • ARM (aarch64) VM using QEMU

    colima start --profile qemu_aarch64
    • You can also use the --arch aarch64 option, but it isn't necessary since it will be the default on an Intel-based Mac.

  • ARM (aarch64) VM using VZ

    colima start --profile vz_aarch64 \
    --vm-type vz
    • You can also use the --arch aarch64 option, but it isn't necessary since it will be the default on an Intel-based Mac.

  • Intel (x86_64) VM using QEMU

    colima start --profile qemu_x86_64 \
    --arch x86_64 --cpu-type max
    • The --cpu-type max option tells QEMU to mirror the x86 CPU capabilities of the underlying host. Without this, the virtualized CPU won't be able to run AlmaLinux/RHEL 9.

  • Intel (x86_64) VM using VZ and Rosetta 2

    colima start --profile vzr_x86_64 \
    --arch x86_64 --cpu-type max \
    --vm-type vz --vz-rosetta
    • The --cpu-type max option tells VZ to mirror the x86 CPU capabilities of the underlying host. Without this, the virtualized CPU won't be able to run AlmaLinux/RHEL 9.

Working with Colima VMs

Colima VMs

The colima list command will show you some basic information about all of the colima VMs on the machine.

$ colima list
PROFILE         STATUS     ARCH       CPUS    MEMORY    DISK      RUNTIME    ADDRESS
default         Running    x86_64     4       4GiB      100GiB    docker
qemu_aarch64    Stopped    aarch64    2       2GiB      60GiB
qemu_x86_64     Stopped    x86_64     2       2GiB      60GiB
vz_aarch64      Running    aarch64    2       2GiB      60GiB     docker

The colima status command will show you which virtualization framework (i.e. QEMU or VZ) the VM is running under, along with the path to the unix socket used by docker or podman commands to talk to the container runtime.

$ colima status --profile vz_x86_64
INFO[0000] colima [profile=vz_x86_64] is running using macOS Virtualization.Framework
INFO[0000] arch: x86_64
INFO[0000] runtime: docker
INFO[0000] mountType: virtiofs
INFO[0000] socket: unix:///Users/jms1/.colima/vz_x86_64/docker.sock

The only way I've found to see more detailed information is to look lat the YAML file colima creates. You will find these as $HOME/.colima/PROFILE/colima.yaml.

SSH

The colima ssh command will SSH directly into the Linux VM that colima creates.

(jms1@M2Air15) 3 ~ $ colima ssh
jms1@colima:/Users/jms1/work$ uname -a
Linux colima 6.8.0-31-generic #31-Ubuntu SMP PREEMPT_DYNAMIC Sat Apr 20 00:40:06 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
jms1@colima:/Users/jms1/work$ exit
logout

If you're using a profile other than default, be sure to specify the profile name in the command.

(jms1@M2Air15) 4 ~ $ colima -p vz_aarch64 ssh
jms1@colima-vzaarch64:/Users/jms1/work$ uname -a
Linux colima-vzaarch64 6.8.0-31-generic #31-Ubuntu SMP PREEMPT_DYNAMIC Sat Apr 20 02:32:42 UTC 2024 aarch64 aarch64 aarch64 GNU/Linux
jms1@colima-vzaarch64:/Users/jms1/work$ exit
logout

I have SSH'd into colima VMs a few times out of curiosity, but the truth is I've never needed to do it. If you're going to do it, be careful not to change anything. Any settings you might need to change, should be changed by editing the VM's $HOME/.colima/PROFILE/colima.yaml file while the VM is stopped, or by deleting the VM and running the colima start command with different options. (I keep the command lines I use to create colima VMs in an Obsidian notebook.)

Using Docker with Specific Colima VMs

If you have multiple colima VMs, you need a way to tell docker commands which VM to talk to. Docker uses "contexts" for this.

At any time, Docker will be a "current" context that all docker commands will use. When colima creates a VM, it also creates a Docker context pointing to that VM (or technically, pointing to a unix socket which is connected to the unix socket where the container runtime within the VM is listening).

If you have multiple contexts and need to control which one a particular docker command uses, you need to "use" the correct context first.

List Contexts

The docker context ls command lists all contexts that the docker command (on the Mac) is aware of.

$ docker context ls
NAME                DESCRIPTION                               DOCKER ENDPOINT                                  ERROR
colima *            colima                                    unix:///Users/jms1/.colima/default/docker.sock
colima-vz_aarch64   colima [profile=vz_aarch64]               unix:///Users/jms1/.colima/vz_aarch64/docker.sock
default             Current DOCKER_HOST based configuration   unix:///var/run/docker.sock

One of the contexts will have a * after the context name. This is the "current" context, which other docker commands will use.

Using a Different Context

The docker context use command will set the context used by other docker commands.

As an example, starting with the following VMs and contexts ...

$ colima list
PROFILE       STATUS     ARCH       CPUS    MEMORY    DISK      RUNTIME    ADDRESS
default       Running    x86_64     4       4GiB      100GiB    docker
vz_aarch64    Running    aarch64    2       2GiB      60GiB     docker

$ docker context ls
NAME                  DESCRIPTION                               DOCKER ENDPOINT                                     ERROR
colima                colima                                    unix:///Users/jms1/.colima/default/docker.sock
colima-vz_aarch64 *   colima [profile=vz_aarch64]               unix:///Users/jms1/.colima/vz_aarch64/docker.sock
default               Current DOCKER_HOST based configuration   unix:///var/run/docker.sock

$ docker run -it --rm alpine:latest uname -m
aarch64

The docker context use command will change which context future docker commands will use.

$ docker context use colima
colima
Current context is now "colima"

$ docker run -it --rm alpine:latest uname -m
x86_64

Changelog

2024-07-29 jms1

  • added info about --cpu-type max option
  • other updates

2024-07-06 jms1

  • copied from Obsidian notes
  • wrote some human-readable descriptions, verified commands for other scenarios

Installing Debian 10 on a Macbook Pro

2021-01-09

I've always used the RedHat-flavoured versions of Linux, usually CentOS. However, Red Hat "acquired" CentOS and has decided to stop maintaining CentOS 8 and declare an early end-of-life for it, and is now offering "CentOS Stream". Instead of following Red Hat Enterprise, CentOS is now essentially a beta-test distro which feeds into RHEL, and anybody who uses CentOS is providing free QA services for Red Hat.

IBM purchased RedHat a few years back, so I can't say I'm totally surprised by this.

Anyway.

At work, the servers we deploy at client sites have been using CentOS 7 for many years, and we were about to start upgrading things from CentOS 7 to CentOS 8. However, with this news we've decided to move away from CentOS entirely.

At the same time, we're also in the process of re-architecting our software to run under Kubernetes, which doesn't really care what distro it's running on, so long as it has a Linux kernel. So moving from CentOS to Debian isn't necessarily a huge deal, except that it means re-writing the systems which build the underlying machines on which Docker and Kubernetes will be installed.

Which is pretty much my job.

Long story short, we've decided to use Debian 10 instead of CentOS.

Actually, other things have changed over time. For a while we were targeting Debian 11, then Debian 12, but the company was acquired, and then that company was acquired, and the new corporate overlords have decreed that we're going to use RHEL 9, and are not open to any kind of discussion about it. The company I started with is just one small part of one division of the current corporation - instead of having 170 employees, we now have 86,000 employees. At this point I'm just glad they remember my name when it comes time to issue paychecks.

This page contains a collection of random notes I made for myself while exploring Debian 10, using a spare 2013 MacBook Pro.

Notes about Debian 10

Moste of these notes also apply to Debian 11 and 12 as well.

Mac Boot Menu

On a Mac, when you hold down the Option key during the start-up chime, it shows a list of all bootable partitions or devices and lets you choose which one to boot from. Debian doesn't appear on this list, however it does boot correctly when you don't use the boot selector.

At some point I'll figure out why it's not showing up there, when I do I'll update this page.

Wifi Driver

This particular machine is going to be used as a server, at least for now, so it doesn't really need wifi support. However, I figure I'm going to need wifi support at some point, so I took a few minutes to figure out how to enable it, so I could include it here.

Broadcomm chipsets (and some others) require that an opaque binary "blob" be uploaded into the card in order to initialize it. These blobs are not open-source, so they cannot be distributed as part of Debian itself. Instead, the debian "contrib" repo contains a package called firmware-b43-installer which, as part of its post-install script, downloads a package full of binary blobs from (somewhere?) to the /lib/firmware/ directory.

Install firmware manually

Edit /etc/apt/sources.list. At the end of every deb and deb-src line which points to deb.debian.org, after main, add contrib.

deb http://deb.debian.org/debian/ buster main contrib
deb-src http://deb.debian.org/debian/ buster main contrib

Once this is done, install the package which downloads and installs the firmware files.

# apt install firmware-b43-installer

When the package finishes installing, you will see it download a file from an external web site, which contains a collection of firmware blobs. This file is expanded into the /lib/firmware/ directory.

Once the file has been downloaded and installed, reboot the machine.

# shutdown -r now

When it boots again, the kernel should load the firmware file, and the wifi interface should be created.

# ip link show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: enp1s0f0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN mode DEFAULT group default qlen 1000
    link/ether xx:xx:xx:xx:xx:xx brd ff:ff:ff:ff:ff:ff
3: wlp2s0b1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether xx:xx:xx:xx:xx:xx brd ff:ff:ff:ff:ff:ff

In this case, "wlp2s0b1" is the new wifi interface.

During the install

The Debian installer recognizes the Wifi chipset as one that it doesn't support without a firmware "blob" file, and provides a mechanism to supply the file on a USB stick (or a floppy, if the machine has one). This seems useful if you need wifi in order to complete the install.

Every time I've seen this, the machine has had a physical ethernet port, so I've never needed to do this. I have no idea what the requirements are, i.e. what filesystem the USB stick can or should use, whether the firmware blob file should be in the root of the stick or within a certain directory, and so forth. If I ever get curious and have the time I'll play around with that and update this page with my findings.

Interface names

Debian 10 has adopted this "Predictable Names" thing, where the network interfaces are given names which are supposed to never change, but which are very tedious for a human to remember or type.

It's not necessarily a bad concept, and on machines where the hardware may change from time to time (i.e. if you have network interfaces which connect via USB) it can make sense. But it doesn't really provide any value for a server, where the hardware never changes.

And I don't particularly care to un-learn almost thirty years' worth of muscle memory, and change from knowing that the ethernet interfaces are eth0, eth1, and so forth, to having to look up the interface names on every machine I touch.

I prefer to use the sensible "old school" names like eth0 and wlan0, so I did the following:

  • Edit /etc/default/grub, add net.ifnames=0 to the kernel's command line.

    GRUB_CMDLINE_LINUX="net.ifnames=0"

  • Run update-grub

  • Reboot.

Once it comes back, you will probably find that none of the interfaces have IP addresses, because their names don't match the names in the /etc/network/interfaces file. To fix this...

  • Run ip link show and note the new names of each interface.

  • Edit /etc/network/interfaces and change any instance of the old interface names, to the corresponding new name.

  • Restart the network.

    TODO: systemctl restart network doesn't work on Debian, need to figure out how to do this.

Timezone

One of the first questions that the Debian installer asks is, what part of the world you're in. Later on it asks what timezone you want the machine to use, however if you selected "United States" as a location, it will only show you the American time zones.

A LOT of people have been complaining about this for years, but apparently the people who maintain the Debian installer don't want to hear that some people build servers and want the systems' clocks to run on UTC.

During install

The only way to get UTC as an option during the install is to lie about your location.

  • Select (UK? Europe? "Etc"?) as location.

  • Then select UTC as the timezone.

Note that doing this may also configure other things on the system, like using the "en_GB" locale instead of "en_US", which may result in using an unexpected console font or keyboard mapping. (Seeing "£" when you're trying to type "#" is always fun.)

After install

The other option is to let the installer do what it wants to do, and then manually configure the timezone after the system is running.

List available time zones: (the list is rather long)

$ timedatectl list-timezones
...
America/New_York
...
Etc/UTC
...

Set the system to use a different time zone:

$ sudo timedatectl set-timezone Etc/UTC

Alternate method: (useful on older Debian machines which don't have a timedatectl command)

$ sudo ln -sf /usr/share/zoneinfo/Etc/UTC /etc/localtime

Changelog

2024-07-06 jms1

  • updated intro text

2024-06-19 jms1

  • moved page to new jms1.info site, updated header
  • minor content updates

2022-01-22 jms1

  • added possible "last updated" field for each page, used this page to test
  • minor tweaks in text
  • fixed Changelog dates (should be 2021, not 2011)

2021-01-12 jms1

  • updated info about how to set time zone
  • better explanation of how firmware-b43-installer works
  • added other general info
  • added to jms1.info site

2021-01-09 jms1

  • initial version (not ready for public consumption yet)

x86_64 Microarchitecture Levels

2024-07-29

Newer versions of the x86_64 processors support instructions that did not exist in the older versions. These newer instructions can perform tasks that would previously have taken multiple instructions using only what was available in the original x86_64 baseline instruction set. These instructions are generally useful for "hardcore" math, graphics, or encryption operations.

If a program tries to run one of these new instructions on a CPU which doesn't support them, you'll get an "Illegal instruction" error and the program will crash.

The x86 CPUs have a set of "flags" which programs can query if they plan to use any of the newer instructions, so they can either fail gracefully (with a better error message), or use different code to do the same job using the older, slower instructions.

ℹ️ On a Linux machine, you can see these flags in the /proc/cpuinfo file.

$ grep ^flags /proc/cpuinfo
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov ...

Each of these codes corresponds to a capability which may or may not exist on an x86 CPU.

Having to query for a whole list of instructions can be rather tedious, so Intel, AMD, Red Hat, and SUSE created a set of "levels" to encapsulate different sets of instructions that were added over time.

  • Original suggestion on the llvm-dev mailing list
  • Wikipedia has a good description of how the levels work and which CPU instructions are included in each level
  • Medium article also explains it fairly clearly
  • Red Hat blog post explains how and why Red Hat built RHEL 9 to require and use the x86-64-v2 capabilities

AlmaLinux (and RHEL) 9

I first ran into this when I tried to run AlmaLinux 9 in a Docker container, under colima on an Apple M2 machine. (I'm currently in the process of migrating my own servers from CentOS 7 to AlmaLinux 9.) AlmaLinux 9 and RHEL 9 require a minimum of x86-64-v2, and the virtual CPU created by QEMU (used by colima to run the x86_64 Linux VM where Docker is actually running) by default only supports x86-64 without any extensions.

At first, when I tried to start an AlmaLinux 9 (or UBI 9) container, I got this error:

$ docker run -it --rm almalinux:9.4
Fatal glibc error: CPU does not support x86-64-v2

This was the problem that made me dig into this issue. I found that adding --cpu-type max when creating the colima VM made the virtual CPU suport the same capabilities as the hardware CPU, which for the Apple M2 includes everything in the x86-64-v3 list.

Detect

If you need to detect which "level" a script is running on, here are a few suggestions:

awk script

⇒ Source

#!/usr/bin/awk -f

BEGIN {
    while (!/flags/) if (getline < "/proc/cpuinfo" != 1) exit 1
    if (/lm/&&/cmov/&&/cx8/&&/fpu/&&/fxsr/&&/mmx/&&/syscall/&&/sse2/) level = 1
    if (level == 1 && /cx16/&&/lahf/&&/popcnt/&&/sse4_1/&&/sse4_2/&&/ssse3/) level = 2
    if (level == 2 && /avx/&&/avx2/&&/bmi1/&&/bmi2/&&/f16c/&&/fma/&&/abm/&&/movbe/&&/xsave/) level = 3
    if (level == 3 && /avx512f/&&/avx512bw/&&/avx512cd/&&/avx512dq/&&/avx512vl/) level = 4
    if (level > 0) { print "CPU supports x86-64-v" level; exit level + 1 }
    exit 1
}

bash script

⇒ Source

Too long to incude here, but it looks like it's doing the same thing as the awk script above.

Changelog

2024-07-29 jms1

  • initial version

macOS

This section includes pages which talk about macOS itself, or about thing which only happen on macOS systems.

Backing up a Time Machine Image

2026-04-29 Wed

I recently ran into a situation where the internal spinning disk in one of my older Mac Minis was dying. This made the machine crash, or just "freeze up", at random intervals. Sometimes it would work for a few days, other times it would work for a few minutes. Over time, the "time between failures" got shorter and shorter, until it finally reached the point where it would consistently crash in less than an hour.

The machine did have a Time Machine backup on an external disk, less than a week old. Nothing had been added or changed on the machine itself that needed to be backed up, so that backup already contained all of the files I needed to save. I was able to physically plug that disk into a newer machine (an M2 Mac Studio) and see the backed-up files, and running Disk Utility's "First Aid" function on the disk didn't report any errors, so the backup was good.

The machine itself (an older Intel Mac Mini) seemed to be okay, the problem was the spinning disk itself. So my plan was to replace the disk with a SATA SSD. However, I didn't want to lose the files,

I wanted to make a DMG disk image, containing the system's full contents, either from the system itself, or from the most recent Time Machine backup.

Things I Tried

  • On the dying machine, use Disk Utility to make an image of the machine, to a different external disk.

    This idea didn't work very well because the machine kept crashing part-way through the process.

    Given the problems I ran into below, I'm not sure if it would have worked any better if I had done it a few months ago, when the hard drive was still capable of running for several hours before crashing.

  • On the newer machine, use Disk Utility to make an image of the folder from the Time Machine backup disk.

    This didn't work. When I tried, about 45 seconds into the operation it stopped and told me "Operation cancelled", without any explanation of why it stopped.

    I tried running "First Aid" on the Time Machine disk, it didn't find any errors, so ... no idea.

  • On the newer machine, using hdiutil to manually make a disk image from the folder on the Time Machine backup disk.

    In theory, this command should have worked:

    sudo hdiutil create \
    -srcfolder  /Volumes/TMDisk/2026-04-09-153426.previous \
    -type       UDZO \
    -layout     NONE \
    -fs         HFS+ \
    -volname    "Old System HD" \
    "$HOME/Desktop/Old System HD"

    This also disn't work. I tried this several times, with different options, after fixing a few oddities with the files in the backup image - files with 0000 permissions, directories whose x permission had been removed, files with strange "flags", etc. I also tried using sudo with the hdiutil command, which did help, but eventually I ran into an error message that didn't explain what the problem was, and about which I couldn't find any useful information on the internet.

  • On the newer machine, using Carbon Copy Cloner to back up the folder on the Time Machine backup disk, to a DMG file. (I figured if anybody knew how to do this, the guys who wrote CCC would.)

    As soon as I pointed CCC to the Time Machine disk as a source, it popped up a dialog saying that it couldn't (and wouldn't even try to) back up Time Machine images, because Apple didn't support it. This is explained in more detail on this page on their support site.

What Finally Worked

I ended up having to make the backup by hand, and I had to use rsync to copy the files, rather than any Apple utility.

At a high level, the process is this:

  • Create a temporary read-write disk image.
  • Use rsync to copy the files from the Time Machine image, into the temporary disk image.
  • Convert the temporary read-write image into a read-only disk image. This image can be compressed and/or encrypted if you like.

This process depends on having enough disk space for two full copies of the data being backed up. The free space doesn't necessarily have to be on the same disk, however if you're going to use two disks (one for the temporary image and one for the final read-only image), the disks containing those locations each need to have enough free space for a full copy of the data being backed up.

Create the Temporary Disk Image

  • Plug in the Time Machine disk, and wait for the disk's icon to appear on the desktop.

    • If the disk is encrypted, you may need to enter its password.
    • If macOS asks about using the disk as a Time Machine backup disk, say no.

  • Find the /Volumes/xxx directory for the Time Machine disk, and save it to a variable.

    $ ls -l /Volumes/
total 0
drwxrwxr-x@  19 root  wheel   192 Apr 27 10:59 Extra Disk
lrwxr-xr-x    1 root  wheel     1 Apr 27 10:59 NewSystem Internal HD -> /
drwxrwxr-x@   6 root  wheel   192 Apr 29 10:32 OldSystem TM

    In this example ...

    • NewSystem Internal HD is the internal disk of the system we're running on.
    • OldSystem TM is the external Time Machine disk.
    • Extra Disk is an extra external disk with enough free space to hold a temporary copy of the data being backed up.
    TM_DISK="/Volumes/OldSystem TM"

    Saving this to a variable will allow you to copy-and-paste the commands below, rather than having to manually type the disk names and worry about quoting.

  • Save the name of the disk image you want to create, to a variable.

    IMG="OldSystem HD"

  • Calculate a few other variables.

    DST="$HOME/Desktop/$IMG"
SRC="$( echo "$TM_DISK"/2* | tail -1 )"
USED="$( sudo du -sk "$SRC" | awk '{print $1}' )"
SIZE_KB="$(( USED + 65536 ))"

    These are:

    • DST is the filename for the final read-only image. This can include .dmg at the end, but if you don't include it, the command which creates the final image will add it automatically.
    • SRC is the Time Machine directory you'll be backing up. Depending on which OS/X or macOS version was used, there may be multiple "dated" directories on the disk. This command returns the name of the most recent directory.
    • USED contains how much disk space is being used by the backup image, in KiB (units of 1024 bytes). We need this so we can create the temporary image with enough space to hold the files.
    • SIZE_KB is the actual size of the temporary disk image we'll be creating. This is the total disk space being used, plus 64 MiB (which is needed to hold the filesystem's data structures).

  • cd into the directory where you want to create the temporary disk image.

    cd "/Volumes/Extra Disk"

    Note that I had to use quotes because the disk name has a space in it.

  • Create and mount the temporary disk image.

    hdiutil create \
    -attach     \
    -size       "${SIZE_KB}k" \
    -type       UDRW \
    -layout     NONE \
    -fs         HFS+ \
    -volname    "$IMG" \
    "$IMG work"

    This created OldSystem HD work.dmg, which is the temporary disk image. It also mounted that image as /Volumes/OldSystem HD Work.

  • Save the "mount point" of the work disk to a variable.

    MPT="/Volumes/OldSystem HD Work"

At this point, the temporary disk image is ready to have files copied into it.

Copy the Files

  • Copy the files from the Time Machine backup, into the temporary disk.

    sudo rsync -av "$SRC/" "$MPT/"

  • Make sure any data which is buffered for the temporary disk image, is written to the image.

    sync

  • Eject the temporary disk.

    hdiutil eject "$MPT"

At this point, OldSystem HD Work.dmg is finished. It is a read-write image, which means if you were to double-click the DMG file to mount it, you would be able to add, change, or remove files.

Convert the Image

This step will convert the temporary image to a read-only image.

The basic command is this:

hdiutil convert \
    -format UDRO \
    -o      "$DST.dmg" \
    "$IMG work.dmg"

Depending on your needs, you may want to modify this command.

  • To make a compressed read-only image, change UDRO to UDZO.

    This will result in a disk image which is somewhat smaller - how much smaller will depend on the files within the image. For the backup which made me need this process, the original Time Machine backup was about 245G, and the compress backup was 224G.

  • To make an encrypted read-only image, add -encryption AES-256 to the command.

    If you do this, the command will prompt you for a passphrase for the encryption. (There is no option at this point to store the passphrase in your Keyring. If you need to do this, you'll be able to do it below. Just don't forget the passphrase between now and then.) 🤔

In my case, the actual command I ended up running (for a compressed and encrypted image) was ...

hdiutil convert \
    -format     UDZO \
    -encryption AES-256 \
    -o          "$DST.dmg" \
    "$IMG work.dmg"

Testing

At this point the final image should be done. You should test it:

  • Double-click the DMG file to mount it.

    If the image is encrypted, you will be asked for the passphrase.

    If you wanted to save the passphrase in your Keyring, this dialog will have a checkbox to let you do that.

  • Make sure the mounted image is read-only.

    If you're looking at it in a Finder window, you should see this in the status bar:

    Obviously, you should not be able to add, remove, or change any files.

  • Eject the mounted disk.

Cleanup

Once you've tested the final disk image, you can ...

  • Delete the temporary disk image.
  • Re-format the external Time Machine disk.
  • Re-format the machine that the Time Machine backup was taken from (or in my case, replace its hard drive with an SSD).

If You're Curious

I replaced the now-dead 500 GB spinning disk with a 512 GB SATA SSD that I had laying around, and used Apple's Internet Recovery mechanism to install the latest macOS.

Short version: I powered the machine on, and while the chime was playing, I pressed ⌥⌘R and held it down until the "spinning globe" graphic appeared. This starts the Internet Recovery process and installs the latest macOS version which is compatible with the hardware.

The machine is back to a "working" state again, in that it hasn't crashed in over a week.

My next step is going to be installing Linux on it. 😎

macOS - SSH Post-Quantum Encryption

2025-12-05

This page explains how I dealt with an issue with having multiple versions of OpenSSH on my macOS workstations.

Background

I use Homebrew to install software that macOS doesn't come with, and also to install newer versions of software that macOS does come with. My "work" workstation (a 2018 Mac Mini with an Intel processor) is currently running macOS 15.7.2, which came with OpenSSH v9.9, but I actually use OpenSSH 10.2 from Homebrew.

I control which version is used by controlling the order in which directories appear in my PATH. As long as Homebrew's bin directory comes before the standard /usr/bin directory, programs installed by Homebrew will "override" programs installed by macOS.

ℹ️ PATH Order for Homebrew

My actual shell profile is a bit more complex than this, but you can do this by adding the following to the END of your .bashrc for (or its equivalent, depending on what shell you're using) ...

PATH="$( brew --prefix )/bin:$PATH"

The Problem

When I use ssh to connect to a server whose sshd is from an older version of OpenSSH, the ssh command prints this warning:

** WARNING: connection is not using a post-quantum key exchange algorithm.
** This session may be vulnerable to "store now, decrypt later" attacks.
** The server may need to be upgraded. See https://openssh.com/pq.html

This is distracting, and in the case of a script which runs an ssh command and processes the output, it can cause problems. The obvious (and more secure) solution would be upgrade sshd on every server I connect to, however 98% of them are running RHEL 7 or RHEL 9, and Red Hat doesn't have OpenSSH packages which support the PQ algorithms yet.

According to the page listed in the warning, I can disable the warning by adding WarnWeakCrypto no to my .ssh/config file to disable this warning. I did this, and the warning no longer appears.

BUT.

When programs are started by launchd or Finder, my .bashrc file isn't used, so I have no way to modify the PATH that the program inherits. This means that if those programs run ssh, they end up running /usr/bin/ssh, which is the older version that comes with macOS. If this older version doesn't recognize the WarnWeakCrypto option, it causes an error.

The Solution

There is an IgnoreUnknown directive, which is recognized by older OpenSSH versions. This tells it to ignore specific unknown option names if they appear in the config.

So, to work around the problem for now, I have done the following in my $HOME/.ssh/config file:

  • At/near top of file (this is actually the first line of my config file)

    IgnoreUnknown   WarnWeakCrypto

  • In the appropriate Host block(s)

    Host *.example.com
    WarnWeakCrypto  no

Other Information

OpenSSH's page explaining the warning

Timeline

  • OpenSSH 9.0 (2022-04)
    • added sntrup761x25519-sha512 as a host key exchange algorithm
  • OpenSSH 9.9 (2024-09)
    • added mlkem768x25519-sha256 as a host key exchange algorithm
  • OpenSSH 10.0 (2025-04)
    • made mlkem768x25519-sha256 the default host key exchange algorithm
  • OpenSSH 10.1 (2025-10)
    • added the warning when connecting to a server which doesn't offer a PQ host key exchange algorithm
    • Also added the WarnWeakCrypto option to disable this warning

mdbook

I'm using mdbook to maintain several sites, including this one.

Creating a "book" with mdbook

2022-03-08

I was in the process of writing a document at work last week, and realized that the Markdown file I was working on already had over 10,000 lines, and it was only about 60% done writing it. When I created a PDF to preview it, the PDF file had about 40 pages so far. I realized that I was "writing a book", and that the document was too long for some people.

I've seen a lot of multi-page "documentation" web sites that all followed a common pattern, with a navigation bar on the left having a set of links to all of the pages making up the documentation. Many of these were hosted with sites like readthedocs.org or GitBook, however I needed a stand-alone tool which produced stand-alone files, because some of the information I'm documenting is proprietary and cannot be hosted outside the company.

I found a couple of programs which automate making these kinds of sites, and mdbook caught my eye. It's written in Rust, and is used by the Rust developers to generate their own documentation.

I tried it out, and found it to be very easy to use - the hardest part for me was figuring out where to logically break that original Markdown file into separate pages. mdbook produced a set of static web pages that made the documentation a LOT easier for readers to navigate.

So now I'm using it at work, and I decided to convert my chicken-scratch notes about how to install it and set up a new book, into a document that I can refer to myself whenever I need to write a book, and which other people may find useful.

Install mdbook

The mdbook documentation explains several different ways to install the software. My personal and work machines both run macOS with Homebrew, so for me the process was very simple:

brew install mdbook

Create a new book

The mdbook init command creates a basic skeleton of the files it needs to build a "book". You can run it in an empty directory and it will create its files there.

mkdir .../xyzzy
cd .../xyzzy
mdbook init --ignore git --title 'Things and Stuff'

It recognizes the following options:

  • --title 'Things and Stuff' = specify a title for the book. If not specified, the command will interactively ask for a title.

  • --ignore git = Create a .gitignore file. If not specified, the command will ask whether or not you want one.

  • --theme = Create a theme/ directory with the files that make up the default theme. This is not normally needed unless you're planning to modify the theme.

This creates the following files:

.gitigore
book/
book.toml
src/
src/SUMMARY.md
src/chapter_1.md

  • book.toml configures the properties of the overall book itself, including any additional processing steps needed while building the book.

  • src/SUMMARY.md contains the "structure" of the book, and is used to build the navigation bar on the left side of every page. The initial contents of the file reference a "Chapter 1".

  • src/chapter_1.md is a sample file. Deleting this file (and removing the reference to it from SUMMARY.md) are usually the first things I do when setting up a new book.

Create a git repo

I use git to track almost everything I work on. When I create a book, I like having the initial commit in the repo contain the exact files generated by "mdbook init".

cd .../xyzzy
git init -b main

At this point the repo has no commits, but you can run git status and see what files are ready to be committed.

$ git status
On branch main

No commits yet

Untracked files:
  (use "git add <file>..." to include in what will be committed)
    .gitignore
    book.toml
    src/

nothing added to commit but untracked files present (use "git add" to track)

No surprises, so use what we have as the initial commit.

$ git add .
$ git status
On branch main

No commits yet

Changes to be committed:
  (use "git rm --cached <file>..." to unstage)
    new file:   .gitignore
    new file:   book.toml
    new file:   src/SUMMARY.md
    new file:   src/chapter_1.md

$ git commit -m 'Initial commit'

I also create an initial tag in each repo, pointing to the very first commit.

$ git tag -sm 'Tagging the initial commit' initial

Set up remote

IF the repo is going to be stored on a remote server, such as Github, Bitbucket, or Keybase...

  • In that remote server's web interface, create an empty repo and get its URL.

  • Keybase doesn't have a web interface, so use the command line to create the repo.

    keybase git create xyzzy

  • On the local machine, add a "remote" pointing to the upstream repo's URL.

    git remote add origin git@github.com:username/xyzzy

    git remote add origin keybase://private/username/xyzzy

  • Push the initial commit and tag.

    git push -u origin main
git push --tags

If the repo doesn't have a remote, you'll need to be a lot more careful about not accidentally deleting the book or its .git/ directory.

Update .gitignore

The .gitignore file created by mdbook init contains the line book, so that git will ignore the book/ directory in the root of the repo. This is fine, but it also makes git ignore other files and directories whose names may be "book". This should be changed so it only ignores the book directory in the root of the repo.

You should also add the names of any other files that git should ignore. I normally use something like this...

/book/
.DS_Store
._*
*~
*.bak

Commit and push the change.

git add .gitignore
git commit -m 'Updated .gitignore with the usual list'
git push

Template

For most of what I do at work, readers need to know which version of a "book" they're looking at. mdbook doesn't have a way to include any kind of version number, but it turns out to not be overly complicated to add this information.

I normally use a "template" to start new "books". This template already includes the modifications to add git information (commit hash and possibly tags) to the pages. It also includes some other cosmetic tweaks I like to have in the documentation I write. This is all documented here:

Template

Working with mdbook

Removing "Chapter 1"

Some people may want to use the src/chapter_1.md file, but I never do.

  • Edit src/SUMMARY.md, remove the appropriate line. It looks like this:

    - [Chapter 1](./chapter_1.md)

  • Stage the file to be committed.

    git add src/SUMMARY.md

  • Use "git rm" to remove the file.

    git rm src/chapter_1.md

  • Commit and push the change.

    $ git status
On branch main
Changes to be committed:
  (use "git restore --staged <file>..." to unstage)
    modified:   src/SUMMARY.md
    deleted:    src/chapter_1.md

$ git commit -m 'Remvoed chapter_1.md'
$ git push

Changelog

2024-06-19 jms1

  • moved page to new jms1.info site, updated header

2022-09-10 jms1

  • This file happened to be on the screen while I was getting ready to move the jms1.info site from Apache to Keybase Sites and a typo caught my eye, so I gave it a quick once-over.
  • Mentioned Keybase as a way to host the git repo.
  • Added info about mdbook-template repo.
  • Other minor tweaks.

2022-03-08 jms1

  • Initial version

Adding git commit/date info to mdbook "books"

2022-03-08

One of the limitations I ran into with mdbook is that it doesn't easily offer a way to automatically embed a version number or date into the generated pages. This makes it difficult for the reader to be sure that the documentation they're reading is the latest version, or that it matches a specific version of the "thing" the documentation is describing. This is something I normally use with any kind of automated documentation system, so I took the time to figure out how to do it.

This will add a section at the bottom of the navigation menu with the git commit and its timestamp, plus the timestamp when the HTML was generated.

This comment had most of the details, however what you see below is different enough to justify making my own write-up instead of just linking to the page.

Pre-requisite

Install jq.

  • macOS

    brew install jq

  • Others - TODO

Template

The first step is to modify the template used for every page, with updated HTML to position and format the information, as well as tokens where the dynamic information (the git commit and timestamps) will be substituted.

Note that the generated pages are all self-contained, there is no "frameset" with different HTML files shown on the left and right sides of the browser window.)

  • In a new directory, run mdbook init --theme to create a dummy book, but with a theme/ directory.

    Note: we're not going to keep this book, we just need somewhere to copy a couple of the default theme's files from.

  • Copy theme/index.hbs from the dummy book, to theme/index-template.hbs in your book.

    This is the template we'll be modifying. Our copy will be used as a template for that template. (Very "meta", I know.) It will need to use a different filename, and a script will be triggered (below) to generate the values we need and substitute them into the contents of this file, to produce the actual index.hbs file that mdbook will use.

  • Edit our new theme/index-template.hbs and add the version placeholders and other formatting within the existing <nav> element, as shown:

        <nav id="sidebar" class="sidebar" aria-label="Table of contents">
        <div class="sidebar-scrollbox">
            {{#toc}}{{/toc}}
<!-- start new content -->
            <hr/><div class="part-title">Version</div>
            <div id="commit" class="version">
                <tt>VERSION_COMMIT_HASH</tt><br/>
                <tt>VERSION_COMMIT_TIME</tt>
            </div>
            <div class="part-title">Generated</div>
            <div id="generated" class="version">
                <tt>VERSION_NOW</tt>
            </div>
<!-- end new content -->
        </div>
        <div id="sidebar-resize-handle" class="sidebar-resize-handle"></div>
    </nav>

  • Add /theme/index.hbs to your .gitignore file. This file will be generated on the fly every time the book is processed.

Stylesheets

The next step is to create a stylesheet, which will format the text added by our template modifications above.

In the root of the repo, create a version-commit.css file with the following contents:

.version {
  font-size: 0.7em;
}

Now we need to tell mdbook to include that stylesheet in the generated pages.

  • Edit your book.toml file.

  • If it doesn't already have one, add an [output.html] section.

  • If this section doesn't already have an additional-css key, add one.

  • Add "version-commit.css" to that list.

The resulting section of the file will look like one of these:

  • If this is the only custom CSS file...

    [output.html]
additional-css = [ "version-commit.css" ]

  • If there are other custom CSS files...

    [output.html]
additional-css = [ "custom.css" , "version-commit.css" ]

The original web page also showed how to not include the version info in any printed output. I'm guessing this is because the "printed output" consists of one big long document containing the entire generated site, as opposed to just the one page you're looking at in the browser, and having the version info in between every page would get redundant.

I'm adding the version info to the "navigation bar" on the left, which already isn't included in printed output, so in my case this isn't necessary. However, if you're doing something different and find that you need this...

  • Create theme/css/print.css with the following contents:

    .version {
    display: none ;
}

Add the preprocessor script

The version-commit script reads the template we copied earlier and substitutes the commit hash, commit time, and current time where the appropriate tokens exist in the template.

I wrote this first as a Perl script, and then tried to re-write it as a shell script. The script itself is a UNIX "filter" (i.e. it reads from STDIN and writes to STDOUT), so it seemed like it should be simple to just calculate the three values, then run a sed command to substitute the values ... but when I tried it, all of the generated output files ended up as zero bytes.

It was already working as a Perl script and I didn't have a lot of time to dig into it, so I left it alone and stuck with that. Maybe in the future if I get curious I'll have another go at making it into a shell script.

  • Copy version-commit to the root of the repo.

  • Set its permissions to allow execution (i.e. chmod 0755 etc.). Make sure you do this before you git commit the file.

Once the script is in the repo, add the following to book.toml:

[preprocessor.generate-version]
renderers       = [ "html" ]
command         = """sh -c 'jq ".[1]"; ./version-commit theme/index-template.hbs > theme/index.hbs'"""

I'm not 100% sure whether the triple-quotes are a TOML thing, or a side effect of whatever code within mdbook parses the file, but this is working.

You will also note, it requires that jq be installed on the machine.

When mdbook runs a preprocessor, it sends a JSON array to the preprocessor's STDIN. This array contains two dictionaries, one being the "context" with information about the job itself, and the other being a JSON structure of the book's sections, chapters, and text. The preprocessor is expected to send a potentially modified version of this "book" JSON structure to its STDOUT.

The jq ".[1]" command simply copies the "book" JSON structure as-is, without making any modifications. In this case, we don't really need to modify anything in the content, we're just using the "preprocessor" to trigger the conversion of the template for the index.hba file, which is then used as the template for rendering the individual pages within the site.

This is why the string used as the command here, runs the jq command first, and then runs the ./version-commit script to process the template and produce the index.hbs file.

Test

I normally leave mdbook serve running while I work on documents, so I can preview my changes immediately in a browser window. I find that this encourages me to "save early, save often", as opposed to something like MacDown which shows a live preview while I'm typing and therefore doesn't force me to save as often.

I tested this by making minor edits to one of the files that mdbook serve is watching - specifically, I added or removed extra empty lines at the end of the src/SUMMARY.md file, and then saved the change.

  • Watch the output that mdbook serve writes while it's running. If there are problems with any of this, that's where any error messages will appear.

  • Obviously, check the browser window where you're previewing the content, to see if the changes you're expecting, appear there.

If you don't use mdbook serve, you can run mdbook build by hand and check the results in a browser window.

Changelog

2024-06-19 jms1

  • moved page to new jms1.info site, updated header

2024-06-19 jms1

  • moved page to new jms1.info site, updated header
  • formtting changes due to Jekyll/mdbook differences

2022-03-08 jms1

  • Initial version

Testing CSS Attributes

This page lists the values of the CSS variables involved in the theme. I use this when testing values for a new theme. I do this by changing the values in the variables.css file, then reloading the browser window, to make sure the colours look the way I want them to.

The page contains an embedded Javascript function which reads the variables' values and inserts them into <span> tags in the tables below. If you change the theme you're using in your browser, you'll need to click one of the buttons (or reload the page) to update the values.

Notes:

  • This is NOT a complete list of the CSS variables used in the pages that mdbook generates. These are only the ones I have found useful when creating different coloured themes.
  • The buttons only run the Javascript function to update the values in the tables. They do not reload the entire page.
  • The list of variables can change between different mdbook versions. The list below is accurate for mdbook v0.5.2, which is the latest version as of the time I'm writing this (2026-03-16).

Examples

Core Background & Text Colours

VariableValueExample
--color-scheme
--bg
--fgtesting
--inline-code-colortesting
--linkstesting
--icons
--icons-hover

This is regular text, with some inline text.

This is a block of pre-formatted text.

VariableValue
--sidebar-active
--sidebar-bg
--sidebar-fg
--sidebar-header-border-color
--sidebar-non-existant
--sidebar-spacer

Blockquotes

VariableValue
--quote-bg
--quote-border

This is a blockquote.

Tables

VariableValue
--table-alternate-bg
--table-border-color
--table-header-bg
 
Extra rows to give the
table enough rows so you
can see the alternate
background colours

Code Highlighting

VariableValue
--code-bg
--code-fg
 
#!/usr/bin/env perl

require 5.005 ;
use strict ;
use warnings ;      # since 'env' won't let us add the '-w' option on the #! line

my $x = 0 ;
while ( $x <= 5 )
{
    $x ++ ;
    print "$x\n" ;
}

Interactive Elements

VariableValue
--scrollbar
--searchbar-border-color
--searchbar-bg
--searchbar-fg
--searchbar-shadow-color
--searchresults-header-fg
--searchresults-border-color
--searchresults-li-bg
--search-mark-bg
--theme-popup-bg
--theme-popup-border
--theme-hover

Embedded in this page

If you look at this page's Markdown source, you'll see that it contains some HTML fragments. I don't normally include HTML when writing Markdown, however I wanted this page to show not only the effects of the different colour values I'm trying, but the actual values as well.

You will also notice (in the Markdown source) that the values in the tables are enclosed with <tt>...</tt> rather than with backticks. This is because part of how mdbook handles backticks involves converting any < and > characters in the monospaced text to &lt; and &gt;, which makes the browser show the <span> tags instead of the values.

To save you the hassle of viewing the source in your browser and digging through that to see how I did this, I'm including visible copies of these HTML fragments here.

Javascript - Show CSS Variable Values

This is a Javascript function which walks through every element on the rendered page which has class='show-value', and sets the element's contents to the value of the CSS variable whose name is in the element's data-var= attribute.

<!-- Update every `<span/>` with `class='show-value'` to have the value of
     the CSS variable named in the `data-var=` attribute, as its content. -->
<script>
function update_values() {
    var elements = document.querySelectorAll( '.show-value' ) ;
    elements.forEach( e => {
        e.innerHTML = getComputedStyle( e ).getPropertyValue( e.dataset.var ) ;
    } )
}
update_values()
</script>

Stylesheet - Do Not Center Tables

The default stylesheets that come with mdbook make tables centered on the page. I don't particularly like this, I think the pages look better with tables aligned to the left (which is HTML's default behaviour), so I'm adding this little block of CSS to override the margin: auto in mdbook's default stylesheet.

<!-- Do not center tables. -->
<style>
table {
    margin : 0 ;
}
</style>

Obsidian

Obsidian is a note-taking app. Individual notes are stored (and written) as Markdown files, stored in a directory tree that it calls a "vault". The files in the vault appear as a "tree view" pane within the app, and clicking on a file there will open that file in an editor pane.

It also tracks links between documents within the vault, and can produce a "graph view" with the documents as nodes and lines connecting them to other documents. I don't use this functionality much, since it requires you to add the links between documents, and my mind doesn't really work that way - to me it makes more sense to organize my documents in a directory structure.

Obsidian is available for macOS and Linux. It's also apparently available for ms-windows as well, if you're into that sort of thing. I don't use ms-windows unless something at $DAYJOB requires it, and that hasn't happened in over a year now.

Obsidian has an API which allows people to write their own plugins to extend or modify how Obsidian works. These plugins can be uploaded to Obsidian's servers, and other users can download and use them from there. It also allows users to make their own themes, which control the visual appearance of the notes you're editing or viewing.

I use a small set of plugins on a regular basis:

  • obsidian-git makes Obsidian automatically commit and push changes to a git repo, as edits are made to the files in a vault. I track own vaults (both personal and for $DAYJOB) using Keybase git repos.

  • Minimal Theme provides more options to customize the appearance of an editing pane, separately from a preview pane. This was the only way I could find to use different fonts for editing and previews.

    The author also provides a plugin to configure its settings.

Quiver

Quiver (macOS App Store link) is a "notebook built for programmers". Its documents are stored as a series of blocks, each of which can be plain text, code, Markdown, or LaTeX. (99% of my documents had a single Markdown block.)

I stopped using Quiver for a few reasons.

  • It has bugs. Nothing serious, I never lost any data - for me it was a tagging feature that was working for a long time, but is no longer working in the most recent release.

  • The last release was 2019-09-29 (over 4½ years now). The only change I've seen since then is that its web site changed from happenapps.comm to yliansoft.com - the site's content appears to be the same.

  • The only avenue for support I could find was an email address. The one time I tried to email the developer about tags not working, I never got any response.

Basically, I'm left with the conclusion that the app itself has been abandoned. Which is sad, because it was almost perfect for my needs.

Logseq

At first glance, Logseq seems to be very similar to Obsidian, however I ended up not going with it for a few reasons:

  • Logseq's documents are structured more as outlines than as free-form Markdown documents. Outlines have their uses, but I've been very happy with OmniOutliner for many years now. If I ever feel the need to use something other than OmniOutliner, Logseq will be on the list.

    The problem I was trying to solve was a way to organize and edit Markdown files, many of which already existed. For this problem, Obsidian was a better fit for me.

  • Logseq's user interface seems to be a lot more heavily centered on the graph view. It's cool, but I found myself constantly thinking about adding links between documents, then checking the graph to make sure I wasn't missing any, then going back into the documents and figuring out how to work links into them.

    In other words, I was spending more time thinking about Logseq than about the contents of the documents I was writing.

mdbook

mdbook is a program which converts a directory tree full of Markdown files, into a web site containing static HTML files. If you're reading this on the jms1.info site, you're looking at mdbook's output.

mdbook is not really an alternative to Obsidian, however until I found Obsidian, I was thinking seriously about using it as part of a workflow to try and provide the things I was using Quiver for.

I am using mdbook to maintain half a dozen internal documentation web sites for $DAYJOB, and I'm starting to use it for my own personal web sites (both public and private) as well.

There is an mdbook section in the menu on the left, where you will find more information about mdbook.

ReText

ReText is an open-source (GPL2) editor for Markdown and reStructuredText. I had never heard of it until I installed Debian 12 on a laptop and ran apt search markdown to see what was available.

From what I've seen it works well enough - it works with individual files rather than having a "vault" mechanism (i.e. no built-in file selector), however I couldn't find an already-packaged version for macOS. It's written in Python and the source code is available on Github, so if I hadn't found Obsidian, I was thinking about maybe packaging it for macOS so I could use it there.

obsidian-git

2024-06-30

obsidian-git is a plugin for Obsidian which automatically tracks changes to a vault in a git repo. If the repo is linked to a "remote" (like Github, Keybase, etc.) the plugin can also push the changes to that remote as they happen.

For me, this serves a few purposes:

  • The git repo, by itself, provides a way to go back and see previous versions of notes which change over time.

  • The git remote serves as a built-in backup mechanism. For non-public vaults I use Keybase git, so it's a cryptographically secure backup - even Keybase themselves can't decrypt what's in the repo.

  • Using a git remote also provides a way to use the same vaults on multiple computers.

I ran into some issues when I started using obsidian-git, because some of the documentation out there wasn't written in a way that "clicked" for me. I went through several iterations of creating dummy vaults, adding the plugin to them, and figuring out how it works. I kept my own notes while I was doing this, and I almost feel like I understand it now.

I'm adding my notes on this site in case somebody else might find them useful. You know, because there isn't enough documentation about it already.

Original - xkcd #927

Adding obsidian-git to an Existing Vault

2024-06-30

This page covers how to add obsidian-git to a vault. It covers both newly created vaults and existing vaults, as long as no git repository exists for them yet. If you already have a git repository for a vault and just need to link a new computer to it, see the "Clone an Existing Vault" page instead.

Note that this page started out as two separate notes - one about creating a new vault with obsidian-git, and one about adding obsidian-git to an existing vault. When I added these notes to the jms1.info site, I noticed that a lot of the content was the same, so I'm combining them into a single page. Depending on when you see this, you might notice some duplicated content.

Creating a New Vault

If you don't already have a vault that you plan on using obsidian-git with, the obvious first step would be to create one. Use Obsidian's standard process of creating a new vault, nothing special.

Assuming Obsidian is already running ...

  • Click the icon in the ribbon on the left.

  • In the dialog that appears, next to "Create new vault", click "Create".

  • Next to "Vault name", enter a name for the new vault.

  • Next to "Location", click "Browse".

  • In the file browser dialog, navigate to the parent directory of the new vault. For example, on macOS I navigate to the $HOME/Documents/Obsidian/ directory. Then click "Open" at the bottom.

  • Back in Obsidian, click "Create".

For the purposes of this document, I created a vault called "sample" in the $HOME/Documents/Obsidian/ directory.

Set up the git repo

This will set up a git repository within the vault. This is all you'll need in order to start tracking changes over time.

Once this is done, you can sync the repo with another location (such as Github, Keybase, or a directory on a shared drive),. This will be covered below - but either way, you need to create the repository within the vault first.

Create the repo itself

In a command line ...

cd ~/Documents/Obsidian/sample/
git init -b main

Create a .gitignore file

This file contains a list of filename patterns that git should not add to, or update in, the repo.

Use a text editor to create a file with the name .gitignore, in the root of the repo (aka the Vault directory - technically, whatever directory you were in when you ran the git init command). I normally use nano or BBEdit for this, others may use editors like vi, vim, or Sublime Text.

The contents of the file will depend on what you do and don't want stored in the git repo. The file I use looks like this:

.DS_Store
._*

.obsidian/*
!.obsidian/app.json
!.obsidian/appearance.json
!.obsidian/config
!.obsidian/community-plugins.json
!.obsidian/core-plugins.json
!.obsidian/graph.json
!.obsidian/hotkeys.json
!.obsidian/plugins
!.obsidian/snippets
!.obsidian/themes

  • The first two entries, .DS_Store and ._*, are files that macOS creates on filesystems which don't support Apple's "resource fork" mechanism.

    I include these two entries in the .gitignore files for every repo I create.

  • The .obsidian/* entry tells git to ignore all files within the vault's .obsidian/ directory.

  • The !.obsidian/xxx entries tell git to NOT ignore those particular files. This means that those files will be tracked over time and possibly sync'ed between computers.

Create the first commit

A git "commit" records a set of changes to the files in the working directory.

git add .

This command will "stage" every file in the current directory (and its children) to be included in the next commit. This will not include any files listed in the .gitignore file, although in this case it will include the .gitignore file itself.

If you're curious, you can run git status to see which files have been staged.

git commit -m 'initial commit'

This command will create a commit, containing the files you just added. Each commit has a message explaining what the changes in the commit are doing, the -m 'initial commit' is supplying that message. Feel free to use a different message if you like.

Note that commit messages are required. If you don't include a -m option, git will open your text editor and ask you to enter a message. If you don't enter one, it will not create a commit.

Set up a remote

If you plan to sync the git repo with any remotes, now is the best time to set that up.

If you're using Github

Using the web interface

If you're using the Github web interface, create a new repo. DO NOT use any of their options to create "starter" files (such as README or .gitignore) automatically, the repo you create needs to be 100% empty.

Using the gh command line tool

If you're using the gh command line tool, the command will look like this:

gh repo create USERNAME/REPONAME --private --disable-issues --disable-wiki

  • You must include either --private or --public. You could also use --internal if the repo is owned by an organization, to make it accessible to members of the organization but not to the entire world.

  • The --disable-issues and --disable-wiki options will turn those features off when the repo is created. Other options may be available, gh repo create -h will show them to you.

Get the new repo's URL

However you created the repo, the output should give you a git remote add command line, or at least a URL which can be used in that command. The URL will look like one of these:

  • For HTTPS: https://github.com/USERNAME/REPONAME
  • For SSH: git@github.com:USERNAME/REPONAME

Github repos can be accessed using HTTPS or SSH. Both of them work, but they handle authentication differently. Using HTTPS involves either having to enter your password for every git operation, or using an "authentication helper" program which may store the password in plain text on the system. (The macOS authentication helper stores it in the system's keychain, which is encrypted.)

ℹ️ Personally I prefer to use SSH, since it uses my SSH keys to authenticate. This means that the SSH agent handles authentication requests automatically. If your SSH secret keys are stored on the computer, you'll have to enter the SSH key's passphrase the first time you use it, but after that the SSH agent will remember the key and not need to ask for the passphrase again.

In my case, the SSH agent passes the requests along to the Yubikey which has my SSH secret key on it. I have to enter the Yubikey's PIN once to unlock it, and it stays unlocked until I remove the Yubikey from the computer. This means I don't have to store my Github password or my SSH secret key on the computer.

If all you have is the URL, you can build the command line. This will be explained in the "Add the remote" section below.

Keybase

Keybase git repos can be owned by an individual user, or by a team. In the case of a user, only that user will have access to the repo. For a team, all members of the team will have access to the repo, and people can gain or lose access to the repo by being added to or removed from the team. In addition, team members with reader access will have read-only access to the repo (i.e. they can clone and fetch commits, but they won't be able to push.)

Note that Keybase's git repos do not offer features like pull requests - just cryptographically secure hosting of the git repos themselves.

Private repo

$ keybase git create obsidian-git-sample
Repo created! You can clone it with:
  git clone keybase://private/USERNAME/obsidian-git-sample
Or add it as a remote to an existing repo with:
  git remote add origin keybase://private/USERNAME/obsidian-git-sample

As you can see, this shows you the git remote add command you'll need below.

Team repo

$ keybase git create --team TEAMNAME obsidian-git-sample
Repo created! You can clone it with:
  git clone keybase://team/TEAMNAME/obsidian-git-sample
Or add it as a remote to an existing repo with:
  git remote add origin keybase://team/TEAMNAME/obsidian-git-sample

As you can see, this shows you the git remote add command you'll need below.

Add the remote to the repo

The next step is to tell the local git repo (the files in the .git/ directory) that the remote exists, and how to find it. This is done using the git remote add command.

When you created the repo, it should have given you either a URL or an actual git remote add command. If you just have a "remote string", you can put git remote add origin in front of it. (The word origin is traditionally the name of the remote, for repos which only have a single remote.)

Examples:

  • git remote add origin git@github.com:USERNAME/obsidian-git-sample (using SSH)
  • git remote add origin https://github.com/USERNAME/obsidian-git-sample (using HTTPS)
  • git remote add origin keybase://private/USERNAME/obsidian-git-sample

Push the commits

After adding the remote, the next step is to push whaetver commits (and tags) are currently in the repo, "up" to the remote.

The first time you do this, you also need to link the local main branch to the main branch you'll be creating when you push the commits.

git push -u origin main

After this, you'll be able to use git push by itself. Because the local main branch is linked to the remote main branch, git push will know which branches to push where automatically.

git push

Keybase: fix the repo on the server

Keybase's git server has a bug where it assumes that every git repo has a branch called master. If the repo doesn't have a branch with this name, then git clone won't be able to clone new copies of the repo without explicitly naming the branch they want to check out at the end of the cloning operation.

Historically, the name "master" was used as the default branch name for newly created repos. The git init command still uses this as the default, but the documentation has a note saying that this will change in the future.

Some people consider the word "master" to be offensive (because it refers to slavery) so many people (myself included) now use main as the name of a git repo's primary branch.

Keybase doesn't provide a command to change the name of a repo's primary branch, however they do provide a mechanism to manually access the files on their "git server", and if you're careful, you can change it by hand.

  • For a private repo

    echo 'ref: refs/heads/main' \
    > /keybase/private/USERNAME/.kbfs_git/obsidian-git-sample/HEAD

  • For a team repo

    echo 'ref: refs/heads/main' \
    > /keybase/team/TEAMNAME/.kbfs_git/obsidian-git-sample/HEAD

If you're using Keybase git, you should do this before anybody else (including yourself) tries to git clone the repo.

ℹ️ If you don't do this, you can specify the correct branch when cloning the repo, like so:

git clone -b main keybase://team/TEAMNAME/REPONAME

Add the obsidian-git plugin

Run Obsidian, and open the vault you want to use the obsidian-git plugin with.

  • Click on the ribbon on the left. (On macOS you can press "⌘," for this.)

  • Select "Community plugins" on the left.

  • Next to "Community plugins", click the "Browse" button.

  • Enter "git" in the search box at the top of the window. Look for the "Git" plugin.

  • Click on the plugin.

  • Click on the "Install" button.

  • After installing it, click the "Enable" button.

  • Close the settings window (the X at the top right)

Configure the plugin

Feel free to use whatever settings you like. These are the settings I'm using.

Note that the list of settings may be different from one version to the next. The list below is from 2.24.3.

Obsidian Settings ( or ⌘, )

  • Under "Community plugins" at the bottom, "git"

    Automatic

    • Split automatic commit and push: NO
    • Vault commit interval (minutes): 5
    • Auto Backup after stopping file edits: YES
    • Auto pull interval (minutes): 0
    • Specify custom commit message on auto backup: NO
    • Commit message on auto backup/commit: auto backup: {{date}} from {{hostname}}

    Commit message

    • Commit message on manual backup/commit: manual backup {{date}} from {{hostname}}
    • {{date}} placeholder format: YYYY-MM-DD HH:mm:ss
    • {{hostname}} placeholder replacement: whatever hostname you use for the current machine
      • This is specific to each machine, not sync'ed to the git repo.
    • Preview commit message
    • List filenames affected by commit in the commit body: YES

    Backup

    • Sync Method: Merge
    • Pull updates on startup: YES
    • Push on backup: YES
    • Pull changes before push: YES

    Line author information

    • Show commit authoring information next to each line: NO

    History View

    • Show Author: Hide
    • Show Date: YES

    Source Control View

    • Automatically refresh Source Control View on file changes: YES
    • Source Control View refresh interval: 7000 (default)

    Miscellaneous

    • Disable notifications: NO
    • Hide notifications for no changes: YES
    • Show status bar: YES
    • Show statge/unstage button in file menu: YES
    • Show branch status bar: YES
    • Show the count of modified files in the status bar: YES

    Commit Author

    • Author name for commit: (empty, I configure this using git config (details)
    • Author email for commit: (empty, I configure this using git config (details)

    Advanced:

    • Update submodules: NO
    • Custom Git binary path: (empty)
    • Additional environment variables: (empty)
    • Additional PATH environment variable paths 
      /usr/local/bin
/opt/homebrew/bin
/opt/keybase/bin
    • Reload with new environment variables: (click this if any changes were made to environment variables)
    • Custom base path (Git repository path): (empty)
    • Custom Git directory path (Instead of '.git'): (empty)
    • Disable on this device: NO
    • Donate: up to you, the link goes here

Clone an Existing Vault which uses obsidian-git

2024-06-30

Get URL of existing vault

  • git@github.com:USERNAME/REPONAME
  • keybase://private/USERNAME/REPONAME
  • keybase://team/TEAMNAME/REPONAME
  • etc.

Clone the repo

mkdir -p ~/Documents/Obsidian
cd ~/Documents/Obsidian
git clone keybase://team/TEAMNAME/REPONAME VAULTNAME

Open vault

  • File → Open Vault...
  • Open folder as vault - click "Open"
  • Navigate to ~/Documents/Obsidian/VAULTNAME, click "Open"
  • When asked about trusting plugins, say yes

Configure plugin

  • Settings - (⌘,) or (Obsidian → Settings)
  • On the left, under "Community Plugins" (bottom), select "Git"
  • Under "Commit message"
    • {{hostname}} placeholder replacement → identifier for this machine (not sync'ed in git repo)
  • Under "Advanced"
    • Additional PATH environment variable paths (especially if you see popups about commands not being recognized, such as gpg or keybase-remote) 
      /usr/local/bin
/opt/homebrew/bin
/opt/keybase/bin
    • Reload with new environment variables → Reload

Copy Obsidian Daily Note

2024-07-14

I use Obsidian's "Daily Notes" feature to keep a daily list of what I want to get done, and what I did get done, at work each day.

Much of my job involves working on things that nobody else can do - or at least, not as well as I can do it. (I'm not trying to brag, I've just been doing this a lot longer than most of my cow-orkers.) Tasks that would take me a few hours might take most other people a few days. Because of this, I tend to manage my own day-to-day tasks. Every morning I use email, chat, and an internal ticketing system to figure out what needs to get done. When it's not obvious, management helps me figure out which tasks need to be prioritized. And once I've got a clear picture of the day, I start working on whatever things need my attention the most, in that order.

Once or twice a week somebody will ask me what I'm working on and what's on my list for the day. This has happened a few times in the last week, and rather than stopping what I was working on to explain it to them, I just copied the current day's "Daily Note" Markdown file to a shared directory on Keybase and told them to look at that. I'm assuming this was enough for whatever they needed, because they didn't ask any more questions at the time.

Since then, when people have asked about it, I just copied the file to the same location - which saves time, but it's still an interruption to have to copy the file when people ask for it. I edit the file in Obsidian while I'm in my morning "planning" phase, and then during the day I'll add notes to the tasks, or if I'm lucky, move items to a "Done" section at the bottom of the page.

In order to avoid the interruptions (and to make sure I don't forget to copy the file after making changes), I automated the process of copying "today's" Daily Notes file to a specific filename on a shared drive. And then I found a different way to do it, that I like better, although both versions work.

Option 1: cron job

Most unix-type systems, including macOS, use a program called cron to run programs in the background on a regular basis. The programs that it runs automatically are referred to as "cron jobs". The cron "engine" program which runs the jobs, uses text files called "crontabs" to configure which commands to run at what times.

I wrote a script which copies "today's" Daily Notes file to a specific filename on a shared drive, and set up a cron job on my macOS workstation to run the script every five minutes. The script also adds a header at the top of the file which tells when the file was copied, so people can tell how up-to-date the file is.

Download the script

Save the script somewhere in your PATH. (I normally have $HOME/bin in my PATH, so I save the script there.)

Wherever you save it, make sure that the file has executable permissions.

chmod +x cron.copy.today

The crontab entry which runs the script is fairly simple. It looks like this.

1/5 * * * *     /Users/jms1/bin/cron.copy.daily

This file makes the machine run the script every five minutes, at 1, 6, 11, etc. minutes after the hour. (Obviously adjust the filename to point to wherever you

TAB characters in crontab files

Back in the day, UNIX systems required that crontab files use a TAB character between the time spec (here 1/5 * * * *) and the command itself. More recent systems allow them, but will work with spaces as well. I'm pretty sure macOS falls in the "more recent systems" category, but I've been in the habit of using TAB characters in crontab files for the past 30+ years, so I do it anyway.

You may want to check your own system's documentation to be sure. If you're going to use TABs, you should also make sure you understand how to make your text editor use TAB characters when you need them.

Option 2: Shell commands plugin

One problem with using a cron job to copy the file is that the copy can only happen at the scheduled time. When the file is updated, it may be up to five minutes before it gets copied. It's possible to change the schedule so the script runs once every minute, but cron itself can't schedule jobs any more often than that.

As it turns out, there's an Obsidian plugin called Shell commands which, as the name suggests, runs shell commands. One of the things it can be configured to do is to run commands when files are saved or updated in a vault.

I wrote a script called copy-daily-note, designed to be called from this plugin with two pieces of information: the name of the file that was updated, and the name of a file that the current day's Daily note should be copied to. When it runs, it does the following:

  • Calculate the filename of the current Daily note.

    By default this will be YYYY-MM-DD.md (for the current date, obviously) in the root directory of the vault, although both the filename and the directory are configurable. Personally, I have a "Daily Notes" folder inside the vault, and my filenames look like 2024-07-14 Sun.md because I find it helpful to have the day of the week in the filename. (The "Date format" value I use for this is YYYY-MM-DD ddd.)

  • If the filename that was updated is not the same as the one we just calculated, exit.

  • If a "checksum file" exists, and contains information about the correct file, and the checksum in that file matches the current Daily note file, exit.

  • Create an output file containing a header with the current Daily note's filename and a line telling when the file was copied, followed by the contents of the file that was updated (which we now know is today's Daily note).

  • Write a checksum file containing the checksum of the file that was updated, so the next time the script runs, it can tell if the file has changed since this time.

After being configured below, the Shell commands plugin will run this script automatically every time a file in the vault changes.

Download the script

You can save the script wherever you like, just be sure it has executable permissions.

chmod +x copy-daily-note

In the vault where I'm using this, I saved it within the vault itself, in a directory called .bin. This lets it "stay with" the vault, plus because I'm also using the obsidian-git plugin with that vault, any updates I make to the script are automatically sync'ed with the rest of the vault's contents.

The examples below will assume you're doing the same thing. If not, you'll need to adjust the path to run the script below.

cd ~/Documents/Obsidian/vaultname/
mkdir -p .bin
cp ~/Downloads/copy-daily-note .bin/
chmod 0755 .bin/copy-daily-note

Options

The script contains a few command line options which can be used to control how the script works. You may or may not need them.

  • -c ___ = Specify a checksum program. (Default sha256sum)

    The script only writes an output file if the input file has changed since the last time the output file was written. The output file is not an exact copy of the input file (it will have an extra header added to it, to tell the reader when the file was copied), so it can't directly compare the two files. Instead, the script writes a second output file containing a checksum of the original input file, and uses that checksum to tell if the input file changed or not.

    This option sets what program is used to generate and verify the checksum files. By default it uses sha256sum, but it can also use sha512sum, sha1sum, or md5sum, depending on what's available on the machines where the script will run.

    The program you specify here needs to support the same -c option that sha256sum has.

  • -d ___ = Specify a custom date format.

    The Daily notes plugin allows the user to configure the filename it builds for each day's note, as well as the directory within the vault where it stores the daily notes. This script needs to build the same filename, in the same directory. It does this by reading the "Daily notes" plugin's configuration and converting the "Date format" setting to a string which can be used with the 'date' command to build the name of "today's" Daily note.

    The library used by the Daily notes plugin uses different "tags" to format the date, than what the date command (used by this script) uses. For example, the plugin's default is YYYY-MM-DD, but the date command would use %Y-%m-%d to format a date the same way. The script has a function to convert the commonly used tags from the plugin's format to the date command's format, but it doesn't cover every possible tag - only the ones that seemed like they would be useful, and for which the date command has corresponding tags for.

    If you're using a custom date format, and the automatic conversion function doesn't work, you can use the -d option to specify the format by hand.

    Documentation

    • moment.js library, used by the "Daily notes" plugin
    • strftime(), used by the date command (and many other programs over the past 40 years)

    Examples

    Daily notes plugin-d optionNotes
    YYYY-MM-DD%Y-%m-%dDefault format
    YYYY-MM-DD ddd%Y-%m-%d %aThe format I use in my vaults

    To test a string, to make sure it looks right:

    $ date '+%Y-%m-%d %a'
2024-07-14 Sun

  • -i = Log ignored files.

    If the script is called with a file that isn't the current daily note, it will exit without doing anything. If this option is given, the script will log (or notify you) that this happened.

  • -l ___ = Specify a log file. (The option is a lowercase "L", not a digit "one".)

    This option will make the script log what it does, every time it runs.

    If the vault is sync'ed to multiple computers (using a plugin like obsidian-git, or by storing it on a sync'ed or shared drive), you should make sure that the filename you use for this option will "work" on every machine. It may be helpful to use a value like $HOME/copy-daily-note.log so the log files on each machine are created in your home directory on that machine.

    The log file should probably NOT be stored within the vault, especially if the vault is sync'ed.

  • -n = Do not include a link to this page when building output files.

    The script normally makes the "Copied from Obsidian" text in the header (before the timestamp) a link to this page, so others can read about how the file was created. If you use this option, that text will be normal text which doesn't link to anything.

Make a note of any options you'd like to use, you will need them below.

Install the Plugin

Run Obsidian, and open the vault you want to use the Shell commands plugin with.

  • Click on the ribbon on the left. (On macOS you can press "⌘," for this.)

  • Select "Community plugins" on the left.

  • Next to "Community plugins", click the "Browse" button.

  • Enter "shell" in the search box at the top of the window. Look for the "Git" plugin.

  • Click on the plugin.

  • Click on the "Install" button.

  • After installing it, click the "Enable" button.

  • Close the settings window (the X at the top right)

Configure the Plugin

  • Click on the ribbon on the left. (On macOS you can press "⌘," for this.)

  • Select "Shell commands" on the left. (It will be near the bottom of the list, under the "Community Plugins" section.)

  • On the right, along the top, make sure the "Shell commands" tab is selected.

  • Click the "New shell command" button. A new row should appear above the button.

  • In the text entry box, enter the following:

    {{vault_path}}/.bin/copy-daily-note {{event_file_path:absolute}} /output/file/name.md

    Any additional options should be added after copy-daily-note and before {{event_file_path:absolute}}.

  • Above the text entry box, click the button. On the menu which appears ...

    Debouncing (experimental)

    • Execute before cooldown: NO
    • Execute after cooldown: YES
    • Cooldown duration (seconds): 3
      • This is how long the plugin waits before running the script, after you stop typing.
    • Prolong cooldown: YES

    Execute this shell command automatically when:

    • File content modified: YES

  • Close the settings windows (the X at the top right, twice)

Changelog

2024-07-14

  • updated to include and explain both scripts
  • moved scripts to be downloads rather than inline on the page
  • added symlinks with .txt in the name, to allow web browsers to view the scripts without downloading them

2024-06-26

  • created this page

Obsidian - Live Preview Pane

2024-06-29

Problem

The first issue I found when I tried Obsidian was the fact that it doesn't have a "live preview" pane. Other Markdown editors I've used all had two panes, with an editor on the left (usually) and a live preview on the right, so that as you're typing in the editor, you see the rendered output on the right, while you're typing.

I couldn't find an option to enable this, so I asked about it on the Obsidian forum. (I am jms1 there, if that isn't obvious).

The responses contained several pieces of information that I found useful, especially the images showing what the various UI elements are called, i.e. "tab groups". However, I finally found the answer to the question on this page in Obsidian's documentation.

Solution

  • When you're looking at a document, at the top right will be an icon, either a pencil or a book, to control whether that tab is an "editing view" or a "reading view".

  • If you hold down ⌘ (or CTRL for Linux) while clicking this icon, it will open a new tab, in a different tab group (so it's visible next to the current tab), showing the same document. The new tab will be "linked" with the old tab, so if you scroll up or down in one tab, the other will scroll itself to keep them "in sync" with each other.

I also found that with the tabs linked, if I select a different document in one tab, the other tab changes itself to show the same document.

It seems pretty simple once you know how Obsidian's UI works, but it took me a few days to figure out because it isn't really obvious, and nobody reads all of the documentation up front.

Other

One "weird" thing I later noticed is that, if I create a new document, it opens a new tab for the new document, but the "live preview" in the other tab group doesn't "follow" it. This is because the new document has its own tab, and is not open in the existing "linked" tab.

My "solution" for this is to close the new tab, then when the existing "linked" tab is active again, click the new note in the file selector to open it.

PGP and GnuPG

The pages in this section have to do with PGP.

I've been using PGP since 1993.

Authentication Subkeys

2019-02-01

This document explains what authentication keys are, and how to add one to an existing GPG key pair which doesn't already have one.

Background

Subkeys and usage flags

A PGP key consists of a primary key, and usually one or more subkeys. Each key has one or more flags to tell what that key's intended uses are. The possible flags are:

  • "E" = Encryption. The public key is shared with the world, so that other people can send you secret messages. The secret key is used to decrypt messages that you receive from others.

  • "S" = Signing. The secret key is used to generate "digital signatures", which can be used to prove that you created a given message. The public key is shared with the world, so they can verify your signatures.

  • "A" = Authentication. This key pair is used to prove your identity when accessing certain types of services, such as SSH. This document talks about creating and using a subkey with this flag.

  • "C" = Certification. This key pair is used for two purposes:

    • Signing other peoples' PGP keys. This is how the "web of trust" works - if somebody "trusts" your key, and you've signed some other key, they would "trust" that other key as well, based on your signature.

    • Issuing new subkeys. All subkeys "under" a given primary key, are signed (certified) by that primary key.

    Only a PGP key's primary key can be flagged for Certification.

The "gpg --list-keys" command will show you which flags are on each of the keys.

$ gpg --list-keys jms1
...
pub   rsa4096/0x6B2EDC90B5C6DC30 2017-05-27 [SC]
      6353320118E1DEA2F38EAE806B2EDC90B5C6DC30
uid                   [ultimate] John M. Simpson <jms1@voalte.com>
uid                   [ultimate] John M. Simpson <jms1@jms1.net>
sub   rsa4096/0x297E5961AB566594 2017-05-27 [E]
...

In this case, the primary key (fingerprint ending with DC30) is flagged with "SC" (signing and certifying), and the subkey (fingerprint ending with 6594) is flagged with "E" (encryption).

Having a separate encryption key like this is a good idea, because if you suspect that the encryption key has been compromised, you can issue a new encryption key without having to create an entire new key pair and get your friends "trust" the new one.

An even more secure way to handle this is to create a separate signing subkey, so that the primary key is only used for certification. If you need to use your "secret key" from more than one machine, you can copy the secret parts of just the subkeys, without copying the secret part of the primary key, and still be able to do most day-to-day PGP tasks, without worrying about your primary key being compromised, even if somebody manges to totally take over the computer.

As an example, this is the key I currently use on a regular basis. The secret halves of the three subkeys are stored in the YubiKey I keep on my keyring, while the secret half of the primary key is only stored on an encrypted USB stick that I only access using Tails on an air-gapped laptop.

$ gpg --list-keys jms1
...
pub   rsa4096 2019-03-21 [SC] [expires: 2022-01-01]
      E3F7F5F76640299C5507FBAA49B9FD3BB4422EBB
uid           [ unknown] John Simpson <jms1@jms1.net>
uid           [ unknown] John Simpson <kg4zow@mac.com>
uid           [ unknown] John Simpson <kg4zow@kg4zow.us>
sub   rsa4096 2019-03-21 [E] [expires: 2022-01-01]
sub   rsa4096 2019-03-21 [S] [expires: 2022-01-01]
sub   rsa4096 2019-03-21 [A] [expires: 2022-01-01]
...

Yubikey

A YubiKey's OpenPGP app has three key storage locations: one for encryption, one for signing, and one for authentication. The only things stored on a Yubikey are the numeric secret key values. It doesn't use, and isn't even aware of, any names, expiration dates, usage flags, or whether a key is a primary or secondary.

When I generate a PGP key that I plan to use on a YubiKey, I specifically generate it with subkeys for encryption, signing, and authentication, all separate from the primary key, and I store the three subkeys on the YubiKey. This allows me to use the YubiKey to do everything except certification operations, without needing the primary secret key at all.

TODO

  • write another page explaining how to generate a new key in this manner

Procedure

Identify the key

In order to modify a key, you need to give the gpg command enough information to uniquely identify the key. If you're like me and have multiple keys with the same "User ID" (name and email), you will need to use the key's fingerprint to identify which key you want to update.

Because of this, I've more or less trained myself to always use fingerprints to identify keys.

Find the fingerprint of the primary key that you want to add the authentication subkey to.

$ gpg --list-keys jms1
...
pub   rsa4096/0x6B2EDC90B5C6DC30 2017-05-27 [SC]
      6353320118E1DEA2F38EAE806B2EDC90B5C6DC30
uid                   [ultimate] John M. Simpson <jms1@voalte.com>
uid                   [ultimate] John M. Simpson <jms1@jms1.net>
sub   rsa4096/0x297E5961AB566594 2017-05-27 [E]
...

This command actually returned four different keys, I'm only showing the one I'm working with below.

For this key, any of the following values can be used as a Key ID. Fingerprints can be specified either with or without "0x" at the beginning:

  • B5C6DC30 (low 32 bits)
  • 6B2EDC90B5C6DC30 (low 64 bits)
  • 6353320118E1DEA2F38EAE806B2EDC90B5C6DC30 (full 160 bits)
  • "jms1", "jms1@jms1.net", "john", or "simpson", if your keyring only contains one key contains that string in the User ID.

Notes

  • The idea is to find something which identifies exactly one key.

  • The fingerprint values shown above are the same, the shorter values are just the "low bits" from the full fingerprint. I normally use the full 160-bit fingerprint, since there are ways for a determined attacker to create a key with the same fingerprint as an existing key that they wish to impersonate.

Generate the authentication subkey

There are two ways to generate subkeys: the quick way, and the normal way. I'm going to show the normal way first. If you're already comfortable with gpg, feel free to skip ahead.

Primary secret key

This process MUST be done on a computer where the secret half of the primary key is available. In my case, this means booting up Tails on the air-gapped laptop, mounting the encrypted USB stick, and importing the backed-up copies of the secret keys I need to work on.

$ gpg --import-secret-key /media/keystore/6B2EDC90B5C6DC30.sec.asc
$ gpg --list-secret-keys jms1
sec   rsa4096/0x6B2EDC90B5C6DC30 2017-05-27 [SC]
      6353320118E1DEA2F38EAE806B2EDC90B5C6DC30
uid                   [ultimate] John M. Simpson <jms1@voalte.com>
uid                   [ultimate] John M. Simpson <jms1@jms1.net>
ssb   rsa4096/0x297E5961AB566594 2017-05-27 [E]

The prefix on the primary key will be one of the following:

  • sec means that the secret key is present in the keyring file(s).

  • sec> means that the secret key is present on a smart card (or YubiKey).

  • sec# means that the secret key is not available.

You will also see the same kinds of suffixes on the subkeys, i.e. "ssb", "ssb>", or "ssb#".

Make sure the secret key is available. You should see "sec" or "sec>".

The normal way

This example walks through how to create a subkey with the "A" flag.

Note that many of the commands and their output look almost identical, so be careful when you follow along with this process.

$ gpg --expert --edit-key 6353320118E1DEA2F38EAE806B2EDC90B5C6DC30
gpg (GnuPG/MacGPG2) 2.2.0; Copyright (C) 2017 Free Software Foundation, Inc.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Secret key is available.

sec  rsa4096/0x6B2EDC90B5C6DC30
     created: 2017-05-27  expires: never       usage: SC
     trust: ultimate      validity: ultimate
ssb  rsa4096/0x297E5961AB566594
     created: 2017-05-27  expires: never       usage: E
[ultimate] (1). John M. Simpson <jms1@voalte.com>
[ultimate] (2)  John M. Simpson <jms1@jms1.net>

It starts off by showing you the current state of the key you're working on. The primary key is marked for both signing and certification, and the one subkey is flagged for encryption.

Start by adding a new RSA subkey, using the option allowing you to set your own capabilities.

gpg> addkey
Please select what kind of key you want:
   (3) DSA (sign only)
   (4) RSA (sign only)
   (5) Elgamal (encrypt only)
   (6) RSA (encrypt only)
   (7) DSA (set your own capabilities)
   (8) RSA (set your own capabilities)
  (10) ECC (sign only)
  (11) ECC (set your own capabilities)
  (12) ECC (encrypt only)
  (13) Existing key
Your selection? 8

You will be asked which flags the new key should have. The "Current allowed actions:" line will show you which flags will be enabled, and entering S, E, or A will toggle that flag.

Turn the flags on and off as needed so that only "Authenticate" is selected, then select "Finished".

Possible actions for a RSA key: Sign Encrypt Authenticate
Current allowed actions: Sign Encrypt

   (S) Toggle the sign capability
   (E) Toggle the encrypt capability
   (A) Toggle the authenticate capability
   (Q) Finished

Your selection? s

Possible actions for a RSA key: Sign Encrypt Authenticate
Current allowed actions: Encrypt

   (S) Toggle the sign capability
   (E) Toggle the encrypt capability
   (A) Toggle the authenticate capability
   (Q) Finished

Your selection? e

Possible actions for a RSA key: Sign Encrypt Authenticate
Current allowed actions:

   (S) Toggle the sign capability
   (E) Toggle the encrypt capability
   (A) Toggle the authenticate capability
   (Q) Finished

Your selection? a

Possible actions for a RSA key: Sign Encrypt Authenticate
Current allowed actions: Authenticate

   (S) Toggle the sign capability
   (E) Toggle the encrypt capability
   (A) Toggle the authenticate capability
   (Q) Finished

Your selection? q

Next you will need to select the key length. The gpg software on your computer itself is able to work with keys with a range of sizes, however the YubiKey is only able to work with keys whose length is exactly 1024, 2048, or 4096, so I normally choose 4096.

RSA keys may be between 1024 and 4096 bits long.
What keysize do you want? (2048) 4096
Requested keysize is 4096 bits

Next you will be asked how long the key should be valid. For this example I chose "key does not expire", however I normally set expiration dates on all of my keys, so that if I later need to revoke a key and not everybody gets the revocation certificate, it will "stop working" after a reasonable length of time. (I normally renew or re-generate new keys every year.)

Please specify how long the key should be valid.
         0 = key does not expire
      <n>  = key expires in n days
      <n>w = key expires in n weeks
      <n>m = key expires in n months
      <n>y = key expires in n years
Key is valid for? (0) 0
Key does not expire at all
Is this correct? (y/N) y

At this point, gpg has the information it needs to create the new key. It will ask for confirmation one last time, and then gen

Really create? (y/N) y
We need to generate a lot of random bytes. It is a good idea to perform
some other action (type on the keyboard, move the mouse, utilize the
disks) during the prime generation; this gives the random number
generator a better chance to gain enough entropy.

sec  rsa4096/0x6B2EDC90B5C6DC30
     created: 2017-05-27  expires: never       usage: SC
     trust: ultimate      validity: ultimate
ssb  rsa4096/0x297E5961AB566594
     created: 2017-05-27  expires: never       usage: E
ssb  rsa4096/0xBA6C2A169C6C0F60
     created: 2017-11-10  expires: never       usage: A
[ultimate] (1). John M. Simpson <jms1@voalte.com>
[ultimate] (2)  John M. Simpson <jms1@jms1.net>

As you can see, there is now a second subkey, with the "A" flag.

The last remaining step is to save the new key out to disk.

gpg> save

The quick way

This one-line command does everything described above.

$ gpg --quick-add-key 6353320118E1DEA2F38EAE806B2EDC90B5C6DC30 rsa4096 auth 2021-12-31

This command will generate a new 4096-bit RSA key and add it to that existing key, with the "A" flag, and a signature expiring on 2021-12-31. You will be prompted for the passphrase of the primary key to which you are adding the subkey.

Notes

  • Subkeys have their own expiration dates, which can be different from the expiration date of the primary key to which they are attached.

  • The "normal" process only allows you to specify an expiration date as "now plus X", while the "quick" process also allows you to specify an exact date/time for the key to expire.

Convert the new key to SSH public key format

In order to use the new key as an SSH key, you need to export the public half of this new key, convert it into the format that SSH needs, and store it in the $HOME/.ssh/authorized_keys file of each machine where you want to be able to SSH in using the key.

Luckily, the gpg --export-ssh-key option does this exact thing.

$ gpg --export-ssh-key 0xBA6C2A169C6C0F60 > sshkey.pub
$ cat sshkey.pub
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC1hMmBJQ+PPYkuFuWxHiv4eV1BDXW4ZxvXkCIeZeKf
LlOc7V2MOMggV4OzHIApEvO4XzSbyuFjiTkrvOHdSrb+J1JhFnpCeYawxRz5UQiZdcN/HJNlIZK6AMvO
hiUuuUKULMhIywL3UQwZknDvUwYrWwjdnjwOZqlyEFu1jUnfVvhGgI4qAcXqd2HBx6juXen6Z2kuP7T5
4N/ZGyB5dEq07iJmXpyQ6cUJdHOY156MG5nb8J2KdY/xn+oWSRyAunDMCNtL7RtjDaaI/4u+UtG5rzGZ
UO/2TeqIubWLDyCgqF1rEhIDqMFl2XkXLoa7fMNNc+njtrwtq5yHy8nzL1NJ0PzW0wTW4h9IICVFKucZ
Yw+2jnBnT+PP7SvNe2uEYxvozb1sJ5A5MwOs7r13X50SWit5n3/Hdg3GPC/GkHWu4plkH+0wRjZLMbOQ
r4opFD/aUZdjpPVodBImfgKZwoVy4DdzZRNJRkOmR/i2iER8L6XOKB3Y7xLHlnTQj48uxaS0mxuagjDu
SrYWY2zOHsjSP78jU2i9cV1yRNa3Jz0Y4sVD5NX+qnQ6yxNOkyBA8IVfig/SnHvfStptkMdsBT4cDGYC
/me2w+OIqFvM5pRhDR1ULW5Mqef0TlALv+clnxDqhdszU7/j4F8yFeaoSD4bz7s/Rfxu5o9toFRmxejr
bw== openpgp:0x9C6C0F60

Note that this output is one big huge long line of text, I've just added line-wrapping so it looks okay in a normal web browser.

This SSH public key should be added to $HOME/.ssh/authorized_keys on each server where you want to be able to log in using the key, just like you would do with any other SSH public key.

Notes:

  • The "gpg --export-ssh-key" command needs the same kind of unique identifier that the "gpg" commands above needed, i.e. a unique portion of a User ID or a segment of the key's fingerprint.

  • In the "gpg --export-ssh-key" command, you can use either the main key ID or the authentication subkey ID, they will both produce the same output. The only time you would need to explicitly list the new authentication subkey ID is if your key pair has multiple subkeys with the "A" flag for some reason.

  • Feel free to change the comment (the "openpgp:0x9C6C0F60" in the example above). I normally change it to something like "jms1@jms1.net 2017-11-10", which tells people later on (usually myself) whose key it is, and when the key was generated.

    However, because the comments can be changed, if you ever find yourself in the position of investigating an unknown SSH public key, you should not trust any comments which may be attached to the key, and should only rely on the key itself (that long string starting with "AAAA").

Changelog

2024-06-19 jms1

  • moved page to new jms1.info site, updated header

2020-12-22 jms1

  • Changed the title to "Authentication Subkeys"
  • Updated much of the "Background" section at the top.

2020-12-20 jms1

  • Moved to jms1.info.
  • Minor formatting changes.
  • Added the "Quick" section with the "gpg --quick-add-key" command.
  • Wrote/updated descriptions for each step in the "Detailed" section.

2019-02-01 jms1

  • Initial content

Make SSH Use gpg‑agent Instead of ssh‑agent

This page contains information about how to make ssh, scp, sftp, and other commands which speak the SSH Agent protocol, plus other commands which use ssh and friends (such as git, rsync, or Thunderbird), use gpg-agent instead of ssh-agent.

The primary reason for doing this would be if your SSH secret keys are physically stored on a Yubikey or a OpenPGP card-compatible smartcard.

ℹ️ GNUPGHOME

In some places below, you will see references to a GNUPGHOME environment variable.

  • If this variable exists, it tells gpg where to find its configuration, keyring, cache, and other files. Changing the GNUPGHOME variable allows you to easily use different sets of PGP keys.
  • If this variable does not exist, gpg will use $HOME/.gnupg/ by default.

You will also see ${GNUPGHOME:-$HOME/.gnupg} in examples. For bash and many other shells, this means "if GNUPGHOME exists, use its value, otherwise use $HOME/.gnupg". If you're using a shell which doesn't handle variable expansions the same way, you may need to manually adjust some commands rather than copy/pasting them.

Setup

The setup procedure is a bit different depending on which operating system you're using. Rather than try to explain it here, I split it off into separate pages.

  • Setup - macOS - I know this is working, it's what I use myself.
  • Setup - Linux - This should be working, but I don't use Linux as a workstation enough to be sure.
  • Setup - ms-windows - I can't help you. I haven't voluntarily used ms-windows since 2002, and the most I remember is preferring "windows 2000" over "windows XP" because XP's eye-candy was ... eXtra Painful. And from what I've seen, it's only gone downhill since then.

Make Sure It Worked

Once you've gone through the setup process above, you should find that ...

  • The SSH_AUTH_SOCK environment variable points to the S.gpg-agent.ssh socket, either directly or indirectly.

    On macOS, SSH_AUTH_SOCK will point to a /var/run/com.apple.launchd.xxxxxx/Listeners file. Assuming you're using the setup procedure linked above, that file will be a symbolic link to the S.gpg-agent.ssh socket, instead of being a UNIX socket.

    $ echo $SSH_AUTH_SOCK
/var/run/com.apple.launchd.eT5Z18IJi4/Listeners
$ ls -l $SSH_AUTH_SOCK
lrwxr-xr-x@ 1 jms1  wheel  34 Mar 26 14:18 /var/run/com.apple.launchd.eT5Z18IJi4/Listeners -> /Users/jms1/.gnupg/S.gpg-agent.ssh

  • gpg-agent is running, listening on the S.gpg-agent.ssh socket.

    $ lsof -nP ${GNUPGHOME:-$HOME/.gnupg}/S.gpg-agent.ssh
COMMAND     PID USER   FD   TYPE             DEVICE SIZE/OFF NODE NAME
gpg-agent 10221 jms1    6u  unix 0xeed0f5322e7e0b39      0t0      /Users/jms1/.gnupg/S.gpg-agent.ssh

Assuming these are both true, you shouldn't need to do anything differently. You can use ssh, scp, sftp, or whatever, the same way you already do. As long as ...

  • Your SSH client works with an agent, and finds that agent using the SSH_AUTH_SOCK environment variable (I've seen a few programs that claim to work with an SSH agent, but need to be configured to find the agent using some other mechanism, like a config file. Bad design decision on their part.)
  • Your YubiKey is physically plugged into the computer
  • An SSH secret key is stored in the YubiKey

... it should "just work", the same as if the SSH secret key were stored on disk. The only difference is, instead of being asked for a passphrase, you may be asked for your YubiKey's PIN.

SSH Agent Forwarding

SSH Agent Forwarding allows processes running on a remote machine that you're SSH'd into, to forward authentication requests back to an SSH Agent process (normally ssh-agent, but for our purposes gpg-agent) running on your local machine. The gpg-agent process handles these requests the same way it handles requests from processes (like ssh) running on the local machine - it forwards the request to the YubiKey, which uses the secret key stored in its "secure element" chip to calculate the correct answer.

Or to put it more simply ... SSH Agent Forwarding allows you to SSH from one remote machine to another, without having to enter any passwords.

Another huge advantage is this: Agent Forwarding means you don't have to store copies of your SSH secret keys on any remote servers. And if your secret key is stored on a YubiKey, you don't need to store the secret key on your local workstation either.

Assuming things are working correctly and your public key is already in your user's $HOME/.ssh/authorized_keys files on the remote machines, you could do something like this:

  • On the local workstation, SSH into a remote machine called alfa.

    (user@local) $ ssh user@alfa

    There will be a prompt for the YubiKey's PIN, unless you've already entered it and the YubiKey is still "unlocked". After this, the YubiKey handles the authentication challenge and you're able to log into alfa.

    So far this is the normal SSH public-key authentication process. and "agent forwarding" isn't involved.

  • On alfa, SSH into another machine called bravo.

    (user@alfa) $ ssh user@bravo

    This is where "agent forwarding" happens.

    • sshd on bravo generates a challenge and sends it back to ssh on alfa.
    • ssh on alfa forwards that challenge back to ssh on the local workstation.
    • ssh on the local workstation forwards the challenge to gpg-agent, who hands it to the Yubikey, who answers it.
    • The response is returned to ssh on alfa, who returns it to sshd on bravo.
    • Access is granted, and you're now logged into bravo.

  • On bravo, SSH into another machine called charlie.

    (user@bravo) $ ssh user@charlie

    The same agent forwarding process happens, but instead of "bravo → alfa → local", it's "charlie → bravo → alfa → local".

Being able to "hop" from one server to another, without having to store secret keys on the intermediate servers, is one of the best reasons to use SSH agent forwarding.

The same agent forwarding mechanism is used to access git repos from a remote machine, when the repo is accessed using SSH, i.e. if the URL is something like ssh://server/repo or git@github.com:user/repo.

Using SSH Agent Forwarding

To use SSH Agent Forwarding ...

  • For a single connection:

    • Use the -A option in your ssh commands, like so:

      ssh -A user@server

  • For all connections to a single hostname:

    • In your $HOME/.ssh/config file, find or add a Host section for that hostname and add ForwardAgent yes below it.

      Host xyzzy
    ForwardAgent    yes

  • For all connections to ALL hosts:

    • In your $HOME/.ssh/config file, find or add a Host * section and add ForwardAgent yes below that.

      Host *
    ForwardAgent    yes

    ❗️ Must be the LAST Host section

    If your $HOME/.ssh/config file has a Host * section, it MUST be the last Host section in the file.

If you have enabled SSH agent forwarding and need to disable it ...

  • For a single connection:

    • Use the -a option in your ssh commands, like so:

      ssh -a user@server

  • For all connections to a single hostname:

    • In your $HOME/.ssh/config file, find or add a Host section for that hostname and add ForwardAgent no below it.

      Host xyzzy
    ForwardAgent    no

  • For all connections to ALL hosts:

    • In your $HOME/.ssh/config file, find or add a Host * section and add ForwardAgent no below that.

      Host *
    ForwardAgent    no

Note that the command line options will override anything in the config file.

Creating $HOME/.ssh/authorized_keys Files

To set up the authorized_keys file on a machine that you'll be SSH'ing into (i.e. a machine which isn't the one with the YubiKey plugged into it) ...

On your local machine (the machine with the YubiKey plugged into it, which is presumably in front of you) ...

  • Make sure the YubiKey is plugged in, and make sure things are set up correctly on the local machine.

    $ ssh-add -L
ssh-rsa AAAAB3NzaC1yc...9toFRmxejrbw== cardno:0006xxxxxxxx

    The comment at the end will contain the serial number of your YubiKey. This tells you that your local machine is using gpg-agent, and that it is talking to the YubiKey correctly. If you don't see this, STOP and figure out why not.

  • Use the ssh-copy-id command to install the public key on a remote machine.

    $ ssh-copy-id user@server.domain.xyz

    If your public key isn't already installed in that user's authorized_keys file on the remote machine, you may be prompted for your password on the remote machine.

  • SSH into that machine, making sure that agent forwarding is enabled.

    ❓ You should not be prompted for a password.

Make sure it works. On the remote machine (the one you just SSH'd into)

  • Run "ssh-add -L". If agent forwarding is working, it should show the SSH public key from your YubiKey.

    $ ssh-add -L
ssh-rsa AAAAB3NzaC1yc...9toFRmxejrbw== cardno:0006xxxxxxxx

    This tells you that agent forwarding is working. If you don't see this, STOP and figure out why agent forwarding isn't working.

  • If you like, you can edit the $HOME/.ssh/authorized_keys file (on the remote machine) and change the comment at the end of the line. If you do this, be careful not to change anything other than the comment.

    # OLD
ssh-rsa AAAAB3NzaC1yc...9toFRmxejrbw== cardno: 12_345_678

    # NEW
ssh-rsa AAAAB3NzaC1yc...9toFRmxejrbw== jms1@jms1.net 2019-03-21 hardware token

    I sometimes do this on machine where multiple SSH keys are listed in the authorized_keys file, to make it easier to tell which key is which.

If you need to set up an authorized_keys file by hand, make sure the files' permissions are correct.

  • The remote user's home directory MUST be owned by that user, and MUST NOT have permissions allowing write access to any other groups or users.

  • The .ssh directory MUST be owned by that user, and MUST NOT have permissions allowing ANY access to other groups or users.

  • The authorized_keys file MUST be owned by that user, and MUST NOT have permissions allowing write access to any other groups or users. It also should not be executable.

  • These two commands are what I use to "fix" the permissions.

    chown -R "$(id -u):$(id -g)" "$HOME/.ssh"
chmod -R a+X,go= "$HOME/.ssh"

Adding Other SSH Keys to the Agent

There may be cases where you need to use multiple SSH keys for accessing different servers, and not all of the keys are stored on YubiKeys.

SSH secret keys which are stored in files on the disk (such as id_something, without .pub at the end) can be added to gpg-agent using the same ssh-add command you would use with ssh-agent.

ssh-add ~/.ssh/id_xxx

However, there are a few things to be aware of.

  • ssh-add sends the actual secret key data to gpg-agent over the same socket that ssh and friends use to request authentication processing. (It uses a different message type.) The gpg-agent process does not read, or even know, the filename that ssh-add is reading from.

  • Because of this, gpg-agent stores its own COPIES of the secret keys.

  • It stores the secret keys in files under the $GNUPGHOME/private-keys-v1.d/ directory. When gpg-agent needs to use one of these keys to answer an authentication request, it reads them from its own storage, rather than from whatever file ssh-add originally read the secret key from.

  • The files are named using the "key grip", which is another kind of fingerprint.

  • These files contain the secret key data, encrypted using passphrases which may be different than the passphrases on the original SSH secret key files they were imported from. (You can use the same passphrases if you like.)

  • When you add a key from an SSH secret key file having a passphrase, there will be multiple passphrase prompts. Read the passphrase prompts carefully.

    • The first prompt is for the existing passphrase for the file containing the secret key. If that file doesn't have a passphrase, you will not be prompted for one.

      This prompt is generated by ssh-add itself, so it may appear on the command line, like this:

      Notice that the prompt is asking for the passphrase for id_rsa_dummy_enc.

    • The second prompt is for a new passphrase to encrypt the file in the private-keys-v1.d/ directory.

      This prompt is generated by gpg-agent, so it will probably be a pop-up window, like this:

      Notice that the prompt explains that the passphrase is "to protect the received secret key XXX within gpg-agent's key storage". (That SHA256:OF/f...TjYk blob in the middle is not the secret key, it's a hash of the secret key.)

    • If the second prompt didn't have two input fields, there may be a third prompt asking you to verify the new passphrase.

  • Once the private-keys-v1.d/ file is written, any time gpg-agent prompts you for a passphrase in order to use the key, you will need to enter the "new passphrase" rather than the original one.

When gpg-agent uses a key stored in a $GNUPGHOME/private-keys-v1.d/ file to process an SSH authentication request, it holds the SSH secret key in memory for a limited amount of time, and then deletes it from memory.

  • The default and maximum cache times can be configured using the default-cache-ttl-ssh and max-cache-ttl-ssh directives in the $GNUPGHOME/gpg-agent.conf file. See the gpg-agent(1) man page (i.e. run man 1 gpg-agent) for more details.

  • These settings do not affect keys stored in a YubiKey/PGPCard. The secret keys are never cached, because they are never present in gpg-agent to begin with.

If you get curious and inspect the files in the $GNUPGHOME/private-keys-v1.d/ directory, you will find that they contain one of ...

  • A private-key element, if the file contains a secret key without a passphrase.
  • A protected-private-key element, if the file contains a secret key with a passphrase.
  • A shadowed-private-key element, if the file contains the serial number of a YubiKey/PGPCard containing the secret key.

Removing SSH Keys from gpg-agent

One problem with gpg-agent is that it does not support deleting keys via the SSH Agent protocol. This means that the ssh-add -d (or -D) command will not remove keys from gpg-agent.

Stored on a YubiKey

If the secret key is stored on a YubiKey, unplug the YubiKey. That's all there is to it.

gpg-agent will remember the serial number of the card containing the secret key, but that's harmless. It won't remember the secret key itself, because it never has the secret key to begin with.

Totally Forget the Key

The file containing the serial numbers are harmless on their own, although they could be used to prove that a paraticular YubiKey/PGPCard was plugged into the computer at some point.

If this is an issue and you need to make gpg-agent totally forget that the keys on that YubiKey ever existed ...

  • Use gpg --card-status --with-keygrip to get the keygrips of the PGP keys stored on that YubiKey.

    $ gpg --card-status --with-keygrip
Reader ...........: Yubico YubiKey FIDO CCID
Application ID ...: D2760001240100000006199999990000
Application type .: OpenPGP
Version ..........: 3.4
Manufacturer .....: Yubico
Serial number ....: 19999999
...
Signature key ....: 77DE BB0C 8C7F BAFF 1E0E  70DC E9E4 4ED3 0E2F 2445
      created ....: 2019-03-21 19:40:23
      keygrip ....: DF5571AFEE0E2A27263ACBE4B01EDCD32DB1A000
Encryption key....: 3C8E C9C7 B067 A4C5 42F9  727D 795C 2CF8 2436 4755
      created ....: 2019-03-21 19:40:11
      keygrip ....: B384E95E0EC2599ECA8BE528FA867C812F114591
Authentication key: 7A6B 95B6 BF89 7A64 9716  5AE4 3682 3233 F8D0 9EB7
      created ....: 2019-03-21 19:40:33
      keygrip ....: 4F2C05E90C31510A194744D3F836149F74D1AE6F
...

    In this case, the YubiKey has three subkeys, with the following keygrips:

    • DF5571AFEE0E2A27263ACBE4B01EDCD32DB1A000
    • B384E95E0EC2599ECA8BE528FA867C812F114591
    • 4F2C05E90C31510A194744D3F836149F74D1AE6F

  • Under the $GNUPGHOME/private-keys-v1.d/ directory, delete the files whose names start with those keygrips.

Note that the next time gpg-agent "sees" that YubiKey/PGPCard, it will create these files again.

Loaded from a File

If you need to delete a secret key that was originally loaded from a file (using ssh-add), the secret key will be stored in one of the files under the $GNUPGHOME/private-keys-v1.d/ directory.

To delete the key from gpg-agent's key storage ...

  • Get the fingerprint of the key you want to delete.

    • Use ssh-add -l to get a list of all SSH keys the agent knows.

      $ ssh-add -l
4096 SHA256:l7CsDA23ENutkRsZ5jhlqJfl2syaiJfHni7b95e8dQ4 cardno:19_999_999 (RSA)
4096 SHA256:OF/fP3dqd07dcWHQ4RR9TyCeGZlZyvBTR2pgg9PTjYk dummy (RSA)

    • Identify the key you want to delete. In this example we're deleting the "dummy" key, the one whose description does not start with cardno:.

    • Remember its fingerprint, in this case SHA256:OF/fP3dqd07dcWHQ4RR9TyCeGZlZyvBTR2pgg9PTjYk.

  • Use gpg-connect-agent to get a list of all SSH keys the agent knows, with their keygrips.

    $ gpg-connect-agent 'keyinfo --ssh-list --ssh-fpr' /bye
S KEYINFO 07E92193AD46C650D8E9BDF0AF7B38D1300300DF D - - - P SHA256:OF/fP3dqd07dcWHQ4RR9TyCeGZlZyvBTR2pgg9PTjYk - S
OK

  • Find the keygrip. This is the long number immediately after KEYINFO, on the same line with the fingerprint you saw in the ssh-add -l output.

    In this example ...

    • SHA256:OF/fP3dqd07dcWHQ4RR9TyCeGZlZyvBTR2pgg9PTjYk is the fingerprint from the ssh-add -l output above.
    • 07E92193AD46C650D8E9BDF0AF7B38D1300300DF is the "key grip".

    Note that this command will not show key grips for keys stored on a YubiKey or PGPCard.

  • Tell the agent to delete the key.

    $ gpg-connect-agent 'delete_key 07E92193AD46C650D8E9BDF0AF7B38D1300300DF' /bye
OK

    When you enter the command, you will get a pop-up dialog asking you to confirm that you want to delete the key:

    When you click "Delete" key, a second pop-up will appear, warning you that the key is flagged for SSH.

    When you click "Delete key" on both of these, the secret key will be deleted.

  • This deletes the $GNUPGHOME/private-keys-v1.d/xxx.key file containing the encrypted key, which is enough to prevent the key from being used.

    $ ssh-add -l
The agent has no identities.
$ cd ${GNUPGHOME:-$HOME/.gnupg}/private-keys-v1.d/
$ ls -la 07E92193AD46C650D8E9BDF0AF7B38D1300300DF*
ls: 07E92193AD46C650D8E9BDF0AF7B38D1300300DF*: No such file or directory

    However, it does not remove the key grip from the $GNUPGHOME/sshcontrol file, which means that gpg-agent will still have a record that a secret key with that grip existed at one point.

    $ cd ${GNUPGHOME:-$HOME/.gnupg}/
$ cat sshcontrol
# List of allowed ssh keys.  Only keys present in this file are used
...
# RSA key added on: 2026-04-03 09:46:59
# Fingerprints:  MD5:13:e7:40:70:aa:f7:49:ab:86:71:b5:72:7d:46:3b:85
#                SHA256:OF/fP3dqd07dcWHQ4RR9TyCeGZlZyvBTR2pgg9PTjYk
07E92193AD46C650D8E9BDF0AF7B38D1300300DF 0

    $ gpg-connect-agent 'keyinfo --ssh-list --ssh-fpr' /bye
S KEYINFO 07E92193AD46C650D8E9BDF0AF7B38D1300300DF - - - - - - - S
OK

    If you want to remove all record of the key grip, edit the $GNUPGHOME/sshcontrol file and remove the line starting with that key grip, plus the comments immediately above it.

    As I mentioned above, if this is the only key grip in the file, you can remove the file. The next time gpg-agent does anything relating to SSH, it will create a new sshcontrol file containing just the comment at the top of the file.

Setting up SSH to use gpg-agent on Linux

🚨 PROCEED WITH CAUTION 🚨

I don't use Linux as a workstation very often, so parts of the information on this page are not as familiar to me as they probably should be.

Treat the information below as "something that worked for me at least once, but I don't really understand well enough to try and teach it to others".

This page covers how to set things up on a Linux machine, so that SSH commands use gpg-agent instead of ssh-agent.

The primary reason for doing this would be if your SSH secret keys are physically stored on a YubiKey or a OpenPGP card-compatible smartcard. Because you will need to insert and remove a YubiKey in the computer itself, the directions below are written with the assumption that you are at or near the machine's console.

Note that the process may be somewhat different for different Linux distributions. This page will focus on the distributions I use on a regular basis, namely CentOS 7, AlmaLinux 10, and Debian 12/13.

Quick Setup

If you're logged into the console of a Linux machine and need to use your YubiKey to authenticate outbound SSH connections, but you don't want to permanently configure the machine ...

CentOS 7

This requires permanently installing a few software packages, but will not permanently change how things work after you log out.

  • Install packages. This will need to be done once on each machine, and will make the software available for every user on the system.

    sudo yum install gnupg2 gnupg2-smime pcsc-lite pinentry

  • The first time you log into the machine, make sure the pcscd service is running. This is needed for gpg-agent to talk to a YubiKey.

    sudo systemctl start pcscd
    • This command will start the pcscd service system-wide. You may want to stop the service (using systemctl stop) when you're finished. If you don't do this, it will continue to run, but it will not start by itself when the machine reboots.

  • Configure the current shell. This will need to be done every time you open a new shell.

    export GPG_TTY="$( tty )"
eval $( gpg-agent --daemon --enable-ssh-support )

    Note that you must export the GPG_TTY variable before running gpg-agent.

AlmaLinux, Rocky, RHEL 10

I use AlmaLinux 10 on my own servers. I'm assuming that whatever I write for AlamLinux 10 will also work for Rocky 10 and RHEL 10 , but I can't guarantee anything - especially not for Rocky, which I have never used.

I don't have a "quick" process for AlmaLinux 10 yet. At first glance, it looks like more extensive changes are needed before gpg-agent will be able to talk to the YubiKey hardware. See the permanent procedure below.

Debian-flavoured distros - Debian, Ubuntu, etc.

  • Install packages. This will need to be done once on each machine, and will make the software available for every user on the system.

    sudo apt install gpg-agent scdaemon

  • The first time you log into the machine, make sure the necessary services are running.

    systemctl --user start gpg-agent.service

    Debian-flavoured systems set up gpg-agent to run as a per-user systemd service. This means that when log out of the last login session using that username, the system will stop the gpg-agent service automatically.

  • Configure the current shell session. This will need to be done every time you open a new shell session.

    export GPG_TTY="$( tty )"
export SSH_AUTH_SOCK="$( gpgconf --list-dirs agent-ssh-socket )"

Permanent Setup

Install Software

The following packages need to be installed on the machine. Note that these are the same packages you need to install if you plan to do this temporarily, as shown above.

  • CentOS 7, Fedora, RHEL 9/10, AlmaLinux 10, etc.

    sudo yum install openssh-clients gnupg2 gnupg2-smime pcsc-lite pinentry

  • Debian, Ubuntu, etc.

    sudo apt install openssh-client gpg-agent scdaemon

Maybe: configure pcscd

There are two different programs that gpg-agent can use to talk to a smartcard reader (or a YubiKey, which emulates a smartcard reader with a single permanently installed card.)

  • scdaemon, which is part of GnuPG.

    From what I remember, Debian 12 sets things up so that gpg-agent uses scdaemon, without pcscd being involved. It also sets up the necessary permissions to find and communicate with the hardware, so there may not be anything to be done for Debian-flavoured systems.

    I'm assuming that Debian 13 does the same, but I haven't installed it on a laptop yet so I can't say for sure.

  • pcscd

    RedHat-flavoured distros (including AlmaLinux, which I use on my own servers) use this.

⚠️ Caution

What I've written below seems to work, or did at one time, possibly only on one machine. The truth, is I don't understand these programs as well as I probably should. It's possible that some of the details below are incomplete or incorrect, or that I entirely missed something important.

FWIW, this page from the Sequoia PGP documentation, has a fairly clear expanation of how scdaemon and pcscd interact and how to make the two work together.

On RedHat-flavoured systems (AlmaLinux 10 etc.), the following steps need to be done, as the root user:

  • Make the pcscd service start automatically when the system boots. (Also start it now, if it isn't already running.)

    systemctl enable --now pcscd.service

  • RHEL 10 and AlmaLinux 10 use Polkit to control which processes are able to connect to pcscd. (This may be true of other distros, I don't know.) On these systems, you need to configure Polkit to allow certain users to access cards managed by pcscd.

    • Create a user group called plugdev and add the appropriate user(s) to it.

      useradd -r plugdev
usermod -a -G plugdev jms1

      If any of these users are currently logged in, they will need to log out and log back in before their shell will know that they belong to the new group.

    • Create /etc/polkit-1/rules.d/90-pcsc.rules with a rule allowing members of the plugdev group, to access pcscd and the cards it manages. The file looks like this:

      cat > /etc/polkit-1/rules.d/90-pcsc.rules <<EOF
// allow members of group 'plugdev' to access pcsc devices
polkit.addRule(function(action, subject) {
    if ( (    action.id === "org.debian.pcsc-lite.access_pcsc"
           || action.id === "org.debian.pcsc-lite.access_card" )
         && subject.isInGroup("plugdev") )
    {
        return polkit.Result.YES;
    }
    return polkit.Result.NOT_HANDLED;
});
EOF

      Note that the filename doesn't have to be exactly this, but the file does need to be in this directory.

      The word debian in this file doesn't mean that the file is only needed on Debian-flavoured systems. This is the exact file I'm using on the AlmaLinux 10 system where I tried this stuff while writing this page.

      The word debian is probably there because they had something to do with ... writing pcsc-lite? writing its integration with Polkit? ... I'm not sure, if anybody knows please let me know.

    • If any YubiKeys are currently pluged in, remove and re-insert them.

To make sure it worked, as your non-root user, run this command:

gpg --card-status

The first few lines of the output should tell you what kind of card reader it is and the "Application ID" of the OpenPGP application on the card.

Reader ...........: Yubico YubiKey FIDO CCID 00 00
Application ID ...: D2760001240100000006199999990000
Application type .: OpenPGP
Version ..........: 3.4
Manufacturer .....: Yubico
Serial number ....: 19999999
...

Configure gpg and gpg-agent

In the home directory of each user who will need to use this ...

  • $HOME/.gnupg/gpg.conf should contain the following line:

    use-agent

    This tells the gpg command to use gpg-agent for operations involving secret keys. This is the default behaviour for GnuPG 2.0 and later, but it doesn't hurt anything to be sure.

  • $HOME/.gnupg/gpg-agent.conf should contain the following line:

    enable-ssh-support

    This tells gpg-agent to always start the extra listener which "speaks" the ssh-agent protocol.

Maybe: XFCE

I normally use XFCE as my desktop envrionment for Linux workstations. Under XUbuntu 18.04 and Debian 12 its default configuration starts ssh-agent automatically while setting up the desktop session (i.e. every time I log into a GUI session). This was causing problems for me, so I ended up configuring XFCE not to do this.

The commands I used were:

xfconf-query -c xfce4-session -p /compat/LaunchGNOME -n -t bool -s false
xfconf-query -c xfce4-session -p /startup/ssh-agent/enabled -n -t bool -s false

This may or may not be necessary for you, I'm including the information here because I need to remember this when setting up a new Debian workstation.

If you're curious, this document was my checklist for setting up Xubuntu, back in 2020. I'm not sure that the entire document is still useful, because my Linux laptops are now running Debian 12 or 13. At some point I plan to update this document for Debian and include it on this site.

Unfortunately Keybase Sites doesn't render Markdown to HTML, but Markdown is pretty easy to read on its own.

Configure your shell

You need to make sure that your shell always sets the GPG_TTY and SSH_AUTH_SOCK variables correctly.

I do this by including something like this in my .bashrc file:

export GPG_TTY="$( tty )"

ZZZ_SOCK="$( gpgconf --list-dirs agent-ssh-socket )"
if [[ -n "${ZZZ_SOCK:-}" ]]
then
    export SSH_AUTH_SOCK="$ZZZ_SOCK"
fi
unset ZZZ_SOCK

Once you have added this, every new interactive shell will have the correct values for these two variables. Note that setting the variables in this way will only affect shells and any processes started from those shells. In particular, it will NOT affect processes started by something other than your shell, such as cron jobs.

If your YubiKey contains an SSH secret key, you can test it all by running ssh-add -L. The output should be your SSH public key, in the same format needed by an authorized_keys file, with the YubiKey's serial number as the comment at the end.

$ ssh-add -L
ssh-rsa AAAAB3NzaC1yc2EAAAAD...evwOVnvi1eTUAkjIPw== cardno:19_999_999

Final Test

Reboot the machine, log in on the console, insert the YubiKey, and run ssh-add -L.

Setting up SSH to use gpg-agent on macOS

These directions will configure your user account on the machine so you don't have to manually do anything after logging in, other than remembering to plug in your YubiKey.

Background

ℹ️ Feel free to skip this part if you like.

In macOS, a "LaunchAgent" is a configuration which starts a process or runs a command automatically, when a user logs in. These are different from a "LaunchDaemon", which starts a process or runs a command when the system boots, regardless of which user logs in (or if a user logs in).

Part of macOS is a LaunchAgent which does the following, every time a user logs in:

  • Creates a UNIX socket with a dynamic name, and sets things up so that ssh-agent is automatically started, listening on on that socket, the first time a process connects to the socket. (If multiple users are logged in, each user will have their own socket and their own ssh-agent process.)

  • Exports an SSH_AUTH_SOCK environment variable whose value is the path to that dynamically generated socket. Every new process started by that user inherits this environment variable.

    This environment variable is used by ssh, scp, sftp, and other commands, to find the UNIX socket it uses to talk to an SSH agent. This is normally the ssh-agent process that launchd starts, the first time a client connects to the socket.

We need to change things around so that the SSH_AUTH_SOCK variable points to a UNIX socket where gpg-agent is listening.

My original idea was to change the value of the SSH_AUTH_SOCK variable itself. I did figure out how to do this automatically when the user logs in, by disabling or removing the built-in LaunchAgent which runs ssh-agent. However, OS X 10.11 "El Capitan" and later introduced some security measures which make this difficult, or (starting with macOS 11 "Big Sur") impossible.

While I was hunting for information about how to disable this LaunchAgent in macOS 10.15 "Catalina", I found this article which explained a different way to solve the problem. Instead of disabling the macOS LaunchAgent (which we cannot do in macOS 11 and later), we can replace the UNIX socket that launchd sets up, with a symbolic link (or "symlink") to the UNIX socket where gpg-agent is listening for SSH agent requests.

This way, any client which uses the SSH_AUTH_SOCK value to connect to an SSH agent, follows the symlink and ends up actually talking to gpg-agent instead.

The other thing is, we need to create this symlink every time the user logs in. The simplest way to do this is using a per-user LaunchAgent, since they run every time the user logs in.

This depends on the system LaunchAgent being run before our new LaunchAgent, since the new LaunchAgent needs the name of the UNIX socket created by the system LaunchAgent.

It looks like launchd always processes the system LaunchAgents before user LaunchAgentsgit status , but I haven't seen any official documentation which says that. While I've never seen it happen, I'm not totally convinced that the new LaunchAgent won't accidentally run before the system LaunchAgent.

Older OS/X or macOS Versions

The process below works with macOS 26. It should also work with earlier macOS versions, however it requires installing packages from Homebrew, which is currently removing support for older macOS versions. See this page for more details.

If you're using an older OS/X or macOS version for which Homebrew is not available, you'll have to find other ways to install GnuPG and a "pinentry" program. Possible options (which I have not tested, or used in several years) include ...

  • GPGTools has a macOS package containing GnuPG. It includes gpg and other command line tools, plus some GUI tools. (I don't know if this includes a "pinentry" program.)

    It also comes with a Mail.app plugin that needs to be paid for if used for more than 30 days. From what I remember, payment is not required if you're not using the Mail.app plugin.

    Their web site says it supports macOS 10.15 "Catalina" and later.

  • MacPorts is another package manager that you might use instead of Homebrew.

    Installers for the latest version (MacPorts 2.12.5, as of 2026-07-25) are available for OS/X 10.5 "Leopard" through macOS 26 "Tahoe".

    Thier web site says that most of the packages they install, target macOS 13 "Ventura" and later.

  • Fink is another possible alternative to Homebrew. Their web site looks like it hasn't been updated in a while, it currently (2026-07-25) says they're "working on" support for macOS 11-13. It looks like the packages that Fink installs are still being updated.

The final option, as always, is to compile the software from source, using a version of XCode which supports your OS/X or macOS version.

I mention this because I own several older Macs which still physically work just fine, even though nobody is writing or updating software for them anymore. (If you're curious, the oldest working machine is a Titanium PowerBook G4.)

Pre-requisites

These steps need to be done once for each machine.

  • Install Homebrew.

    Homebrew is a "package manager" for macOS, similar to systems like yum or apt for Linux. It makes installing other software a LOT easier.

    /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"

  • Install the gnupg and pinentry-mac Homebrew packages.

    brew install gnupg pinentry-mac

  • Configure gpg-agent.

    mkdir -p $HOME/.gnupg
cat > $HOME/.gnupg/gpg-agent.conf <<EOF
enable-ssh-support
pinentry-program        $( which pinentry-mac )
EOF

Manual Process

The following things need to be done every time you log into the Mac.

  • Configure gpg-agent.

    • Get the full path to the pinentry-mac executable.

      which pinentry-mac

      You may need this in the next step.

    • Check the ${GNUPGHOME:-$HOME/.config}/gpg-agent.conf file. It should contain the two lines listed in the Pre-requisites section above.

      enable-ssh-support
pinentry-program        /path/to/pinentry-mac

    • If gpg-agent is running, restart it.

      gpg-connect-agent killagent /bye
gpg-connect-agent /bye

  • Set up the SSH_AUTH_SOCK symlink.

    • Make sure the SSH_AUTH_SOCK variable is not empty.

      echo "$SSH_AUTH_SOCK"

      It will normally point to a /var/run/com.apple.launchd.xxxxxx/Listeners file.

    • Create or fix the SSH_AUTH_SOCK symlink.

      rm -f "$SSH_AUTH_SOCK"
ln -s "${GNUPGHOME:-$HOME/.gnupg}/S.gpg-agent.ssh" "$SSH_AUTH_SOCK"

Automated Process

I've written a script called ssh-gpg-agent which automates the manual process above.

It can also create a "Launch Agent" which runs the script when you first log in.

ℹ️ Why create a LaunchAgent?

You might think that adding a line to your .bashrc file which runs the script, would solve the problem. You would be close, but not entirely correct.

The .bashrc file is processed when an interactive shell starts. This means when you first log into your Mac, the .bashrc file isn't processed until you start a Terminal (or in my case, iTerm2) session, and then it's processed again every time you open a new shell in a new window or tab.

LaunchAgents are "Login Items", which are executed one time, as part of the login process, without the user having to do anything. This is the same mechanism used by a lot of other programs which start automatically when you log in. If you install a LaunchAgent, you will see it in the list of Login Items, in the System Preferences app, and can disable it there if needed.

Download and install the script

  • Download the ssh-gpg-agent script. Save it in a directory which is in your PATH.

    • I use $HOME/bin/ for this, since my .bashrc already adds this directory to my PATH. Another commonly used directory for this is $HOME/.local/bin/. It doesn't really matter where you store it, as long as it's in your PATH so you can run it by hand when needed.

  • Make sure the file is marked as executable.

    chmod 0744 "$HOME/bin/ssh-gpg-agent"

Using the script

Once the script is installed, there are three ways to run it.

  • Running the script with no options will configure gpg-agent and set up the SSH_AUTH_SOCK symlink, as explained above.

  • Running the script with the -i option will install a LaunchAgent which runs the script as part of the login process.

  • Running the script with the -r option will remove the LaunchAgent, if that becomes necessary. Note that this will not undo the symlink change or "un-configure" gpg-agent - it just means that the next time you log in, the script's changes (primarily the symlink change) will not happen, and that login session will use ssh-agent (unless you run the script by hand, of course).

  • Running the script with the -h option will show a list of the other options the script supports.

There is also a -l option which lets you specify a log file where the script will record what it's doing. If you combine this with the -i option, the installed LaunchAgent will record its activities to that log file.

ssh-gpg-agent -il "$HOME/ssh-gpg-agent.log"

Note that the LaunchAgent (the $HOME/Library/LaunchAgents/net.jms1.ssh-gpg-agent.plist file) will contain the full path to wherever you installed the script. If you need to move it later, remove and re-add the LaunchAgent.

Test

If you already have a YubiKey with an SSH key loaded, you should verify that things are working.

  • If you installed the LaunchAgent, log out and log back into your Mac. Rebooting will also accomplish this.

  • Open a shell (Terminal, iTerm2, etc.)

  • If you didn't install the LaunchAgent, run the script with no options.

  • If the SSH key stored on the YubiKey is also stored in id_xxx files, move them to a different directory for now. Once the test is working and you're comfortable with it, you can remove that other directory so the secret key isn't present on the machine at all (which is, after all, the whole point of storing your secret keys on a YubiKey to begin with).

    $ mkdir ~/hold-keys
$ mv -v ~/.ssh/id_* ~/hold-keys/
$ chmod -R go= ~/hold-keys/

  • With the YubiKey NOT inserted, make sure the SSH agent doesn't see any keys. (In this command, -l is a lowercase "L", not the digit "one".)

    $ ssh-add -l
The agent has no identities.

    If there are any keys on the list, they will be keys loaded from id_xxx files.

  • Insert the YubiKey, and make sure the agent DOES see the key stored on the YubiKey. (In this command, -l is a lowercase "L", not the digit "one".)

    $ ssh-add -l
4096 SHA256:l7CsDA23ENutkRsZ5jhlqJfl2syaiJfHni7b95e8dQ4 cardno:19_999_999 (RSA)

    If there were any keys on the first list (with the YubiKey unplugged), the same keys will be present on the second list. What you're looking for is the key whose comment starts with cardno: ... it should not be there when the YubiKey is unplugged, and it should be there when the YubiKey is plugged in.

You may notice that the SSH public key doesn't have the same "comment" that you might normally see after a key. The YubiKey only stores the secret key, and the public key is derived from that. It doesn't store any kind of metadata about the key. The comment it shows will contain the serial number of the YubiKey.

Puppet

I've been using Puppet to build and manage servers at work since 2013.

Puppet - ensure_packages()

2022-02-24

I've been using Puppet to manage systems at work for about nine years now. We use Puppet to install pretty much everything on the servers we sell to our customers, as well as managing our internal infrastructure systems.

Our servers have several scripts on them, written in Perl and Python. Most of these scripts require external libraries to run. For the most part, these libraries can be installed using OS packages. such as .rpm or .deb files, depending on the OS.

One problem I've been running into for years is, how to manage installing these packages along with the scripts which need them. Note that installing the packages is actually pretty simple, it's just a normal package declaration:

  package { 'perl-Digest-SHA' :
    ensure => present ,
  }

Puppet only allows each package to be declared once, across all of the classes which are used to build the machine's catalog. Each script needs its own set of libraries, and each machine may need a different collection of scripts. Ultimately I need to make sure that each machine has the right collection of libraries for the scripts which that machine uses.

History

(Feel free to skip this section if you like.)

When I started, I just declared the library packages with the scripts. At the time all of the scripts were declared in the same class, so it was fairly simple - until we started needing to build a different type of server which used some of the scripts but not all of them (and in fact, needed to not contain some of the others for security reasons).

Try #1: Libraries Class

My first "solution" was to move all of the library package declarations to a single libraries class, which is used on every type of machine we manage. The resulting machines work, however ...

  • Any time a script is added, and it has a new dependency, that dependency ends up being installed on every machine we manage, whether that package is needed on the machine or not.

    Our production machines run in environments that I will only describe as "highly regulated". Whenever Puppet installs or updates software on a server, it shows up in that system's logs, and the facilities' IT security people ask questions. There have been a few times times when I've been given a list of every package installed on a machine and told to provide an explanation of why each package is needed. Trying to explain to a client that a package was installed on their servers, because some other client's servers needed it ... not the most pleasant conversation.

  • Any time a script is added, or an existing script is updated and has a new list of required libraries, I have to look in two placess to update things - the class which installs the script, and the libraries class which installs all of the library packages.

  • When a script is removed, I have to do one of two things:

    • Figure out which library packages the script was using, and then figure out if any other scripts needed that particular package, in order to tell if it's safe to delete that particular package declaration from the Puppet code.

      I tried to leave comments in the Puppet code in both classes, to help me figure this out later, but when other people are writing the scripts and don't tell you that an update added a dependency, because it happened to have been one which was already being installed by the libraries class, I never get to update the comments for that library package to show that the "xyz script" now also needs that particular package.

    • Leave the package declaration in the "libraries" class forever, even if nothing really needs it anymore. This is what I eventually ended up doing, however it has resulted in a lot of unnecessary packages on the servers. Luckily the packages are not huge, and the systems involved have enough storage so it isn't a problem.

As you can see, it's a lot of manual management that I've always thought, shouldn't be needed.

Try #2: Manually check before declaring

Another thing I tried was having the Puppet code explicitly check whether the package had already been included in the catalog, before declaring it. The package declarations ended up looking like this:

  if ( ! defined( Package['perl-Digest-SHA'] ) ) {
    package { 'perl-Digest-SHA' :
      ensure => present ,
    }
  }

This works, and can be used to "declare if needed" each package, in the same class where the script itself is declared. However, converting to this scheme would involve adding these blocks for every package, for every script, across all of the classes, and then removing the libraries package. This is possible, but it would be tedious and time-consuming, and I'm the only person doing Puppet programming. Unless it's something that makes an improvement that our clients can see, I can't justify spending the time on doing it.

The ensure_packages() function

Earlier today I asked about this (without so much detail) in the "Puppet Community" slack server, and some kind person pointed me to the ensure_packages() function in the puppetlabs/stdlib module. I vaguely remember being aware of this function several years ago, but the description at the time didn't really explain much, and I was looking for something else, so it didn't really "click" - and as a result, I didn't realize how useful it could be. (The current description looks to be identical to the one I saw years ago, i.e. "still not great", which is part of why I'm wrting this page.)

⚠️ Removed

It looks like the ensure_packages() function has been removed from the puppetlabs/stdlib module, some time after version 8.1.0.

The function is actually a wrapper around another function called ensure_resources(), which does more or less what the "Manually check before declaring" code shown above does - it checks the catalog-in-process and, if the given resource hasn't already been declared, it adds it to the catalog, just as if it had been declared in the Puppet code.

It's probably easier to see with an example, so ... let's assume that we're going to install two scripts, "/usr/local/bin/abc" and "/usr/local/bin/xyz". They're written in Perl. One contains the following "use" lines at the top, which "link in" the libraries when the script starts ...

# from /usr/local/bin/abc

use Digest::SHA qw ( hmac_sha256_hex ) ;
use IO::Socket::SSL ;
use JSON ;
use LWP ;
use Sys::Hostname ;

The other script contains these "use" lines:

# from /usr/local/bin/xyz

use IO::Socket::SSL ;
use JSON ;
use LWP ;
use Sys::Hostname ;

In order for these script to run, the libraries need to be installed. We don't have to worry about Sys::Hostname since it's a "core" library, installed as part of the perl package, however the other libraries do need to be installed.

Ideally, we'd like to be able to have something in the same Puppet class which installs the script itself, and we'd like to be able to list all of the packages each script needs, even if multiple scripts happen to need the same packages.

We could do something like this, i.e. "Manually check before declaring" ...

  ########################################
  # Install the abc script and its dependencies

  $pkg_abc = [ 'perl-Digest-SHA' , 'perl-IO-Socket-SSL' , 'perl-JSON' , 'perl-libwww-perl' ]

  $pkg_abc.each |$x| {
    if ( ! defined( Package[$x] ) ) {
      package { $x :
        ensure => present ,
      }
    }
  }

  file { '/usr/local/bin/abc' :
    ensure => file ,
    owner  => 'root' ,
    mode   => '0744' ,
    source => 'puppet:///modules/${module_name}/usr/local/bin/abc' ,
  }

  ########################################
  # Install the xyz script and its dependencies

  $pkg_xyz = [ 'perl-IO-Socket-SSL' , 'perl-JSON' , 'perl-libwww-perl' ]

  $pkg_xyz.each |$x| {
    if ( ! defined( Package[$x] ) ) {
      package { $x :
        ensure => present ,
      }
    }
  }

  file { '/usr/local/bin/xyz' :
    ensure => file ,
    owner  => 'root' ,
    mode   => '0744' ,
    source => 'puppet:///modules/${module_name}/usr/local/bin/xyz' ,
  }

While this works, having those each constructs above every single script gets kind of tedious, and if you're not careful it makes it really easy to make mistakes (ask me how I know).

Instead, we can do this ...

  ########################################
  # Install the abc script and its dependencies

  ensure_packages( 'perl-Digest-SHA' )
  ensure_packages( 'perl-IO-Socket-SSL' )
  ensure_packages( 'perl-JSON' )
  ensure_packages( 'perl-libwww-perl' )

  file { '/usr/local/bin/abc' :
    ensure => file ,
    owner  => 'root' ,
    mode   => '0744' ,
    source => 'puppet:///modules/${module_name}/usr/local/bin/abc' ,
  }

  ########################################
  # Install the xyz script and its dependencies

  ensure_packages( 'perl-IO-Socket-SSL' )
  ensure_packages( 'perl-JSON' )
  ensure_packages( 'perl-libwww-perl' )

  file { '/usr/local/bin/xyz' :
    ensure => file ,
    owner  => 'root' ,
    mode   => '0744' ,
    source => 'puppet:///modules/${module_name}/usr/local/bin/xyz' ,
  }

This is easier to see, it's easier to understand, and it's easier for a junior person (or a "programmer but not a Puppet programmer") to maintain without having to constantly worry about typos.

Changelog

2024-06-19 jms1

  • moved page to new jms1.info site, updated header
  • added note about ensure_packages() no longer existing

2022-02-24 jms1

  • Initial version

Random

Upside-Down Characters

2020-12-24

A friend saw some "upside down writing" on a social networking site and wanted to know how to do the same thing.

˙ʇxǝʇ uʍop-ǝpᴉsdn ɟo ǝldɯɐxǝ uɐ sᴉ sᴉɥ┴

I've seen this kind of thing before, it's just substituting characters which "look" upside down (like "Y" and "⅄"). The trick is knowing what the upside-down version of each character is to begin with.

I pointed my friend to an online converter (I think it was http://www.upsidedowntext.com but there are several out there), but at the same time I decided that I wanted a list of them for myself as well, in case I want to embed them into a web page or something... hence this page.

If it isn't obvious, all codes below are hexadecimal.

OrigCharUpsideDnCode HexCode DecNormal
21!¡&#x00A1;&#161;
22",&#x002C;&#x002C;&#44;&#44;,,
23##&#x0023;&#35;#
24$$&#x0024;&#36;$
25%%&#x0025;&#37;%
26&ⅈ&#x2148;&#8520;
27',&#x002C;&#44;,
28()&#x0029;&#41;)
29)(&#x0028;&#40;(
2A**&#x002A;&#42;*
2B++&#x002B;&#43;+
2C,'&#x0027;&#39;'
2D--&#x002D;&#45;-
2E.˙&#x02D9;&#729;
2F//&#x002F;&#47;/
3000&#x0030;&#48;0
311Ɩ&#x0196;&#406;
322ᄅ&#x1105;&#4357;
333Ɛ&#x0190;&#400;
344ㄣ&#x3123;&#12579;
355ϛ&#x03DB;&#987;
3669&#x0039;&#57;9
377ㄥ&#x3125;&#12581;
3888&#x0038;&#56;8
3996&#x0036;&#54;6
3A::&#x003A;&#58;:
3B;;&#x003B;&#59;;
3C<>&#x003E;&#62;>
3D==&#x003D;&#61;=
3E><&#x003C;&#60;<
3F?¿&#x00BF;&#191;
40@@&#x0040;&#64;@
41A∀&#x2200;&#8704;
42Bq&#x0071;&#113;q
43CƆ&#x0186;&#390;
44Dp&#x0070;&#112;p
45EƎ&#x018E;&#398;
46FℲ&#x2132;&#8498;
47Gפ&#x05E4;&#1508;
48HH&#x0048;&#72;H
49II&#x0049;&#73;I
4AJſ&#x017F;&#383;
4BKʞ&#x029E;&#670;
4CL˥&#x02E5;&#741;
4DMW&#x0057;&#87;W
4ENN&#x004E;&#78;N
4FOO&#x004F;&#79;O
50PԀ&#x0500;&#1280;
51QQ&#x0051;&#81;Q
52Rɹ&#x0279;&#633;
53SS&#x0053;&#83;S
54T┴&#x2534;&#9524;
55U∩&#x2229;&#8745;
56VΛ&#x039B;&#923;
57WM&#x004D;&#77;M
58XX&#x0058;&#88;X
59Y⅄&#x2144;&#8516;
5AZZ&#x005A;&#90;Z
5B[]&#x005D;&#93;]
5C\\&#x005C;&#92;\
5D][&#x005B;&#91;[
5E^^&#x005E;&#94;^
5F_‾&#x203E;&#8254;
60```,&#x002C;&#44;,
61aɐ&#x0250;&#592;
62bq&#x0071;&#113;q
63cɔ&#x0254;&#596;
64dp&#x0070;&#112;p
65eǝ&#x01DD;&#477;
66fɟ&#x025F;&#607;
67gƃ&#x0183;&#387;
68hɥ&#x0265;&#613;
69iᴉ&#x1D09;&#7433;
6Ajɾ&#x027E;&#638;
6Bkʞ&#x029E;&#670;
6Cll&#x006C;&#108;l
6Dmɯ&#x026F;&#623;
6Enu&#x0075;&#117;u
6Foo&#x006F;&#111;o
70pd&#x0064;&#100;d
71qb&#x0062;&#98;b
72rɹ&#x0279;&#633;
73ss&#x0073;&#115;s
74tʇ&#x0287;&#647;
75un&#x006E;&#110;n
76vʌ&#x028C;&#652;
77wʍ&#x028D;&#653;
78xx&#x0078;&#120;x
79yʎ&#x028E;&#654;
7Azz&#x007A;&#122;z
7B{}&#x007D;&#125;}
7C``|&#x007C;&#124;
7D}{&#x007B;&#123;{
7E~~&#x007E;&#126;~

Changelog

2024-06-19 jms1

  • moved page to new jms1.info site, updated header

2021-01-02 jms1

  • wrote a script to generate the table (saved a LOT of copy/pasting)
  • added decimal codes
  • moved "normal" upside down characters to their own column

2020-12-26 jms1

  • added "normal" characters where possible

2020-12-24 jms1

  • initial version

Shell Scripting

I've been writing shell scripts since the early 1990's. Over the years I've built up a collection of useful scripts, functions and one-liners, that others might find useful.

My Shell Setup

I use bash on all of my machines, mostly because I've been using it for the past 20+ years and it's what I'm used to. I also find it useful to be able to be able to type the same commands I use in my scripts, and know that the shell I'm typing them into will do exactly what the shell that executes the script will do.

My .bashrc file is fairly small, however it includes this:

for F in "$HOME"/.bashrc.d/*
do
    if [[ -f "$F" ]]
    then
        source "$F"
    fi
done

This lets me add files to my $HOME/.bashrc.d/ directory, where they will be included while the $HOME/.bashrc file is being processed. The filenames within the directory are automatically sorted by name when the shell expands the * into a list of filenames, so the filenames control what order the files are processed.

cheat

My memory isn't perfect, I'm finding I have to look up the documentation for commands I use all the time, just to remember what the command line options are or what they do. Just as an example, the rsync command has half a dozen different "delete" options, based on whether you want to delete files on the source or on the destination, and to do the deleting before or after copying the file.

I found a program called cheat which lets you create "cheat sheets" with quick reminders of just the details you need for the commands you need them for, and it has become a real time-saver for me.

They also have a collection of cheatsheets, maintained by the community, with information that others find useful.

The program was designed to hold information about command line options that people might forget, however it can be used for just about anything that can be stored as plain text. As an example, you could use it to store a list of peoples' email addresses, phone numbers, social media names, or other contact information.

Example

As a more concrete example ... I store my shell scripting template as a cheatsheet file. When I need to start a new shell script, I run something like this:

cheat template > new-script

This creates the "skeleton" of a shell script, with most of the functions I end up using in 95% of my scripts, arleady in place. This lets me concentrate on making the script do what I need it to do, and then once it's working, I remove the bits I ended up not needeing.

My Cheatsheets

I've started a small collection of cheatsheets. In addition to recording command line options, I'm also using it to store functions that I use when writing shell and Perl scripts.

In particular, my template cheatsheet is what I use to start a new shell script, it already contains most of the functions I end up using for shell scripts. Whatever I end up not needing for the script I'm writing, I normally remove when the script is finished.

I don't know that they'll be useful to anybody else, but if you're curious, there's a copy of them on Github.

⇒ kg4zow/jms1-cheatsheets

My Configuration

The cheat program itself can be configured with multiple collections of cheatsheets, and each one can be flagged as "read-only" or "read-write". I find it useful to use both the community cheatsheets and a collection of my own cheatsheets. I set this up on my workstations using ...

########################################
# clone community cheatsheets

mkdir -p ~/git/cheat
git clone https://github.com/cheat/cheatsheets ~/git/cheat/cheatsheets

########################################
# clone my own cheatsheets and set up github additional target

git clone foks://(redacted)/cheatsheets ~/git/jms1-cheatsheets
cd ~/git/jms1-cheatsheets/
git remote set-url --add origin git@github.com:kg4zow/jms1-cheatsheets

Once the repos are cloned, I set up the config file for cheat using the following:

cheatpaths:
  - name     : community
    path     : /Users/jms1/git/cheat/cheatsheets
    tags     : [ community ]
    readonly : true

  - name     : jms1
    path     : /Users/jms1/git/jms1-cheatsheets
    tags     : [ jms1 ]
    readonly : false

With this configuration ...

  • When I type cheat xxx, it looks for a cheatsheet called xxx in both collections.

    • If it only exists in one collection, it shows the contents of that file.
    • If it exists in both collections, it shows the contents of the file from my personal collection.

    I couldn't find any documentation which explicitly says this, but it looks like it uses the last matching file.

  • When I type cheat -e xxx (to edit a cheatsheet file) ...

    • If the file exists in my personal collection, it runs my editor (normally BBEdit) against that file.
    • If the file exists in the community collection, it copies that file to my personal collection and opens my copy for editing.
    • If the file doesn't exist in either, it opens the editor with the filename where the file belongs in my personal collection. When the editor saves the file, that creates it in my collection.

Other Notes

  • When I edit cheatsheets, I need to remember to commit and push the changes in my git repos, and then run git pull on my other workstations.

    Part of this could be automated using a cron job that runs git pull, however committing and pushing has to be a manual process, since I don't want to commit/push until I know the changes are correct. (I make typos just like everybody else.)

  • On my work machines, there is a third cheatsheet collection, shared with members of the team I work with. cheat has a -p option to specify which collection to use, especially when creating or editing cheatsheet files.

    You can use cheat -h to see all of its options. Or run cheat cheat, if the community cheatsheets repo is part of your configuration. 😎

My Cheatsheets

The "master copy" of my cheatsheets are stored in an encrypted FOKS git repo, with a Github repo as an additional push target, so that whenever I push changes to FOKS, Github is updated at the same time.

$ cd ~/git/jms1-cheatsheets
$ git remote -v
origin  foks://(redacted)/cheatsheets (fetch)
origin  foks://(redacted)/cheatsheets (push)
origin  git@github.com:kg4zow/jms1-cheatsheets (push)

If you're curious, you're welcome to look at the Github repo.

Coloured line functions

This is a collection of shell functions which print coloured lines of text.

Coloured lines can make it easy to tell where a piece of text starts or ends.

ANSI Control Sequences

These functions assume that your terminal handles ANSI control sequences, also known as ANSI esecape codes. These are explained in more detail on the Colors page.

Functions

These could be stand-alone scripts, however I found it easier to write them as functions and source them into my shell when it starts.

The first function, coloured_line, does the heavy lifting. It requires one parameter, and may have more.

  • The first parameter specifies the colour. This is a set of numbers and ; characters which are inserted into the ANSI command sequence which sets the colour.

  • If there are more parameters, they are essentially joined into a single string with newlines between them, and then that is split into lines. Each line is printed with the appropriate control sequences before and after the text of that line.

  • If there are no more parameters, the function reads lines from STDIN and prints each one with the appropriate control sequences before and after the text of that line.

Joining the parameters and splitting the lines allows the caller to provide multiple lines, either as multiple parameters or as multi-line strings.

function coloured_line {
    local COL P IFS LINE LINES

    COL="${1:?no colour code specified}"
    shift

    if [[ -n "$1" ]]
    then
        ########################################
        # each parameter is a line

        for P in "$@"
        do
            IFS=$'\n' read -a LINES -d '' -r <<< "$P" || true
            for LINE in "${LINES[@]}"
            do
                printf "\e[%sm%s\e[0K\e[0m\n" "$COL" "$LINE"
            done
        done
    else
        ########################################
        # STDIN will contain one or more lines of text

        while IFS='' read -r LINE
        do
            printf "\e[%sm%s\e[0K\e[0m\n" "$COL" "$LINE"
        done
    fi
}

The other functions are wrappers around the coloured_line function. These specify the appropriate colour code and pass their input parameters (if any) along without modification. Note that the colour codes all start with 0;, which resets the colour to the defaults. This ensures that the output won't "inherit" any attributes from whatever may have been active before.

function redline {
    coloured_line '0;1;37;41' "$@"
}

function yellowline {
    coloured_line '0;30;43' "$@"
}

function blueline {
    coloured_line '0;1;37;44' "$@"
}

function greenline {
    coloured_line '0;1;37;42' "$@"
}

function cyanline {
    coloured_line '0;1;37;46' "$@"
}

function purpleline {
    coloured_line '0;1;37;45' "$@"
}

function whiteline {
    coloured_line '0;1;37;47' "$@"
}

Examples

Variations

These are different versions of the same functions. Depending on your needs, you may find these more useful.

Simpler verions

These functions are simpler, but they may not handle multi-line strings correctly.

function redline {
    printf "\e[0;1;37;41m%s\e[0K\e[0m\n" "$*"
}

(not showing all colours)

Handle output when STDOUT is not a terminal

One thing to note is that the functions, as written above, will always print the codes to generate coloured lines. You can change them so that they only print the "colour codes" when run in a terminal, and print some kind of plain-text version, without the colour codes. As an example ...

function redline {
    if [[ -t 1 ]]
    then
        printf "\e[0;1;37;41m%s\e[0K\e[0m\n" "$*"
    else
        echo "***** $* *****"
    fi
}

fail Function

In many of my scripts, when an error occurs I want to draw the user's eye to it. I do this using the fail function, which is pretty simple - it uses redline to print the message it received in a red line, then exits from the script.

This function is useful within scripts, but you probably don't want to include it in your .bashrc file. If you were to type fail in a shell, it would print the message and then immediately exit from the shell.

function fail {
    redline "$@"
    exit 1
}

Colors

This is a script I threw together as a quick way to see what different ANSI colour combinations look like on whatever terminal I'm using at the moment.

(Screenshot from Apple Terminal.app under macOS 26.0.1)

Download

⇒ colors

Control Sequences

Most of the values you're seeing above are used as part of an ANSI Control Sequence (these are also known as ANSI escape codes). These are sequences of control characters which, when "printed" to a terminal, control the terminal's behaviour. In the case of the colors script, they control the colour used to draw the characters sent after the control sequence.

These control sequences were originally used by the DEC VT-100 terminal (which I used in high school. The computer lab had several different kinds of terminals, but I preferred the VT-100's because of how the keyboard "felt" under my fingers.

This is the first time I remember thinking about how keyboards feel, and was the beginning of what made me prefer mechanical keyboards to this day.

A "control sequence" is just a specific set of characters, sent to the terminal, by the "host" it's connected to. In most cases (these days), the terminal is a window on your desktop, and the host is either the computer in front of you, or a computer that you may be connected to remotely (via SSH, for example).

These characters are ...

  • ESC is the ANSI "ESC" control character, ASCII 27 or 0x1B.

    The C printf() function uses the notation \e to represent this character. Many other languages and utilities (including echo -e) use the same notation. I'll be using the same notation below.

  • The sequence \e[ (i.e. ESC followed by [) tells the terminal that an ANSI control sequence is starting. Some documentation will refer to this sequence as CSI, which means "command sequence initiator".

  • There may or may not be a sequence of numbers, separated by ;.

  • After the numbers will be a character that tells the terminal which command is being sent.

Setting Text Attributes

The \e[...m command tells the terminal that any characters printed after the sequence, should be drawn using certain colours or other visual attributes. The numbers between [ and m control what they should look like.

The numbers used to draw the first line are ...

  • 0 means to reset the text back to "standard" colours, typically white text on a black background.
  • 1 means to make the foreground colour "bold".
  • 2 means to make the foreground colour "faint".
  • 3 means to draw the text in italics.
  • 4 means to draw the text with an underline.
  • 5 means to make the text blink slowly.
  • 6 means to make the text blink quickly.
  • 7 means to draw the text in "reverse video", i.e. with the foreground and background colours reversed.
  • 8 means to "conceal" the text. (What this actually means will depend on the terminal software.)
  • 9 means to draw the text "crossed out", i.e. with a horizontal line through it.

Note that not all terminal programs support all of these codes. As an example, none of the terminal programs I use know how to make text blink. For the most part, terminal programs will ignore codes they don't understand. This is actually why I wrote the colors script, so I could quickly tell how different terminal programs handle different visual attributes.

Setting Colours

Setting text colours uses the same \e[...m command shown above, but uses a different set of numbers to specify the colours.

  • 30-37 set the foreground colour.
  • 40-47 set the background colour.

Setting Text Height and Width

The original VT-100 and later VT-series terminals were able to draw characters twice as high and/or twice as wide as normal. Not all terminals were (or are) able to do this, so the control sequences to show text this way are not part of the ANSI X3.41 standard. They are proprietary to DEC (Digital Equipment Corporation, who made the VT-series terminals), however some terminal emulation software also supports them.

Instead of starting with ESC [, these control sequences start with ESC #. In these sequences, the character after the # controls the sizes of all characters on that line rather than any individual characters on the line, which means you can't mix different character sizes within the same line.

In addition, double-height characters need to be drawn on two lines - one line containing the top half of each character, and the next line containing the bottom half.

The control sequences for this are ...

  • ESC # 3 means that the characters on the current line should be drawn as the top half of double-height, double-width versions of those characters.
  • ESC # 4 means that the characters on the current line should be drawn as the bottom half of double-height, double-width versions of those characters.
  • ESC # 5 means that the characters on the current line should be drawn as "normal height, normal width" characters (i.e. the way they would normally be drawn anyway).
  • ESC # 6 means that the characters on the current line should be drawn as double-width, but normal height, versions of those characters.

This is easier to understand when you can see it, so these screenshots show the difference between a terminal which does support these sequences (Apple Terminal.app), and a terminal which does not support them (iTerm2), both using "Andale Mono 13 Regular" as the font.

(Screenshot from Apple Terminal.app v2.15(464) under macOS 26.0.1)

(Screenshot from iTerm2 v3.6.5 under macOS 26.0.1)

set_x and show_x functions

These functions simulate what set -x does, but only for specific commands.

Functions

set_x

The set_x function simulates set -x for just the one command. I use this in a lot of the scripts I write.

###############################################################################
#
# Maybe print a command before executing it

SET_X="${SET_X:-false}"

function set_x {
    if [[ "${SET_X:-false}" == "true" ]]
    then
        local IFS=$' '
        echo "$PS4$*" 1>&2
    fi
    "$@"
}

A one-liner to accomplish the same thing looks like this:

set -x ; COMMAND ; { set +x ; } 2>/dev/null

ℹ️ Colouring Output

I've started standardizing how I write my scripts, by starting new scripts with a common template which contains the colored line functions and the set_x and show_x functions, and then deleting the functions I end up not needing for that script. As part of this, I'm using the cyanline function to print the output generated by set_x and show_x.

function set_x {
    if [[ "${SET_X:-false}" == "true" ]]
    then
        local IFS=$' '
        cyanline "$PS4$*" 1>&2
    fi
    "$@"
}

If you write a lot of scripts, you may want to do the same.

show_x

This shows the same output that set_x shows, but doesn't actually run the command.

###############################################################################
#
# Show the same output that set_x would show,
# but don't actually run the command

function show_x {
    if [[ "${SET_X:-false}" == "true" ]]
    then
        local IFS=$' '
        echo "$PS4$*" 1>&2
    fi
}

As you can see, it's identical to set_x other than the "$@" at the end.

Example

#!/bin/bash

###############################################################################
#
# Usage message. Every good script should have one.

function usage {
    MSG="${1:-}"

    cat <<EOF
$0 [options]

Example program.

-x      Show commands being executed.

-h      Show this help message.

EOF

    if [[ -n "$MSG" ]]
    then
        echo "$MSG"
        exit 1
    fi

    exit 0
}

###############################################################################
#
# Maybe print a command before executing it

SET_X="${SET_X:-false}"

function set_x {
    if [[ "${SET_X:-false}" == "true" ]]
    then
        local IFS=$' '
        echo "$PS4$*" 1>&2
    fi
    "$@"
}

###############################################################################
###############################################################################
###############################################################################

SET_X=false

while getopts ':hx' OPT
do
    case $OPT in
        h)  usage
            ;;
        x)  SET_X=true
            ;;
        *)  echo "ERROR: unknown option '$OPTARG'"
            exit 1
    esac
done
shift $(( OPTIND - 1 ))

########################################
# Examples

set_x echo hello

ID="$( set_x gh pr list ... )"

Template

For many years I've kept a collection of pieces of shell code that I end up using when I start a new script. At one point I had to create about ten scripts in the same day, and realized that for each script I was copying a basic getopts loop, then copying the set_x function, then copying the coloured line functions, and realized that it would be easier if I had a "template" with these things already in place, so I started a template.txt file in my home directory.

A few months later I discovered the cheat program, and stored the template as a cheatsheet. When I start a new script, I type something like this:

cheat template > new-script

... and then edit the new-script file from there.

Download

My cheatsheet collection is backed up to Github. You can download the template cheatsheet from this link.

Yubikey

A Yubikey is a physical USB security key. These are commonly used as a "second factor" for authentication (2FA). The Yubikey 5 series supports several protocols, including FIDO2, U2F, and OpenPGP.

Load PGP keys into a YubiKey

2017-12-13

This document covers how to load PGP keys into a YubiKey.

Background

The YubiKey is actually a tiny computer, powered by the USB port (or via NFC). It contains several tiny "apps" which provide the functionality of the YubiKey. In this case, we're going to be talking about the OpenPGP app.

Most smartcards, including the Yubikey, require some kind of authentication before they will agree to do anything. When you first plug the YubiKey into a USB slot, the OpenPGP app will be in a "locked" state, and the user needs to enter a PIN number to unlock the YubiKey before the app can be used.

The YubiKey has three different codes which can be entered, which allow different types of operations to be performed.

  • The Personal Identification Number, or "PIN", is used to allow operations which involve using the secrets stored within the YubiKey. This includes decrypting data, creating digital signatures, and (with the right kind of key) performing SSH authentication.

    If the wrong PIN is entered three times, the YubiKey will "lock" itself and not allow itself to be unlocked at all, even if the correct PIN is entered.

  • The Personal Unblocking Key, or "PUK", is used to "unlock" the YubiKey after the wrong PIN has been entered three times.

    If the wrong PUK is entered too many times, the PUK function will also be locked.

  • The Admin PIN is the "master key" for the OpenPGP app. It is used to load new secret keys, set or change PINs, and "unlock" the OpenPGP app after the wrong PIN (or PUK) has been entered too many times.

    I think the Admin PIN will also unlock normal key-use functions like the regular PIN does. However, you shouldn't use the Admin PIN on an everyday basis like this, for a few reasons:

    • If somebody happens to see you entering the PIN, and then "borrows" the YubiKey (with or without your knowledge), they would be able to change any/all of the PINs and/or load new keys without your knowledge.

    • If you forget the Admin PIN and lock the card, you won't have a way to unlock it, other than totally resetting it and loading new keys.

    If the Admin PIN is entered incorrectly three times, the OpenPGP app will permanently lock itself. The only way to recover from this is to totally reset the OpenPGP app, which deletes any secret keys which were prevously stored in the YubiKey.

Note that there is no way to download the secret keys from the YubiKey, even if you have all three PIN codes.

PIN Requirements

PIN codes are generally a string of digits, however...

  • YubiKeys do not require that PINs can only be digits.

    However, most smartcards require PINs to be digits, because they may need to be used with a card reader with an integrated PIN pad, and entering letters or other characters using a ten-key keypad can be a bit of a pain. And because of this, some computers' PGPCard implementations may assume that only digits are allowed, and only allow digits to be entered.

    Unless you are 100% sure that every system where you will ever use the YubiKey will support non-digits in the PIN codes, I recommend that you stick with digits.

  • The YubiKey OpenPGP app has a lower limit of 6 characters. This is different from the smartcards embedded in most credit/debit cards, which only require 4 characters (and which may not allow more than 4 characters).

  • The YubiKey OpenPGP app has an uppper limit of 127 characters. However, some computers may limit how many characters the user can enter, which means that if your PIN is ten digits but the computer only allows you to enter eight, you won't be able to use it at all.

My own PINs have more than eight characters, and I haven't had any problems using them with macOS, Linux, or one time with Windows (it was a work thing, and I have changed the PIN since this happened - I don't know if the corporate IT overlords were recording keystrokes at the time or not.)

Default PIN codes

When a Yubikey arrives from the factory, or if its OpenPGP app has been reset, the default PIN codes are:

  • PIN: 123456
  • PUK: (none, which means that the Admin PIN must be used to unlock the PIN)
  • Admin PIN: 12345678

Set Yubikey OpenPGP PINs

If you have not already done so, you should set your own PIN and Admin PIN codes.

While you're setting the PINs, you may also want to set the cardholder name, language preference, public key URL, and login data. These are all optional, however I normally do this with my YubiKeys, so you will see these steps below.

$ gpg --card-edit

Reader ...........: Yubico Yubikey 4 OTP U2F CCID
Application ID ...: D2760001240102010006069404470000
Version ..........: 2.1
Manufacturer .....: Yubico
Serial number ....: 06940447
Name of cardholder: [not set]
Language prefs ...: [not set]
Sex ..............: unspecified
URL of public key : [not set]
Login data .......: [not set]
Signature PIN ....: not forced
Key attributes ...: rsa4096 rsa4096 rsa4096
Max. PIN lengths .: 127 127 127
PIN retry counter : 3 0 3
Signature counter : 0
Signature key ....: [none]
Encryption key....: [none]
Authentication key: [none]
General key info..: [none]

gpg/card> admin
Admin commands are allowed

gpg/card> passwd
gpg: OpenPGP card no. D2760001240102010006069404470000 detected

1 - change PIN
2 - unblock PIN
3 - change Admin PIN
4 - set the Reset Code
Q - quit

Your selection? 3

At this point your workstation will ask for the following, usually as separate prompts:

  • The current Admin PIN (it will probably just say "the Admin PIN"). Again, the default for a new (or newly reset) YubiKey is "12345678".
  • The new Admin PIN
  • The new Admin PIN again, to verify that you typed it correctly
PIN changed.

Next, set the PIN you'll use on a regular basis in order to generate signatures, decrypt messages, or perform SSH authentication.

1 - change PIN
2 - unblock PIN
3 - change Admin PIN
4 - set the Reset Code
Q - quit

Your selection? 1
  • Enter the current PIN. (Again, the default is "123456".)
  • Enter the new PIN.
  • Enter the new PIN again.

Note that if the current PIN was wrong, this command will fail and you will receive a "Error changing the PIN: Bad PIN" error.

PIN changed.

Once the PIN is set, if you want to set a separate PUK you can use the "unblock PIN" setting. Personally I don't have one, but if I were managing YubiKeys for a company and might need to help a user who locked their YubiKey by entering the wrong PIN too many times, and they weren't able to physically bring the YubiKey to me, I would definitely want to be able to give them a code which unlocks the PIN without giving them full access to change everything on the card.

1 - change PIN
2 - unblock PIN
3 - change Admin PIN
4 - set the Reset Code
Q - quit

Your selection? 2
  • Enter the Admin PIN (which you just set above)
  • Enter the new PUK
  • Enter the new PUK again
PIN changed.

When you're finished setting the PIN codes, use "q" to leave that menu and go back to the "gpg/card>" prompt.

1 - change PIN
2 - unblock PIN
3 - change Admin PIN
4 - set the Reset Code
Q - quit

Your selection? q

gpg/card>

Next, enter some basic info about the "owner" of the card, along with their preferred language. This information is stored on the YubiKey, and will be visible to anybody who runs "gpg --card-info" or "gpg --card-edit" while the YubiKey is plugged in.

gpg/card> name
Cardholder's surname: Simpson
Cardholder's given name: John

gpg/card> login
Login data (account name): jms1@jms1.net

gpg/card> lang
Language preferences: en

You can also enter a URL where the corresponding public key can be downloaded. Doing this allows you to use the "gpg --edit-card" command's "fetch" sub-command to load your public keys into a new computer's keyring. PGPCards only hold secret keys - they don't hold public keys, user IDs, signatures, or expiration dates.

This is not required. If you don't have, or don't want, a copy of your public key saved on a web site somewhere, feel free to skip this step.

gpg/card> url
URL to retrieve public key: https://jms1.net/6B2EDC90B5C6DC30.pub.asc

You can also set a flag which tells the YubiKey to require the PIN to be entered, every time a signature is generated. Without this, you will be asked for the PIN the first time you generate one, and the YubiKey will "stay unlocked" and generate more signatures as requested, until it is unplugged from the computer.

gpg/card> forcesig

To see the updated state of the card, just hit RETURN at the "gpg/card>" prompt.

gpg/card>

Reader ...........: Yubico Yubikey 4 OTP U2F CCID
Application ID ...: D2760001240102010006069404470000
Version ..........: 2.1
Manufacturer .....: Yubico
Serial number ....: 06940447
Name of cardholder: John Simpson
Language prefs ...: en
Sex ..............: unspecified
URL of public key : https://jms1.net/6B2EDC90B5C6DC30.pub.asc
Login data .......: jms1@jms1.net
Signature PIN ....: forced
Key attributes ...: rsa4096 rsa4096 rsa4096
Max. PIN lengths .: 127 127 127
PIN retry counter : 3 0 3
Signature counter : 0
Signature key ....: [none]
Encryption key....: [none]
Authentication key: [none]
General key info..: [none]

When you're happy with the settings, use "q" to exit the "gpg --card-edit" command. Note that this isn't "saving" anything, the changes you made were saved to the YubiKey immediately.

gpg/card> q

Remove and re-insert the Yubikey.

From this point forward, you will need to enter the PIN in order to make use of any keys, and you will need to enter the Admin PIN in order to load keys or change settings.

Notes

  • The "URL of public key" is used by the "fetch" command (under "gpg --card-edit") to retrieve the public key when using the YubiKey on a machine which doesn't already have the public key in its keyring.

Load keys on Yubikey

$ gpg --edit-key 6353320118E1DEA2F38EAE806B2EDC90B5C6DC30
gpg (GnuPG/MacGPG2) 2.2.0; Copyright (C) 2017 Free Software Foundation, Inc.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Secret key is available.

sec  rsa4096/0x6B2EDC90B5C6DC30
     created: 2017-05-27  expires: never       usage: SC
     trust: ultimate      validity: ultimate
ssb  rsa4096/0x297E5961AB566594
     created: 2017-05-27  expires: never       usage: E
ssb  rsa4096/0xBA6C2A169C6C0F60
     created: 2017-11-10  expires: never       usage: A
[ultimate] (1). John M. Simpson <jms1@voalte.com>
[ultimate] (2)  John M. Simpson <jms1@jms1.net>

Select the Authentication sub-key.

  • Each "ssb" line is a sub-key. They are numbered in the order shown here (even though the program doesn't show the numbers.)

  • You will see an asterisk appear next to a sub-key when it is selected.

  • It is possible to select more than one key. The same key command which selects a key will also de-select a key. (You will see this below.)

Make sure the Authentication sub-key is the only one selected.

gpg> key 2

sec  rsa4096/0x6B2EDC90B5C6DC30
     created: 2017-05-27  expires: never       usage: SC
     trust: ultimate      validity: ultimate
ssb  rsa4096/0x297E5961AB566594
     created: 2017-05-27  expires: never       usage: E
ssb* rsa4096/0xBA6C2A169C6C0F60
     created: 2017-11-10  expires: never       usage: A
[ultimate] (1). John M. Simpson <jms1@voalte.com>
[ultimate] (2)  John M. Simpson <jms1@jms1.net>

Send the selected sub-key to the Authentication slot on the Yubikey (or the "card", as gpg calls it.)

gpg> keytocard
Please select where to store the key:
   (3) Authentication key
Your selection? 3

sec  rsa4096/0x6B2EDC90B5C6DC30
     created: 2017-05-27  expires: never       usage: SC
     trust: ultimate      validity: ultimate
ssb  rsa4096/0x297E5961AB566594
     created: 2017-05-27  expires: never       usage: E
ssb* rsa4096/0xBA6C2A169C6C0F60
     created: 2017-11-10  expires: never       usage: A
[ultimate] (1). John M. Simpson <jms1@voalte.com>
[ultimate] (2)  John M. Simpson <jms1@jms1.net>

Now select the Encryption sub-key, and un-select the Authentication sub-key.

gpg> key 1

sec  rsa4096/0x6B2EDC90B5C6DC30
     created: 2017-05-27  expires: never       usage: SC
     trust: ultimate      validity: ultimate
ssb* rsa4096/0x297E5961AB566594
     created: 2017-05-27  expires: never       usage: E
ssb* rsa4096/0xBA6C2A169C6C0F60
     created: 2017-11-10  expires: never       usage: A
[ultimate] (1). John M. Simpson <jms1@voalte.com>
[ultimate] (2)  John M. Simpson <jms1@jms1.net>

gpg> key 2

sec  rsa4096/0x6B2EDC90B5C6DC30
     created: 2017-05-27  expires: never       usage: SC
     trust: ultimate      validity: ultimate
ssb* rsa4096/0x297E5961AB566594
     created: 2017-05-27  expires: never       usage: E
ssb  rsa4096/0xBA6C2A169C6C0F60
     created: 2017-11-10  expires: never       usage: A
[ultimate] (1). John M. Simpson <jms1@voalte.com>
[ultimate] (2)  John M. Simpson <jms1@jms1.net>

Send the selected sub-key to the Encryption slot on the Yubikey.

gpg> keytocard
Please select where to store the key:
   (2) Encryption key
Your selection? 2

sec  rsa4096/0x6B2EDC90B5C6DC30
     created: 2017-05-27  expires: never       usage: SC
     trust: ultimate      validity: ultimate
ssb* rsa4096/0x297E5961AB566594
     created: 2017-05-27  expires: never       usage: E
ssb  rsa4096/0xBA6C2A169C6C0F60
     created: 2017-11-10  expires: never       usage: A
[ultimate] (1). John M. Simpson <jms1@voalte.com>
[ultimate] (2)  John M. Simpson <jms1@jms1.net>

Now un-select all sub-keys, which results in the main key being selected.

gpg> key 1

sec  rsa4096/0x6B2EDC90B5C6DC30
     created: 2017-05-27  expires: never       usage: SC
     trust: ultimate      validity: ultimate
ssb  rsa4096/0x297E5961AB566594
     created: 2017-05-27  expires: never       usage: E
ssb  rsa4096/0xBA6C2A169C6C0F60
     created: 2017-11-10  expires: never       usage: A
[ultimate] (1). John M. Simpson <jms1@voalte.com>
[ultimate] (2)  John M. Simpson <jms1@jms1.net>

Send the main key to the Signature slot on the Yubikey.

gpg> keytocard
Really move the primary key? (y/N) y
Please select where to store the key:
   (1) Signature key
   (3) Authentication key
Your selection? 1

sec  rsa4096/0x6B2EDC90B5C6DC30
     created: 2017-05-27  expires: never       usage: SC
     trust: ultimate      validity: ultimate
ssb  rsa4096/0x297E5961AB566594
     created: 2017-05-27  expires: never       usage: E
ssb  rsa4096/0xBA6C2A169C6C0F60
     created: 2017-11-10  expires: never       usage: A
[ultimate] (1). John M. Simpson <jms1@voalte.com>
[ultimate] (2)  John M. Simpson <jms1@jms1.net>

We're done here, BUT ... we need to be careful. The next command will quit out of the "gpg --card-edit" command, and it will ask if you want to save changes. IF YOU SAY YES, the secret keys you just installed on the YubiKey will be REMOVED from the secret keyring file on the computer.

Unless you are 100% sure that's what you want to do (i.e. if you have a known-good backup of the secret keys), BE SURE TO SAY NO.

gpg> q
Save changes? (y/N) n
Quit without saving? (y/N) y

Now if you query the card, you will see the keys in the three slots.

$ gpg --card-status

Reader ...........: Yubico Yubikey 4 OTP U2F CCID
Application ID ...: D2760001240102010006069404470000
Version ..........: 2.1
Manufacturer .....: Yubico
Serial number ....: 06940447
Name of cardholder: John Simpson
Language prefs ...: en
Sex ..............: unspecified
URL of public key : https://jms1.net/6B2EDC90B5C6DC30.pub.asc
Login data .......: jms1@jms1.net
Signature PIN ....: not forced
Key attributes ...: rsa4096 rsa4096 rsa4096
Max. PIN lengths .: 127 127 127
PIN retry counter : 3 0 3
Signature counter : 0
Signature key ....: 6353 3201 18E1 DEA2 F38E  AE80 6B2E DC90 B5C6 DC30
      created ....: 2017-05-27 22:28:31
Encryption key....: 0660 766F 2768 F41F D4B9  1DB7 297E 5961 AB56 6594
      created ....: 2017-05-27 22:28:31
Authentication key: BBA5 C6BB 23D2 B53B 0D0F  6C0B BA6C 2A16 9C6C 0F60
      created ....: 2017-11-10 23:29:14
General key info..: pub  rsa4096/0x6B2EDC90B5C6DC30 2017-05-27 John M. Simpson <jms1@voalte.com>
sec   rsa4096/0x6B2EDC90B5C6DC30  created: 2017-05-27  expires: never
ssb   rsa4096/0x297E5961AB566594  created: 2017-05-27  expires: never
ssb   rsa4096/0xBA6C2A169C6C0F60  created: 2017-11-10  expires: never

Notes

  • The output you see from the commands above may differ slightly based on the version of the gpg software and how it's configured.

  • The Yubikey does not store public keys, it only stores private keys. Private keys are just numbers, they don't have attributes like names or expire dates. Everything after the fingerprints, such as the name and email, and the created: and expires: dates, all came from the keyring on the machine. If you query the card from a machine which doesn't have the public keys available, all you will see is the fingerprints.

As an example, this is a different version of gpg, looking at a different Yubikey, with different key loaded, and for this example I manually changed the GNUPGHOME variable to point to an empty directory so the command won't recognize the key...

$ gpg --card-status
Reader ...........: Yubico YubiKey OTP FIDO CCID
Application ID ...: D2760001240102010006063013830000
Version ..........: 2.1
Manufacturer .....: Yubico
Serial number ....: 06301383
Name of cardholder: John Simpson
Language prefs ...: en
Sex ..............: unspecified
URL of public key : https://jms1.net/A7EC1FBAB3B50007.pub.asc
Login data .......: jms1@jms1.net
Signature PIN ....: forced
Key attributes ...: rsa4096 rsa4096 rsa4096
Max. PIN lengths .: 127 127 127
PIN retry counter : 3 0 3
Signature counter : 74
Signature key ....: AF6C 0A45 953A 0881 06A7  D254 CE57 35E1 E04C 1374
      created ....: 2017-11-27 00:53:45
Encryption key....: BF37 34CD 9834 B3B4 7A8E  D70E 2A9E B3A6 20A1 C087
      created ....: 2017-11-27 00:36:27
Authentication key: 5761 1969 0CC7 57A4 7300  57C3 A634 470E CECC 41E0
      created ....: 2017-11-27 01:00:17
General key info..: [none]

Changelog

2024-06-19 jms1

  • moved page to new jms1.info site, updated header

2021-01-22 jms1

  • added more detail about setting the user data (name, email, etc.)
  • added more detail about setting PINs, including PUK

2020-12-24 jms1

  • added note about not saving changes when quitting out of gpg --card-edit

2020-12-20 jms1

  • moved to jms1.info
  • added the "Background" section at the top, moved "Changelog" to the end
  • tweaked formatting

2018-03-06 jms1

  • tweaked the formatting
  • last version on jms1.net site

2017-12-13 jms1

  • first version

Reset the YubiKey OpenPGP App

2017-12-13

How to reset the OpenPGP app on a YubiKey Neo

Yubico's official procedure

Yubico now has an officially documented procedure for resetting the OpenPGP applet on a YubiKey device.

The procedure documented below seems to have worked for me with a YubiKey Neo in the past, however I don't know if it will also work with other YubiKey hardware, and to be honest I'm not 100% sure exactly what it's doing. I just combined information from a few different web sites until I found something that worked for me at the time.

Please use Yubico's officially documented procedure instead of using the procedure below.

Old content

$ gpg-connect-agent <<EOF
/hex
scd serialno
scd apdu 00 20 00 81 08 40 40 40 40 40 40 40 40
scd apdu 00 20 00 81 08 40 40 40 40 40 40 40 40
scd apdu 00 20 00 81 08 40 40 40 40 40 40 40 40
scd apdu 00 20 00 81 08 40 40 40 40 40 40 40 40
scd apdu 00 20 00 83 08 40 40 40 40 40 40 40 40
scd apdu 00 20 00 83 08 40 40 40 40 40 40 40 40
scd apdu 00 20 00 83 08 40 40 40 40 40 40 40 40
scd apdu 00 20 00 83 08 40 40 40 40 40 40 40 40
scd apdu 00 44 00 00
scd apdu 00 e6 00 00
/bye
EOF

Remove and insert YubiKey.

$ gpg --card-status
gpg: selecting openpgp failed: Operation not supported by device
gpg: OpenPGP card not available: Operation not supported by device

$ gpg-connect-agent <<EOF
/hex
scd serialno undefined
scd apdu 00 a4 04 00 06 d2 76 00 01 24 01
scd apdu 00 44 00 00
scd apdu 00 e6 00 00
/bye
EOF

$ gpg --card-status
gpg: selecting openpgp failed: Conflicting use
gpg: OpenPGP card not available: Conflicting use

Remove and insert YubiKey.

$ gpg --card-status
Reader ...........: Yubico Yubikey 4 OTP U2F CCID
Application ID ...: D2760001240102010006069404470000
Version ..........: 2.1
Manufacturer .....: Yubico
Serial number ....: 06940447
Name of cardholder: [not set]
Language prefs ...: [not set]
Sex ..............: unspecified
URL of public key : [not set]
Login data .......: [not set]
Signature PIN ....: not forced
Key attributes ...: rsa2048 rsa2048 rsa2048
Max. PIN lengths .: 127 127 127
PIN retry counter : 3 0 3
Signature counter : 0
Signature key ....: [none]
Encryption key....: [none]
Authentication key: [none]
General key info..: [none]

The OpenPGP app is now "empty" - no keys, PINs reset to default values, etc.

Changelog

2024-06-19 jms1

  • moved page to new jms1.info site, updated header

2020-12-20 jms1

  • moved to jms1.info, moved Changelog to end of file
  • minor formatting updates

2019-03-23 jms1

  • added info about Yubico's supported process for resetting the OpenPGP applet
  • last version on jms1.net

2017-12-13 jms1

  • initial version

Using a Yubikey for SSH keys

2024-06-30

This page will cover how I'm storing my SSH (and PGP) keys on a Yubikey. It will combine information from other notes that I've written over the years.

It will also contain links to other pages on this site, where parts of this process are already explained.

Finally, note that I haven't "finished" this page yet. It has enough information for me to know what's going on, but there are parts that I just haven't had time to "flesh out" yet. I don't plan to leave it this way forever, but I at least wanted to get this page mostly done and on the site, because at least two different people are waiting for it.

Background

In order to understand how this works, you'll need to have a basic understanding of a few other things first. If you already have a basic understanding of these things, or if you're impatient and don't want to have to read through them, feel free to skip over them.

ℹ️ These explanations are deliberately leaving out a lot of detail. I promise, I'm not trying to make this page any longer than it needs to be.

Public-Key Encryption

Traditional encryption systems (or "cryptosystems") use the same key to encrypt and decrypt each message. If you encrypt a message using one key, the recipient needs the same key to decrypt it - and anybody else who manages to get a copy of that key will also be able to decrypt it. These are known as "shared-key" or "symmetric-key" systems, because the same key is used for both operations.

In a public-key encryption system, each party has a pair of keys. These keys are used with algorithms where a message encrypted using one of those keys, can only be decrypted using the other key in the same pair. Each user generates a key pair, shares one key with the world (known as a "public key"), and keeps the other key to themself (known as a "private key" or "secret key").

Public-key algorithms use a LOT more resources (CPU, RAM, and time) than symmetric-key algorithms. Because of this, most cryptosystems (including PGP and SSH) will generate and use a random key (or "ephemeral" key) with a symmetric-key algorithm to encrypt the message itself, and then use a public-key algorithm to encrypt that ephemeral key. The recipient uses their secret key to decrypt the portion containing the ephemeral key, then uses the ephemeral key to decrypt the actual message. This way the "more expensive" operations are used at the beginning, but the "less expensive" operations are used for the bulk of the message. This is especially useful for larger messages.

PGP and GnuPG

PGP, or "Pretty Good Privacy", is an encryption system written by Phil Zimmermann in the early 1990's. The source code was available online, although it didn't quite have what we know today as an "open source" license.

At the time he wrote it, the US had restrictions on exporting cryptographic software, and he was subjected to a three-year criminal investigation. (Those export restrictions were later removed when Daniel J. Bernstein filed, and won, a series of lawsuits against the United States.)

Zimmermann later started a company called PGP, Inc. to try and commercialize the pgp software. It had some success, and was later sold to Network Associates, who then sold it to Symantec, who still owns the name and the "intellectual property" but doesn't appear to be doing anything with it.

PGP's message format was standardized as the "OpenPGP Message Format" in RFC 4880. Several programs implement this standard.

GnuPG is the most common implementation of PGP. This is an open-source software package which is available for pretty much every operating system out there, including macOS and Linux. (There is a related project called gpg4win for ms-windows.)

PGP Keys and Subkeys

When a user generates a PGP key, they are actually generating a set of key pairs. Each key pair has flags describing which operations that key is meant to be used for. One of the keys is designated as the "primary" key, and most "PGP keys" have one or more "subkeys". Each individual key or subkey is flagged to be used for specific operations.

The default configuration of a "PGP key" has ...

  • A primary key, flagged with [C] for certifying (signing other PGP keys), and usually [S] for signing messages. You normally see these combined as [SC].

  • A subkey, flagged with [E] for encrypting messages.

It is possible, and in some cases can be useful, to create a primary key with only the [C] flag. This can be useful if you need a key that will never be used to sign messages, and should only be used by others to encrypt message to you.

There is a fourth capability, [A] for authenticating. Most PGP users aren't even aware that it exists, but we're going to use it below.

SSH Key-based Authentication

For SSH, each user has a key pair. These are commonly stored in files with matched filenames, such as id_rsa for a secret key, and id_rsa.pub for the corresponding public key. However, if you're able to store the secret key somewhere else (like in a Yubikey), there's no need for the secret key to exist on the computer at all - which makes it very hard for an attacker to steal the secret key. (They can't steal what isn't there in the first place.)

SSH key-based authentication works like this:

  • On each server that a user might need to log into, they store copies of their SSH public key(s) in their $HOME/.ssh/authorized_keys file.

  • When the user wants to log into the server, the server sends the client a challenge containing a block of random data (also known as a "nonce").

  • The client answers the challenge by "signing" the nonce (encrypting it using the SSH secret key) and sending the result back to the server.

  • The server tries to decrypt the client's response using the public keys in the user's authorized_keys file. If one of them successfully decrypts the response, authentication succeeds and the incoming connection is logged in as that user.

  • Otherwise, authentication fails and the client is not allowed to log in.

Most systems use an "SSH agent" to perform the nonce-signing. This is a process which holds secret keys in memory, and offers an interface which allows clients to ask for nonces to be signed. This interface is implemented using a "unix socket", which is only accessible from processes on the same machine.

OpenSSH is the standard SSH implementation for macOS and Linux. It uses a program called ssh-agent to perform the agent function, however any program which offers the same interface can do the same thing.

GnuPG comes with a program called gpg-agent which serves a similar function for PGP secret keys, and can be configured to "speak" the SSH agent protocol. Part of the solution we'll be building below will involve configuring your SSH clients to talk to a pgp-agent process.

Yubikey

A Yubikey is a small USB device that fits on your keychain. It can be used as a "second factor" for authentication, and is available with USB-A, USB-C, or "Lightning" (used on many Apple devices) connectors, as well as NFC (short-range wireless) connectivity.

Yubikey devices are miniature computers. They run their own "apps", which are loaded by Yubico during manufacturing, and which for security reasons, cannot be upgraded, deleted, or modified. Each Yubikey has a "secure element" which stores encryption keys in a way that they cannot be extracted, even by an attacker who physically disassembles the Yubikey and attaches wires directly to the right chips on the board.

The Yubikey Neo, 4, and 5 series have an OpenPGP app. This app implements the OpenPGP Card standard, which allows it to work with GnuPG's smart card support. The OpenPGP app can store three secret keys in the Yubikey's secure element, and can use those keys to perform OpenPGP functions which require them (signing and decrypting messages).

The app does not have a way to export the secret keys, so once a secret key is loaded into (or generated on) a Yubikey, it cannot be extracted.

Tails

Tails is a Linux system designed around privacy.

  • It boots from a USB memory stick and runs from a RAM disk. Any data saved to the RAM disk is deleted when Tails shuts down.

  • Almost all network traffic is routed through Tor.

  • The software we're going to need, including GnuPG, is already installed.

Tails can set up an encrypted Persistent Storage partition on the USB stick, where files can be saved permanently. We will be using this functionality to hold the only copy of your secret keys.

Recap

This is what we're actually going to do:

  • Under Tails with Persistent Storage

    • Generate a new PGP key (or import an existing key).
    • Add a subkey with the authentication flag.
    • Generate the corresponding SSH public key for this new subkey.
    • Load that subkey into a Yubikey.

  • On each machine you want to be able to SSH into (i.e. the servers you need to SSH into)

    • Add the generated SSH public key, to your $HOME/.ssh/authorized_keys file.

  • On each machine where you want this to work (i.e. workstations)

    • Configure pgp-agent to support the SSH agent protocol.
    • Make your SSH clients talk to pgp-agent as the SSH agent.

If anything on this list doesn't make sense to you at all, please go back up and re-read the information above. If it still doesn't make any sense, there's a chance that I'm forgetting something. Please let me know if this is the case, so I can update this page.

Boot into Tails with Persistent Storage

Most of the procedures below will take place in a running Tails system.

Rather than try and explain Tails here, I'm going to point you to the documentation on their web site.

Make sure you boot into Tails, with Persistent Storage unlocked, and the listed categories enabled, BEFORE you continue.

These directions will involve using the command line. In Tails, you can access the command line using:

  • Applications → Utilities → Terminal

Create or Import a PGP Key

The goal of this section is to have a PGP key pair, with an authentication subkey, in the GnuPG keyring in your Tails Persistent Storage. This will include both the public and secret keys.

The idea is that this Tails stick, with Persistent Storage unlocked, will be the only place you'll be able to use the PGP key without a Yubikey.

Create a PGP Key

If you don't already have a PGP key, you'll need to create one.

  • gpg --gen-key
  • gpg --quick-generate-key 'Name <email>' rsa4096 default 20250101T000000

Because some older PGP software may not be able to handle ed25519 keys, I use rsa4096 for my primary key. You can do this and still have an ed25519 authentication subkey for SSH (and actually, the PGP key I use for $DAYJOB has two authentication subkeys - one rsa4096 and one ed25519.)

Import a PGP Key

If you already have a PGP key, you'll need to import both the public and secret keys into the GnuPG keyring within Tails.

Export from your current PGP software

The mechanics of exporting keys will depend on your current PGP software. If you're using GnuPG, you'll want to create an encrypted USB stick (using LUKS or VeraCrypt, so Tails will be able to mount it) and store the exported files there. DO NOT store your exported secret key where anybody else will be able to access it.

The process will look something like this:

cd /mnt/encrypted
gpg -a --export KEYID > KEYID.pub
gpg -a --export-secret-keys KEYID > KEYID.sec

Import into GnuPG

This process will look something like this:

gpg --import KEYID.pub
gpg --import KEYID.sec

You will probably want to set the key's "trust" to "ultimate" as well.

$ gpg --edit-key KEYID
...
gpg> trust
...
Please decide how far you trust this user to correctly verify other users' keys
(by looking at passports, checking fingerprints from different sources, etc.)

  1 = I don't know or won't say
  2 = I do NOT trust
  3 = I trust marginally
  4 = I trust full
  5 = I trust ultimately
  m = back to the main menu

Your decision? 5
Do you really want to set this key to ultimate trust? (y/N) y
...
Please note that the shown key validity is not necessarily correct
until you restart the program.

gpg> q

⚠️ Do not use the "ultimate" trust level for any keys other than your own.

Add an Authentication Subkey

⇒ Authentication Subkeys

  • couldn't find a way to import existing SSH secret key as an authentication subkey
    • "monkeysphere" sounds like it might be able to do it, but it looks like it's been abandoned (web site appears to have been taken over by a domain squatter)

Export SSH public key

Use the Key ID of the authentication subkey for this. If your PGP key only has one authentication subkey, you can also use the Key ID of the primary key for this. The software will find and use the authentication subkey automatically.

In this example, I'm using my primary Key ID.

$ gpg --export-ssh-key E3F7F5F76640299C5507FBAA49B9FD3BB4422EBB > id_rsa_yubikey.pub
$ cat id_rsa_yubikey.pub
ssh-rsa AAAAB3Nz...AkjIPw== openpgp:0xF8D09EB7

Edit the comment as needed. I normally use my name, email, the date the subkey was generated, and which Yubikey(s) will contain that key. This way when they appear in a file with other keys, it's easy to recognize which key is which.

$ cat id_rsa_yubikey.pub
ssh-rsa AAAAB3Nz...AkjIPw== John Simpson <jms1@jms1.net> 2019-03-21 Yubikey Blue

I have different coloured stickers on my Yubikeys, so I can tell which Yubikeys have which PGP/SSH keys on them. My personal PGP/SSH keys are on Yubikeys with the plain blue stickers.

Send the key to an "outside" system.

Load Keys into the Yubikey

⇒ Load Keys into a Yubikey

Add SSH key to authorized_keys files

  • standard process, just like adding any other key
  • can be done with ssh-copy-id

Set up Workstation

⇒ Make SSH use gpg-agent

  • manual process
  • Tails?

Changelog

2024-07-07 jms1

  • published what I have so far to jms1.info
  • included a note explaining that I'll add more human-readable info when I have time

2024-06-30 jms1

  • started this page, pulling in info from several other pages
Generated using mdbook v0.5.4
2026-07-28 06:25:32 +0000
initial-74-g80468f1
2026-07-28 06:25:10 +0000
Keybase - jms1 Mastodon - jms1@mastodon.social Buy Me A Coffee Hacker Logo RSS Feed