colourful words and phrases

techy tangents and general life chatter from a tired sysadmin

On encryption and jails

TLS FreeBSD BSD Jails

Note: This post was written when the Let's Encrypt private beta first launched, and most of the tooling related to Let's Encrypt has completely changed since.

It was reported recently that Let's Encrypt had recently launched their private beta. As someone who does their best to ensure encryption is as widely-available as possible, I was excited.

Yesterday, I got my email stating that the domain maff.scot had been whitelisted for issuance using the letsencrypt utility. Awesome! Let's see how you get it set up. To the best of my knowledge, there was no FreeBSD port or package for the utility, which meant I had to git clone their repo and set things up. Depending on how your system's set up, this might be a breeze, or it might be a challenge.

A few things to note, going in:

  • FreeBSD by default ships with csh as the default shell, and no sudo.
  • I very strongly embraced the BSD way of doing things, so that was still the case.
  • The letsencrypt utility has some questionable defaults, in that it will attempt to run its own server temporarily for use in verifying ownership of your domain. It can also futz with your Apache/nginx config files.
  • It doesn't take into account what platform it's running on, and as such, flies in the face of the FreeBSD filesystem hierarchy, documented in hier(7), defaulting to /etc/letsencrypt for configuration and storage, and /var/lib/letsencrypt for its working directory.
  • The documentation was a bit confusing on what I actually had to do to get things up and running in the event that a binary package was not available.

With all these in mind, I felt it might be prudent to just set it all up inside a FreeBSD Jail. This would allow all dependencies to be satisifed from the letsencrypt client's point of view, and would allow it to place files where it pleased, without making a mess of my main server.

Preparing to run Jails

I already had a jail system set up on my server, but I'll document the process here anyway. To simplify things, I used the EzJail system to get everything set up for me, and did final configuration and such using a flavour.

To start off, I ran pkg install ezjail to install ezjail. While the package itself is called "ezjail", the utility used to actually work with it is ezjail-admin.

After installing the package, the "base system", what all jails will draw from, must be installed. This can be compiled from source, but if you're running a RELEASE version of FreeBSD, it's perfectly fine (and easier) to simply use the same distribution bundles that you installed your host FreeBSD system from. To install the base system, just run ezjail-admin install, adding -m and -p to include man pages and the ports tree, respectively. This will download the FreeBSD base and userland, manpages (if desired), and will invoke portsnap to download the ports tree, and install it all to /usr/jails/basejail, by default.

After this, I set up my flavour for the jails I'd be running. A flavour is a set of files and configuration options that will be added to a jail you're creating, and allows for decent customisation; specific flavours can have certain packages preinstalled, services pre-enabled or pre-configured. I just set up a default flavour, which disabled services like sendmail, and pre-populated /etc/resolv.conf. This can be easily done by copying the example flavour bundled with ezjail, and editing the name and files where appropriate.

Since the machine I'm running on only has one IPv4 address, I needed to NAT traffic from my jails, by adding the following rules to my pf.conf:

jails="10.0.1.0/24"
wan="re0"
wan_ip="0.0.0.0" # replaced with your real IP, obviously

nat pass on $wan from $jails to any -> $wan_ip
pass in from $jails to $jails # to allow inter-jail trafficCode language: PHP (php)

It's also a good idea to give jails their own loopback interface, separate from the host's, by adding cloned_interfaces="lo1" to your rc.conf, and restart networking.

Setting up the Let's Encrypt jail

Now, finally, we're ready to create the jail itself, by running ezjail-admin create -f yourflavour letsencrypt 're0|10.0.1.10,lo1|127.0.1.10', which will create a jail named 'letsencrypt' using the flavour 'yourflavour', and will give it a private IP of 10.0.1.10 and a loopback IP of 127.0.1.10. These IPs haven't been pre-configured on your machine, but will be added on-the-fly when the jail is started; hence why the interface names re0 and lo1 are specified.

I then ran ezjail-admin start letsencrypt to actually start the jail. At this point, you may receive a warning about using jail.conf instead of JAIL_ variables if you're running FreeBSD 10; this is a fault in ezjail, but has no impact on the operation of the actual jail, as it's converted on-the-fly.

Now you're able to enter into the jail using ezjail-admin console letsencrypt, but before doing that, we need to install packages for the jail. To prepare it for running letsencrypt, install base dependencies using pkg -j letsencrypt install pkg git sudo bash on the host. This will install git and pkg inside the jail, so that letsencrypt-auto can install all packages it needs to operate. It will also install bash, necessary for running letsencrypt-auto, and sudo, necessary because letsencrypt expects to be run as an unprivileged user with sudo access.

Entering the jail, installing Let's Encrypt

Now we can enter the jail with ezjail-admin console letsencrypt. This'll give you a root shell inside your jail, from which we can create our unprivileged user with pw user add -n leuser -m -s /usr/local/bin/bash -G wheel, followed by passwd leuser to set the password you'll enter when sudo prompts for one. Then, set up sudo by running visudo and adding %wheel ALL=(ALL) ALL. You'll also need to run mkdir -p /etc/letsencrypt/www to create the directory which will be used for proving to the Let's Encrypt systems that you own the domain you're requesting a certificate for.

Drop down to the unprivileged user with su - leuser, which gives us a clean login environment as the user, then pull down the Let's Encrypt client with git clone https://github.com/letsencrypt/letsencrypt, and cd into the directory. From here, just run ./letsencrypt-auto, which will install all necessary packages, python modules, and finally sort out the letsencrypt binary itself.

The documentation then dictates that you swap out all instances of the letsencrypt command with ./letsencrypt-auto, but this isn't necessary every time, as letsencrypt-auto will check each and every time for updates to packages and python modules before actually doing anything, which can be very irritating to wait on. After running it once, you can simply make a symbolic link to the actual letsencrypt program by running ln -s $HOME/.local/share/letsencrypt/bin/letsencrypt $HOME/letsenc. From then on, you can simply run sudo ./letsenc ... from the unprivileged user's home directory.

Getting the host system ready for Let's Encrypt

As I mentioned, I wasn't comfortable with letting the letsencrypt client run its own server for domain validation. Thankfully, an alternate authentication method was made available to me: webroot. Using this, the letsencrypt client will simply create a file in the webroot directory, which will then be requested by the validation service, at http://your-domain/.well-known/acme/. It will also expect the webroot path to remain the same, irrespective of whether you're requesting a certificate for multiple domains.

Thankfully, since I know it expects the file to be served at /.well-known/acme/, and I run nginx, I was able to set up a redirect across all vhosts in the same manner that a standard 404 page is served in nginx: Adding a root location directive, beneath nginx's standard location 403.html, 404.html, 50x.html directive.

location ~^/\.well-known\/acme-challenge\/.*$ {
	root /usr/local/www/jail_symlinks/letsencrypt;
}

Followed by creating the symbolic link nginx would look for, via ln -s /usr/jails/letsencrypt/etc/letsencrypt/www /usr/local/www/jail_symlinks/letsencrypt. As I was using jails for serving web applications, I had a separate folder specifically for access to jails, but you can change this to any path you desire, of course.

Using Let's Encrypt to get a certificate

Now that all setup is out of the way, we can finally get our certificate. To do this, I ran sudo ./letsenc --agree-dev-preview --server https://acme-v01.api.letsencrypt.org/directory certonly -a webroot --webroot-path /etc/letsencrypt/www. This presented me with a few dialogs, prompting me for an email address and list of domains to request certificates for. Once this was done, it performed validation that I owned the domains in question. This succeeded, and the certificates were issued and stored into /etc/letsencrypt/live. From here, you can simply use symlinks to provide nginx with your fullchain.pem and privkey.pem files, which contain the certificate and chain, and private key, respectively.

At this point, I've successfully got Let's Encrypt set-up, and a certificate issued. After adding ssl_certificate /path/to/fullchain.pem and ssl_certificate_key /path/to/privkey.pem to my site's server block in nginx, I was up and running. You can also use the chain.pem file provided by letsencrypt to easily set up OCSP stapling. After doing this, your server block should contain something like the following (providing you also run ln -s /usr/jails/letsencrypt/etc/letsencrypt/live /etc/ssl/letsencrypt):

ssl_certificate /etc/ssl/letsencrypt/maff.scot/fullchain.pem;
ssl_certificate_key /etc/ssl/letsencrypt/maff.scot/privkey.pem;
ssl_trusted_certificate /etc/ssl/letsencrypt/maff.scot/chain.pem;
ssl_stapling on;
ssl_stapling_verify on;
resolver 8.8.8.8 valid=300;
resolver_timeout 10s;

A few notes on the resulting setup

Having gone through all this, setting up new domains with a Let's Encrypt certificate should be quite painless, as is renewing certificates. Let's Encrypt issues certificates with a 90-day validity period, and they recommend renewing within 60 days of issuing, such that if something goes wrong, you have a month's time to notice and fix it.

While looking at how renewal works, I noticed there was a letsencrypt-renewer program included with letsencrypt. I'm not sure if this is fully-functioning as of yet, as the documentation states that an automated renewal system is not yet in-place.