This site, in its infancy, was running Debian on Linode. Then I moved ¹ to FreeBSD on Vultr. Today marks a day of another migration: hello from OpenBSD running on OpenBSD Amsterdam.²

OpenBSD

OpenBSD is one of the three most popular BSD distributions. While NetBSD focuses on running on obscure hardware⁵, and FreeBSD has ZFS as its killer feature, OpenBSD is all about security⁶. I was very happy with FreeBSD, but at the same time, I was never fully confident in my ability to configure it securely. Not that my server hosts anything of real value⁷, but I still wouldn’t like a machine I administer to become a cog in some botnet. Between learning forensics and a new OS, the latter seems nicer.

OpenBSD’s official project goal⁸ states that even though they aim to provide the most secure OS, each developer has their own goals and can freely pursue them as long as the project adheres to these goals. It’s a very different approach to what we see anywhere else. There is no 10-year roadmap and constant consultations. Instead, we have a hacker-oriented⁹ culture. This resulted in multiple projects having their inception in OpenBSD, like OpenSSH or LibreSSL.

OpenBSD ships with a secure by-default mindset. All non-essential services are disabled, and those running are using sensible configurations. For example, I had huge problems configuring a firewall on FreeBSD, especially for IPv6¹⁰. On OpenBSD, it was much simpler.

OpenBSD being a BSD, provides a complete system - system and user space are developed together. No GNU tools are needed, as everything comes together. At the same time, BSDs come with a lot of surprising things out of the box. FreeBSD wowed me with Jails¹¹.

All in all, a lot of things I’ve learned on FreeBSD are easily transplantable to OpenBSD. They say that all BSDs are separate OSes, a stark difference from distributions of GNU/Linux. I fail to see it, as so much works the same. The package manager of FreeBSD may be more modern, and the separation between system space and user sapce¹² is not so evident here, but so many things work the same. I can not pretend to be a pro-BSDer, but I fail to see evidence of them diverging so narrowly to call them completely different OSes. But then again, maybe it’s just my poor judgment and love for POSIX.

And still no SystemD(1) in sight. I don’t have enough willpower to learn forensics or Rust, not even to mention an OS-level complex PID1 process.

OpenBSD Amsterdam

I had a similar exodus of server providers. First, it was Linode, then Vultr. Linode became useless when I wanted to try BSD. Vultr was great as it provided images of FreeBSD and OpenBSD for its VMs. But why stop halfway? Vultr doesn’t use BSD as the base system. While it may not be a big deal, I’ve recently learned of OpenBSD Amsterdam.¹³

OpenBSD Amsterdam is a small company based in (to the surprise of everyone reading this) Amsterdam. What’s even better is that they serve OpenBSD VMS from OpenBSD hosts via vmm(4) and vmd(8) - a small virtualization driver baked into OpenBSD. Cool.

What’s even cooler is that they give a significant part of their earnings to the OpenBSD Fundation.

I could not resist, and a day after learning about them, I had already paid for a full year.

Httpd(8) and Relayd(8)

So here we are: OpenBSD VM. What now? Let’s configure a webserver!

OpenBSD comes with three great tools out of the box:

• httpd(8) - an HTTP daemon
• relayd(8) - a relay daemon
• acme-client(1) - a client for Automatic Certificate Management Environment (ACME)

With those free things, we can serve static webpages over TLS. While you most likely already use NGINX or oApache¹⁴, those solutions are complex. They work amazingly in enterprise environments where you have people with doctorates in NGINX configuration, but most real-world examples don’t need that complexity. A static blog most likely doesn’t.

Let’s set it up.

Due to security concerns, OpenBSD comes with doas(1) instead of sudo(1). Copy /etc/examples/doas.conf file to /etc/doas.conf. For all intends, and purposes, from now on doas(1) will work the same as sudo(1).

When the system boots for the very first time, ports 80 and 443 are closed, and only the SSH port is open. This alone was a nice surprise for me. But it gets better: since all utilities are part of the OSes, they work together perfectly.

Assuming your domain is already pointing at the correct IPs, let’s start listening for unencrypted HTTP traffic. I will use “michal.sapka.me” as the domain in all examples.

First, Open /etc/httpd.conf in your favorite editor and add

1server "michal.sapka.me" {
2   listen on * port 80
3   root "/htdocs/michal-sapka-me"
4}

Then create a simple HTML file under /var/www/htdocs/michal-sapka-me/index.html.

Httpd(8) works chrooted to /var/www/, so it threats this directory as root. This makes the “root” option shorter to write, but it also means that the process doesn’t have access to anything outside of /var/www/. Even if an attacker can break in via the daemon, he will be locked in the www folder, so there is no risk to the rest of the system. As I said, OpenBSD is secure by default¹⁵.

All we need to do now it to enable the daemon via the handy rcctl(8) tool.

$ doas rcctl enable httpd

and to start it

$ doas rcctl start httpd

And boom. Opening http://michal.sapka.me shows on our site both on IPv4 and IPv6. One thing to note here is the limitation of up to HTTP 1.1. HTTP 2 is not supported.

Let’s add TLS, so we have this cute lock icon. For this, we will request a certificate from Let’s Encrypt using acme-client(1). If you used certbot, this will look familiar - just tidier.

First, let’s add config to /etc/acme-client.conf

 1authority letsencrypt {
 2        api url "https://acme-v02.api.letsencrypt.org/directory"
 3        account key "/etc/acme/letsencrypt-privkey.pem"
 4}
 5
 6authority letsencrypt-staging {
 7        api url "https://acme-staging.api.letsencrypt.org/directory"
 8        account key "/etc/acme/letsencrypt-staging-privkey.pem"
 9}
10
11domain michal.sapka.me {
12       domain key "/etc/ssl/private/michal.sapka.me.key"
13       domain full chain certificate "/etc/ssl/michal.sapka.me.crt"
14       sign with letsencrypt
15}

Lines 1-9 tell our acme-client(1) how to talk with Let’s Encrypt, while lines 11-15 allow us to request a certificate for our domain. OpenBSD comes preconfigured for Let’s Encrypt, so we just enable provided settings.

Nice! Next, we need to allow Let’s Encrypt challenges. Acme-client(1) will manage all required files, and Let’s Encrypt can read them via httpd(8). Again, like cogs in a well-oiled machine. By default, acme-client(1) will write to /var/www/acme, so we need to redirect /.well-known/acme-challenge/* there. Let’s change our httpd.conf:

1server "michal.sapka.me" {
2   listen on * port 80
3   root "/htdocs/michal-sapka-me"
4   
5   location "/.well-known/acme-challenge/*" {
6      root "/acme"
7      request strip 2
8    }
9}

We can now either restart httpd(8) or reload it. Let’s for the latter.

$ doas rcctl reload httpd

Now we can request the certificates

$ doas acme-client -v michal.sapka.me

OpenBSDs supplied tools don’t print unnecessary information to the user, so we add the -v to see what’s happening. Assuming everything went fine, let’s start serving the page with TLS!

For this, we will use relayd(8). We could use only httpd(8), but moving it one layer up is easier. Relayd(8) also gives us nice options for changing headers or moving some locations to a different process, like we will do with Plaroxy soon. This also shows us the big difference between this simple solution and NGINX: while NGINX shovels everything into one process and config, OpenBSD splits it into narrow focus areas.

Let’s open /etc/relayd.conf and add:

 1table <httpd> { 127.0.0.1 }
 2
 3http protocol "https" {
 4    tls keypair "michal.sapka.me"
 5
 6    match request quick header "Host" value "michal.sapka.me" forward to <httpd>
 7}
 8
 9relay "https" {
10    listen on 0.0.0.0 port 443 tls
11    protocol https
12    forward to <httpd> port 8080
13
14}
15relay "https6" {
16    listen on :: port 443 tls
17    protocol https
18    forward to <httpd> port 8080
19}

Now, I won’t go into much detail here, but what happens here is:

1. We create two relays, one for ipv4 and one for ipv6. One relay can listen on a single port for given IP. Each relay uses protocol “https” to modify and steer the request to a given process.
2. Both relays set up forwarding to httpd (IP taken from the table on the head of the file) on port 8080.
3. https protocol adds a TLS key pair for the session. We’ve got the files from Let’s Encrypt in the step above.
4. We then test each request, and if the host matches “michal.sapka.me” it will be forwarded to httpd(8).

You can also see that relayd(8) can listen on a given IP or all IPs (:: in case of IPv6)

But our httpd(8) listens only on port 80! Let’s fix that by changing the httpd.conf file:

1server "michal.sapka.me" {
2   listen on * port 8080

We also need to redirect HTTP to HTTPS. Since we use Relayd(8) only for HTTPS, this will be done in httpd(8). Let’s add a second server to our httpd.conf:

1server "michal.sapka.me" {
2        listen on * port 80
3        location * {
4                block return 301 "https://$HTTP_HOST$REQUEST_URI"
5        }
6}

Now, when the user enters the site, the flow will look like:

1. httpd(8) will respond to :80 requests and return a 301 redirect to HTTPS
2. relayd(8) will catch the request to :443 and forward it on port :8080 to httpd(8)
3. httpd(8) will serve our site and pass the response to relayd(8) again
4. relayd(8) can modify headers before returning the response to the client.

Talking about modifying headers, let’s apply some extra security! We can expand our https protocol with the following:

 1 # Return HTTP/HTML error pages to the client
 2    return error
 3    match request header set "X-Forwarded-For" value "$REMOTE_ADDR"
 4    match request header set "X-Forwarded-By" value "$SERVER_ADDR:$SERVER_PORT"
 5    match response header remove "Server"
 6    match response header append "Strict-Transport-Security" value "max-age=31536000; includeSubDomains"
 7    match response header append "X-Frame-Options" value "SAMEORIGIN"
 8    match response header append "X-XSS-Protection" value "1; mode=block"
 9    match response header append "X-Content-Type-Options" value "nosniff"
10    match response header append "Referrer-Policy" value "strict-origin"
11    match response header append "Content-Security-Policy" value "default-src https:; style-src 'self' \
12      'unsafe-inline'; font-src 'self' data:; script-src 'self' 'unsafe-inline' 'unsafe-eval'"
13    match response header append "Permissions-Policy" value "accelerometer=(), camera=(), \
14      geolocation=(), gyroscope=(), magnetometer=(), microphone=(), payment=(), usb=()"
15
16    # set recommended tcp options
17    tcp { nodelay, sack, socket buffer 65536, backlog 100 }
18
19    # set up certs
20    tls { no tlsv1.0, ciphers "HIGH:!aNULL:!SSLv3:!DSS:!ECDSA:!RSA:-ECDH:ECDHE:+SHA384:+SHA256" }

I won’t discuss details here as each header has a dedicated MDM webdoc. Most of the headers here are considered a standard.

Besides adding headers, we configure TLS here, disabling weak ciphers and old TLS versions and adding some standard config.

Lastly, we can automate refreshing the certificate via cron(8):

0~59 0~23 * * 1 acme-client michal.sapka.me &&  rcctl reload relayd

It looks almost like a normal cron. The “0~59” and “0~29” parts are unique to OpenBSD: Cron(8) will evenly split all tasks between specified time boxes so that no two jobs run simultaneously.

We now have created a fully working web server without any 3rd party packages. All OpenBSD provided, all secure, all simple, all cool as ice.

To further your knowledge, you can challenge the assumption that BSD has the best doc and read man pages for httpd.conf(5), relayd.conf(5), and acme-client.conf(5).

I also can’t recommend enough “Httpd and Relayd Mastery” by Michael W. Lucas¹⁶

Plaprox

One thing that OpenBSD doesn’t provide (yet?) is an HTTP proxy. I use Plausible for basic visitor analytics ¹⁷ here, and one of the cool things you can do is to break all adblockers via serving Plausible from my own domain¹⁸

After two evenings of failed attempts, I reminded myself that I am a programmer, and I wrote one myself. You can find it on my VCS page. It was a great learning exercise and a chance to touch Golang¹⁹ for the first time.

Assuming you have it running (it works on my machine!), let’s adjust our relayd(8). Plaprox listens on port 9090, and we want to relay all requests to /js/script.js there.

Let’s add it to our relays in relayd.conf:

 1table <plausibleproxyd> { 127.0.0.1 }
 2
 3http protocol "https" {
 4    
 5   # all our previous content omitted
 6
 7    match request quick path "/js/script.js"  forward to <plausibleproxyd>
 8    match request quick path "/api/event"  forward to <plausibleproxyd>
 9}
10
11
12relay "https" {
13    listen on 0.0.0.0 port 443 tls
14    protocol https
15    forward to <httpd> port 8080
16    forward to <plausibleproxyd> port 9090
17}
18relay "https6" {
19    listen on :: port 443 tls
20    protocol https
21    forward to <httpd> port 8080
22    forward to <plausibleproxyd> port 9090
23}

You can also move the port number to a table.

Remember that in Relayd(8) last one wins. We already have a match for the domain and added another matcher for the path. The request will be forwarded to the last marching matcher - so we put our new matchers at the end of the protocol definition.

Next

What are my next steps? It looks like OpenBSD much better supports the hardware of my laptop than FreeBSD, so I’ll try to migrate it.

Updates

2023-07-28: remove wrong information abot PF. 2023-07-30: fix invalid cron format