Chapter 1: The OpenBSD Storage Stack

“I came to OpenBSD for a week, and got stuck for twenty-eight years.”

—The Hitchhiker’s Guide to OpenBSD

While many Unix-like operating systems dump everything onto the disk and hope things work out, OpenBSD uses a meticulous partitioning scheme to improve confidentiality, integrity, and availability. Additionally, OpenBSD’s multi–architecture design demands abstraction layers between the storage hardware and the user–visible file system. Taking a little time to understand these two systems and how they interact will greatly simplify your work as a system administrator.

Filesystems and disks are complicated in that before you understand how any one piece works, you must understand every other component of the system. We will walk through each piece slowly, but you can’t skip around this chapter hoping to learn the topic. Everything is built on everything else. Relax and follow along.

Storage Layers

Like all operating systems, OpenBSD uses abstraction layers to separate the user–visible file system from the underlying hardware. OpenBSD keeps its storage stack as simple as possible. You do not get the fancy storage transformations available in the Logical Volume Managers used by other operating systems, but you are protected from the accompanying headaches.

The bottom layer of any storage stack is the physical hardware. Yes, all modern hard drives run their own operating system, but they are invisible to OpenBSD.3 Each device is assigned a number, starting at zero.

Your hardware architecture expects a certain partitioning method, which I’ll call a hardware partition. Hardware partitions are defined for the convenience of the hardware. Systems descended from the primordial IBM PC rely on either the Master Boot Record (MBR) or GUID Partition Tables (GPT). Newer platforms like arm64 have also adopted GPT. If you are running OpenBSD on less common hardware, check that platform’s manuals and the OpenBSD documentation to learn about its hardware–level partitioning. Don’t try to use partitioning methods the hardware doesn’t support.

While the previous two layers are all hardware–specific, we now have the first OpenBSD-specific layer: the disklabel. A disklabel is placed on a hardware partition, and subdivides the disk into OpenBSD partitions. Yes, the word partition refers to a subdivision in multiple layers of the disk. Be clear which layer you are working at.

An OpenBSD disklabel describes the whole disk, not a hardware partition. A disklabel can include references to physical partitions other than that reserved for OpenBSD. A GPT boot disk includes a FAT partition for the boot loader. In a standard install, the disklabel includes an entry for that partition. The hardware uses the native partitions to find the operating system, while OpenBSD uses the disklabel to manage storage.

OpenBSD represents both hardware and disklabel partitions by a device node. A device node lets the sysadmin declare “I am running this command on this exact partition.” You can mount disks by using their device node.

Decades ago, disk numbering remained constant. If the disk the operating system saw as sd0 contained the root file system, you could be confident that that drive would be sd0 every time you booted the system. Removable media, like floppy and optical drives, were numbered separately from hard disks. This is no longer the case. Drive numbering can shuffle around a system, especially when you consider hot–pluggable storage like USB flash drives. Every disklabel includes a Disklabel Unique Identifier (DUID), a unique hexadecimal number that OpenBSD uses to identify a partition. Configuration files reference the DUID rather than a device node.

The problem with DUIDs is that they’re unpronounceable and unmemorable. The most common name you’ll see for partitions is the device node. We will start by discussing device nodes so that we have a common language, and then move our way up through the storage stack.

Device Nodes

A device node is a file that provides a logical interface to a piece of hardware. Reading from a device node extracts data from the hardware. Writing to the device node sends data to the hardware. What happens to the data depends on the hardware. Not all device nodes do both: it makes no sense to read data from a speaker, for example. Disks are pretty straightforward—you read and write to them. OpenBSD puts device nodes in /dev and forbids them everywhere else.4

Device nodes are a common feature across Unixes. Many programs expect to be given a device node as an argument. Unfortunately, device node names range from cryptic to occult. For every meaningful /dev/speaker we have dozens of entries like /dev/rvnd3q. The device node names used on one Unix variant often bear no resemblance to those used on another Unix, and similar-looking ones might refer to entirely different hardware. Don’t let muscle memory from another Unix betray you on OpenBSD.

Each device node starts with a name giving the type of hardware the node represents, often followed by a number indicating which piece of equipment this is. /dev/speaker represents the computer’s speakers. Most computers have only one set of speakers at a time. The /dev/sd devices represent modern hard drives, and so are followed by a number. /dev/sd0 is the first hard drive the system found at boot, /dev/sd1 is the second, and so on.

The top of the script /dev/MAKEDEV lists all device nodes.

Common Device Names

While the sd device is the most common today, you will also encounter a few others. Here are the device node names for OpenBSD disk–type devices as of OpenBSD 7.2.

cd      optical drives

fd      floppy disks

rd      memory disks

wd      IDE disks, and SATA disks that pretend to be IDE

sd      SCSI, SAS, SATA, RAID, NVMe, and other non—IDE disks

vnd      filesystem images

Each type of device node has a man page. If you want to know about /dev/vnd devices, run man vnd.

You will also see letters after drive device nodes. These represent partitions, as we will see later this chapter.

Raw and Block Devices

Disks expect the operating system device driver to request data by entire sectors. If the operating system wants to read the disk’s MBR partition table, it commands the disk to send the contents of LBA sector zero. The hardware is optimized to process requests on a sector–by–sector basis. The standard device node is called a block device, or occasionally a cooked device.5

Sometimes, you don’t want to be efficient. Sometimes, a program needs to dump data to the disk using a very specific buffering or in a particular pattern. You are accessing the disk as a raw device. This is increasingly rare, but occasionally you will encounter a program that expects to use the raw device. Raw devices have the same device name as the block device, with an r in front of the name. Device sd0 is a block device. Device rsd0 represents the exact same hardware, accessed in raw mode. Partition /dev/sd0a represents the exact same partition as /dev/rsd0a. Choose the mode you access the device in by picking the device node.

Use the raw device node only if a program demands it. Almost all modern software is written for block devices. OpenBSD’s device nodes are generated at install and/or upgrade, and OpenBSD provides the raw devices just in case they are needed. Their presence does not mean they should be used.

Insufficient Devices Nodes

OpenBSD does not dynamically create device nodes. It has a set of defaults that are good enough for most users, but if you need additional device nodes you must create them. It ships with device nodes sd0 through sd9, supporting ten disks, and does not dynamically create new device nodes. If you have more than ten disks, you’ll run out of device nodes. In Chapter 7 I’ll boot from the install disk and have the same problem for disk sd0.

#...