What is iPXE?

What is iPXE?

It comes up often, especially at Equinix Metal, where it is one way you can deploy servers with a custom operating system.

So, what is it?

Let's take a tour inside our computers, and maybe a bit of history to go along with it. Keep in mind that this isn't a detailed guide to debugging a boot process, let alone designing a new one. Rather, it is a simplified description to put iPXE in context.

The Boot Process

We begin with power.

When you turn on a computer, you are providing it with power. That power runs through the computer, and it does... nothing. It just sits there, powered on, but not doing anything. Electricity is just that: voltage and amperage, but not much else.

Every computer needs a way to start up, to find its operating system code to run. When you turn on that computer, whether it's your super-powerful smartphone, laptop, or server, it needs to find that code to run.

On top of the CPU and memory, there are a few other chips on the computer. One of them contains the firmware. All computers have firmware. Firmware is low-level special software, embedded in one of those chips, that always runs and has direct access to the hardware. Most importantly, it runs on power-on. This firmware doesn't have anything like the capabilities of a full operating system. What it does have is enough knowledge to configure the hardware and find the operating system code to run.

Your computer gets power, which starts the CPU. The CPU is built to look for the firmware on startup.

In the early days, and for many years, Basic Input Output System, or BIOS, was the firmware for Intel-based computers. As the name suggests, basic input and output was pretty much what it did.

In those days, the firmware knew how to find the operating system code on a floppy disk or hard drive. For those who had early computers, you knew that you pressed some key combination, often F2, to get the "firmware menu." Somewhere in that menu were choices of which device you wanted to use to boot, or sometimes a boot order: try this one, then that one, then the next one.

The BIOS was supplied by one of a handful of companies, like American Megatrands (AMI), Phoenix, or Award.

The problem?

It required you to physically visit each computer to update the operating system, or even run a different one for a single boot. When all computers were being connected to networks, this seemed a rather serious limitation.

Each chipset manufacturer started to build its own standard for booting off the network. The computer would be provided with configuration information on how to find its operating system off of the network, while a network server would speak the right protocol to "dish up" the operating system. Conceptually, this isn't that different than what you do when browsing the Internet. Your browser knows how to speak http, you tell it where to go, and it downloads that great page that you want to see from a server, which knows how to speak the same protocol.

The process worked like this:

1. You configured the computer's firmware just once, assuming it supported some form of network booting, to point it at a boot server
2. You booted the computer
3. The computer reached out to the network to get its operating system
4. The configured server returned everything needed to boot
5. The computer booted

Introducing PXE

In 1998, the Preboot Execution Environment, or PXE (pronouned "PIK-see"), specification was released by Intel. PXE is a set of protocols to enable network booting of compatible servers. PXE was based on the protocols available at the time:

1. DHCP
2. TFTP

The computer booting used DHCP -- the well-known protocol for a computer broadcasting to the network, "Here I am! May I please have a network address?" -- and specifically its extensions for additional data, to get an address for itself, but also the address to a server with its operating system.

It then used the provided server address and the TFTP protocol to download the operating system.

You can read all about PXE at the Wikipedia article.

This all worked pretty well, if somewhat slowly.

The first problem? It didn't scale. It was designed before the massive growth of the internet, both as a public facility and as internal networks inside companies and data centres. It also stuck to some pretty old protocols, especially TFTP. In an era when you can stream an entire HD movie from across the internet, this was slow and rigid.

The second problem? PXE was closed-source. You only could use it on compatible licensed chipsets. Sure, the protocol was open, but the implementation was not. This meant that if you wanted to do something different, you had to convince its maintainers to change it. And if you wanted to use it on other devices, you had to write it yourself.

Enter iPXE

iPXE is an open-source implementation of PXE, with a lot more on top. In addition to the basic capabilities of PXE, iPXE supports booting off of many other protocols, including HTTP(S), iSCSI and Fibre Channel over Ethernet (FCoE). In some cases, it supports booting over WiFi or cellular networks.

Why does this matter?

Let's look at the PXE process, and then compare it with iPXE. For PXE:

1. I need a chip from a vendor with PXE support built in and licensed.
2. I need to configure the firmware to use PXE.
3. I need to set up a DHCP server, both to deliver IP addresses and to deliver the TFTP server address.
4. I need to set up a TFTP server, with the operating system I want to boot.

For iPXE:

1. I need a computer with a network card that supports iPXE.
2. I need to configure the firmware to use network boot, and the URL to boot from.
3. I need to set up a DHCP server, to deliver my IP address; this is optional, if I have a fixed IP.
4. I need a Web server with the operating system I want to boot. I am pretty sure there are lots of those lying around.

iPXE is faster, more flexible, and easier to work with.

Why does all of this matter when deploying bare metal servers, like with Equinix Metal?

With bare metal, you need some way to get the operating system you want to the server. If that server is in a data centre, you likely have very limited, if any, access to it, and flying to Singapore to plug in a USB stick is not an option. You need to be able to do it remotely.

You also need to be able to do it at scale. If you are deploying 100 servers, you don't want to have to visit each one to install the operating system.

With iPXE, you can configure the servers remotely to boot iPXE, providing a URL of a Web server that you control or one managed as a service, for example netboot.xyz.

Equinix Metal makes your life easier by providing a slew of supported operating systems to choose from. As a true cloud provider, you can deploy your server by just picking the operating system in the console, or via the API, and Equinix Metal makes sure that OS is deployed on your server.

What if you want an OS not provided by Equinix Metal? Or a customized variant, your own "golden image"?

Set it up on a Web server, and use iPXE to boot it for you.

iPXE is a powerful tool for deploying bare metal servers, and Equinix Metal makes it easy to use.

When to use iPXE

So when should you use iPXE to boot your computer?

Let's start with when not to.

You won't need to use iPXE if your computer already has the operating system you want set up and configured on its local drive. You definitely don't want to use it if you are running disconnected from a network, where you won't be able to boot it that way at all.

Finally, if your cloud provider has the operating system you want ready to run, chances are your life is going to be much easier just using that image. Equinix Metal has more than a dozen of the most popular and useful operating systems and their variant versions available and supported, just by selecting it when deploying a server. Almost all other cloud providers have the same setup.

When should you use it?

When the exact setup you need isn't available. If you need not just Ubuntu 22.04, and not just 22.04 configured via cloud-init, but tweaked just right at the operating system level before booting, then iPXE is your solution.

Taken to the next step, if you need an operating system that isn't offered, iPXE allows you to boot any compatible operating system you want. For example, if you purpose-build a minimal Linux operating system using linuxkit, then you can be sure no cloud provider will have it. Build your image, place it where the server can get to it, and boot via iPXE.

And how do you know that iPXE not only is flexible for your needs, but highly reliable?

Ironically, when Equinix Metal boots its supported operating systems, it actually is booting iPXE! It is just pointing that boot configuration towards its supported ready-to-run operating systems.

References

All about PXE and iPXE:

• Preboot Execution Environment (PXE) on Wikipedia
• iPXE home page
• OSDev's PXE page

Using iPXE and PXE hosting services mentioned:

• netboot.xyz
• official Equinix Metal documentation on booting custom operating systems with iPXE
• SmartOS on Equinix Metal - good example of iPXE in action