Create a Custom Firecracker Image

In the previous post I confirmed that I can run a Firecracker instance locally. However, it’s using a hello-world image, and I need to customize it. Today, I’m putting a custom image on a Firecraker.

Basically, you have to do two things:

Make a Linux kernel. An official tutorial how to do this: https://github.com/firecracker-microvm/firecracker/blob/master/docs/rootfs-and-kernel-setup.md
Make an ext4 filesystem image

Create a Kernel image

Here’s a quick step-by-step guide to building your own kernel that Firecracker can boot:

Get the Linux source code:

$ git clone https://github.com/torvalds/linux.git
$ cd linux.git

Check out the Linux version you want to build (e.g. we’ll be using v5.10 here):
```
$ git checkout v5.10
```
You will need to configure your Linux build. You can start from the recommended config by copying it to .config (under the Linux sources dir). You can make interactive config adjustments using:
```
$ make menuconfig
```
Build the uncompressed kernel image, depending on your .config file and may need to answer a few questions:
```
$ make vmlinux
```
This is going to take some time, so take a moment and pour yourself some tea.
Upon a successful build, you can find the uncompressed kernel image under ./vmlinux.
```
$ cp vmlinux /tmp
```

Create a rootfs image

A rootfs image is just a file system image, that hosts at least an init system. For instance, this guide uses an EXT4 FS image with OpenRC as an init system. Note that, whichever file system you choose to use, support for it will have to be compiled into the kernel, so it can be mounted at boot time.

To build an EXT4 image:

Prepare a properly-sized file. We’ll use 50MiB here, but this depends on how much data you’ll want to fit inside:
```
$ cd /tmp
$ dd if=/dev/zero of=rootfs.ext4 bs=1M count=50
```
Create an empty file system on the file you created:
```
$ mkfs.ext4 rootfs.ext4
```

You now have an empty EXT4 image in rootfs.ext4, so let’s prepare to populate it. First, you’ll need to mount this new file system, so you can easily access its contents:

$ mkdir /tmp/my-rootfs
$ sudo mount rootfs.ext4 /tmp/my-rootfs

The minimal init system would be just an ELF binary, placed at /sbin/init. The final step in the Linux boot process executes /sbin/init and expects it to never exit. More complex init systems build on top of this, providing service configuration files, startup / shutdown scripts for various services, and many other features.

For the sake of simplicity, let’s set up an Alpine-based rootfs, with OpenRC as an init system. To that end, we’ll use the official Docker image for Alpine Linux:

First, let’s start the Alpine container, bind-mounting the EXT4 image created earlier, to /my-rootfs:
```
$ docker run -it --rm -v /tmp/my-rootfs:/my-rootfs alpine
```
Then, inside the container, install the OpenRC init system, and some basic tools:
```
# apk add openrc
# apk add util-linux
```

And set up userspace init (still inside the container shell):

# # Set up a login terminal on the serial console (ttyS0):
# ln -s agetty /etc/init.d/agetty.ttyS0
# echo ttyS0 > /etc/securetty
# rc-update add agetty.ttyS0 default

# # Make sure special file systems are mounted on boot:
# rc-update add devfs boot
# rc-update add procfs boot
# rc-update add sysfs boot

# # Then, copy the newly configured system to the rootfs image:
# for d in bin etc lib root sbin usr; do tar c "/$d" | tar x -C /my-rootfs; done
# for dir in dev proc run sys var; do mkdir /my-rootfs/${dir}; done

# # All done, exit docker shell
# exit

Finally, unmount your rootfs image:
```
$ sudo umount /tmp/my-rootfs
```

You should now have a kernel image (vmlinux) and a rootfs image (rootfs.ext4), that you can boot with Firecracker.

Now, you can run Firecracker with your custom images:

$ firectl --kernel=/tmp/vmlinux --root-drive=/tmp/rootfs.ext4 --kernel-opts="console=ttyS0 noapic reboot=k panic=1 pci=off nomodules rw"

Let’s do some automation and pre-create a Dockerfile and then copy the contents of the container into a filesystem image. For the testing, I’m also instal python3 and therefore increased the image to 150MiB.

Dockerfile:

FROM alpine:3.7

RUN apk add openrc
RUN apk add util-linux
RUN apk add python3

# Set up a login terminal on the serial console (ttyS0):
RUN ln -s agetty /etc/init.d/agetty.ttyS0
RUN echo ttyS0 > /etc/securetty
RUN rc-update add agetty.ttyS0 default

# Make sure special file systems are mounted on boot:
RUN rc-update add devfs boot
RUN rc-update add procfs boot
RUN rc-update add sysfs boot

# Change root password
RUN echo 'root:root' | chpasswd

Here’s a tiny script by Julia Evans from this post https://jvns.ca/blog/2021/01/23/firecracker--start-a-vm-in-less-than-a-second/, which creates a rootfs:

IMG_ID=$(docker build -q .)
CONTAINER_ID=$(docker run -td $IMG_ID /bin/bash)

MOUNTDIR=mnt
FS=rootfs.ext4

mkdir $MOUNTDIR
qemu-img create -f raw $FS 150M
mkfs.ext4 $FS
sudo mount $FS $MOUNTDIR
sudo docker cp $CONTAINER_ID:/ $MOUNTDIR
sudo umount $MOUNTDIR

Run a Firecracker:

$ firectl --kernel=/tmp/vmlinux --root-drive=./rootfs.ext4 --kernel-opts="console=ttyS0 noapic reboot=k panic=1 pci=off nomodules rw"
...
Welcome to Alpine Linux 3.7
Kernel 5.10.0 on an x86_64 (ttyS0)

(none) login: root
Password: 
Welcome to Alpine!

The Alpine Wiki contains a large amount of how-to guides and general
information about administrating Alpine systems.
See <http://wiki.alpinelinux.org>.

You can setup the system with the command: setup-alpine

You may change this message by editing /etc/motd.

login[612]: root login on 'ttyS0'
(none):~# python3 -c "print(40 + 2)"
[   15.635148] random: python3: uninitialized urandom read (24 bytes read)
42

I’m still not sure whether I like Docker setup to create rootfs, but as it works I’ll stick with it for now. I guess, it must be a different way how to set it up, and I’ll leave till next time.

Hopefully, tomorrow I can find some extra time and figure out how to connect Firecracker guest to the network.