Run Firecracker on DigitalOcean

Apparently, DigitalOcean supports nested virtualization on their instances, so it’s worth checking how many Firecracker instances I can run. I’m starting with the cheapest option $5/mo running Ubuntu 20.04 LTS.

Setup an instance:

$ # copy vmlinux and rootfs.ext4 that we built in the last post to the instance
$ apt-get update
$ apt install golang make
$ git clone
$ cd firectl
$ make
$ cp firectl /usr/local/bin
$ cd ../
$ mkdir firecracker
$ cd firecraker
$ wget
$ tar -zxf firecracker-v0.24.2-x86_64.tgz
$ cp firecracker-v0.24.2-x86_64 /usr/local/bin/firecracker
$ cp jailer-v0.24.2-x86_64 /usr/local/bin/jailer
$ firectl --kernel=./vmlinux --root-drive=./rootfs.ext4 --kernel-opts="console=ttyS0 noapic reboot=k panic=1 pci=off nomodules rw"

The first thing that I noticed is it takes between 2.4-2.9sec to start a Firecracker instance. Each instance takes about 60MiB RAM, so I should be able to launch around 13 virtual machines on the 5$/mo instance. Everything seems stable enough. I managed to run a simple Python application on each instance without any noticeable side-effects.

I’m kinda torn that between Kubernetes cluster (which doesn’t work as expected in DigitalOcean and bare-metal offerings are quite expensive) and creating a home-grown basic orchestrator; it is likely going to be a rabbit’s hole, I need to research whether there’re any options I can use as an alternative.

DigitalOcean provides premium instances that run on the second generation Intel Xeon processor, so it’s interesting to see if there’re any benefits of using them.

KVM is not enabled (but it’s supported) on these instances by default, so I followed instructions from to enable it.

$ apt-get install qemu-kvm libvirt-daemon-system libvirt-clients bridge-utils
$ adduser `id -un` libvirt
$ adduser `id -un` kvm
$ virsh list --all
 Id   Name   State

$ rmmod kvm
$ modprobe -a kvm
$ mkdir /dev/kvm
$ chown root:libvirt /dev/kvm
$ apt install cpu-checker
$ kvm-ok 
INFO: /dev/kvm exists
KVM acceleration can be used
$ lsmod | grep -i kvm
kvm                   663552  0
$ ls /lib/modules/$(uname -r)/kernel/arch/x86/kvm/*
/lib/modules/5.4.0-51-generic/kernel/arch/x86/kvm/kvm-amd.ko    /lib/modules/5.4.0-51-generic/kernel/arch/x86/kvm/kvm.ko
$ modprobe kvm-intel
modprobe: ERROR: could not insert 'kvm_intel': Input/output error

Ouch, kvm_intel is not enabled and cannot be loaded. Is it that nested virtualization is not enabled on these instances or is it a defect in the kernel it’s running? I’ll try another distro.

Ubuntu 18.04 has an old go package, so before compiling firectl we need to install at least 1.16:

$ add-apt-repository ppa:longsleep/golang-backports
$ apt update
$ apt install golang-go

Also, kvm_intel is loaded, and it looks promising:

$ lsmod | grep -i kvm
kvm_intel             217088  0
kvm                   614400  1 kvm_intel
irqbypass              16384  1 kvm

After I followed firecracker setup instructions, I was able to start an instance. I didn’t notice any visible performance improvements. I didn’t run an extensive test, so it could be that the devil is in the detail and CPU-wise, it could be faster. However, for now, my concern is boot time.

As the last experiment for today, I tried another $60/mo general-purpose instance, which improved boot time to ~2sec. Perhaps, I need some ansible scripts to automate this process to validate other offerings.