FreeBSD on ARM64


So I’ve been thinking about this since 2017, but only yesterday signed up for an account and played around with the ARM64 offering.
Turns out it’s pretty great! KVM boots into UEFI, there’s a local VirtIO disk attached, no NBD junk required. So we can definitely run FreeBSD.

I managed to “depenguinate” a running instance, the notes are below.
Would be great if Scaleway offered an official image instead :wink:

For some reason, unlike on x86, mounting additional volumes is not allowed on ARM64 instances. So we’ll have to move the running Linux to a ramdisk using pivot_root and then we can do whatever to our one and only disk.

Spin up an instance with Ubuntu Zesty and ssh in.

Prepare the system and change the root to a tmpfs:

apt install gdisk
mount -t tmpfs tmpfs /tmp
cp -r /bin /sbin /etc /dev /root /home /lib /run /usr /var /tmp
mkdir /tmp/proc /tmp/sys /tmp/oldroot
mount /dev/vda /tmp/oldroot
mount --make-rprivate /
pivot_root /tmp /tmp/oldroot
for i in dev proc sys run; do mount --move /oldroot/$i /$i; done
systemctl daemon-reload
systemctl restart sshd

Now reconnect to ssh from a second terminal (note: rm the connection file if you use ControlPersist in ssh config), then exit the old session. Kill the old sshd process, restart or stop the rest of the stuff using the old disk:

pkill -f notty
sed -ibak 's/RefuseManualStart.*$//g' /lib/systemd/system/dbus.service
systemctl daemon-reload
systemctl restart dbus
systemctl daemon-reexec
systemctl stop user@0 ntp cron systemd-logind
systemctl restart systemd-journald systemd-udevd
pkill agetty
pkill rsyslogd

Check that nothing is touching /oldroot:

lsof | grep oldroot

There will probably be an old dbus-daemon, kill it.

And finally, unmount the old root and overwrite the hard disk with a memstick image:

umount -R /oldroot
xzcat FreeBSD-12.0-CURRENT-arm64-aarch64-20180719-r336479-mini-memstick.img.xz | dd if=/dev/stdin of=/dev/vda bs=1M

(Look for the newest snapshot here, don’t copy paste the July 19 link above if you’re reading this in the future. Actually maybe use a release instead of CURRENT…)

Now, fix the GPT: move the secondary table to the end of the disk and resize the table.
It’s important to resize here, as FreeBSD does not do that and silently creates partitions that won’t persist across reboots :smiley:

gdisk /dev/vda

And reboot. (You might actually want to hard reboot here: for some reason on the first reboot from Linux, pressing the any-key to enter the prompt in the loader hangs the console for me.)

I didn’t have to go into the ESC menu and choose the local disk in the boot manager, it seems to boot from disk automatically.

Now we’re in the FreeBSD EFI loader.
For some reason, the (recently fixed?) serial autodetection from EFI is not working correctly. Or something.
So you don’t get console output by default.
To fix, you have to run these commands in the boot loader command prompt:

set console=comconsole,efi

(Ignore the warning about comconsole not being a valid console.
Since there’s at least one (efi) that the loader thinks is valid, it sets the whole variable.)

(UPD: shouldn’t be necessary in the next snapshot)

Now it’s a regular installation process!
When asked about partitioning, choose Shell, and manually add a partition and set up a root filesystem:

gpart add -t freebsd-zfs -a 4k -l zroot vtbd0
zpool create -R /mnt -O mountpoint=none -O atime=off zroot /dev/gpt/zroot
zfs create -o canmount=off -o mountpoint=none zroot/ROOT
zfs create -o mountpoint=/ zroot/ROOT/default
zfs create -o mountpoint=/usr zroot/ROOT/default/usr
zfs create -o mountpoint=/var zroot/ROOT/default/var
zfs create -o mountpoint=/var/log zroot/ROOT/default/var/log
zfs create -o mountpoint=/usr/home zroot/home
zpool set bootfs=zroot/ROOT/default zroot

(In this example, I set up ZFS with a beadm-compatible layout which allows me to use Boot Environments.)

In the post-install chroot shell, fix some configs like so:

echo 'opensolaris_load="YES"' >> /boot/loader.conf
echo 'console="comconsole,efi"' >> /boot/loader.conf
echo 'vfs.zfs.arc_max="512M"' >> /boot/loader.conf
sysrc zfs_enable=YES

(Yeah, for some reason, the loader does not load zfs.ko's dependency opensolaris.ko automatically here. idk what even. It does on my desktop and laptop.)

Now you can reboot into the installed system!!

Here’s how you can set up IPv6 (and root’s ssh key) auto configuration on boot:

pkg install curl
curl > /usr/local/bin/scw-metadata
chmod +x /usr/local/bin/scw-metadata
echo '#\!/bin/sh' > /etc/rc.local
echo 'PATH=/usr/local/bin:$PATH' >> /etc/rc.local
echo 'eval $(scw-metadata)' >> /etc/rc.local
echo 'echo $SSH_PUBLIC_KEYS_0_KEY > /root/.ssh/authorized_keys' >> /etc/rc.local
echo 'chmod 0400 /root/.ssh/authorized_keys' >> /etc/rc.local
echo 'ifconfig vtnet0 inet6 $IPV6_ADDRESS/$IPV6_NETMASK' >> /etc/rc.local
echo 'route -6 add default $IPV6_GATEWAY' >> /etc/rc.local
mkdir /run
mkdir /root/.ssh
sh /etc/rc.local

One remaining issue: incoming TCP connections (like ssh, or simply testing with nc) from the public IPv4 address don’t work — I see incoming packets in tcpdump but no replies o_0 (IPv6 works perfectly, at least)

And to fix incoming TCP connections, configure the DHCP client to change the broadcast address:

echo 'interface "vtnet0" { supersede broadcast-address; }' >> /etc/dhclient.conf
killall dhclient
dhclient vtnet0

Other random notes:

  • keep in mind that -CURRENT snapshots come with a debugging kernel by default, which limits syscall performance by a lot, you might want to build your own with config GENERIC-NODEBUG
  • also disable heavy malloc debugging features by running ln -s 'abort:false,junk:false' /etc/malloc.conf (yes that’s storing config in a symlink)
  • you can reuse the installer’s partition for swap

P.S. Thanks to for good information about Linux pivot_root!

FreeBSD on x86
How to run a custom kernel on a VPS (VC1S, etc.)

@myfreeweb great job! :slight_smile:

Have you tried changing the broadcast-addess in /etc/dhclient.conf ?

interface "vtnet0" {
  supersede broadcast-address;

It works on x86.

1 Like

Thanks a lot!! Looks like I still have some things to learn about networking basics :smiley:


UPDATE: fix for disk performance: add


to /boot/loader.conf.

Somehow the PCIe controller was blacklisted from using MSI-X (even though all I can find in the list is old Intel/AMD/VMware, nothing about QEMU/KVM or Cavium). And the machine was using legacy interrupts, which turned out to be super slow.


Unfortunately, Scaleway have broken this. At the end of December 2018, they “upgraded” their systems so that arm64 instances are forceably PXE-booted using one of their limited selection of Linux kernel images.

When I raised a ticket saying that I didn’t need or want to PXE boot, they said that wasn’t supported and therefore they wouldn’t assist.


I guess I shouldn’t ever reboot my server now? D:

WTF. How did they even do this? Does mashing Esc on the console just not work anymore?


Local boot would be useful also for other images (like Fedora IoT)


hmm! The local boot documentation says “It will become available for our ARM64 virtual cloud server in a few days.”

Seems like these days are taking forever…

#9 STILL says

Local boot is currently available for all our x86_64 based virtual cloud instances. It will become available for our ARM64 virtual cloud instances in a few days.

This is just FALSE ADVERTISING at this point, it’s been HALF A YEAR instead of “a few days” and it’s still broken!

As much as I dislike Amazon’s business practices, and as expensive as it is, AWS EC2 offers aarch64 instances (with much more powerful Cortex-A72 cores, but fewer of them) and FreeBSD now has official AMIs for them. You can see the AMI IDs in freebsd-snapshots posts, e.g.: