KVM virtualization with Allwinner A20 on Debian: libre, low-power, low-cost
Introduction
Various cheap ARM boards based on the Allwinner A20 SoC are available already for a few years. The first EOMA68 computer [1] will be also based on this chipset. Not many users know that the Allwinner A20 supports hardware-supported virtualization as well. Its Cortex A7 cores allow running hardware-accelerated ARM virtual machines (guests) using KVM or Xen.
While Allwinner has been blamed to violate the GPL for years [2], their A20 SoC is imho one of the best choices today when it comes to building a small and libre server for SOHO use (thanks to the hard work of the Allwiner-independent Linux-Sunxi community). While many SoCs found on popular boards like those from the Raspberry Pi family require proprietary blobs, the A20 works with a free bootloader and requires no proprietary drivers or firmware for basic operation.
The virtualization on A20 hosts works out of the box on Debian Jessie with the stock kernel and official packages in main — without cross-compiling, patching or other tinkering (this was not the case in the past, see [3]). This also means that updating your host and guests later will be easy and painless. Creating and managing guests can be done with virt-manager [4] – a secure and comfortable graphical user interface licensed under GPLv3.
After first discussing some A20 hardware options, this guide takes the example of the Olimex “A20-OLinuXIno-LIME2″ board [5] and shows how to turn it into a virtualization host. Then shows how create and manage guest-VMs on the virtualization host. The guide assumes that you are running a a GNU/Linux-based desktop system from which you want to manage the A20 device.
Disclaimer
All data and information provided in this article is for informational purposes only. The author makes no representations as to accuracy, completeness, currentness, suitability, or validity of any information on this article and will not be liable for any errors, omissions, or delays in this information or any losses, injuries, or damages arising from its display or use. All information is provided on an as-is basis.
In no event the author we be liable for any loss or damage including without limitation, indirect or consequential loss or damage, or any loss or damage whatsoever arising from loss of data or profits arising out of, or in connection with, the use of this article.
Hardware choices
There are plenty of boards with the Allwinner A20. However, only few are known to work out of the box on Debian Jessie. The particular page on the Debian Wiki [6] mentions the following boards in particular:
- Cubietech Cubieboard2
- Cubietech Cubieboard3 (“Cubietruck”)
- LeMaker Banana Pro
- Olimex A20-OLinuXino-LIME
- Olimex A20-OLinuXino-LIME2 (only the regular one, not the eMMC variant!)
- Olimex A20-Olinuxino Micro
While some of these boards feature Gigabit ethernet and SATA, only the Cubieboard 3 has 2 GB of RAM. To me, this seems to be the best choice for a A20-based KVM virtualization host. Since I only had a spare Olimex A20-OLinuXino-LIME2 board at hand, this guide uses this board as example.
Beware: The “A20-OLinuXino-LIME2″ and the “A20-OLinuXino-LIME2-eMMC” are not the same! Debian provides no firmware for the “A20-OLinuXino-LIME2-eMMC” and I could not get it to work at all on Debian. Although I thought that they would be the same except for the eMMC flashg, the firmware for the regular “”A20-OLinuXino-LIME2″ did NOT work for me at all!
Base installation
The article in the Debian wiki provides the necessary information on installing Debian Jessie using the text-based Debian-Installer. Make sure you have a microSD card with a good 4K random I/O performance or the installation will take forever and your A20 system will run terribly slow afterwards (see my article comparing performance of various microSD cards).
If you don’t have a serial cable and want to install using the HDMI output, you need to use the installer images from unstable. The easiest way to do is to fetch the firmware file from unstable and the partition image from Jessie. Then write them to your microSD card (replace /dev/sdX with your particular device):
$ zcat firmware.A20-OLinuXino-Lime2.img.gz partition.img.gz > /dev/sdX
Next, insert the microSD card into your device, connect your device to your LAN and power it up. Then install Debian as usual using the text-based installer. During the installation, sure to create a root account (needed for KVM) and a ext2 boot partition (the safest method here is to use the guided installer). When tasksel gets called, make sure to install the tasks/packages “SSH Server” and “Standard system utilities”.
Note for users of the German mirrors: Using the mirror “ftp.de.debian.org” will break your installation as something seems to be missing there as of 2016-11-05. Using “ftp2.de.debian.org” works fine.
Installing the KVM virtualization
By default, interactive root logins are not allowed on Debian. Therefore, make sure you copy over your SSH public key to your a20-box or simply enable interactive root logins over SSH by changing the following option in /etc/ssh/sshd_config:
#PermitRootLogin without-password PermitRootLogin yes
Then restart the SSH server:
# service ssh restart
Now you should be able to log in directly as root. Next, install the virtualization packages:
# apt install libvirt-daemon-system
Reading package lists... Done Building dependency tree Reading state information... Done The following extra packages will be installed: ... 0 upgraded, 105 newly installed, 0 to remove and 0 not upgraded. Need to get 44.4 MB of archives. After this operation, 182 MB of additional disk space will be used. Do you want to continue? [Y/n]
Now fire up virt-manager on your desktop and make sure you can connect to your a20-box:
Creating and installing a guest
For running ARM virtual machines you need a kernel and DTBs which support the VExpress-A15 chipset (the ARM reference board usually emulated on ARM). This is already provided in stock Debian, so there is no need to compile anything yourself.
Regarding the guest, you can choose any Linux you want. In the following example, we will install a Debian Jessie guest using the Debian installer. Therefore we need to download the to the Virtualization host. This time, we don’t need a partition image but can use the usual the initrd installer-Image from the Debian server. SSH into the virtualization host and download it:
wget http://ftp.uk.debian.org/debian/dists/jessie/main/installer-armhf/current/images/netboot/initrd.gz -O initrd-installer-jessie.gz
For the installation, you will also need a different kernel because in the Kernel installed on the host the network drivers are in initrd, but the Installer’s initrd assumes they are in the kernel. Therefore, fetch a kernel for the installer:
wget http://ftp.uk.debian.org/debian/dists/jessie/main/installer-armhf/current/images/netboot/vmlinuz -O vmlinuz-installer-jessie
Now, fire up virt-manager on your desktop and connect to the Virtualization host. Then, start the wizard for creating guests using “create new virtual machine”. On the first screen, change the machine type to “vexpress-a15″:
On the next screen, specify a storage (just create one using the dialog following “Browse”), and also use “Browse” to locate the kernel and initrd images so you specify the ones we just downloaded. For the DTB, we’ll use the one that is part of Debian’s stock kernel and resides under /usr/lib/linux-image-3.16.0-4-armmp-armmp-lpae/vexpress-v2p-ca15-tc1.dtb (make sure it corresponds to the version on your a20-host! TODO: Is there any symlink which points to the current version?)). The kernel args are also very important, or you will not get any output. For this line, specify the following:
root=/dev/vda1 console=ttyAMA0,115200 rootwait
Finally, select OS type and version appropiately. Your dialog should look like this:
Then, specify RAM (e.g. 256MB) and the number of CPUs (e.g. 1) you want to give the guest and jump to the last screen. Here, give your guest a nice name and make sure you check the “Customize configuration before install” checkbox before you click “Finish”:
Otherwise, you would end up with an error message like this:
Unable to complete install: 'internal error: early end of file from monitor: possible problem: kvm_init_vcpu failed: Invalid argument
In the configuration of the VM, under “Processor”, change the configuration from “Hypervisor Default” to “Application Default”:
To get better performance, also change the BUS of your virtual disk to “VIRTIO” (by default, it would emulate an SD card):
And do the same for the network adapter:
Finally, fire up the guest using “Begin installation”. If everything goes fine, you should see the kernel boot and be presented with the welcome screen of the installer. For jessie, it should look like this:
If you selected the kernel and initrd from stretch/sid you should get a nicer color screen (make sure you set the baudrate of the console to 115200 or you will get a disorted output!):
When partitioning the guest, just create a single root partition spanning the whole (virtual) device. The guest will always boot using externally specified kernels, dtbs and initrds, therefore there is no use in creating a /boot partition as the “guided install” would do.
Near the end of the installation, you will be notified that no bootloader could be installed. You can safely ignore this message:
After finishing the installation, the system will boot again into the installer because the initrd is still active. To change this, power off the guest (“Force Off”) and specify in the boot options to use the kernel and initrd image of your A20 host instead (whenever they will be updated on the host, the guests will also get the update on their next boot):
Now your guest should finally succeed to boot up:
And you can check that it indeed uses the current A20 kernel on the host and virtualizes the VExpress15 SoC:
Benchmarks
Finally, I want to provide some benchmarks so you can get a feeling about the impact of the virtualization. The benchmarks were done using a guest with 2 CPUs and 512MB memory assigned.
IO/Performance
For a first I/O benchmark, I used hdparm.
On the host:
$ hdparm -tT /dev/mmcblk0
/dev/mmcblk0: Timing cached reads: 814 MB in 2.00 seconds = 406.33 MB/sec Timing buffered disk reads: 66 MB in 3.01 seconds = 21.93 MB/sec
On the guest:
$ hdparm -tT /dev/vda
/dev/vda: Timing cached reads: 694 MB in 2.00 seconds = 346.49 MB/sec Timing buffered disk reads: 30 MB in 3.15 seconds = 9.52 MB/sec
CPU processing
For benchmarking processing, I used the openssl suite to do a few simple AES benchmarks:
$ openssl speed aes
On the host:
... The 'numbers' are in 1000s of bytes per second processed. type 16 bytes 64 bytes 256 bytes 1024 bytes 8192 bytes aes-128 cbc 20267.83k 22390.70k 23325.10k 23575.89k 23642.11k aes-192 cbc 17594.13k 19464.20k 19956.57k 20102.83k 20146.86k aes-256 cbc 15727.25k 17158.89k 17592.58k 17706.67k 17738.41k
On the guest:
... The 'numbers' are in 1000s of bytes per second processed. type 16 bytes 64 bytes 256 bytes 1024 bytes 8192 bytes aes-128 cbc 19784.01k 22100.48k 22697.56k 23272.20k 23288.29k aes-192 cbc 17363.72k 19097.02k 19643.68k 19786.41k 19800.53k aes-256 cbc 15455.28k 16939.28k 17374.44k 17415.85k 17504.58k
Conclusion
With one of the Allwinner A20 boards supported by Debian, you can easily build a tiny virtualization host that can handle a few simple VMs and draws only 2-3W of power. While this process was pretty cumbersome in the past (you had to cross-compile kernels etc.), thanks to the efforts of the Debian project and Linux-Sunxi community, it is now pretty straight-forward with only few caveats involved. This might also be an interesting option if you want to run a low-power virtualization cluster on fully libre software down to the firmware level.
References
[1] https://www.crowdsupply.com/eoma68/micro-desktop
[2] http://linux-sunxi.org/GPL_Violations
[3] http://blog.flexvdi.com/2014/07/28/enabling-kvm-virtualization-on-arm-allwinner-a20/
[4] https://virt-manager.org/
[5] https://www.olimex.com/Products/OLinuXino/A20/A20-OLinuXIno-LIME2/
[6] https://wiki.debian.org/InstallingDebianOn/Allwinner
