Paul Boddie's Free Software-related blog
Paul's activities and perspectives around Free Software
Porting L4Re and Fiasco.OC to the Ben NanoNote (Summary)
As promised, here is a summary of the work involved in porting L4Re and Fiasco.OC to the Ben NanoNote. First of all, a list of all the articles with some brief descriptions of what they cover:
- Familiarisation with L4Re and Fiasco.OC on the MIPS Creator CI20, adding some missing pieces
- Setting up and introducing a suitable compiler for the Ben, also describing the hardware in the kernel
- Handling instructions unsupported by the JZ4720 (the Ben’s SoC) in the kernel
- Describing the Ben and dealing with unsupported instructions in the L4Re portion of the system
- Configuring the memory layout and attempting to bootstrap the kernel
- Making the kernel support the MIPS architecture revision used by the JZ4720, also fixing the interrupt system description
- Investigating context/thread switching and fixing an inadvertently-introduced fault in the unsupported instruction handling
- Configuring user space examples and getting a simple framebuffer demonstration working
- Getting the framebuffer driver, GUI multiplexer, and “spectrum” example working
As I may have noted a few times in the articles, this work just builds on previous work done by a number of people over the years, obviously starting with the whole L4 microkernel effort, the development of Fiasco.OC, L4Re and their predecessors, and the work done to port these components to the MIPS architecture. On the l4-hackers mailing list, Adam Lackorzynski was particularly helpful when I ran into obstacles, and Sarah Hoffman provided some insight into problems with the CI20 just as it was needed.
You really don’t have to read all the articles or even any of them! The point of this article is to summarise the work and perhaps make similar porting efforts a bit more approachable for others in the same position: anyone having a vague level of familiarity with L4Re/Fiasco.OC or similar systems, also having a device that might be supported, and being somewhat familiar with writing code that drives hardware.
Practical Details
It might be useful to give certain practical details here, if only to indicate the nature of the development and testing routine employed in this endeavour. First of all, I have been using a chroot containing the Debian “unstable” distribution for the i386 architecture. Although this was essential for a time when building the software for the CI20 and trying to take advantage of Debian’s cross-compiler packages, any fairly recent version of Debian would probably be fine because I ended up using a Buildroot toolchain to be able to target the Ben. You could probably choose any Free Software distribution and reproduce what I have done.
The distribution of patches contains instructions regarding preparation and the building of the software. It isn’t too useful to repeat that information here, but the following things need doing:
- Installing packages for build tools
- Obtaining or building a cross-compiler
- Checking out the source code for L4Re and Fiasco.OC from its repository
- Applying the patches
- Configuring and building the kernel
- Configuring and building the runtime environment
- Copying the payload to a memory card
- Booting the device
Some scripts have been included in the patch distribution, one of which should do the tricky job of applying patches to the repository checkout according to the chosen device configuration. Because a centralised version control system (Subversion) has been used to publish the L4Re and Fiasco.OC sources, I had to find a way of working with my own local changes. Consequently, I wrote a few scripts to maintain bundles of changes associated with certain files, and I then managed these bundles in a different version control system. Yes, this effectively meant versioning the changes themselves!
Things would be simpler with a decentralised version control system because local commits would be convenient, and upstream updates would be incorporated into the repository separately and merged with local changes in a controlled fashion. One of the corporate participants has made a Git repository for Fiasco.OC available, which may alleviate some issues, although I am increasingly finding larger Git repositories to be unusable on my modest hardware, and I also tend to disagree with everybody deciding to put everything on GitHub.
Fixing and Building
Needing to repeatedly build, test, go back and fix, I found myself issuing the same command sequences a lot. When working with the kernel, I tended to enter the kernel build directory, which I called “mybuild”, edit the kernel sources, and then re-run the make command:
cd mybuild vi ../src/kern/mips/exception.S # edit a familiar file with vim make
Having built a new kernel, I would then need to build a new payload to deploy, which meant ascending the directory hierarchy and building an image in the L4Re section of the sources:
cd ../../../l4 make O=mybuild uimage E=mips-qi_lb60-spectrum-example
Given a previously-built “user space”, this would bundle the new kernel together with code that might be able to test it. Of particular importance is the bootstrap code which launches the kernel: without that, there is no point in even trying to test the kernel!
I found that re-building L4Re components seemed to require a general build to be performed:
make O=mybuild
If that proved successful, an image would then be built and tested. In general, focusing on either the kernel or some user space component meant that there was rarely a need to build a new kernel and then build much of the user space.
Work Summary
The patches accumulated during this process cover a range of different areas of functionality. Looking at them organised by functional area, instead of in the more haphazard fashion presented throughout the series of articles, allows for a more convenient review of the work actually needed to get the job done.
Build System Adjustments and Various Fixes
As early as my experiments with the CI20, I experienced the need to fix some things that didn’t work on my system, either due to some Debian peculiarities or differences in compiler behaviour:
- l4util-mips-thread.diff (fixes a symbol visibility issue with certain compiler versions)
- mips-gcc-cpload.diff (fixes the initialisation of certain L4Re components)
- no-at.diff (allows the build to work on Debian for the i386 architecture)
Other adjustments are required to let the build system do its job, setting paths for other components and for the toolchains:
- conf-makeconf-boot.diff (lets the L4Re build system find things like the kernel, modules and hardware descriptions)
- qi_lb60-gcc-buildroot-fiasco.diff (changes the compiler and architecture settings)
- qi_lb60-gcc-buildroot-l4re.diff (changes the compiler, architecture and soft-float settings)
The build system also needs directing towards new drivers, and various files need to be excluded or changed:
- ingenic-mips-drivers-top.diff (enables drivers added by this work)
- qi_lb60-fbdrv.diff (changes the splash image for the framebuffer driver)
- qi_lb60-l4re.diff (includes a temporary fix disabling a Mag plugin)
The first of these is important to remember when adding drivers since it changes the l4/pkg/drivers/Control file and defines the driver “packages” provided by each of the driver libraries. These package definitions help the build system work out which other parts of the system need to be consulted when building a particular driver.
Supporting MIPS32r1 Devices
Throughout the kernel and L4Re, changes need making to support the earlier architecture version provided by the JZ4720. The bulk of the following patch files deals with such changes:
- qi_lb60-fiasco.diff
- qi_lb60-l4re.diff
Maybe I will try and break out the architecture version changes into specific patch files, provided this does not result in the original source files ending up being patched by multiple patch files. My aim has been to avoid patches having to be applied in a particular order, and that starts to happen when multiple patches modify the same file.
Describing the Ben NanoNote
The kernel needs some knowledge of the Ben with regard to timers and interrupts. Meanwhile, L4Re needs to set the Ben up correctly when booting. Both sections of the system need an awareness of how memory is going to be used, and extra configuration options need to be provided to merely allow the selection of the Ben for building. Currently, the following patch files include things concerned with such matters:
- qi_lb60-fiasco.diff (contains timer, interrupt and memory details, plus configuration system changes)
- qi_lb60-l4re.diff (contains bootstrap and memory details, plus configuration system changes)
- qi_lb60-platform.diff (platform definitions for the Ben in L4Re)
One significant objective here is to be able to offer the Ben as a “first class” configuration option and have the build system do the right thing, setting up all the components and code regions that the Ben needs to function.
Introducing Driver Code
To be able to activate the framebuffer on the Ben, driver code needs introducing for a few peripherals provided by the JZ4720: CPM (clock/power management), GPIO (general-purpose input/output) and LCD (liquid crystal display, or similar). A few different patch files cover these areas:
- ingenic-mips-cpm.diff (CPM support for JZ4720 and JZ4780)
- ingenic-mips-gpio.diff (GPIO support for JZ4720 and JZ4780)
- qi_lb60-lcd.diff (LCD support for JZ4720)
The JZ4780 support is intended for the CI20 and will not be used with the Ben. However, it is convenient to incorporate support for these different platforms in the same patch file in each instance.
Meanwhile, the LCD driver should work with a range of JZ4700-series devices (labelled as JZ4740 in the patches). While focusing on getting things working, the only panel supported by this work was that provided by the Ben. Since then, support has been made slightly more general, just as was done with the Linux kernel support for products employing this particular SoC family and, subsequently, for panels in general. (Linux has moved towards a “device tree” approach for specifying things like panels and displays, although this is arguably just restating things that were once C-coded structures in another, rather peculiar, format.)
To support these drivers, some useful code has been copied from elsewhere in L4Re:
- drivers_frst-register-block.diff
This provides a convenient abstraction for registers that is exposed via an include directive:
#include <l4/drivers/hw_mmio_register_block.h>
Indeed, it is worth focusing on the LCD driver briefly. The code has its origins in existing driver code written for the Ben that I adapted to get working as part of a simple “bare metal” payload. I have maintained a separation between the more intricate hardware configuration and aspects that deal with the surrounding software. As part of L4Re, the latter involves obtaining access to memory using the appropriate API calls and invoking other drivers.
In L4Re, there is a kind of framework for LCD drivers, and the existing drivers seem to be written in C rather than C++. Reminiscent of Linux, there is a mechanism for exporting driver operations using a well-defined data structure, and this permits the “probing” of drivers to see if they can be enabled and if meaningful information can be obtained about things like the supported resolution, colour depth and pixel format. To make the existing code compatible with L4Re, a fair amount of the work involves translating the information already known (and used) in the hardware configuration activity to a form that other L4Re components can understand and use.
Originally, for the GPIO driver, I had intended it to operate as part of the Io server framework. Components using GPIO functionality would then employ the appropriate API to configure and interact with the exposed input and output pins. Unfortunately, this proved rather cumbersome, and so I decided to take a simpler approach of providing the driver as an abstraction that a program would use together with explicitly-requested memory. I did decide to preserve the general form of the API for this relocated abstraction, however, meaning that various classes and methods are provided that behave in the same way as those “left behind” in the Io server framework.
Thus, a program would itself request access to the GPIO-related memory, and it would then use GPIO-related abstractions to “do the right thing” with this memory. One would envisage that such a program would not be a “normal”, unprivileged program as such, but instead be more like a server or driver in its own right. Indeed, the LCD driver employs these abstractions to use SPI-based signalling with the LCD panel, and it uses the same techniques to configure the LCD clock frequencies using the CPM-related memory and CPM-related abstractions.
Although the GPIO driver follows existing conventions, the CPM driver has no obvious precedent in L4Re, but I adopted some of the conventions employed in the GPIO driver, adding more specialised methods and functions to expose functionality specific to the SoC. Since I had previously written a CPM driver for the JZ4780, the main objective was to make the JZ4720/JZ4740 driver resemble the existing driver as much as possible.
Introducing and Configuring Example Programs
Throughout the series of articles, I was working towards running one specific example program, making some new ones on the way for testing purposes. These additional programs are provided together with their configuration, accompanied by suitable configurations for existing examples and components, by the following patch files:
- ingenic-mips-modules.diff (example program definitions)
- qi_lb60-examples.diff (example program implementations and configuration details)
The additional programs (defined in l4/conf/modules.list) are as follows:
- mips-qi_lb60-lcd-example (implemented by qi_lb60_lcd, configured by the mips-qi_lb60-lcd files)
- mips-qi_lb60-lcd-driver-example (implemented by qi_lb60_lcd_driver, configured by the mips-qi_lb60-lcd-driver files)
Configurations are provided for the existing examples and components as follows:
- mips-qi_lb60-fbdrv-example (configured by the mips-qi_lb60-fbdrv files)
- mips-qi_lb60-spectrum-example (configured by the mips-qi_lb60-spectrum files)
All configurations reside in the l4/conf/examples directory. All new examples reside in the l4/pkg/examples/misc directory.
Further Work
In the final article in the series, I mentioned a few ideas for further work based on that described above:
- Porting to other JZ4700-series devices and products
- Writing drivers for other hardware peripherals
- Investigating related frameworks such as Genode, given its support for Fiasco.OC
- Contemplating the relevance of these technologies to the Hurd
Since completing the above work, I have already made some progress on the first two of these topics. More on that in an upcoming post!
Some Updates
Since writing this article, a few things are worth adding. First of all, the patches produced in the initial effort described by this series of articles are now available in an “initial archive” via the Web page documenting the effort. In contrast, a “current archive” provides patches for the current state of the work, with the aim being to focus these patches only on essential support for these devices within L4Re and Fiasco.OC, and with future development being done elsewhere.
Another couple of observations have been made since completing this initial effort that qualify or correct some information provided here. On the topic of the memory map needed to support the Ben NanoNote and its bootloader, it turned out that a fairly conventional arrangement was feasible after all and that only the “exception base” might be a problem. Here is the more conventional arrangement:
0x80600000 payload load address when copied by bootm 0x802d0000 bootstrap start address 0x80010000 kernel load address 0x80001000 exception handlers
The kernel load address and bootstrap start address are now the same as for other MIPS platforms. The exception handlers are positioned 4 kilobytes above the normal exception base just in case overwriting them might upset the bootloader.
Previously, I had experienced problems with Mag and its mag-input-libinput plugin, causing me to disable that plugin to get things working. This can be avoided by just providing a capability called “vbus” when starting Mag.