Looking for a secure smartphone running only free software? Good luck!

Motivation

On many occasions, I have seen myself and other members of the FSFE being asked which smartphones and mobile operating systems they can recommend to users who are looking for a free (owner-controlled) and secure smartphone that respects their freedom and privacy. I would like to share some of my thoughts about this complex topic. Please be warned, that it might be disappointing for those who might expect to get clear recommendations in the conclusion.

Operating Systems

In the following I want to briefly discussing pros and cons of a few mostly free mobile operating systems (skipping completely proprietary systems like iOS or Windows Phone). Mostly dead projects like B2G/FirefoxOS or Ubuntu Phone are not discussed either.

Stock Android

This is the binary Android distribution from Google and comes pre-installed on Google/Pixel devices.

  • Verified boot with locked bootloaders (but you have to trust the vendor)
  • Regular updates, once per month, available directly from upstream
  • Only available for Nexus/Pixel devices
  • Limited support period (18 months for Nexus devices, 3 years for Pixel devices) but clearly communicated
  • All supported devices require vast amounts of binary-only, proprietary firmware
  • Google Apps and Services are pre-installed and not removable.
  • Standard Android Security
  • Vendor-modified Android

    This is the Android distribution from typical OEMs like Samsung, HTC, LG, Sony, Huawei etc. that is based on Stock Android and customized by the OEM and in case of “branded” devices also by the network operator.

  • Ultimate choice of devices
  • Verified boot with locked bootloaders (but you have to trust the OEM), at least for most devices
  • Google Apps and Services are pre-installed and not removable.
  • Updates highly dependant on the OEM; often irregular, delayed or incomplete, sometimes no updates at all
  • Additional customizations by network operators often lead to further delayed updates
  • Often unclear support periods (depending on the vendor) and no formal EOL declaration
  • Almost all devices require binary-only, proprietary firmware (a few devices require not that much firmware, but most devices require a lot)
  • Standard Android Security
  • In many cases, the whole distribution is not directly available as a “factory image” from the vendor (depends on the vendor, but most of them do not provide such downloads). In most cases, flashing requires non-free software by the OEM that in turn only runs on non-free operating Systems such as Windows or OS X.
  • AOSP

    This is the source distribution of Android from Google.

  • Does not include Google Apps and Services
  • Regular updates, once per month, usually released shortly after new Stock Android builds
  • Limited support period (18 months for Nexus devices, 3 years for Pixel devices) but clearly communicated
  • Has to be self-compiled by the user (instructions only available for Nexus/Pixel devices as well as few 3rd party devices, e.g. some from Sony)
  • Verified boot with locked bootloaders and user-supplied keys (but only on Nexus/Pixel devices)
  • Almost all supported devices require vast amounts of binary-only, proprietary firmware
  • Copperhead OS

    A custom Android distribution provided by a community-oriented company. Based on AOSP with a strong focus on security.

  • Does not include Google Apps and Services
  • Some apps from AOSP have been replaced by more privacy-oriented alternatives (e.g. the calendar).
  • Much more than standard Android security: Hardened kernel and userland, privacy-friendly defaults
  • Binary downloads on website only for Nexus/Pixel devices and a development board
  • Regular updates, once per month (vendor promises to make them available within 24 hours after updates in AOSP became available)
  • Limited support period (18 months for Nexus devices, 3 years for Pixel devices) but clearly communicated
  • Verified boot with locked bootloaders and user-supplied keys (but only on Nexus/Pixel devices)
  • All supported smartphones require vast amounts of binary-only, proprietary firmware (the Hikey 960 does not, but it is not a smartphone)
  • All modifications are open source, but commercial redistribution is prohibited. Thus, the FSFE would not consider it free software.
  • Lineage OS

    A custom Android distribution based on AOSP but with heavy modifications. Developed by a large community of volunteers.

  • Does not include Google Apps and Services by default
  • Regular updates, but security patches are sometimes delayed or incomplete
  • Verified boot with locked bootloaders and user-supplied keys (but only on Nexus/Pixel devices) theoretically possible (but afaik no official instructions provided)
  • Backports security patches to Android versions which are not supported by Google anymore (but not forever)
  • Supports devices that are EOL by their original vendors since years, however, actual support period by LineageOS is not clear in advance and official EOL has often not been clearly communicated in the past.
  • Lower security than on standard Android due to userdebug builds by default
  • Broad support for a huge number of devices
  • All supported devices require binary-only, proprietary firmware (a few devices require not that much non-free firmware, but most devices require a lot)
  • OmniROM

    A custom Android distribution based on AOSP with moderate modifications. Developed by a community of volunteers.

  • Similar to Lineage OS with respect to the discussed properties
  • Slightly higher security because uses “eng” builds by default
  • Limited device support
  • Provides security updates usually only for the most recent branch
  • Replicant

    A custom Android distribution based on Lineage OS 13. Developed by very few community volunteers but strongly backed by many free software enthusiasts.

  • Does not include Google Apps and Services by default
  • Irregular updates, security patches are heavily delayed
  • All supported devices run completely without binary-only, proprietary firmware on the main processor (but on no device functionality such as GPS, Bluetooth, 3D acceleration is available and Wifi works only with one selected model of external USB adapters). Non-free firmware for baseband processor is still required.
  • None of the supported devices supports verified boot (all require an unlocked bootloader)
  • Even with a closed bootloader, many of the supported devices have severe security issues (e.g. storage is directly accessible via the insecure odin protocol)
  • postmarketOS

    A “classic” Linux distribution targeted also on legacy smartphones vastly considered obsolete. Developed by volunteeers. Based on Alpine Linux, thus very lightweight.

  • Nice and clear architectural design
  • Classic distro packages instead of app store
  • Not really useable yet, under heavy development
  • Regular updates, just like a normal Linux distribution
  • Choice between several UIs (most promising to me: Plasma Mobile), however, none of them seems Production-ready yet
  • Partial support for a large number of devices, but not a single device seems ready for daily usage as a smartphone
  • Verified boot with locked bootloaders and user-supplied keys theoretically possible (only on Nexus/Pixel devices) but afaik not implemented yet.
  • pureOS

    (I have no personal experience with this OS, thus the following information is just from my research)

    Another “classic” Linux distribution but primarily targeted at “Librem” devices by purism. Based on Debian Testing.

  • Convergence approach
  • Under heavy development, not available yet for phones
  • Support for GNOME and KDE Plasma Mobile planned
  • No information about planned support period available yet
  • Nothing concrete known about security features such as verified boot yet
  • Sailfish OS

    (I have no personal experience with this OS, thus the following information is just from my research)

    A Linux-based operating system that supports Android apps through an compatibility layer. Has an open core but a closed source UI that is based on QT5.

  • Not all parts are open source, using vendor-provided binaries requires purchasing a license
  • Very limited number of devices officially supported (as of now, only one Sony device is globally purchaseable)
  • Community-backed ports for many devices such as the Fairphone 2 are available
  • All supported devices (officially and by the community) require binary-only, proprietary firmware (a few devices require not that much non-free firmware, but most devices require a lot)
  • OS-independent freedom and security concerns

    Firmware vulnerabilities

    If you run non-free firmware that is required to operate most smartphones (which is always required except if you run Replicant and can live with severely degraded functionality) you are the mercy of the vendor to provide you with updates. However, there are critical vulnerabilities in some of these firmwares. But even in extreme cases such as the “Broadpwn” exploit (affects BCM43xx wifi chipsets found on more than a billion of devices), vendors often refuse to provide updated firmware if the affected devices are outside of the official support period.

    Bootloader freedom

    As far as I know, there are no modern smartphones which come preinstalled with a free bootloader or where the proprietary bootloader can be replaced. Few exception may be some (quite dated) devices supported by postmarketOS such as the Nokia N900.

    Hardware-backed security firmware

    Modern SoCs come with a TEE (Trusted Execution Environment) as part of their chip design. These separate systems are used for securely processing/storing credentials such as fingerprints or (parts of) device encryption keys. Usually, these subsystems are closed source. Thus, their proper operation is difficult to audit (although the developers of CopperheadOS claim to have succeeded in doing this [2]).

    Separation of SoC and baseband

    Many modern do not have a clear separation between the baseband processor (that runs the non-free radio firmware) and the SoC. This means, that you actually have to trust the firmware of the baseband firmware vendor. There is at least one known case of a backdoor built into the layer above that was detected on older Samsung devices and mitigated by Replicant developers [3].

    Other partitions with non-free binaries and data

    Some people may think that by re-flashing a ROM or factory image they return their device to the original state. However, todays smartphones often contain a bunch of partitions, but most of them remained untouched during flashing. For instance, here is a listing of partition names and corresponding flash partitions from a Nexus 5:

    DDR	->	/dev/block/mmcblk0p24
    aboot	->	/dev/block/mmcblk0p6
    abootb	->	/dev/block/mmcblk0p11
    boot	->	/dev/block/mmcblk0p19
    cache	->	/dev/block/mmcblk0p27
    crypto	->	/dev/block/mmcblk0p26
    fsc	->	/dev/block/mmcblk0p22
    fsg	->	/dev/block/mmcblk0p21
    grow	->	/dev/block/mmcblk0p29
    imgdata	->	/dev/block/mmcblk0p17
    laf	->	/dev/block/mmcblk0p18
    metadata	->	/dev/block/mmcblk0p14
    misc	->	/dev/block/mmcblk0p15
    modem	->	/dev/block/mmcblk0p1
    modemst1	->	/dev/block/mmcblk0p12
    modemst2	->	/dev/block/mmcblk0p13
    pad	->	/dev/block/mmcblk0p7
    persist	->	/dev/block/mmcblk0p16
    recovery	->	/dev/block/mmcblk0p20
    rpm	->	/dev/block/mmcblk0p3
    rpmb	->	/dev/block/mmcblk0p10
    sbl1	->	/dev/block/mmcblk0p2
    sbl1b	->	/dev/block/mmcblk0p8
    sdi	->	/dev/block/mmcblk0p5
    ssd	->	/dev/block/mmcblk0p23
    system	->	/dev/block/mmcblk0p25
    tz	->	/dev/block/mmcblk0p4
    tzb	->	/dev/block/mmcblk0p9
    userdata	->	/dev/block/mmcblk0p28
    

    Even if you flash the factory ROM provided by Google, it only touches a few of these partitions. The other parts remain unchanged. It is hard to find documentation on what the purpose of these partitions is and, depending on the device, there are many different partitions. Also, for partitions that usually don’t change I haven’t seen any list of hashes from OEMs. Therefore, if you buy a used phone you have to be aware that there is (1) no official way to reset these partitions to their factory state and (2) it’s hard to tell if any of these partitions have been modified. Most of these partitions can be modified if you have root access, therefore, if you ever executed something with root rights it could have tampered with one of those.

    Conclusion and recommendations

    Given the current situation, providing recommendations is hard as it highly depends on these factors:

    • your willingness to run non-free software (esp. firmware)
    • your trust in particular SoC vendors
    • your threat perceptions (e.g. do you see more threats from local or from remote attackers?)

    The options I would recommend are as follows (order is arbitrary and SailfishOS was not considered due to my lack of experience with it):

    • Get a device supported by Replicant (Samsung Galaxy S2, Galaxy S3 or Galaxy Note 1), live with the degraded functionality and the devices’ trivial local exploitability. If you can, support development towards newer LineageOS versions to get OS-level security updates with less delays in the future.
    • Get a recent (still-supported) Nexus/Pixel devices and run CopperheadOS on it. You will be secured as good as possible from local and OS-level remote attacks and get OS and firmware updates every month. However, you have to fully trust the SoC platform and its (non-separated) baseband firmware. In addition, you have to pollute the environment by switching to newer devices once your device runs out of support.
    • Support the Librem 5 campaign and hope purism will keep their promises for choosing libre-friendly hardware (if you ask me, they should concentrate on these issues, abandon pureOS and support postmarketOS instead).
    • Recycle an old device and support postmarketOS and other free software projects like Plasma Mobile to help building a free alternative to Android. This might become interesting if we succeed in lowering the dependencies on non-free firmware with the availability of devices like the Librem 5 (provided the vendor keeps their promises).

    References

    [1] https://blogs.fsfe.org/kuleszdl/2018/03/31/securing-copperheados-by-using-separate-encryptionlockscreen-passphrases/
    [2] https://github.com/copperhead/bugtracker/issues/451
    [3] https://www.fsf.org/blogs/community/replicant-developers-find-and-close-samsung-galaxy-backdoor

    Further securing Nexus devices running CopperheadOS by using separate Encryption/Lockscreen passphrases

    Please note that this article has undergone a major revision. After receiving negative feedback about mixing different aspects in this article from the Copperhead CTO [1], I decided to discuss the more opinionated comparison with other systems in a separate article. [2]

    Motivation

    Some of you may be familiar with CopperheadOS. CopperheadOS is a security-oriented custom Android distribution that is based directly on AOSP (not on LineageOS or similar “heavy” forks). It is free from Google Apps and aims to be more secure than stock Android by incorporating various hardening patches.

    When CopperheadOS was announced in 2015, one great feature they wanted to provide was support for using separate passphrases/pins for device encryption and the lockscreen. Unfortunately, with the move to Android 7 (or 8 ) supporting this feature got infeasible. However, providing a long passphrase each time you want to unlock your phone is extremely incovinient. On the other hand, using a short passphrase or simple PIN is insecure, because if an attacker succeeds to read your key from the (unauditable hard to audit) TEE, it is trivial to get the key using brute force attacks.

    Officially, there is no way for using a separate encryption passphrase in neither AOSP or CopperheadOS atm. Luckily, I found a (rather complicated) method how it can be achieved on Nexus devices (according to discussion with CopperheadOS devs it is not applicable for Pixel devices). This article briefly describes this method (that in principle should also work for AOSP).

    Disclaimer

    All data and information provided in this article is for informational purposes only. The author makes no representations as to accuracy, completeness, currentness, suitability, or validity of any information on this article and will not be liable for any errors, omissions, or delays in this information or any losses, injuries, or damages arising from its display or use. All information is provided on an as-is basis.

    In no event the author we be liable for any loss or damage including without limitation, indirect or consequential loss or damage, or any loss or damage whatsoever arising from loss of data or profits arising out of, or in connection with, the use of this article.

    Enabling a secure boot time password in CopperheadOS

    In all Android versions since 4.0, a separate passphrase could be enabled via the terminal or ADB using the “vdc” command. There are also some apps in the F-Droid store which allow to to this even more comfortably. However, both approaches require root access. Since CopperheadOS has decided (for good reasons!) not to provide root access at all, it’s not possible to use the vdc call (or even those less secure apps that require a superuser service installed) in the vendor-supported CopperheadOS builds.

    You can overcome this limitation by using a self-compiled “userdebug”-build that allows root access via adb. However, you want to run such a build only temporarily and this is where things get more complicated. I spent way too much time in figuring out how to achieve a desirable result, thus, I can only share the principal steps how to achieve this for now:

    1. Modify the boardconfig for your device and edit the CMDLINE parameter so that it tries to disable SELinux (todo: provide the necessary string here). Note that this will be only effective in “userdebug” builds but ignored in “user” builds. There also seems to be a bug that it will still display “enforcing” in the system menu.
    2. Compile CopperheadOS using the official documentation and own release signing keys; backup the keys and the generated images.
    3. Compile second “release” build (using the same release keys!) but choose the variant “userdebug” to obtain an insecure, debug-enabled build. Backup the generated images as well. It is important that your “user” build is newer, since OTA downgrades are not allowed in many bootloaders.
    4. Unlock the bootloader (your data partition will get wiped!).
    5. Flash the userdebug build using the “flash-all” method.
    6. Re-flash the recovery partition, replacing it by the one from the normal “user” build. This is needed because the “userdebug” recovery is insecure as it provides root access without authentication! (Turned out to be not needed)
    7. Lock the bootloader (your data partition will get wiped again on the next boot).
    8. Boot into the insecure “userdebug” build.
    9. Set the simple lockscreen protection you would like to have for everyday use PIN/password (the lockscreen is rate-limited, so you can even use a simple PIN).
    10. Enable adb
    11. Connect via adb and use “su” to become root.
    12. Run the vdc command to set a complex passphrase for the boot-time password (vdc cryptfs changepw password your-new-super-secure-extraa-long-password).
    13. Reboot the device and verify that everything is working as expected.
    14. Reboot to recovery
    15. Verify that you have the secure recovery installed: (1) It welcomes you just with “no command” and you have to hold power+volup to get a menu. (2) The menu you get does not have the insecure options such as “mount /system”.
    16. Sideload the OTA image of your secure “user” build.
    17. Reboot to the normal OS (do not wipe!)

    To make sure you are running the secured build, you can enable adb, log in via adb and try calling “su” to make sure it does not exist.

    If I find the time I will update this article with more detailed instructions and describe how to work around the several build issues I encountered in CopperheadOS (you have to overcome them in order to build it) as well.

    Limitations

    Please be aware of the following limitations:

    • To change your password or key you have to build a new userdebug build and a new user build and flash them via OTA because the recovery does not accept older builds.
    • To do other operations which require root (e.g. a full backup via one of the methods described in one of my earlier posts [3]) you also have to go through this procedure of making two new builds.
    • Even if you have a separate and more secure boot password, your key remains in memory if you lock the device and could be subject to various attack vectors (unlike the newer FDE approach on Pixel devices, where the key is not in memory when your screen is locked)

    References

    [1] https://github.com/copperhead/bugtracker/issues/451
    [2] https://blogs.fsfe.org/kuleszdl/2018/04/02/looking-for-a-secure-smartphone-running-only-free-software-good-luck/
    [3] http://blogs.fsfe.org/kuleszdl/2016/11/02/android-data-backup/

    How a single unprivileged app can brick the whole Android system

    This article is highly subjective and only states the author’s opinion based on actual observations and “wild” assumptions. Better explanations and corrections are warmly welcome!

    Motivation

    After updating an App from the F-Droid store (OpenCamera), my Android device was completely unusable. In this state, the only feasible option for a typical end-user to recover the device (who does not know how to get to safe mode in order to remove or downgrade the app [5]) would have been to wipe data in recovery, loosing all data.

    How can such a disaster happen? In this article, I argue why I have serious doubts about the memory management approach taken in Android.

    The failure

    After updating the OpenCamera app to the recently released version 1.42, my Android device ran into a bootloop that was hard to recover from. I was able to repeatingly reproduce the failure on a different device, namely the following:

    • Device: Samsung Galaxy S3 (i9300)
    • ROM: Lineage OS 13 (Android 6.0), freshly built from latest sources, commit 42f4b851c9b2d08709a065c3931f6370fd78b2b0 [1]

    Steps to reproduce:

    1. wipe all data and caches
    2. newly configure the device using the first-use wizard
    3. install the F-Droid store
    4. search for “Open Camera”
    5. install Open Camera version 1.42

    Expected:

    The install completes and the app is available. If installation fails (for whatever reason), an error message is shown but the device is still working

    Actual:

    The install freezes, the LineageOS splash screen appears and re-initializes all apps; this happens several times and after aprox 10-15 minutes the device is back “working”; when trying to start apps they crash or even the launcher (“Trebuchet”) crashes. After rebooting the device, it is stuck in an infinite loop initializing apps.

    The fault (what happens under the hood?)

    When installing OpenCamera, the following is printed in the log:

    12-10 14:48:30.915  4034  5483 I ActivityManager: START u0 {act=org.fdroid.fdroid.installer.DefaultInstaller.action.INSTALL_PACKAGE dat=file:///data/user/0/org.fdroid.fdroid/files/Open Camera-1.42.apk cmp=org.fdroid.fdroid/.installer.DefaultInstallerActivity (has extras)} from uid 10070 on display 0
    12-10 14:48:30.915  4034  5483 W ActivityManager: startActivity called from non-Activity context; forcing Intent.FLAG_ACTIVITY_NEW_TASK for: Intent { act=org.fdroid.fdroid.installer.DefaultInstaller.action.INSTALL_PACKAGE dat=file:///data/user/0/org.fdroid.fdroid/files/Open Camera-1.42.apk cmp=org.fdroid.fdroid/.installer.DefaultInstallerActivity (has extras) }
    12-10 14:48:30.925  4034  5483 D lights  : set_light_buttons: 2
    12-10 14:48:30.955  4034  5649 I ActivityManager: START u0 {act=android.intent.action.INSTALL_PACKAGE dat=file:///data/user/0/org.fdroid.fdroid/files/Open Camera-1.42.apk cmp=com.android.packageinstaller/.PackageInstallerActivity (has extras)} from uid 10070 on display 0
    12-10 14:48:31.085  6740  6740 W ResourceType: Failure getting entry for 0x7f0c0001 (t=11 e=1) (error -75)
    12-10 14:48:31.700  4034  4093 I ActivityManager: Displayed com.android.packageinstaller/.PackageInstallerActivity: +724ms (total +758ms)
    12-10 14:48:36.770  4034  4362 D lights  : set_light_buttons: 1
    12-10 14:48:36.840  4034  4938 I ActivityManager: START u0 {dat=file:///data/user/0/org.fdroid.fdroid/files/Open Camera-1.42.apk flg=0x2000000 cmp=com.android.packageinstaller/.InstallAppProgress (has extras)} from uid 10018 on display 0
    12-10 14:48:36.850  3499  3895 D audio_hw_primary: select_output_device: AUDIO_DEVICE_OUT_SPEAKER
    12-10 14:48:36.955  6863  6874 D DefContainer: Copying /data/user/0/org.fdroid.fdroid/files/Open Camera-1.42.apk to base.apk
    12-10 14:48:37.100  4034  4093 I ActivityManager: Displayed com.android.packageinstaller/.InstallAppProgress: +251ms
    12-10 14:48:37.155  6740  6753 D OpenGLRenderer: endAllStagingAnimators on 0x486226f0 (RippleDrawable) with handle 0x48604d28
    12-10 14:48:37.170  4034  4100 W ResourceType: Failure getting entry for 0x7f0c0001 (t=11 e=1) (error -75)
    12-10 14:48:37.465  4034  4100 I PackageManager.DexOptimizer: Running dexopt (dex2oat) on: /data/app/vmdl872450731.tmp/base.apk pkg=net.sourceforge.opencamera isa=arm vmSafeMode=false debuggable=false oatDir = /data/app/vmdl872450731.tmp/oat bootComplete=true
    12-10 14:48:37.585  7205  7205 I dex2oat : Starting dex2oat.
    12-10 14:48:37.585  7205  7205 E cutils-trace: Error opening trace file: No such file or directory (2)
    12-10 14:48:42.405  7205  7205 I dex2oat : dex2oat took 4.815s (threads: 4) arena alloc=5MB java alloc=2023KB native alloc=13MB free=1122KB
    12-10 14:48:42.415  4034  4100 D lights  : set_light_buttons: 2
    12-10 14:48:42.680  4034  4100 V BackupManagerService: restoreAtInstall pkg=net.sourceforge.opencamera token=3 restoreSet=0
    12-10 14:48:42.680  4034  4100 W BackupManagerService: Requested unavailable transport: com.google.android.gms/.backup.BackupTransportService
    12-10 14:48:42.680  4034  4100 W BackupManagerService: No transport
    12-10 14:48:42.680  4034  4100 V BackupManagerService: Finishing install immediately
    12-10 14:48:42.705  4034  4100 W Settings: Setting install_non_market_apps has moved from android.provider.Settings.Global to android.provider.Settings.Secure, returning read-only value.
    12-10 14:48:42.705  4034  4100 I art     : Starting a blocking GC Explicit
    12-10 14:48:42.805  4034  4100 I art     : Explicit concurrent mark sweep GC freed 52637(2MB) AllocSpace objects, 20(424KB) LOS objects, 33% free, 14MB/21MB, paused 2.239ms total 96.416ms
    12-10 14:48:42.835  4034  4363 I InputReader: Reconfiguring input devices.  changes=0x00000010
    12-10 14:48:42.935  5420  5420 D CarrierServiceBindHelper: Receive action: android.intent.action.PACKAGE_ADDED
    12-10 14:48:42.940  5420  5420 D CarrierServiceBindHelper: mHandler: 3
    12-10 14:48:42.940  5420  5420 D CarrierConfigLoader: mHandler: 9 phoneId: 0
    12-10 14:48:42.945  4034  4034 F libc    : invalid address or address of corrupt block 0x120 passed to dlfree
    12-10 14:48:42.945  4034  4034 F libc    : Fatal signal 11 (SIGSEGV), code 1, fault addr 0xdeadbaad in tid 4034 (system_server)
    12-10 14:48:42.950  3496  3496 I DEBUG   : property debug.db.uid not set; NOT waiting for gdb.
    12-10 14:48:42.950  3496  3496 I DEBUG   : HINT: adb shell setprop debug.db.uid 100000
    12-10 14:48:42.950  3496  3496 I DEBUG   : HINT: adb forward tcp:5039 tcp:5039
    12-10 14:48:42.975  3496  3496 F DEBUG   : *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
    12-10 14:48:42.975  3496  3496 F DEBUG   : LineageOS Version: '13.0-20171125-UNOFFICIAL-i9300'
    12-10 14:48:42.975  3496  3496 F DEBUG   : Build fingerprint: 'samsung/m0xx/m0:4.3/JSS15J/I9300XXUGMJ9:user/release-keys'
    12-10 14:48:42.975  3496  3496 F DEBUG   : Revision: '0'
    12-10 14:48:42.975  3496  3496 F DEBUG   : ABI: 'arm'
    12-10 14:48:42.975  3496  3496 F DEBUG   : pid: 4034, tid: 4034, name: system_server  >>> system_server <<<
    12-10 14:48:42.975  3496  3496 F DEBUG   : signal 11 (SIGSEGV), code 1 (SEGV_MAPERR), fault addr 0xdeadbaad
    12-10 14:48:43.030  3496  3496 F DEBUG   : Abort message: 'invalid address or address of corrupt block 0x120 passed to dlfree'
    12-10 14:48:43.030  3496  3496 F DEBUG   :     r0 00000000  r1 00000000  r2 00000000  r3 00000002
    12-10 14:48:43.030  3496  3496 F DEBUG   :     r4 00000120  r5 deadbaad  r6 404e0f38  r7 40005000
    12-10 14:48:43.030  3496  3496 F DEBUG   :     r8 00000128  r9 bee01b0c  sl 40358be3  fp 40358bec
    12-10 14:48:43.030  3496  3496 F DEBUG   :     ip 404db5d8  sp bee019f8  lr 404abfab  pc 404abfaa  cpsr 60070030
    12-10 14:48:43.045  3496  3496 F DEBUG   :
    12-10 14:48:43.045  3496  3496 F DEBUG   : backtrace:
    12-10 14:48:43.045  3496  3496 F DEBUG   :     #00 pc 00030faa  /system/lib/libc.so (dlfree+1285)
    12-10 14:48:43.045  3496  3496 F DEBUG   :     #01 pc 000158df  /system/lib/libandroidfw.so (_ZN7android13ResStringPool6uninitEv+38)
    12-10 14:48:43.045  3496  3496 F DEBUG   :     #02 pc 0001662b  /system/lib/libandroidfw.so (_ZN7android10ResXMLTree6uninitEv+12)
    12-10 14:48:43.045  3496  3496 F DEBUG   :     #03 pc 00016649  /system/lib/libandroidfw.so (_ZN7android10ResXMLTreeD1Ev+4)
    12-10 14:48:43.045  3496  3496 F DEBUG   :     #04 pc 00013373  /system/lib/libandroidfw.so (_ZN7android12AssetManager10getPkgNameEPKc+258)
    12-10 14:48:43.045  3496  3496 F DEBUG   :     #05 pc 000133cf  /system/lib/libandroidfw.so (_ZN7android12AssetManager18getBasePackageNameEj+62)
    12-10 14:48:43.045  3496  3496 F DEBUG   :     #06 pc 00088b33  /system/lib/libandroid_runtime.so
    12-10 14:48:43.045  3496  3496 F DEBUG   :     #07 pc 72cb9011  /data/dalvik-cache/arm/system@framework@boot.oat (offset 0x1f78000)
    12-10 14:48:50.095  3496  3496 F DEBUG   :
    12-10 14:48:50.095  3496  3496 F DEBUG   : Tombstone written to: /data/tombstones/tombstone_00
    12-10 14:48:50.185  1912  1912 I ServiceManager: service 'statusbar' died
    12-10 14:48:50.185  1912  1912 I ServiceManager: service 'netstats' died
    12-10 14:48:50.185  1912  1912 I ServiceManager: service 'power' died
    12-10 14:48:50.185  1912  1912 I ServiceManager: service 'media_projection' died
    12-10 14:48:50.185  1912  1912 I ServiceManager: service 'network_management' died
    12-10 14:48:50.185  1912  1912 I ServiceManager: service 'window' died
    12-10 14:48:50.185  1912  1912 I ServiceManager: service 'consumer_ir' died
    12-10 14:48:50.185  1912  1912 I ServiceManager: service 'telecom' died
    12-10 14:48:50.185  1912  1912 I ServiceManager: service 'cmpartnerinterface' died
    12-10 14:48:50.185  1912  1912 I ServiceManager: service 'package' died
    12-10 14:48:50.185  1912  1912 I ServiceManager: service 'user' died

    Since Open Camera needs some background service and is started on bootup, I assume that after installation the system tries to restart this service. However, it appears that there is some memory issue with the app, as it requests so much memory that Android starts killing other apps to make this memory available. In case Android does not manage to provide this space, the device is rebooted. Since OpenCamera is started at bootup, it again tries to allocate (too much) memory and the device is stuck in an infinite loop.

    Looking at Android’s memory management

    I expected that the following excerpt from the log above might lead to some useful hints:

    12-10 14:48:42.945  4034  4034 F libc    : invalid address or address of corrupt block 0x120 passed to dlfree
    12-10 14:48:42.945  4034  4034 F libc    : Fatal signal 11 (SIGSEGV), code 1, fault addr 0xdeadbaad in tid 4034 (system_server)

    After searching on the net, I found an interesting discussion [2] suggesting the following:


    “A likely cause of this is that you have ran out of memory, maybe because a memory leak or simply used up all memory. This can be caused by a bug you are using in a plugin that uses native C/C++ code through NDK.”

    To rule out hardware issues, I also exchanged the storage (I run /data from sdcard) and compiled memtester [3] to test the device’s RAM. When experimenting with memtester, I noticed a striking difference between running memtester on a regular GNU/Linux system and running it on Android/LineageOS. When giving memtester less memory than actually available, there is no difference. However, when giving memtester *more* RAM than acutally available, the following happens on GNU/Linux:

    # free -h
                  total        used        free      shared  buff/cache   available
    Mem:            28G        124M         28G        8.5M        219M         28G
    Swap:            0B          0B          0B
    # memtester 40G
    memtester version 4.3.0 (64-bit)
    Copyright (C) 2001-2012 Charles Cazabon.
    Licensed under the GNU General Public License version 2 (only).
    
    pagesize is 4096
    pagesizemask is 0xfffffffffffff000
    want 40960MB (42949672960 bytes)
    got  29075MB (30488387584 bytes), trying mlock ...Killed
    #
    Killed
    [1]+  Stopped                 sh

    While on Android the device suddenly reboots after trying to mlock the memory:

    root@i9300:/ # free -h
                    total        used        free      shared     buffers
    Mem:             828M        754M         74M           0        1.3M
    -/+ buffers/cache:           752M         75M
    Swap:            400M         18M        382M
    
    root@i9300:/ # /sbin/memtester 2G
    memtester version 4.3.0 (32-bit)
    Copyright (C) 2001-2012 Charles Cazabon.
    Licensed under the GNU General Public License version 2 (only).
    
    pagesize is 4096
    pagesizemask is 0xfffff000
    want 2048MB (2147483648 bytes)
    got  2008MB (2105921536 bytes), trying mlock ...

    This is what is printed to logcat:

    01-01 01:10:29.485  4933  4933 D su      : su invoked.
    01-01 01:10:29.485  4933  4933 E su      : SU from: shell
    01-01 01:10:29.490  4933  4933 D su      : Allowing shell.
    01-01 01:10:29.490  4933  4933 D su      : 2000 /system/bin/sh executing 0 /system/bin/sh using binary /system/bin/sh : sh
    01-01 01:10:29.490  4933  4933 D su      : Waiting for pid 4934.
    01-01 01:10:44.840  2478  3264 D LightsService: Excessive delay setting light: 81ms
    01-01 01:10:44.925  2478  3264 D LightsService: Excessive delay setting light: 82ms
    01-01 01:10:45.010  2478  3264 D LightsService: Excessive delay setting light: 82ms
    01-01 01:10:45.090  2478  3264 D LightsService: Excessive delay setting light: 82ms
    01-01 01:10:45.175  2478  3264 D LightsService: Excessive delay setting light: 82ms
    01-01 01:10:45.260  2478  3264 D LightsService: Excessive delay setting light: 82ms
    01-01 01:10:45.340  2478  3264 D LightsService: Excessive delay setting light: 82ms
    01-01 01:10:50.735  2478  2538 I PowerManagerService: Going to sleep due to screen timeout (uid 1000)...
    01-01 01:10:50.785  2478  2538 E         : Device driver API match
    01-01 01:10:50.785  2478  2538 E         : Device driver API version: 29
    01-01 01:10:50.785  2478  2538 E         : User space API version: 29
    01-01 01:10:50.785  2478  2538 E         : mali: REVISION=Linux-r3p2-01rel3 BUILD_DATE=Tue Aug 26 17:05:16 KST 2014
    01-01 01:10:52.000  2478  2538 V KeyguardServiceDelegate: onScreenTurnedOff()
    01-01 01:10:52.040  2478  2538 E libEGL  : call to OpenGL ES API with no current context (logged once per thread)
    01-01 01:10:52.045  2478  2536 I DisplayManagerService: Display device changed: DisplayDeviceInfo{"Integrierter Bildschirm": uniqueId="local:0", 720 x 1280, modeId 1, defaultModeId 1, supportedModes [{id=1, width=720, height=1280, fps=60.002}], colorTransformId 1, defaultColorTransformId 1, supportedColorTransforms [{id=1, colorTransform=0}], density 320, 304.8 x 306.71698 dpi, appVsyncOff 0, presDeadline 17666111, touch INTERNAL, rotation 0, type BUILT_IN, state OFF, FLAG_DEFAULT_DISPLAY, FLAG_ROTATES_WITH_CONTENT, FLAG_SECURE, FLAG_SUPPORTS_PROTECTED_BUFFERS}
    01-01 01:10:52.060  1915  1915 D SurfaceFlinger: Set power mode=0, type=0 flinger=0x411dadf0
    01-01 01:10:52.160  2478  2538 I PowerManagerService: Sleeping (uid 1000)...
    01-01 01:10:52.165  2478  3231 D WifiConfigStore: Retrieve network priorities after PNO.
    01-01 01:10:52.170  1938  3241 E bt_a2dp_hw: adev_set_parameters: ERROR: set param called even when stream out is null
    01-01 01:10:52.170  2478  3231 E native  : do suspend false
    01-01 01:10:52.175  2478  3231 D WifiConfigStore: No blacklist allowed without epno enabled
    01-01 01:10:52.190  3846  4968 D NfcService: Discovery configuration equal, not updating.
    01-01 01:10:52.435  2478  3231 D WifiConfigStore: Retrieve network priorities before PNO. Max priority: 0
    01-01 01:10:52.435  1938  1938 E bt_a2dp_hw: adev_set_parameters: ERROR: set param called even when stream out is null
    01-01 01:10:52.440  2478  3231 E WifiStateMachine:  Fail to set up pno, want true now false
    01-01 01:10:52.440  2478  3231 E native  : do suspend true
    01-01 01:10:52.670  2478  3231 D WifiStateMachine: Disconnected CMD_START_SCAN source -2 3, 4 -> obsolete
    01-01 01:10:54.160  2478  2538 W PowerManagerService: Sandman unresponsive, releasing suspend blocker
    01-01 01:10:55.825  2478  3362 D CryptdConnector: SND -> {3 cryptfs getpw}
    01-01 01:10:55.825  1903  1999 D VoldCryptCmdListener: cryptfs getpw
    01-01 01:10:55.825  1903  1999 I Ext4Crypt: ext4 crypto complete called on /data
    01-01 01:10:55.825  1903  1999 I Ext4Crypt: No master key, so not ext4enc
    01-01 01:10:55.830  1903  1999 I Ext4Crypt: ext4 crypto complete called on /data
    01-01 01:10:55.830  1903  1999 I Ext4Crypt: No master key, so not ext4enc
    01-01 01:10:55.830  2478  2798 D CryptdConnector: RCV  {4 cryptfs clearpw}
    01-01 01:10:55.835  1903  1999 D VoldCryptCmdListener: cryptfs clearpw
    01-01 01:10:55.835  1903  1999 I Ext4Crypt: ext4 crypto complete called on /data
    01-01 01:10:55.835  1903  1999 I Ext4Crypt: No master key, so not ext4enc
    01-01 01:10:55.835  2478  2798 D CryptdConnector: RCV <- {200 4 0}
    01-01 01:10:55.925  3417  3417 D PhoneStatusBar: disable:
    01-01 01:10:56.020  3417  3417 D PhoneStatusBar: disable:
    01-01 01:10:56.330  3417  3417 D PhoneStatusBar: disable:
    01-01 01:11:44.875  2478  4667 I ActivityManager: Process com.android.messaging (pid 4607) has died
    01-01 01:11:44.920  2478  4667 D ActivityManager: cleanUpApplicationRecord -- 4607
    01-01 01:11:45.860  2478  3356 W art     : Long monitor contention event with owner method=void com.android.server.am.ActivityManagerService$AppDeathRecipient.binderDied() from ActivityManagerService.java:1359 waiters=0 for 907ms
    01-01 01:11:45.890  2478  3356 I ActivityManager: Process org.cyanogenmod.profiles (pid 4593) has died
    01-01 01:11:45.900  2478  3356 D ActivityManager: cleanUpApplicationRecord -- 4593
    01-01 01:11:45.955  2478  2529 W art     : Long monitor contention event with owner method=void com.android.server.am.ActivityManagerService$AppDeathRecipient.binderDied() from ActivityManagerService.java:1359 waiters=1 for 914ms
    01-01 01:11:45.960  1913  1913 E lowmemorykiller: Error opening /proc/3662/oom_score_adj; errno=2
    01-01 01:11:45.970  2478  2529 I ActivityManager: Process com.android.exchange (pid 3662) has died
    01-01 01:11:45.970  2478  2529 D ActivityManager: cleanUpApplicationRecord -- 3662
    01-01 01:11:45.985  2478  3943 W art     : Long monitor contention event with owner method=void com.android.server.am.ActivityManagerService$AppDeathRecipient.binderDied() from ActivityManagerService.java:1359 waiters=2 for 611ms
    01-01 01:11:45.995  2478  3943 I ActivityManager: Process com.android.calendar (pid 4415) has died
    01-01 01:11:45.995  2478  3943 D ActivityManager: cleanUpApplicationRecord -- 4415
    01-01 01:11:46.000  2478  2532 W art     : Long monitor contention event with owner method=void com.android.server.am.ActivityManagerService$AppDeathRecipient.binderDied() from ActivityManagerService.java:1359 waiters=3 for 537ms
    01-01 01:11:46.025  2478  3362 W art     : Long monitor contention event with owner method=void com.android.server.am.ActivityManagerService$AppDeathRecipient.binderDied() from ActivityManagerService.java:1359 waiters=4 for 378ms
    01-01 01:11:46.045  2478  3362 I ActivityManager: Process org.lineageos.updater (pid 4449) has died
    01-01 01:11:46.045  2478  3362 D ActivityManager: cleanUpApplicationRecord -- 4449
    01-01 01:11:46.045  1913  1913 E lowmemorykiller: Error writing /proc/3938/oom_score_adj; errno=22
    01-01 01:11:46.050  2478  3413 W art     : Long monitor contention event with owner method=void com.android.server.am.ActivityManagerService$AppDeathRecipient.binderDied() from ActivityManagerService.java:1359 waiters=5 for 372ms
    01-01 01:11:46.505  2478  3232 D WifiService: Client connection lost with reason: 4
    01-01 01:11:47.165  2478  4666 D GraphicsStats: Buffer count: 3
    01-01 01:11:47.400  2478  2532 W art     : Long monitor contention event with owner method=int com.android.server.am.ActivityManagerService.broadcastIntent(android.app.IApplicationThread, android.content.Intent, java.lang.String, android.content.IIntentReceiver, int, java.lang.String, android.os.Bundle, java.lang.String[], int, android.os.Bundle, boolean, boolean, int) from ActivityManagerService.java:17497 waiters=0 for 667ms
    01-01 01:11:47.465  2478  4664 W art     : Long monitor contention event with owner method=int com.android.server.am.ActivityManagerService.broadcastIntent(android.app.IApplicationThread, android.content.Intent, java.lang.String, android.content.IIntentReceiver, int, java.lang.String, android.os.Bundle, java.lang.String[], int, android.os.Bundle, boolean, boolean, int) from ActivityManagerService.java:17497 waiters=1 for 858ms
    01-01 01:11:47.465  2478  3412 W art     : Long monitor contention event with owner method=int com.android.server.am.ActivityManagerService.broadcastIntent(android.app.IApplicationThread, android.content.Intent, java.lang.String, android.content.IIntentReceiver, int, java.lang.String, android.os.Bundle, java.lang.String[], int, android.os.Bundle, boolean, boolean, int) from ActivityManagerService.java:17497 waiters=2 for 859ms
    01-01 01:11:47.475  2478  4665 I ActivityManager: Process com.android.providers.calendar (pid 4434) has died
    01-01 01:11:47.480  2478  4665 D ActivityManager: cleanUpApplicationRecord -- 4434
    01-01 01:11:47.545  1913  1913 E lowmemorykiller: Error opening /proc/3938/oom_score_adj; errno=2
    01-01 01:11:47.545  1913  1913 E lowmemorykiller: Error opening /proc/4014/oom_score_adj; errno=2
    01-01 01:11:47.550  1913  1913 E lowmemorykiller: Error opening /proc/4542/oom_score_adj; errno=2
    01-01 01:11:47.550  2478  3943 W art     : Long monitor contention event with owner method=int com.android.server.am.ActivityManagerService.broadcastIntent(android.app.IApplicationThread, android.content.Intent, java.lang.String, android.content.IIntentReceiver, int, java.lang.String, android.os.Bundle, java.lang.String[], int, android.os.Bundle, boolean, boolean, int) from ActivityManagerService.java:17497 waiters=3 for 894ms
    01-01 01:11:47.560  2478  3943 I ActivityManager: Process org.cyanogenmod.themes.provider (pid 3497) has died
    01-01 01:11:47.560  2478  3943 D ActivityManager: cleanUpApplicationRecord -- 3497
    01-01 01:11:47.560  2478  2529 W art     : Long monitor contention event with owner method=int com.android.server.am.ActivityManagerService.broadcastIntent(android.app.IApplicationThread, android.content.Intent, java.lang.String, android.content.IIntentReceiver, int, java.lang.String, android.os.Bundle, java.lang.String[], int, android.os.Bundle, boolean, boolean, int) from ActivityManagerService.java:17497 waiters=4 for 673ms
    01-01 01:11:47.570  2478  2529 I ActivityManager: Process com.svox.pico (pid 4014) has died
    01-01 01:11:47.570  2478  2529 D ActivityManager: cleanUpApplicationRecord -- 4014
    01-01 01:11:48.325  2478  2529 W ActivityManager: Scheduling restart of crashed service com.svox.pico/.PicoService in 1000ms

    Verdict

    I wasted lots of time with this issue, but was finally able to reproduce it and to recover all of my data. At least, I have an explanation now for various random reboots I experienced in the past in similar low-memory conditions.

    Overall, I am really shocked that a simple, unprivileged Android app that is scheduled to start on bootup can ruin a working system so badly. Further research indicates that there are more apps known to cause such behavior [4]. I hope that a device based on a GNU/Linux system instead of Android (such as the announced Librem5) will not suffer from such a severe flaw.

    References

    [1] https://review.lineageos.org/#/c/197305/
    [2] https://stackoverflow.com/questions/25069186/invalid-address-passed-to-dlfree
    [3] https://github.com/royzhao/memtester4Android
    [4] https://gitlab.com/fdroid/fdroiddata/issues/979
    [5] https://gitlab.com/fdroid/fdroiddata/issues/979#note_48990149

    KVM-virtualization on ARM using the “virt” machine type

    Introduction

    A while ago, I described how to run KVM-based virtual machines on libre, low-end virtualization hosts on Debian Jessie [1]. For emulating the ARM board, I used the vexpress-a15 which complicates things as it requires the specification of compatible DTBs. Recently, I stepped over Peter’s article [2] that describes how to use the generic “virt” machine type instead of vexpress-a15. This promises to give some advantages such as the ability to use PCI devices and makes the process of creating VMs much easier.

    As it was also reported to me that my instructions caused trouble on Debian Stretch (virt-manager generates incompatible configs when choosing the vexpress-a15 target). So I spent some time trying to find out how to run VMs using the “virt” machine type using virt-manager (Peter’s article only described the manual way using command-line calls). This included several traps, so I decided to write up this article. It gives a brief overview how to create a VM using virt-manager on a ARMv7 virtualization host such as the Cubietruck or the upcoming EOMA68-A20 computing card.

    Disclaimer

    All data and information provided in this article is for informational purposes only. The author makes no representations as to accuracy, completeness, currentness, suitability, or validity of any information on this article and will not be liable for any errors, omissions, or delays in this information or any losses, injuries, or damages arising from its display or use. All information is provided on an as-is basis.

    In no event the author we be liable for any loss or damage including without limitation, indirect or consequential loss or damage, or any loss or damage whatsoever arising from loss of data or profits arising out of, or in connection with, the use of this article.

    Tested VMs

    I managed to successfully create and boot up the following VMs on a Devuan (Jessie) system:

    • Debian Jessie Installer
    • Debian Stretch Installer
    • Debian Unstable Installer
    • Fedora Core 27 (see [3] for instructions how to obtain the necessary files)
    • Arch Linux, using the latest ARMv7 files available at [4]
    • LEDE 17.0.1.4

    I was able to reproduce the steps for the Debian guests on a Debian Stretch system as well (I did not try with the other guests).

    Requirements / Base installation

    This article assumes you have setup a working KVM virtualization host on ARM. If you don’t, please work through my previous article [1].

    Getting the necessary files

    Depending on the system you want to run in your Guest, you typically need an image of the kernel and the initrd. For the Debian-unstable installer, you could get the files like this:

    wget http://http.us.debian.org/debian/dists/unstable/main/installer-armhf/current/images/netboot/vmlinuz -O vmlinuz-debian-unstable-installer

    and

    wget http://http.us.debian.org/debian/dists/unstable/main/installer-armhf/current/images/netboot/vmlinuz -O initrd-debian-unstable-installer.gz

    Creating the Guest

    Now, fire up virt-manager and start the wizard for creating a new Guest. In the first step, select “Import existing disk image” and the default settings, which should use the Machine Type “virt” already:

    In the second step, choose a disk image (or create one) and put the paths to the Kernel and Initrd that you downloaded previously. Leave the DTB path blank and put “console=ttyAMA0″ as kernel arguments. Choose an appropriate OS type of just leave the default (may negatively impact the performance of your guest, or other things may happen such as your virtual network card may not be recognized by the installer):

    Next, select memory and CPU settings as required by your Guest:

    Finally, give the VM a proper name and select the “Customize configuration before install” option:

    In the machine details, make sure the CPU model is “host-passthrough” (enter it manually if you can’t select it in the combo box):

    In the boot options tab, make sure the parameter “console=ttyAMA0″ is there (otherwise you will not get any output on the console). Depending on your guest, you might also need more parameters, such as for setting the rootfs:

    Finally, click “begin installation” and you should see your VM boot up:

    Post-Installation steps

    Please note, that after installing your guest, you must extract the kernel and initrd from the installed guest image (you want to boot the real system, not the installer) and change your VM configuration to use these files instead.

    Eventually, I will provide instructions how to do this for a few guest types in the future. Meanwhile, you can obtain instructions for extracting the files from a Debian guest from Peter’s article [2].

    [1] https://blogs.fsfe.org/kuleszdl/2016/11/06/installing-a-libre-low-power-low-cost-kvm-virtualization-host/
    [2] https://translatedcode.wordpress.com/2016/11/03/installing-debian-on-qemus-32-bit-arm-virt-board
    [3] https://fedoraproject.org/wiki/QA:Testcase_Virt_ARM_on_x86
    [4] http://os.archlinuxarm.org/os/ArchLinuxARM-armv7-latest.tar.gz

    Benchmarking microSD cards and more

    It’s worth re-visiting this blog-post as I am going to do more benchmarks and add more results over time.

    Motivation

    If you ever tried using a flash drive (such a SD, microSD, USB drive or CF) for your root or home filesystem on a small computing device or smartphone, you probably have noticed that flash cards are in most cases a lot slower than integreted eMMC flash. Since most filesystems use 4k blocks, the random write/read performance using 4k blocks is what matters most in such scenarios. And while flash cards don’t come close to internal flash in these disciplines, there are significant differences between the models.

    Jeff Geerling [1,2] has already benchmarked the performance of various microSD cards on different models of the “blobby” Raspberry Pi. I had a number of different microSD cards at hand and I tried to replicate his results on my sample. In addition, I extended the comparison to regular SD cards and USB drives.

    Disclaimer


    All data and information provided in this article is for informational purposes only. The author makes no representations as to accuracy, completeness, currentness, suitability, or validity of any information on this article and will not be liable for any errors, omissions, or delays in this information or any losses, injuries, or damages arising from its display or use. All information is provided on an as-is basis.

    In no event the author will be liable for any loss or damage including without limitation, indirect or consequential loss or damage, or any loss or damage whatsoever arising from loss of data or profits arising out of, or in connection with, the use of this article.

    Environment and tools

    I encourage every reader to replicate my benchmarks and share their results. Therefore, I first want to describe how the measures were taken.

    General equipment

    For the benchmarks, I used the following hardware on my workstation (listing only relevant parts):

    • Renesas Technology Corp. uPD720201 USB 3.0 Host Controller (rev 03)
    • Logilink CR0034A USB 3.0 card reader [5], used for (m)SD cards
    • Apacer ? USB 2.0 card reader, used for CF cards
    • Debian 9 “Stretch”

    In the past, I also did iozone benchmarks using a Logilink CR0015 USB 2.0 (m)SD card reader [4] which are provided at the end of the article for reference.

    iozone Benchmarks

    For the first tests, Just like Jeff, I used the open source (but non-free) tool “iozone” [3] in the current stable version that is available on Debian Stretch (3.429).

    I disabled caches and benchmarked on raw devices to avoid measuring filesystem overhead. Therefore, I used the following call to iozone to run the benchmarks (/dev/sde is my sdcard):

    $ iozone -e -I -a -s 100M -r 4k -r 16M -i 0 -i 1 -i 2 -f /dev/sdX
    

    (replacing sdX by the actual device name)

    ddrescue benchmarks

    I used GNU ddrescue (1.21) in synchronous mode to overwrite a device with zeroes and measured the performance there. For this purpose, I used the following call to do a write benchmark:

    $ ddrescue -D --force /dev/zero /dev/sdX
    

    (replacing sdX by the actual device name)

    Similarly, to measure the read performance I used (for media > 50G the test was aborted when 50G were reached; tmp was mounted as tmpfs):

    $ ddrescue -D --force /dev/sdX /tmp/foo
    

    (replacing sdX by the actual device name)

    Results

    iozone

    Here are the iozone results:

    Manufacturer Make / model Type Speed Class Capacity 16M seq. read (MB/s) 16M seq. write (MB/s) 4K rand. read (MB/s) 4K rand. write (MB/s)
    Samsung Pro+ (2017) mSDXC U3 64 GB 42.33 39.75 8.83 2.39
    Samsung Evo+ (2017) mSDXC U3 64 GB 42.47 40.10 8.48 2.29
    Samsung Evo+ mSDHC U1 32 GB 42,53 34,48 8.01 3.97
    Sandisk Ultra mSDXC C10 64 GB 18.21 19.19 5.35 1.06
    Sandisk Cruzer USB 2.0 4 GB 22.89 4.05 6.86 0.44
    Kingston Data Traveller G3 USB 2.0 8 GB 21.52 4.96 6.96 0.08
    Super Talent Pico C USB 2.0 8 GB 36.72 13.77 DNF DNF
    Sandisk Ultra USB 3.0 32 GB 148.46 52.97 6.41 2.27
    Samsung ? USB 2.0 1 GB 21.48 13.47 11.10 0.23
    Transcend ? CF 80x 1 GB 20.19 12.56 7.56 0.19
    Data Power ? (SLC) CF ? 1 GB 12.91 4.97 6.45 0.003
    Toshiba ? CF ? 256 MB 7.52 2.86 3.78 0.003
    Samsung Evo SDHC U1 32 GB 46.34 17.50 9.29 4.01
    Emtec ? mSDHC C10 4 GB 22.06 9.21 4.08 0.29
    Sandisk Ultra mSDXC A1 200 GB 42.75 36.80 8.54 5.13
    ? mSDHC ? ? 8 GB tbd tbd tbd tbd

    ddrescue

    Here are the results using the USB 3.0 card reader or the device directly (for usb drives):

    Manufacturer Make / model Type Rating Capacity Avg read (MB/s) Avg write (MB/s)
    Samsung Pro+ (2017) mSDHC U3 64 GB tbd 34.77
    Samsung Evo+ mSDHC U1 32 GB 41.95 21.54
    Sandisk Cruzer USB 2.0 4 GB 36.37 2.64
    Super Talent Pico C USB 2.0 8 GB 36.29 19.80
    Sandisk Ultra USB 3.0 32 GB 115.00 24.62
    Samsung ? USB 2.0 1 GB 21.63 14.62
    Kingston Data Traveller G3 USB 3.0 8 GB 20.64 5.07
    Transcend ? CF 80x 1 GB 19.26 12.60
    Data Power ? (SLC) CF ? 1 GB 12.68 5.02
    Toshiba ? CF ? 256 MB 7.33 2.82
    Samsung Evo SDHC U1 32 GB 45,39 19.39
    ? mSDHC ? ? 8 GB 15.64 5.45
    Emtec ? mSDHC C10 4 GB 17.93 11.78
    Sandisk Ultra mSDXC A1 200 GB 41.96 32.35

    Legacy USB 2.0 iozone results

    Previously, I ran the iozone benchmarks using the USB 2.0 card reader. They might be relevant, if you are looking for a storage device which you want to put in a computing device with a slow card reader:

    Manufacturer Make / model Type Speed Class Capacity 16M seq. read (MB/s) 16M seq. write (MB/s) 4K rand. read (MB/s) 4K rand. write (MB/s)
    Adata ? mSD C4 32 GB 9.58 2.97 2.62 0.64
    Samsung Evo mSDHC U1 32 GB 17.80 10.13 4.45 1.32
    Samsung Evo+ mSDHC U1 32 GB 18.10 14.27 5.28 2.86
    Samsung Evo+ (2017) mSDXC U3 64 GB 16,98 8,03 7,99 2,99
    Sandisk Extreme mSDXC U3 64 GB 18.44 11.22 5.20 2.36
    Sandisk Ultra mSDHC C10 64 GB 18.16 8.69 3.73 0.80
    Toshiba Exceria mSDXC U3 64 GB 16.10 6.60 3.82 0.09
    Kingston ? mSD C4 8 GB 14.69 3.71 3.97 0.18

    Verdict and future work

    I can confirm Jeff’s results about microSD cards and would also recommend the Evo+ (both the old and the 2017 model) which have the best 4K random write performance of the sample. On the other hand, I am very disappointed about the Toshiba Exceria card. Actually running a device on this card with a very sluggish performance was the reason why I took this benchmark initiative. And indeed, after switching to the Evo+, the device feels much snappier now.

    Of course, it would be worthwhile to add more cards and drives to this benchmark. Also, using fio instead of the non-free iozone might be interesting. Furthermore, doing the benchmarks internally on the device or using a other USB 3.0 card readers might be also interesting.

    References

    [1] http://www.pidramble.com/wiki/benchmarks/microsd-cards
    [2] http://www.jeffgeerling.com/blogs/jeff-geerling/raspberry-pi-microsd-card
    [3] http://www.iozone.org/
    [4] http://www.logilink.eu/Products_LogiLink/Notebook-Computer_Accessories/Card_Reader/Cardreader_USB_20_Stick_external_for_SD-MMC_CR0015.htm
    [5] http://www.logilink.org/Produkte_LogiLink/Notebook-Computerzubehoer/Kartenleser/Cardreader_USB_30_SD-SD-HC-Micro_SD-Micro_SD-HC_CR0034A.htm?seticlanguage=en

    KVM virtualization with Allwinner A20 on Debian: libre, low-power, low-cost

    Introduction

    Various cheap ARM boards based on the Allwinner A20 SoC are available already for a few years. The first EOMA68 computer [1] will be also based on this chipset. Not many users know that the Allwinner A20 supports hardware-supported virtualization as well. Its Cortex A7 cores allow running hardware-accelerated ARM virtual machines (guests) using KVM or Xen.

    While Allwinner has been blamed to violate the GPL for years [2], their A20 SoC is imho one of the best choices today when it comes to building a small and libre server for SOHO use (thanks to the hard work of the Allwiner-independent Linux-Sunxi community). While many SoCs found on popular boards like those from the Raspberry Pi family require proprietary blobs, the A20 works with a free bootloader and requires no proprietary drivers or firmware for basic operation.

    The virtualization on A20 hosts works out of the box on Debian Jessie with the stock kernel and official packages in main — without cross-compiling, patching or other tinkering (this was not the case in the past, see [3]). This also means that updating your host and guests later will be easy and painless. Creating and managing guests can be done with virt-manager [4] – a secure and comfortable graphical user interface licensed under GPLv3.

    After first discussing some A20 hardware options, this guide takes the example of the Olimex “A20-OLinuXIno-LIME2″ board [5] and shows how to turn it into a virtualization host. Then shows how create and manage guest-VMs on the virtualization host. The guide assumes that you are running a a GNU/Linux-based desktop system from which you want to manage the A20 device.

    Disclaimer


    All data and information provided in this article is for informational purposes only. The author makes no representations as to accuracy, completeness, currentness, suitability, or validity of any information on this article and will not be liable for any errors, omissions, or delays in this information or any losses, injuries, or damages arising from its display or use. All information is provided on an as-is basis.

    In no event the author we be liable for any loss or damage including without limitation, indirect or consequential loss or damage, or any loss or damage whatsoever arising from loss of data or profits arising out of, or in connection with, the use of this article.

    Hardware choices

    There are plenty of boards with the Allwinner A20. However, only few are known to work out of the box on Debian Jessie. The particular page on the Debian Wiki [6] mentions the following boards in particular:

    • Cubietech Cubieboard2
    • Cubietech Cubieboard3 (“Cubietruck”)
    • LeMaker Banana Pro
    • Olimex A20-OLinuXino-LIME
    • Olimex A20-OLinuXino-LIME2 (only the regular one, not the eMMC variant!)
    • Olimex A20-Olinuxino Micro

    While some of these boards feature Gigabit ethernet and SATA, only the Cubieboard 3 has 2 GB of RAM. To me, this seems to be the best choice for a A20-based KVM virtualization host. Since I only had a spare Olimex A20-OLinuXino-LIME2 board at hand, this guide uses this board as example.

    Beware: The “A20-OLinuXino-LIME2″ and the “A20-OLinuXino-LIME2-eMMC” are not the same! Debian provides no firmware for the “A20-OLinuXino-LIME2-eMMC” and I could not get it to work at all on Debian. Although I thought that they would be the same except for the eMMC flashg, the firmware for the regular “”A20-OLinuXino-LIME2″ did NOT work for me at all!

    Base installation

    The article in the Debian wiki provides the necessary information on installing Debian Jessie using the text-based Debian-Installer. Make sure you have a microSD card with a good 4K random I/O performance or the installation will take forever and your A20 system will run terribly slow afterwards (see my article comparing performance of various microSD cards).

    If you don’t have a serial cable and want to install using the HDMI output, you need to use the installer images from unstable. The easiest way to do is to fetch the firmware file from unstable and the partition image from Jessie. Then write them to your microSD card (replace /dev/sdX with your particular device):

    $ zcat firmware.A20-OLinuXino-Lime2.img.gz partition.img.gz > /dev/sdX
    

    Next, insert the microSD card into your device, connect your device to your LAN and power it up. Then install Debian as usual using the text-based installer. During the installation, sure to create a root account (needed for KVM) and a ext2 boot partition (the safest method here is to use the guided installer). When tasksel gets called, make sure to install the tasks/packages “SSH Server” and “Standard system utilities”.

    Note for users of the German mirrors: Using the mirror “ftp.de.debian.org” will break your installation as something seems to be missing there as of 2016-11-05. Using “ftp2.de.debian.org” works fine.

    Installing the KVM virtualization

    By default, interactive root logins are not allowed on Debian. Therefore, make sure you copy over your SSH public key to your a20-box or simply enable interactive root logins over SSH by changing the following option in /etc/ssh/sshd_config:

    #PermitRootLogin without-password
    PermitRootLogin yes
    

    Then restart the SSH server:

    # service ssh restart
    

    Now you should be able to log in directly as root. Next, install the virtualization packages:

    # apt install libvirt-daemon-system
    
    Reading package lists... Done
    Building dependency tree
    Reading state information... Done
    The following extra packages will be installed:
    
    ...
    
    0 upgraded, 105 newly installed, 0 to remove and 0 not upgraded.
    Need to get 44.4 MB of archives.
    After this operation, 182 MB of additional disk space will be used.
    Do you want to continue? [Y/n]
    

    Now fire up virt-manager on your desktop and make sure you can connect to your a20-box:

    Creating and installing a guest

    For running ARM virtual machines you need a kernel and DTBs which support the VExpress-A15 chipset (the ARM reference board usually emulated on ARM). This is already provided in stock Debian, so there is no need to compile anything yourself.

    Regarding the guest, you can choose any Linux you want. In the following example, we will install a Debian Jessie guest using the Debian installer. Therefore we need to download the to the Virtualization host. This time, we don’t need a partition image but can use the usual the initrd installer-Image from the Debian server. SSH into the virtualization host and download it:

    wget http://ftp.uk.debian.org/debian/dists/jessie/main/installer-armhf/current/images/netboot/initrd.gz -O initrd-installer-jessie.gz

    For the installation, you will also need a different kernel because in the Kernel installed on the host the network drivers are in initrd, but the Installer’s initrd assumes they are in the kernel. Therefore, fetch a kernel for the installer:

    wget http://ftp.uk.debian.org/debian/dists/jessie/main/installer-armhf/current/images/netboot/vmlinuz -O vmlinuz-installer-jessie

    Now, fire up virt-manager on your desktop and connect to the Virtualization host. Then, start the wizard for creating guests using “create new virtual machine”. On the first screen, change the machine type to “vexpress-a15″:

    On the next screen, specify a storage (just create one using the dialog following “Browse”), and also use “Browse” to locate the kernel and initrd images so you specify the ones we just downloaded. For the DTB, we’ll use the one that is part of Debian’s stock kernel and resides under /usr/lib/linux-image-3.16.0-4-armmp-armmp-lpae/vexpress-v2p-ca15-tc1.dtb (make sure it corresponds to the version on your a20-host! TODO: Is there any symlink which points to the current version?)). The kernel args are also very important, or you will not get any output. For this line, specify the following:

    root=/dev/vda1 console=ttyAMA0,115200 rootwait
    

    Finally, select OS type and version appropiately. Your dialog should look like this:

    Then, specify RAM (e.g. 256MB) and the number of CPUs (e.g. 1) you want to give the guest and jump to the last screen. Here, give your guest a nice name and make sure you check the “Customize configuration before install” checkbox before you click “Finish”:

    Otherwise, you would end up with an error message like this:

    Unable to complete install: 'internal error: early end of file from monitor: possible problem:
    kvm_init_vcpu failed: Invalid argument
    

    In the configuration of the VM, under “Processor”, change the configuration from “Hypervisor Default” to “Application Default”:

    To get better performance, also change the BUS of your virtual disk to “VIRTIO” (by default, it would emulate an SD card):

    And do the same for the network adapter:

    Finally, fire up the guest using “Begin installation”. If everything goes fine, you should see the kernel boot and be presented with the welcome screen of the installer. For jessie, it should look like this:

    If you selected the kernel and initrd from stretch/sid you should get a nicer color screen (make sure you set the baudrate of the console to 115200 or you will get a disorted output!):

    When partitioning the guest, just create a single root partition spanning the whole (virtual) device. The guest will always boot using externally specified kernels, dtbs and initrds, therefore there is no use in creating a /boot partition as the “guided install” would do.

    Near the end of the installation, you will be notified that no bootloader could be installed. You can safely ignore this message:

    After finishing the installation, the system will boot again into the installer because the initrd is still active. To change this, power off the guest (“Force Off”) and specify in the boot options to use the kernel and initrd image of your A20 host instead (whenever they will be updated on the host, the guests will also get the update on their next boot):

    Now your guest should finally succeed to boot up:

    And you can check that it indeed uses the current A20 kernel on the host and virtualizes the VExpress15 SoC:

    Benchmarks

    Finally, I want to provide some benchmarks so you can get a feeling about the impact of the virtualization. The benchmarks were done using a guest with 2 CPUs and 512MB memory assigned.

    IO/Performance

    For a first I/O benchmark, I used hdparm.

    On the host:

    $ hdparm -tT /dev/mmcblk0
    
    /dev/mmcblk0:
     Timing cached reads:   814 MB in  2.00 seconds = 406.33 MB/sec
     Timing buffered disk reads:  66 MB in  3.01 seconds =  21.93 MB/sec
    

    On the guest:

    $ hdparm -tT /dev/vda
    
    /dev/vda:
     Timing cached reads:   694 MB in  2.00 seconds = 346.49 MB/sec
     Timing buffered disk reads:  30 MB in  3.15 seconds =   9.52 MB/sec
    

    CPU processing

    For benchmarking processing, I used the openssl suite to do a few simple AES benchmarks:

    $ openssl speed aes
    

    On the host:

    ...
    The 'numbers' are in 1000s of bytes per second processed.
    type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes
    aes-128 cbc      20267.83k    22390.70k    23325.10k    23575.89k    23642.11k
    aes-192 cbc      17594.13k    19464.20k    19956.57k    20102.83k    20146.86k
    aes-256 cbc      15727.25k    17158.89k    17592.58k    17706.67k    17738.41k
    

    On the guest:

    ...
    The 'numbers' are in 1000s of bytes per second processed.
    type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes
    aes-128 cbc      19784.01k    22100.48k    22697.56k    23272.20k    23288.29k
    aes-192 cbc      17363.72k    19097.02k    19643.68k    19786.41k    19800.53k
    aes-256 cbc      15455.28k    16939.28k    17374.44k    17415.85k    17504.58k
    

    Conclusion

    With one of the Allwinner A20 boards supported by Debian, you can easily build a tiny virtualization host that can handle a few simple VMs and draws only 2-3W of power. While this process was pretty cumbersome in the past (you had to cross-compile kernels etc.), thanks to the efforts of the Debian project and Linux-Sunxi community, it is now pretty straight-forward with only few caveats involved. This might also be an interesting option if you want to run a low-power virtualization cluster on fully libre software down to the firmware level.

    References

    [1] https://www.crowdsupply.com/eoma68/micro-desktop
    [2] http://linux-sunxi.org/GPL_Violations
    [3] http://blog.flexvdi.com/2014/07/28/enabling-kvm-virtualization-on-arm-allwinner-a20/
    [4] https://virt-manager.org/
    [5] https://www.olimex.com/Products/OLinuXino/A20/A20-OLinuXIno-LIME2/
    [6] https://wiki.debian.org/InstallingDebianOn/Allwinner

    Backing up and restoring data on Android devices directly via USB (Howto)

    Motivation

    I was looking for a simple way to backup data on Android devices directly to a device running GNU/Linux connected over a USB cable (in my case, a desktop computer).

    Is this really so unique that it’s worth writing a new article about it? Well, in my case, I did not want to buffer the data on any “intermediate” devices such as storage cards connected via microSD or USB-OTG. Also, I did not want to use any proprietary tools or formats. Instead, I wanted to store my backups in “oldschool” formats such as dd-images or tar archives. I did not find a comprehensive howto for that, so I decided to write this article.

    Disclaimer


    All data and information provided in this article is for informational purposes only. The author makes no representations as to accuracy, completeness, currentness, suitability, or validity of any information on this article and will not be liable for any errors, omissions, or delays in this information or any losses, injuries, or damages arising from its display or use. All information is provided on an as-is basis.

    In no event the author we be liable for any loss or damage including without limitation, indirect or consequential loss or damage, or any loss or damage whatsoever arising from loss of data or profits arising out of, or in connection with, the use of this article.

    Overview

    This article describes two different approaches. Both have their pros and cons:

    • Block-level: Doing 1:1 block-level backups (above the file system) is an imaging approach that corresponds to doing dd-style backups.
    • Filesystem-level: Doing filesystem-level backups (on the file system) corresponds to tar-style backups.

    An important factor when doing backups is also performance. Filesystem-level backups are usually faster on block devices which are only filled up to a small degree. However, due to the file system overhead they have a lower “raw” throughput rate — especially when backing up data on flash media such as microSD cards. Here, typical filesystems such as ext4 or f2fs operating with a 4K block size are a major bottleneck as these media often have horrible 4k write/read performance.

    The following instructions for applying these approaches assume that you already have a “liberated” Android device which can boot into TWRP (a free Android recovery) or CWM. I am using the example of a Nexus-S running Replicant 4.2.0004 and TWRP 2.8.7.0, but the approaches also work with most other Android distributions and recoveries.

    Getting familiar with the block devices on your Android device

    First of all, you should know which block device you actually want to backup. The internal flash on Android devices is usually partitioned in about 15-25 partitions, depending on the device. To get a first overview, you can try the following (I am using adb shell on the desktop):

    $ adb shell cat /proc/partitions
    

    major minor #blocks name

    31 0 2048 mtdblock0
    31 1 1280 mtdblock1
    31 2 8192 mtdblock2
    31 3 8192 mtdblock3
    31 4 480768 mtdblock4
    31 5 13824 mtdblock5
    31 6 6912 mtdblock6
    179 0 15552512 mmcblk0
    179 1 524288 mmcblk0p1
    179 2 1048576 mmcblk0p2
    179 3 13978607 mmcblk0p3
    179 16 1024 mmcblk0boot1
    179 8 1024 mmcblk0boot0

    To find out what the partitions are about you can inspect the directory /dev/block/platform//by-Name/ which contains symlinks to the actual partitions. In my case, the Nexus-S has two flash chips and I am listing the partitions of the one where the userdata-partition resides on:

    $ adb shell ls -l /dev/block/platform/s3c-sdhci.0/by-name
    

    lrwxrwxrwx root root 2016-11-02 19:51 media -> /dev/block/mmcblk0p3
    lrwxrwxrwx root root 2016-11-02 19:51 system -> /dev/block/mmcblk0p1
    lrwxrwxrwx root root 2016-11-02 19:51 userdata -> /dev/block/mmcblk0p2

    Please note that unlike the Nexus-S, most newer Android devices only have a single eMMC flash chip and don't use MTD devices anymore.

    Block-level approach

    Block-level backups take up a lot of space (without compression) and extracting single files is cumbersome (especially when talking about encrypted data partitions or backups of the whole flash). On the other hand, "just" restoring a full backup is easy.

    Backing up a single partition

    Now that you know which block devices you want to backup, you can directly create a 1:1 image via adb pull as you would normally do by using dd. In our case:

    $ adb pull /dev/block/platform/s3c-sdhci.0/by-name/userdata
    

    7942 KB/s (1073741824 bytes in 132.027s)

    On your workstation, you will obtain a file named userdata which contains the whole partition/filesystem as an image. If you didn't enable encryption on your Android device, you can directly mount this file as loopback device and access its contents:

    $ mount userdata /mnt
    

    Restoring a single partition


    BE CAREFUL! RESTORING THE WRONG IMAGE OR WRITING TO THE WRONG BLOCK DEVICE CAN RUIN YOUR DATA OR EVEN BRICK YOUR ANDROID DEVICE!

    To restore your backup, you can simply use adb push. In my case:

    $ adb push userdata /dev/block/platform/s3c-sdhci.0/by-name/userdata
    

    failed to copy 'userdata' to '/dev/block/platform/s3c-sdhci.0/by-name/userdata': No space left on device

    Alternative: Operating on the whole block device

    Instead of backing up just a single partition, it is also possible to backup the whole flash device including all partitions. Example:

    $ adb pull /dev/block/mmcblk0
    

    Remarks:

    • On some devices, not all partitions are readable and, thus, cannot be backed up.
    • Please be careful with restores!
    • Accessing files inside this image is not straight-forward (but doable).

    Filesystem-level approach

    Filesystem-level backups only work for single partitions, take up as much space as the files on the particular filesystem you backup and it is easy to access individual files in them. I am using a combination of adb, netcat and tar to create and restore these backups.

    Backing up your data

    First, connect to your device via an adb shell:

    $ adb shell
    

    Then, change to the directory from where you want to create your backup. If your device was not automatically mounted, you have to do it first:

    # mount /dev/block/platform/s3c-sdhci.0/by-name /data
    

    Now change to this directory:

    # cd /data
    

    Now, start the netcat process:

    # tar -cvp . | busybox nc -lp 5555
    

    On the receiver side (desktop), set up adb port forwarding:

    $ adb forward tcp:4444 tcp:5555
    

    Then, start the process to receive the tar file:

    $ nc -w 10 localhost 4444 > userdata.tar
    

    You should see your files being packed up on the Android side:

    ./
    ./lost+found/
    ./dontpanic/
    ./misc/
    ./misc/adb/
    ./misc/audit/
    ./misc/audit/audit.log
    ...

    Now wait for the process to exit.

    Restoring your data

    Again, on your receiver side (Android device), mount /data if it was not mounted yet and change in there:

    # mount /dev/block/platform/s3c-sdhci.0/by-name /data
    
    # cd /data
    

    Now, start the tar extraction process:

    # busybox nc -lp 5555 | tar -xpvf -
    

    On the sender side (desktop), again, set up adb port forwarding:

    $ adb forward tcp:4444 tcp:5555
    

    And send the tar file:

    $ cat userdata.tar | nc -q 2 localhost 4444
    

    Now you should be able to see your previously backed up files getting restored...

    References

    For the filesystem-level part of the article I used and adapted the following sources:

    • http://www.screenage.de/blog/2007/12/30/using-netcat-and-tar-for-network-file-transfer/
    • http://stackoverflow.com/questions/15278587/pipe-into-adb-shell