How a single unprivileged app can brick the whole Android system

This article is highly subjective and only states the author’s opinion based on actual observations and “wild” assumptions. Better explanations and corrections are warmly welcome!

Motivation

After updating an App from the F-Droid store (OpenCamera), my Android device was completely unusable. In this state, the only feasible option for a typical end-user to recover the device (who does not know how to get to safe mode in order to remove or downgrade the app [5]) would have been to wipe data in recovery, loosing all data.

How can such a disaster happen? In this article, I argue why I have serious doubts about the memory management approach taken in Android.

The failure

After updating the OpenCamera app to the recently released version 1.42, my Android device ran into a bootloop that was hard to recover from. I was able to repeatingly reproduce the failure on a different device, namely the following:

  • Device: Samsung Galaxy S3 (i9300)
  • ROM: Lineage OS 13 (Android 6.0), freshly built from latest sources, commit 42f4b851c9b2d08709a065c3931f6370fd78b2b0 [1]

Steps to reproduce:

  1. wipe all data and caches
  2. newly configure the device using the first-use wizard
  3. install the F-Droid store
  4. search for “Open Camera”
  5. install Open Camera version 1.42

Expected:

The install completes and the app is available. If installation fails (for whatever reason), an error message is shown but the device is still working

Actual:

The install freezes, the LineageOS splash screen appears and re-initializes all apps; this happens several times and after aprox 10-15 minutes the device is back “working”; when trying to start apps they crash or even the launcher (“Trebuchet”) crashes. After rebooting the device, it is stuck in an infinite loop initializing apps.

The fault (what happens under the hood?)

When installing OpenCamera, the following is printed in the log:

12-10 14:48:30.915  4034  5483 I ActivityManager: START u0 {act=org.fdroid.fdroid.installer.DefaultInstaller.action.INSTALL_PACKAGE dat=file:///data/user/0/org.fdroid.fdroid/files/Open Camera-1.42.apk cmp=org.fdroid.fdroid/.installer.DefaultInstallerActivity (has extras)} from uid 10070 on display 0
12-10 14:48:30.915  4034  5483 W ActivityManager: startActivity called from non-Activity context; forcing Intent.FLAG_ACTIVITY_NEW_TASK for: Intent { act=org.fdroid.fdroid.installer.DefaultInstaller.action.INSTALL_PACKAGE dat=file:///data/user/0/org.fdroid.fdroid/files/Open Camera-1.42.apk cmp=org.fdroid.fdroid/.installer.DefaultInstallerActivity (has extras) }
12-10 14:48:30.925  4034  5483 D lights  : set_light_buttons: 2
12-10 14:48:30.955  4034  5649 I ActivityManager: START u0 {act=android.intent.action.INSTALL_PACKAGE dat=file:///data/user/0/org.fdroid.fdroid/files/Open Camera-1.42.apk cmp=com.android.packageinstaller/.PackageInstallerActivity (has extras)} from uid 10070 on display 0
12-10 14:48:31.085  6740  6740 W ResourceType: Failure getting entry for 0x7f0c0001 (t=11 e=1) (error -75)
12-10 14:48:31.700  4034  4093 I ActivityManager: Displayed com.android.packageinstaller/.PackageInstallerActivity: +724ms (total +758ms)
12-10 14:48:36.770  4034  4362 D lights  : set_light_buttons: 1
12-10 14:48:36.840  4034  4938 I ActivityManager: START u0 {dat=file:///data/user/0/org.fdroid.fdroid/files/Open Camera-1.42.apk flg=0x2000000 cmp=com.android.packageinstaller/.InstallAppProgress (has extras)} from uid 10018 on display 0
12-10 14:48:36.850  3499  3895 D audio_hw_primary: select_output_device: AUDIO_DEVICE_OUT_SPEAKER
12-10 14:48:36.955  6863  6874 D DefContainer: Copying /data/user/0/org.fdroid.fdroid/files/Open Camera-1.42.apk to base.apk
12-10 14:48:37.100  4034  4093 I ActivityManager: Displayed com.android.packageinstaller/.InstallAppProgress: +251ms
12-10 14:48:37.155  6740  6753 D OpenGLRenderer: endAllStagingAnimators on 0x486226f0 (RippleDrawable) with handle 0x48604d28
12-10 14:48:37.170  4034  4100 W ResourceType: Failure getting entry for 0x7f0c0001 (t=11 e=1) (error -75)
12-10 14:48:37.465  4034  4100 I PackageManager.DexOptimizer: Running dexopt (dex2oat) on: /data/app/vmdl872450731.tmp/base.apk pkg=net.sourceforge.opencamera isa=arm vmSafeMode=false debuggable=false oatDir = /data/app/vmdl872450731.tmp/oat bootComplete=true
12-10 14:48:37.585  7205  7205 I dex2oat : Starting dex2oat.
12-10 14:48:37.585  7205  7205 E cutils-trace: Error opening trace file: No such file or directory (2)
12-10 14:48:42.405  7205  7205 I dex2oat : dex2oat took 4.815s (threads: 4) arena alloc=5MB java alloc=2023KB native alloc=13MB free=1122KB
12-10 14:48:42.415  4034  4100 D lights  : set_light_buttons: 2
12-10 14:48:42.680  4034  4100 V BackupManagerService: restoreAtInstall pkg=net.sourceforge.opencamera token=3 restoreSet=0
12-10 14:48:42.680  4034  4100 W BackupManagerService: Requested unavailable transport: com.google.android.gms/.backup.BackupTransportService
12-10 14:48:42.680  4034  4100 W BackupManagerService: No transport
12-10 14:48:42.680  4034  4100 V BackupManagerService: Finishing install immediately
12-10 14:48:42.705  4034  4100 W Settings: Setting install_non_market_apps has moved from android.provider.Settings.Global to android.provider.Settings.Secure, returning read-only value.
12-10 14:48:42.705  4034  4100 I art     : Starting a blocking GC Explicit
12-10 14:48:42.805  4034  4100 I art     : Explicit concurrent mark sweep GC freed 52637(2MB) AllocSpace objects, 20(424KB) LOS objects, 33% free, 14MB/21MB, paused 2.239ms total 96.416ms
12-10 14:48:42.835  4034  4363 I InputReader: Reconfiguring input devices.  changes=0x00000010
12-10 14:48:42.935  5420  5420 D CarrierServiceBindHelper: Receive action: android.intent.action.PACKAGE_ADDED
12-10 14:48:42.940  5420  5420 D CarrierServiceBindHelper: mHandler: 3
12-10 14:48:42.940  5420  5420 D CarrierConfigLoader: mHandler: 9 phoneId: 0
12-10 14:48:42.945  4034  4034 F libc    : invalid address or address of corrupt block 0x120 passed to dlfree
12-10 14:48:42.945  4034  4034 F libc    : Fatal signal 11 (SIGSEGV), code 1, fault addr 0xdeadbaad in tid 4034 (system_server)
12-10 14:48:42.950  3496  3496 I DEBUG   : property debug.db.uid not set; NOT waiting for gdb.
12-10 14:48:42.950  3496  3496 I DEBUG   : HINT: adb shell setprop debug.db.uid 100000
12-10 14:48:42.950  3496  3496 I DEBUG   : HINT: adb forward tcp:5039 tcp:5039
12-10 14:48:42.975  3496  3496 F DEBUG   : *** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
12-10 14:48:42.975  3496  3496 F DEBUG   : LineageOS Version: '13.0-20171125-UNOFFICIAL-i9300'
12-10 14:48:42.975  3496  3496 F DEBUG   : Build fingerprint: 'samsung/m0xx/m0:4.3/JSS15J/I9300XXUGMJ9:user/release-keys'
12-10 14:48:42.975  3496  3496 F DEBUG   : Revision: '0'
12-10 14:48:42.975  3496  3496 F DEBUG   : ABI: 'arm'
12-10 14:48:42.975  3496  3496 F DEBUG   : pid: 4034, tid: 4034, name: system_server  >>> system_server <<<
12-10 14:48:42.975  3496  3496 F DEBUG   : signal 11 (SIGSEGV), code 1 (SEGV_MAPERR), fault addr 0xdeadbaad
12-10 14:48:43.030  3496  3496 F DEBUG   : Abort message: 'invalid address or address of corrupt block 0x120 passed to dlfree'
12-10 14:48:43.030  3496  3496 F DEBUG   :     r0 00000000  r1 00000000  r2 00000000  r3 00000002
12-10 14:48:43.030  3496  3496 F DEBUG   :     r4 00000120  r5 deadbaad  r6 404e0f38  r7 40005000
12-10 14:48:43.030  3496  3496 F DEBUG   :     r8 00000128  r9 bee01b0c  sl 40358be3  fp 40358bec
12-10 14:48:43.030  3496  3496 F DEBUG   :     ip 404db5d8  sp bee019f8  lr 404abfab  pc 404abfaa  cpsr 60070030
12-10 14:48:43.045  3496  3496 F DEBUG   :
12-10 14:48:43.045  3496  3496 F DEBUG   : backtrace:
12-10 14:48:43.045  3496  3496 F DEBUG   :     #00 pc 00030faa  /system/lib/libc.so (dlfree+1285)
12-10 14:48:43.045  3496  3496 F DEBUG   :     #01 pc 000158df  /system/lib/libandroidfw.so (_ZN7android13ResStringPool6uninitEv+38)
12-10 14:48:43.045  3496  3496 F DEBUG   :     #02 pc 0001662b  /system/lib/libandroidfw.so (_ZN7android10ResXMLTree6uninitEv+12)
12-10 14:48:43.045  3496  3496 F DEBUG   :     #03 pc 00016649  /system/lib/libandroidfw.so (_ZN7android10ResXMLTreeD1Ev+4)
12-10 14:48:43.045  3496  3496 F DEBUG   :     #04 pc 00013373  /system/lib/libandroidfw.so (_ZN7android12AssetManager10getPkgNameEPKc+258)
12-10 14:48:43.045  3496  3496 F DEBUG   :     #05 pc 000133cf  /system/lib/libandroidfw.so (_ZN7android12AssetManager18getBasePackageNameEj+62)
12-10 14:48:43.045  3496  3496 F DEBUG   :     #06 pc 00088b33  /system/lib/libandroid_runtime.so
12-10 14:48:43.045  3496  3496 F DEBUG   :     #07 pc 72cb9011  /data/dalvik-cache/arm/system@framework@boot.oat (offset 0x1f78000)
12-10 14:48:50.095  3496  3496 F DEBUG   :
12-10 14:48:50.095  3496  3496 F DEBUG   : Tombstone written to: /data/tombstones/tombstone_00
12-10 14:48:50.185  1912  1912 I ServiceManager: service 'statusbar' died
12-10 14:48:50.185  1912  1912 I ServiceManager: service 'netstats' died
12-10 14:48:50.185  1912  1912 I ServiceManager: service 'power' died
12-10 14:48:50.185  1912  1912 I ServiceManager: service 'media_projection' died
12-10 14:48:50.185  1912  1912 I ServiceManager: service 'network_management' died
12-10 14:48:50.185  1912  1912 I ServiceManager: service 'window' died
12-10 14:48:50.185  1912  1912 I ServiceManager: service 'consumer_ir' died
12-10 14:48:50.185  1912  1912 I ServiceManager: service 'telecom' died
12-10 14:48:50.185  1912  1912 I ServiceManager: service 'cmpartnerinterface' died
12-10 14:48:50.185  1912  1912 I ServiceManager: service 'package' died
12-10 14:48:50.185  1912  1912 I ServiceManager: service 'user' died

Since Open Camera needs some background service and is started on bootup, I assume that after installation the system tries to restart this service. However, it appears that there is some memory issue with the app, as it requests so much memory that Android starts killing other apps to make this memory available. In case Android does not manage to provide this space, the device is rebooted. Since OpenCamera is started at bootup, it again tries to allocate (too much) memory and the device is stuck in an infinite loop.

Looking at Android’s memory management

I expected that the following excerpt from the log above might lead to some useful hints:

12-10 14:48:42.945  4034  4034 F libc    : invalid address or address of corrupt block 0x120 passed to dlfree
12-10 14:48:42.945  4034  4034 F libc    : Fatal signal 11 (SIGSEGV), code 1, fault addr 0xdeadbaad in tid 4034 (system_server)

After searching on the net, I found an interesting discussion [2] suggesting the following:


“A likely cause of this is that you have ran out of memory, maybe because a memory leak or simply used up all memory. This can be caused by a bug you are using in a plugin that uses native C/C++ code through NDK.”

To rule out hardware issues, I also exchanged the storage (I run /data from sdcard) and compiled memtester [3] to test the device’s RAM. When experimenting with memtester, I noticed a striking difference between running memtester on a regular GNU/Linux system and running it on Android/LineageOS. When giving memtester less memory than actually available, there is no difference. However, when giving memtester *more* RAM than acutally available, the following happens on GNU/Linux:

# free -h
              total        used        free      shared  buff/cache   available
Mem:            28G        124M         28G        8.5M        219M         28G
Swap:            0B          0B          0B
# memtester 40G
memtester version 4.3.0 (64-bit)
Copyright (C) 2001-2012 Charles Cazabon.
Licensed under the GNU General Public License version 2 (only).

pagesize is 4096
pagesizemask is 0xfffffffffffff000
want 40960MB (42949672960 bytes)
got  29075MB (30488387584 bytes), trying mlock ...Killed
#
Killed
[1]+  Stopped                 sh

While on Android the device suddenly reboots after trying to mlock the memory:

root@i9300:/ # free -h
                total        used        free      shared     buffers
Mem:             828M        754M         74M           0        1.3M
-/+ buffers/cache:           752M         75M
Swap:            400M         18M        382M

root@i9300:/ # /sbin/memtester 2G
memtester version 4.3.0 (32-bit)
Copyright (C) 2001-2012 Charles Cazabon.
Licensed under the GNU General Public License version 2 (only).

pagesize is 4096
pagesizemask is 0xfffff000
want 2048MB (2147483648 bytes)
got  2008MB (2105921536 bytes), trying mlock ...

This is what is printed to logcat:

01-01 01:10:29.485  4933  4933 D su      : su invoked.
01-01 01:10:29.485  4933  4933 E su      : SU from: shell
01-01 01:10:29.490  4933  4933 D su      : Allowing shell.
01-01 01:10:29.490  4933  4933 D su      : 2000 /system/bin/sh executing 0 /system/bin/sh using binary /system/bin/sh : sh
01-01 01:10:29.490  4933  4933 D su      : Waiting for pid 4934.
01-01 01:10:44.840  2478  3264 D LightsService: Excessive delay setting light: 81ms
01-01 01:10:44.925  2478  3264 D LightsService: Excessive delay setting light: 82ms
01-01 01:10:45.010  2478  3264 D LightsService: Excessive delay setting light: 82ms
01-01 01:10:45.090  2478  3264 D LightsService: Excessive delay setting light: 82ms
01-01 01:10:45.175  2478  3264 D LightsService: Excessive delay setting light: 82ms
01-01 01:10:45.260  2478  3264 D LightsService: Excessive delay setting light: 82ms
01-01 01:10:45.340  2478  3264 D LightsService: Excessive delay setting light: 82ms
01-01 01:10:50.735  2478  2538 I PowerManagerService: Going to sleep due to screen timeout (uid 1000)...
01-01 01:10:50.785  2478  2538 E         : Device driver API match
01-01 01:10:50.785  2478  2538 E         : Device driver API version: 29
01-01 01:10:50.785  2478  2538 E         : User space API version: 29
01-01 01:10:50.785  2478  2538 E         : mali: REVISION=Linux-r3p2-01rel3 BUILD_DATE=Tue Aug 26 17:05:16 KST 2014
01-01 01:10:52.000  2478  2538 V KeyguardServiceDelegate: onScreenTurnedOff()
01-01 01:10:52.040  2478  2538 E libEGL  : call to OpenGL ES API with no current context (logged once per thread)
01-01 01:10:52.045  2478  2536 I DisplayManagerService: Display device changed: DisplayDeviceInfo{"Integrierter Bildschirm": uniqueId="local:0", 720 x 1280, modeId 1, defaultModeId 1, supportedModes [{id=1, width=720, height=1280, fps=60.002}], colorTransformId 1, defaultColorTransformId 1, supportedColorTransforms [{id=1, colorTransform=0}], density 320, 304.8 x 306.71698 dpi, appVsyncOff 0, presDeadline 17666111, touch INTERNAL, rotation 0, type BUILT_IN, state OFF, FLAG_DEFAULT_DISPLAY, FLAG_ROTATES_WITH_CONTENT, FLAG_SECURE, FLAG_SUPPORTS_PROTECTED_BUFFERS}
01-01 01:10:52.060  1915  1915 D SurfaceFlinger: Set power mode=0, type=0 flinger=0x411dadf0
01-01 01:10:52.160  2478  2538 I PowerManagerService: Sleeping (uid 1000)...
01-01 01:10:52.165  2478  3231 D WifiConfigStore: Retrieve network priorities after PNO.
01-01 01:10:52.170  1938  3241 E bt_a2dp_hw: adev_set_parameters: ERROR: set param called even when stream out is null
01-01 01:10:52.170  2478  3231 E native  : do suspend false
01-01 01:10:52.175  2478  3231 D WifiConfigStore: No blacklist allowed without epno enabled
01-01 01:10:52.190  3846  4968 D NfcService: Discovery configuration equal, not updating.
01-01 01:10:52.435  2478  3231 D WifiConfigStore: Retrieve network priorities before PNO. Max priority: 0
01-01 01:10:52.435  1938  1938 E bt_a2dp_hw: adev_set_parameters: ERROR: set param called even when stream out is null
01-01 01:10:52.440  2478  3231 E WifiStateMachine:  Fail to set up pno, want true now false
01-01 01:10:52.440  2478  3231 E native  : do suspend true
01-01 01:10:52.670  2478  3231 D WifiStateMachine: Disconnected CMD_START_SCAN source -2 3, 4 -> obsolete
01-01 01:10:54.160  2478  2538 W PowerManagerService: Sandman unresponsive, releasing suspend blocker
01-01 01:10:55.825  2478  3362 D CryptdConnector: SND -> {3 cryptfs getpw}
01-01 01:10:55.825  1903  1999 D VoldCryptCmdListener: cryptfs getpw
01-01 01:10:55.825  1903  1999 I Ext4Crypt: ext4 crypto complete called on /data
01-01 01:10:55.825  1903  1999 I Ext4Crypt: No master key, so not ext4enc
01-01 01:10:55.830  1903  1999 I Ext4Crypt: ext4 crypto complete called on /data
01-01 01:10:55.830  1903  1999 I Ext4Crypt: No master key, so not ext4enc
01-01 01:10:55.830  2478  2798 D CryptdConnector: RCV  {4 cryptfs clearpw}
01-01 01:10:55.835  1903  1999 D VoldCryptCmdListener: cryptfs clearpw
01-01 01:10:55.835  1903  1999 I Ext4Crypt: ext4 crypto complete called on /data
01-01 01:10:55.835  1903  1999 I Ext4Crypt: No master key, so not ext4enc
01-01 01:10:55.835  2478  2798 D CryptdConnector: RCV <- {200 4 0}
01-01 01:10:55.925  3417  3417 D PhoneStatusBar: disable:
01-01 01:10:56.020  3417  3417 D PhoneStatusBar: disable:
01-01 01:10:56.330  3417  3417 D PhoneStatusBar: disable:
01-01 01:11:44.875  2478  4667 I ActivityManager: Process com.android.messaging (pid 4607) has died
01-01 01:11:44.920  2478  4667 D ActivityManager: cleanUpApplicationRecord -- 4607
01-01 01:11:45.860  2478  3356 W art     : Long monitor contention event with owner method=void com.android.server.am.ActivityManagerService$AppDeathRecipient.binderDied() from ActivityManagerService.java:1359 waiters=0 for 907ms
01-01 01:11:45.890  2478  3356 I ActivityManager: Process org.cyanogenmod.profiles (pid 4593) has died
01-01 01:11:45.900  2478  3356 D ActivityManager: cleanUpApplicationRecord -- 4593
01-01 01:11:45.955  2478  2529 W art     : Long monitor contention event with owner method=void com.android.server.am.ActivityManagerService$AppDeathRecipient.binderDied() from ActivityManagerService.java:1359 waiters=1 for 914ms
01-01 01:11:45.960  1913  1913 E lowmemorykiller: Error opening /proc/3662/oom_score_adj; errno=2
01-01 01:11:45.970  2478  2529 I ActivityManager: Process com.android.exchange (pid 3662) has died
01-01 01:11:45.970  2478  2529 D ActivityManager: cleanUpApplicationRecord -- 3662
01-01 01:11:45.985  2478  3943 W art     : Long monitor contention event with owner method=void com.android.server.am.ActivityManagerService$AppDeathRecipient.binderDied() from ActivityManagerService.java:1359 waiters=2 for 611ms
01-01 01:11:45.995  2478  3943 I ActivityManager: Process com.android.calendar (pid 4415) has died
01-01 01:11:45.995  2478  3943 D ActivityManager: cleanUpApplicationRecord -- 4415
01-01 01:11:46.000  2478  2532 W art     : Long monitor contention event with owner method=void com.android.server.am.ActivityManagerService$AppDeathRecipient.binderDied() from ActivityManagerService.java:1359 waiters=3 for 537ms
01-01 01:11:46.025  2478  3362 W art     : Long monitor contention event with owner method=void com.android.server.am.ActivityManagerService$AppDeathRecipient.binderDied() from ActivityManagerService.java:1359 waiters=4 for 378ms
01-01 01:11:46.045  2478  3362 I ActivityManager: Process org.lineageos.updater (pid 4449) has died
01-01 01:11:46.045  2478  3362 D ActivityManager: cleanUpApplicationRecord -- 4449
01-01 01:11:46.045  1913  1913 E lowmemorykiller: Error writing /proc/3938/oom_score_adj; errno=22
01-01 01:11:46.050  2478  3413 W art     : Long monitor contention event with owner method=void com.android.server.am.ActivityManagerService$AppDeathRecipient.binderDied() from ActivityManagerService.java:1359 waiters=5 for 372ms
01-01 01:11:46.505  2478  3232 D WifiService: Client connection lost with reason: 4
01-01 01:11:47.165  2478  4666 D GraphicsStats: Buffer count: 3
01-01 01:11:47.400  2478  2532 W art     : Long monitor contention event with owner method=int com.android.server.am.ActivityManagerService.broadcastIntent(android.app.IApplicationThread, android.content.Intent, java.lang.String, android.content.IIntentReceiver, int, java.lang.String, android.os.Bundle, java.lang.String[], int, android.os.Bundle, boolean, boolean, int) from ActivityManagerService.java:17497 waiters=0 for 667ms
01-01 01:11:47.465  2478  4664 W art     : Long monitor contention event with owner method=int com.android.server.am.ActivityManagerService.broadcastIntent(android.app.IApplicationThread, android.content.Intent, java.lang.String, android.content.IIntentReceiver, int, java.lang.String, android.os.Bundle, java.lang.String[], int, android.os.Bundle, boolean, boolean, int) from ActivityManagerService.java:17497 waiters=1 for 858ms
01-01 01:11:47.465  2478  3412 W art     : Long monitor contention event with owner method=int com.android.server.am.ActivityManagerService.broadcastIntent(android.app.IApplicationThread, android.content.Intent, java.lang.String, android.content.IIntentReceiver, int, java.lang.String, android.os.Bundle, java.lang.String[], int, android.os.Bundle, boolean, boolean, int) from ActivityManagerService.java:17497 waiters=2 for 859ms
01-01 01:11:47.475  2478  4665 I ActivityManager: Process com.android.providers.calendar (pid 4434) has died
01-01 01:11:47.480  2478  4665 D ActivityManager: cleanUpApplicationRecord -- 4434
01-01 01:11:47.545  1913  1913 E lowmemorykiller: Error opening /proc/3938/oom_score_adj; errno=2
01-01 01:11:47.545  1913  1913 E lowmemorykiller: Error opening /proc/4014/oom_score_adj; errno=2
01-01 01:11:47.550  1913  1913 E lowmemorykiller: Error opening /proc/4542/oom_score_adj; errno=2
01-01 01:11:47.550  2478  3943 W art     : Long monitor contention event with owner method=int com.android.server.am.ActivityManagerService.broadcastIntent(android.app.IApplicationThread, android.content.Intent, java.lang.String, android.content.IIntentReceiver, int, java.lang.String, android.os.Bundle, java.lang.String[], int, android.os.Bundle, boolean, boolean, int) from ActivityManagerService.java:17497 waiters=3 for 894ms
01-01 01:11:47.560  2478  3943 I ActivityManager: Process org.cyanogenmod.themes.provider (pid 3497) has died
01-01 01:11:47.560  2478  3943 D ActivityManager: cleanUpApplicationRecord -- 3497
01-01 01:11:47.560  2478  2529 W art     : Long monitor contention event with owner method=int com.android.server.am.ActivityManagerService.broadcastIntent(android.app.IApplicationThread, android.content.Intent, java.lang.String, android.content.IIntentReceiver, int, java.lang.String, android.os.Bundle, java.lang.String[], int, android.os.Bundle, boolean, boolean, int) from ActivityManagerService.java:17497 waiters=4 for 673ms
01-01 01:11:47.570  2478  2529 I ActivityManager: Process com.svox.pico (pid 4014) has died
01-01 01:11:47.570  2478  2529 D ActivityManager: cleanUpApplicationRecord -- 4014
01-01 01:11:48.325  2478  2529 W ActivityManager: Scheduling restart of crashed service com.svox.pico/.PicoService in 1000ms

Verdict

I wasted lots of time with this issue, but was finally able to reproduce it and to recover all of my data. At least, I have an explanation now for various random reboots I experienced in the past in similar low-memory conditions.

Overall, I am really shocked that a simple, unprivileged Android app that is scheduled to start on bootup can ruin a working system so badly. Further research indicates that there are more apps known to cause such behavior [4]. I hope that a device based on a GNU/Linux system instead of Android (such as the announced Librem5) will not suffer from such a severe flaw.

References

[1] https://review.lineageos.org/#/c/197305/
[2] https://stackoverflow.com/questions/25069186/invalid-address-passed-to-dlfree
[3] https://github.com/royzhao/memtester4Android
[4] https://gitlab.com/fdroid/fdroiddata/issues/979
[5] https://gitlab.com/fdroid/fdroiddata/issues/979#note_48990149

KVM-virtualization on ARM using the “virt” machine type

Introduction

A while ago, I described how to run KVM-based virtual machines on libre, low-end virtualization hosts on Debian Jessie [1]. For emulating the ARM board, I used the vexpress-a15 which complicates things as it requires the specification of compatible DTBs. Recently, I stepped over Peter’s article [2] that describes how to use the generic “virt” machine type instead of vexpress-a15. This promises to give some advantages such as the ability to use PCI devices and makes the process of creating VMs much easier.

As it was also reported to me that my instructions caused trouble on Debian Stretch (virt-manager generates incompatible configs when choosing the vexpress-a15 target). So I spent some time trying to find out how to run VMs using the “virt” machine type using virt-manager (Peter’s article only described the manual way using command-line calls). This included several traps, so I decided to write up this article. It gives a brief overview how to create a VM using virt-manager on a ARMv7 virtualization host such as the Cubietruck or the upcoming EOMA68-A20 computing card.

Disclaimer

All data and information provided in this article is for informational purposes only. The author makes no representations as to accuracy, completeness, currentness, suitability, or validity of any information on this article and will not be liable for any errors, omissions, or delays in this information or any losses, injuries, or damages arising from its display or use. All information is provided on an as-is basis.

In no event the author we be liable for any loss or damage including without limitation, indirect or consequential loss or damage, or any loss or damage whatsoever arising from loss of data or profits arising out of, or in connection with, the use of this article.

Tested VMs

I managed to successfully create and boot up the following VMs on a Devuan (Jessie) system:

  • Debian Jessie Installer
  • Debian Stretch Installer
  • Debian Unstable Installer
  • Fedora Core 27 (see [3] for instructions how to obtain the necessary files)
  • Arch Linux, using the latest ARMv7 files available at [4]
  • LEDE 17.0.1.4

I was able to reproduce the steps for the Debian guests on a Debian Stretch system as well (I did not try with the other guests).

Requirements / Base installation

This article assumes you have setup a working KVM virtualization host on ARM. If you don’t, please work through my previous article [1].

Getting the necessary files

Depending on the system you want to run in your Guest, you typically need an image of the kernel and the initrd. For the Debian-unstable installer, you could get the files like this:

wget http://http.us.debian.org/debian/dists/unstable/main/installer-armhf/current/images/netboot/vmlinuz -O vmlinuz-debian-unstable-installer

and

wget http://http.us.debian.org/debian/dists/unstable/main/installer-armhf/current/images/netboot/vmlinuz -O initrd-debian-unstable-installer.gz

Creating the Guest

Now, fire up virt-manager and start the wizard for creating a new Guest. In the first step, select “Import existing disk image” and the default settings, which should use the Machine Type “virt” already:

In the second step, choose a disk image (or create one) and put the paths to the Kernel and Initrd that you downloaded previously. Leave the DTB path blank and put “console=ttyAMA0″ as kernel arguments. Choose an appropriate OS type of just leave the default (may negatively impact the performance of your guest, or other things may happen such as your virtual network card may not be recognized by the installer):

Next, select memory and CPU settings as required by your Guest:

Finally, give the VM a proper name and select the “Customize configuration before install” option:

In the machine details, make sure the CPU model is “host-passthrough” (enter it manually if you can’t select it in the combo box):

In the boot options tab, make sure the parameter “console=ttyAMA0″ is there (otherwise you will not get any output on the console). Depending on your guest, you might also need more parameters, such as for setting the rootfs:

Finally, click “begin installation” and you should see your VM boot up:

Post-Installation steps

Please note, that after installing your guest, you must extract the kernel and initrd from the installed guest image (you want to boot the real system, not the installer) and change your VM configuration to use these files instead.

Eventually, I will provide instructions how to do this for a few guest types in the future. Meanwhile, you can obtain instructions for extracting the files from a Debian guest from Peter’s article [2].

[1] https://blogs.fsfe.org/kuleszdl/2016/11/06/installing-a-libre-low-power-low-cost-kvm-virtualization-host/
[2] https://translatedcode.wordpress.com/2016/11/03/installing-debian-on-qemus-32-bit-arm-virt-board
[3] https://fedoraproject.org/wiki/QA:Testcase_Virt_ARM_on_x86
[4] http://os.archlinuxarm.org/os/ArchLinuxARM-armv7-latest.tar.gz

Benchmarking microSD cards and more

It’s worth re-visiting this blog-post as I am going to do more benchmarks and add more results over time.

Motivation

If you ever tried using a flash drive (such a SD, microSD, USB drive or CF) for your root or home filesystem on a small computing device or smartphone, you probably have noticed that flash cards are in most cases a lot slower than integreted eMMC flash. Since most filesystems use 4k blocks, the random write/read performance using 4k blocks is what matters most in such scenarios. And while flash cards don’t come close to internal flash in these disciplines, there are significant differences between the models.

Jeff Geerling [1,2] has already benchmarked the performance of various microSD cards on different models of the “blobby” Raspberry Pi. I had a number of different microSD cards at hand and I tried to replicate his results on my sample. In addition, I extended the comparison to regular SD cards and USB drives.

Disclaimer


All data and information provided in this article is for informational purposes only. The author makes no representations as to accuracy, completeness, currentness, suitability, or validity of any information on this article and will not be liable for any errors, omissions, or delays in this information or any losses, injuries, or damages arising from its display or use. All information is provided on an as-is basis.

In no event the author will be liable for any loss or damage including without limitation, indirect or consequential loss or damage, or any loss or damage whatsoever arising from loss of data or profits arising out of, or in connection with, the use of this article.

Environment and tools

I encourage every reader to replicate my benchmarks and share their results. Therefore, I first want to describe how the measures were taken.

General equipment

For the benchmarks, I used the following hardware on my workstation (listing only relevant parts):

  • Renesas Technology Corp. uPD720201 USB 3.0 Host Controller (rev 03)
  • Logilink CR0034A USB 3.0 card reader [5], used for (m)SD cards
  • Apacer ? USB 2.0 card reader, used for CF cards
  • Debian 9 “Stretch”

In the past, I also did iozone benchmarks using a Logilink CR0015 USB 2.0 (m)SD card reader [4] which are provided at the end of the article for reference.

iozone Benchmarks

For the first tests, Just like Jeff, I used the open source (but non-free) tool “iozone” [3] in the current stable version that is available on Debian Stretch (3.429).

I disabled caches and benchmarked on raw devices to avoid measuring filesystem overhead. Therefore, I used the following call to iozone to run the benchmarks (/dev/sde is my sdcard):

$ iozone -e -I -a -s 100M -r 4k -r 16M -i 0 -i 1 -i 2 -f /dev/sdX

(replacing sdX by the actual device name)

ddrescue benchmarks

I used GNU ddrescue (1.21) in synchronous mode to overwrite a device with zeroes and measured the performance there. For this purpose, I used the following call to do a write benchmark:

$ ddrescue -D --force /dev/zero /dev/sdX

(replacing sdX by the actual device name)

Similarly, to measure the read performance I used (I had more than 100GB RAM left for tmpfs):

$ ddrescue -D --force /dev/sdX /tmp/foo

(replacing sdX by the actual device name)

Results

iozone

Here are the iozone results:

Manufacturer Make / model Type Speed Class Capacity 16M seq. read (MB/s) 16M seq. write (MB/s) 4K rand. read (MB/s) 4K rand. write (MB/s)
Samsung Pro+ (2017) mSDXC U3 64 GB 42.33 39.75 8.83 2.39
Samsung Evo+ (2017) mSDXC U3 64 GB 42.47 40.10 8.48 2.29
Samsung Evo+ mSDHC U1 32 GB 42,53 34,48 8.01 3.97
Sandisk Ultra mSDXC C10 64 GB 18.21 19.19 5.35 1.06
Sandisk Cruzer USB 2.0 4 GB 22.89 4.05 6.86 0.44
Kingston Data Traveller G3 USB 2.0 8 GB 21.52 4.96 6.96 0.08
Super Talent Pico C USB 2.0 8 GB 36.72 13.77 DNF DNF
Sandisk Ultra USB 3.0 32 GB 6.41 2.27 148.46 52.97
Samsung ? USB 2.0 1 GB 21.48 13.47 11.10 0.23
Transcend ? CF 80x 1 GB 20.19 12.56 7.56 0.19
Data Power ? (SLC) CF ? 1 GB 12.91 4.97 6.45 0.003
Toshiba ? CF ? 256 MB 7.52 2.86 3.78 0.003
Samsung Evo SDHC U1 32 GB 9.29 4.01 46.34 17.50
? mSDHC ? ? 8 GB tbd tbd tbd tbd

ddrescue

Here are the results using the USB 3.0 card reader or the device directly (for usb drives):

Manufacturer Make / model Type Rating Capacity Avg read (MB/s) Avg write (MB/s)
Samsung Pro+ (2017) mSDHC U3 64 GB tbd 34.77
Samsung Evo+ mSDHC U1 32 GB 41.95 21.54
Sandisk Cruzer USB 2.0 4 GB 36.37 2.64
Super Talent Pico C USB 2.0 8 GB 36.29 19.80
Sandisk Ultra USB 3.0 32 GB 115.00 24.62
Samsung ? USB 2.0 1 GB 21.63 14.62
Kingston Data Traveller G3 USB 3.0 8 GB 20.64 5.07
Transcend ? CF 80x 1 GB 19.26 12.60
Data Power ? (SLC) CF ? 1 GB 12.68 5.02
Toshiba ? CF ? 256 MB 7.33 2.82
Samsung Evo SDHC U1 32 GB 45,39 19.39
? mSDHC ? ? 8 GB 15.64 5.45

Legacy USB 2.0 iozone results

Previously, I ran the iozone benchmarks using the USB 2.0 card reader. They might be relevant, if you are looking for a storage device which you want to put in a computing device with a slow card reader:

Manufacturer Make / model Type Speed Class Capacity 16M seq. read (MB/s) 16M seq. write (MB/s) 4K rand. read (MB/s) 4K rand. write (MB/s)
Adata ? mSD C4 32 GB 9.58 2.97 2.62 0.64
Samsung Evo mSDHC U1 32 GB 17.80 10.13 4.45 1.32
Samsung Evo+ mSDHC U1 32 GB 18.10 14.27 5.28 2.86
Samsung Evo+ (2017) mSDXC U3 64 GB 16,98 8,03 7,99 2,99
Sandisk Extreme mSDXC U3 64 GB 18.44 11.22 5.20 2.36
Sandisk Ultra mSDHC C10 64 GB 18.16 8.69 3.73 0.80
Toshiba Exceria mSDXC U3 64 GB 16.10 6.60 3.82 0.09
Kingston ? mSD C4 8 GB 14.69 3.71 3.97 0.18

Verdict and future work

I can confirm Jeff’s results about microSD cards and would also recommend the Evo+ (both the old and the 2017 model) which have the best 4K random write performance of the sample. On the other hand, I am very disappointed about the Toshiba Exceria card. Actually running a device on this card with a very sluggish performance was the reason why I took this benchmark initiative. And indeed, after switching to the Evo+, the device feels much snappier now.

Of course, it would be worthwhile to add more cards and drives to this benchmark. Also, using fio instead of the non-free iozone might be interesting. Furthermore, doing the benchmarks internally on the device or using a other USB 3.0 card readers might be also interesting.

References

[1] http://www.pidramble.com/wiki/benchmarks/microsd-cards
[2] http://www.jeffgeerling.com/blogs/jeff-geerling/raspberry-pi-microsd-card
[3] http://www.iozone.org/
[4] http://www.logilink.eu/Products_LogiLink/Notebook-Computer_Accessories/Card_Reader/Cardreader_USB_20_Stick_external_for_SD-MMC_CR0015.htm
[5] http://www.logilink.org/Produkte_LogiLink/Notebook-Computerzubehoer/Kartenleser/Cardreader_USB_30_SD-SD-HC-Micro_SD-Micro_SD-HC_CR0034A.htm?seticlanguage=en

KVM virtualization with Allwinner A20 on Debian: libre, low-power, low-cost

Introduction

Various cheap ARM boards based on the Allwinner A20 SoC are available already for a few years. The first EOMA68 computer [1] will be also based on this chipset. Not many users know that the Allwinner A20 supports hardware-supported virtualization as well. Its Cortex A7 cores allow running hardware-accelerated ARM virtual machines (guests) using KVM or Xen.

While Allwinner has been blamed to violate the GPL for years [2], their A20 SoC is imho one of the best choices today when it comes to building a small and libre server for SOHO use (thanks to the hard work of the Allwiner-independent Linux-Sunxi community). While many SoCs found on popular boards like those from the Raspberry Pi family require proprietary blobs, the A20 works with a free bootloader and requires no proprietary drivers or firmware for basic operation.

The virtualization on A20 hosts works out of the box on Debian Jessie with the stock kernel and official packages in main — without cross-compiling, patching or other tinkering (this was not the case in the past, see [3]). This also means that updating your host and guests later will be easy and painless. Creating and managing guests can be done with virt-manager [4] – a secure and comfortable graphical user interface licensed under GPLv3.

After first discussing some A20 hardware options, this guide takes the example of the Olimex “A20-OLinuXIno-LIME2″ board [5] and shows how to turn it into a virtualization host. Then shows how create and manage guest-VMs on the virtualization host. The guide assumes that you are running a a GNU/Linux-based desktop system from which you want to manage the A20 device.

Disclaimer


All data and information provided in this article is for informational purposes only. The author makes no representations as to accuracy, completeness, currentness, suitability, or validity of any information on this article and will not be liable for any errors, omissions, or delays in this information or any losses, injuries, or damages arising from its display or use. All information is provided on an as-is basis.

In no event the author we be liable for any loss or damage including without limitation, indirect or consequential loss or damage, or any loss or damage whatsoever arising from loss of data or profits arising out of, or in connection with, the use of this article.

Hardware choices

There are plenty of boards with the Allwinner A20. However, only few are known to work out of the box on Debian Jessie. The particular page on the Debian Wiki [6] mentions the following boards in particular:

  • Cubietech Cubieboard2
  • Cubietech Cubieboard3 (“Cubietruck”)
  • LeMaker Banana Pro
  • Olimex A20-OLinuXino-LIME
  • Olimex A20-OLinuXino-LIME2 (only the regular one, not the eMMC variant!)
  • Olimex A20-Olinuxino Micro

While some of these boards feature Gigabit ethernet and SATA, only the Cubieboard 3 has 2 GB of RAM. To me, this seems to be the best choice for a A20-based KVM virtualization host. Since I only had a spare Olimex A20-OLinuXino-LIME2 board at hand, this guide uses this board as example.

Beware: The “A20-OLinuXino-LIME2″ and the “A20-OLinuXino-LIME2-eMMC” are not the same! Debian provides no firmware for the “A20-OLinuXino-LIME2-eMMC” and I could not get it to work at all on Debian. Although I thought that they would be the same except for the eMMC flashg, the firmware for the regular “”A20-OLinuXino-LIME2″ did NOT work for me at all!

Base installation

The article in the Debian wiki provides the necessary information on installing Debian Jessie using the text-based Debian-Installer. Make sure you have a microSD card with a good 4K random I/O performance or the installation will take forever and your A20 system will run terribly slow afterwards (see my article comparing performance of various microSD cards).

If you don’t have a serial cable and want to install using the HDMI output, you need to use the installer images from unstable. The easiest way to do is to fetch the firmware file from unstable and the partition image from Jessie. Then write them to your microSD card (replace /dev/sdX with your particular device):

$ zcat firmware.A20-OLinuXino-Lime2.img.gz partition.img.gz > /dev/sdX

Next, insert the microSD card into your device, connect your device to your LAN and power it up. Then install Debian as usual using the text-based installer. During the installation, sure to create a root account (needed for KVM) and a ext2 boot partition (the safest method here is to use the guided installer). When tasksel gets called, make sure to install the tasks/packages “SSH Server” and “Standard system utilities”.

Note for users of the German mirrors: Using the mirror “ftp.de.debian.org” will break your installation as something seems to be missing there as of 2016-11-05. Using “ftp2.de.debian.org” works fine.

Installing the KVM virtualization

By default, interactive root logins are not allowed on Debian. Therefore, make sure you copy over your SSH public key to your a20-box or simply enable interactive root logins over SSH by changing the following option in /etc/ssh/sshd_config:

#PermitRootLogin without-password
PermitRootLogin yes

Then restart the SSH server:

# service ssh restart

Now you should be able to log in directly as root. Next, install the virtualization packages:

# apt install libvirt-daemon-system
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following extra packages will be installed:

...

0 upgraded, 105 newly installed, 0 to remove and 0 not upgraded.
Need to get 44.4 MB of archives.
After this operation, 182 MB of additional disk space will be used.
Do you want to continue? [Y/n]

Now fire up virt-manager on your desktop and make sure you can connect to your a20-box:

Creating and installing a guest

For running ARM virtual machines you need a kernel and DTBs which support the VExpress-A15 chipset (the ARM reference board usually emulated on ARM). This is already provided in stock Debian, so there is no need to compile anything yourself.

Regarding the guest, you can choose any Linux you want. In the following example, we will install a Debian Jessie guest using the Debian installer. Therefore we need to download the to the Virtualization host. This time, we don’t need a partition image but can use the usual the initrd installer-Image from the Debian server. SSH into the virtualization host and download it:

wget http://ftp.uk.debian.org/debian/dists/jessie/main/installer-armhf/current/images/netboot/initrd.gz -O initrd-installer-jessie.gz

For the installation, you will also need a different kernel because in the Kernel installed on the host the network drivers are in initrd, but the Installer’s initrd assumes they are in the kernel. Therefore, fetch a kernel for the installer:

wget http://ftp.uk.debian.org/debian/dists/jessie/main/installer-armhf/current/images/netboot/vmlinuz -O vmlinuz-installer-jessie

Now, fire up virt-manager on your desktop and connect to the Virtualization host. Then, start the wizard for creating guests using “create new virtual machine”. On the first screen, change the machine type to “vexpress-a15″:

On the next screen, specify a storage (just create one using the dialog following “Browse”), and also use “Browse” to locate the kernel and initrd images so you specify the ones we just downloaded. For the DTB, we’ll use the one that is part of Debian’s stock kernel and resides under /usr/lib/linux-image-3.16.0-4-armmp-armmp-lpae/vexpress-v2p-ca15-tc1.dtb (make sure it corresponds to the version on your a20-host! TODO: Is there any symlink which points to the current version?)). The kernel args are also very important, or you will not get any output. For this line, specify the following:

root=/dev/vda1 console=ttyAMA0,115200 rootwait

Finally, select OS type and version appropiately. Your dialog should look like this:

Then, specify RAM (e.g. 256MB) and the number of CPUs (e.g. 1) you want to give the guest and jump to the last screen. Here, give your guest a nice name and make sure you check the “Customize configuration before install” checkbox before you click “Finish”:

Otherwise, you would end up with an error message like this:

Unable to complete install: 'internal error: early end of file from monitor: possible problem:
kvm_init_vcpu failed: Invalid argument

In the configuration of the VM, under “Processor”, change the configuration from “Hypervisor Default” to “Application Default”:

To get better performance, also change the BUS of your virtual disk to “VIRTIO” (by default, it would emulate an SD card):

And do the same for the network adapter:

Finally, fire up the guest using “Begin installation”. If everything goes fine, you should see the kernel boot and be presented with the welcome screen of the installer. For jessie, it should look like this:

If you selected the kernel and initrd from stretch/sid you should get a nicer color screen (make sure you set the baudrate of the console to 115200 or you will get a disorted output!):

When partitioning the guest, just create a single root partition spanning the whole (virtual) device. The guest will always boot using externally specified kernels, dtbs and initrds, therefore there is no use in creating a /boot partition as the “guided install” would do.

Near the end of the installation, you will be notified that no bootloader could be installed. You can safely ignore this message:

After finishing the installation, the system will boot again into the installer because the initrd is still active. To change this, power off the guest (“Force Off”) and specify in the boot options to use the kernel and initrd image of your A20 host instead (whenever they will be updated on the host, the guests will also get the update on their next boot):

Now your guest should finally succeed to boot up:

And you can check that it indeed uses the current A20 kernel on the host and virtualizes the VExpress15 SoC:

Benchmarks

Finally, I want to provide some benchmarks so you can get a feeling about the impact of the virtualization. The benchmarks were done using a guest with 2 CPUs and 512MB memory assigned.

IO/Performance

For a first I/O benchmark, I used hdparm.

On the host:

$ hdparm -tT /dev/mmcblk0
/dev/mmcblk0:
 Timing cached reads:   814 MB in  2.00 seconds = 406.33 MB/sec
 Timing buffered disk reads:  66 MB in  3.01 seconds =  21.93 MB/sec

On the guest:

$ hdparm -tT /dev/vda
/dev/vda:
 Timing cached reads:   694 MB in  2.00 seconds = 346.49 MB/sec
 Timing buffered disk reads:  30 MB in  3.15 seconds =   9.52 MB/sec

CPU processing

For benchmarking processing, I used the openssl suite to do a few simple AES benchmarks:

$ openssl speed aes

On the host:

...
The 'numbers' are in 1000s of bytes per second processed.
type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes
aes-128 cbc      20267.83k    22390.70k    23325.10k    23575.89k    23642.11k
aes-192 cbc      17594.13k    19464.20k    19956.57k    20102.83k    20146.86k
aes-256 cbc      15727.25k    17158.89k    17592.58k    17706.67k    17738.41k

On the guest:

...
The 'numbers' are in 1000s of bytes per second processed.
type             16 bytes     64 bytes    256 bytes   1024 bytes   8192 bytes
aes-128 cbc      19784.01k    22100.48k    22697.56k    23272.20k    23288.29k
aes-192 cbc      17363.72k    19097.02k    19643.68k    19786.41k    19800.53k
aes-256 cbc      15455.28k    16939.28k    17374.44k    17415.85k    17504.58k

Conclusion

With one of the Allwinner A20 boards supported by Debian, you can easily build a tiny virtualization host that can handle a few simple VMs and draws only 2-3W of power. While this process was pretty cumbersome in the past (you had to cross-compile kernels etc.), thanks to the efforts of the Debian project and Linux-Sunxi community, it is now pretty straight-forward with only few caveats involved. This might also be an interesting option if you want to run a low-power virtualization cluster on fully libre software down to the firmware level.

References

[1] https://www.crowdsupply.com/eoma68/micro-desktop
[2] http://linux-sunxi.org/GPL_Violations
[3] http://blog.flexvdi.com/2014/07/28/enabling-kvm-virtualization-on-arm-allwinner-a20/
[4] https://virt-manager.org/
[5] https://www.olimex.com/Products/OLinuXino/A20/A20-OLinuXIno-LIME2/
[6] https://wiki.debian.org/InstallingDebianOn/Allwinner

Backing up and restoring data on Android devices directly via USB (Howto)

Motivation

I was looking for a simple way to backup data on Android devices directly to a device running GNU/Linux connected over a USB cable (in my case, a desktop computer).

Is this really so unique that it’s worth writing a new article about it? Well, in my case, I did not want to buffer the data on any “intermediate” devices such as storage cards connected via microSD or USB-OTG. Also, I did not want to use any proprietary tools or formats. Instead, I wanted to store my backups in “oldschool” formats such as dd-images or tar archives. I did not find a comprehensive howto for that, so I decided to write this article.

Disclaimer


All data and information provided in this article is for informational purposes only. The author makes no representations as to accuracy, completeness, currentness, suitability, or validity of any information on this article and will not be liable for any errors, omissions, or delays in this information or any losses, injuries, or damages arising from its display or use. All information is provided on an as-is basis.

In no event the author we be liable for any loss or damage including without limitation, indirect or consequential loss or damage, or any loss or damage whatsoever arising from loss of data or profits arising out of, or in connection with, the use of this article.

Overview

This article describes two different approaches. Both have their pros and cons:

  • Block-level: Doing 1:1 block-level backups (above the file system) is an imaging approach that corresponds to doing dd-style backups.
  • Filesystem-level: Doing filesystem-level backups (on the file system) corresponds to tar-style backups.

An important factor when doing backups is also performance. Filesystem-level backups are usually faster on block devices which are only filled up to a small degree. However, due to the file system overhead they have a lower “raw” throughput rate — especially when backing up data on flash media such as microSD cards. Here, typical filesystems such as ext4 or f2fs operating with a 4K block size are a major bottleneck as these media often have horrible 4k write/read performance.

The following instructions for applying these approaches assume that you already have a “liberated” Android device which can boot into TWRP (a free Android recovery) or CWM. I am using the example of a Nexus-S running Replicant 4.2.0004 and TWRP 2.8.7.0, but the approaches also work with most other Android distributions and recoveries.

Getting familiar with the block devices on your Android device

First of all, you should know which block device you actually want to backup. The internal flash on Android devices is usually partitioned in about 15-25 partitions, depending on the device. To get a first overview, you can try the following (I am using adb shell on the desktop):

$ adb shell cat /proc/partitions

major minor #blocks name

31 0 2048 mtdblock0
31 1 1280 mtdblock1
31 2 8192 mtdblock2
31 3 8192 mtdblock3
31 4 480768 mtdblock4
31 5 13824 mtdblock5
31 6 6912 mtdblock6
179 0 15552512 mmcblk0
179 1 524288 mmcblk0p1
179 2 1048576 mmcblk0p2
179 3 13978607 mmcblk0p3
179 16 1024 mmcblk0boot1
179 8 1024 mmcblk0boot0

To find out what the partitions are about you can inspect the directory /dev/block/platform//by-Name/ which contains symlinks to the actual partitions. In my case, the Nexus-S has two flash chips and I am listing the partitions of the one where the userdata-partition resides on:

$ adb shell ls -l /dev/block/platform/s3c-sdhci.0/by-name

lrwxrwxrwx root root 2016-11-02 19:51 media -> /dev/block/mmcblk0p3
lrwxrwxrwx root root 2016-11-02 19:51 system -> /dev/block/mmcblk0p1
lrwxrwxrwx root root 2016-11-02 19:51 userdata -> /dev/block/mmcblk0p2

Please note that unlike the Nexus-S, most newer Android devices only have a single eMMC flash chip and don't use MTD devices anymore.

Block-level approach

Block-level backups take up a lot of space (without compression) and extracting single files is cumbersome (especially when talking about encrypted data partitions or backups of the whole flash). On the other hand, "just" restoring a full backup is easy.

Backing up a single partition

Now that you know which block devices you want to backup, you can directly create a 1:1 image via adb pull as you would normally do by using dd. In our case:

$ adb pull /dev/block/platform/s3c-sdhci.0/by-name/userdata

7942 KB/s (1073741824 bytes in 132.027s)

On your workstation, you will obtain a file named userdata which contains the whole partition/filesystem as an image. If you didn't enable encryption on your Android device, you can directly mount this file as loopback device and access its contents:

$ mount userdata /mnt

Restoring a single partition


BE CAREFUL! RESTORING THE WRONG IMAGE OR WRITING TO THE WRONG BLOCK DEVICE CAN RUIN YOUR DATA OR EVEN BRICK YOUR ANDROID DEVICE!

To restore your backup, you can simply use adb push. In my case:

$ adb push userdata /dev/block/platform/s3c-sdhci.0/by-name/userdata

failed to copy 'userdata' to '/dev/block/platform/s3c-sdhci.0/by-name/userdata': No space left on device

Alternative: Operating on the whole block device

Instead of backing up just a single partition, it is also possible to backup the whole flash device including all partitions. Example:

$ adb pull /dev/block/mmcblk0

Remarks:

  • On some devices, not all partitions are readable and, thus, cannot be backed up.
  • Please be careful with restores!
  • Accessing files inside this image is not straight-forward (but doable).

Filesystem-level approach

Filesystem-level backups only work for single partitions, take up as much space as the files on the particular filesystem you backup and it is easy to access individual files in them. I am using a combination of adb, netcat and tar to create and restore these backups.

Backing up your data

First, connect to your device via an adb shell:

$ adb shell

Then, change to the directory from where you want to create your backup. If your device was not automatically mounted, you have to do it first:

# mount /dev/block/platform/s3c-sdhci.0/by-name /data

Now change to this directory:

# cd /data

Now, start the netcat process:

# tar -cvp . | busybox nc -lp 5555

On the receiver side (desktop), set up adb port forwarding:

$ adb forward tcp:4444 tcp:5555

Then, start the process to receive the tar file:

$ nc -w 10 localhost 4444 > userdata.tar

You should see your files being packed up on the Android side:

./
./lost+found/
./dontpanic/
./misc/
./misc/adb/
./misc/audit/
./misc/audit/audit.log
...

Now wait for the process to exit.

Restoring your data

Again, on your receiver side (Android device), mount /data if it was not mounted yet and change in there:

# mount /dev/block/platform/s3c-sdhci.0/by-name /data
# cd /data

Now, start the tar extraction process:

# busybox nc -lp 5555 | tar -xpvf -

On the sender side (desktop), again, set up adb port forwarding:

$ adb forward tcp:4444 tcp:5555

And send the tar file:

$ cat userdata.tar | nc -q 2 localhost 4444

Now you should be able to see your previously backed up files getting restored...

References

For the filesystem-level part of the article I used and adapted the following sources:

  • http://www.screenage.de/blog/2007/12/30/using-netcat-and-tar-for-network-file-transfer/
  • http://stackoverflow.com/questions/15278587/pipe-into-adb-shell