HOME | GIT Overview | Script-Archive: (docs) : (wiki) : (git) | Android: (articles) : (files) | ...
Also see the Android directory of the script-archive.
Besides some scripts, this directory contains a mothballed snapshot of my working Android environment intended to complement the articles below (android-setup-example-files.tgz).
In it you find the scripts for the Android shell, a collection of static armel binaries, the tree structure of the chroot on the Galaxy note, as well as scripts and configuration files used for monitoring, backup, etc from a remote Linux server via cron.
Last changes:
20120222 abin/CHECK + CHROOT.mount, system/xbin/CHROOT.umount
Current versions of the more interesting scripts will be placed in above directory (if Android-specific) or elsewhere in the script-archive (parts of my bash/ksh setup or generic Unix stuff).
(or at least as much as we can get away with, given Google's Linux kernel patches and dumbed-down kernel configurations)
For alternative native setups like OPTWARE of NSLU2 fame, see article 1b Setup Alternatives. OPTWARE is a good companion / alternative to using terminal ide for an improved native shell environment. The page also lists alternate package feeds, candidates for chroot, and finally some (incomplete) hacks to dualboot or run X11 natively.
State described: Android 2.3.6 (gingerbread)
Last change: 20120309
License: CC-BY-SA.
Most of the article should also hold for other recent gingerbread devices, the generics also for all newer Android devices. When I've installed ICS on the Note (middle 2012 I'd assume), I will add ICS errata at the end of each part.
This article sketches a sane setup of GNU/Linux on an ordinary 1-GB-RAM DUAL-CORE PC (henceforth called the armel architecture Android device Samsung Galaxy Note GT-N7000).
This hardware is more than enough for a moderate desktop PC that is not used for heavy gaming, virtualization or video editing. In this article, I'm not interested in audio/video input/output nor phone/s-pen/sensor/camera use.
Web Resources of interest:
I/O options of the Note include
Flash considerations: Choose a quality sdhc microsd flash, with a size of 16 or 32GB. Read speeds for the internal flash are 50+ MB/s and 18MB/s for the external (as measured with hdparm for my Sandisk class 4).
After bad experiences with both Class 10 Transcend and Lexar, I've now settled on a reliable Sandisk class 4 16GB microsd flash. As Sandisk chooses labels according to the minimum write rate, it is nearly as fast as the others. One advantage is that I no longer find the chroot remounted read-only, as this means to shutdown/restart the chroot and running fsck (no loss of data though). When switching I had a look at the external sdcard's FAT filesystem and my titanium backup files: Massive data corruption. Flash problems may well have been worsened by whatever timeout changes Google patched into the kernel.
Power considerations: to save battery,
Known pitfalls to avoid:
Besides getting faster access to upstream security and bug fixes than by waiting for something to trickle down from Google to Hardware Maker to Provider to OTA distribution, rooting enables more options for
Depending on how you root the device, the flash counter may increase (though that 301K resistor field testing jig may help). But in case of mailing in the device for service, you need to reflash a stock firmware rom anyway. Thus for me, the conclusion was: so what.
ROMs are very device specific, so Android ecosystem and community are extremely fragmented: Good community support and custom ROMs are IMHO a pre-requirement when buying Android devices. Some custom roms reuse vendor stock kernels and/or stock roms, some use custom kernel builds (usually based on the Android Open Source Project/AOSP; e.g. cyanogenmod).
Bad makers or lack of community lead to early obsolescence of the device and wasted money (consider the disastrous Motorola Milestone). Depending on the number of non-standard chips, various firmware-blobs might be required to get a fully working kernel and/or "rom". With some makers not even supporting devices at start of sale, the initial release kernel might well be the only Linux kernel ever working on a device that supports all hardware (again the Milestone example: blobs issue plus a signed bootloader). This will also worsen community support and shorten availability of community firmware for this maker's devices (as any rom would have to reuse the old, unmaintained and probably also buggy kernel). Some of this is already visible for the Note as well: the unofficial CM9 builds currently still lack blobs for camera and accelerated display.
(state 201112-201201, Note gingerbread ROMs)
At the end of the research phase, you should have selected a target custom ROM, and maybe one or more intermediate steps if you start out with a stock ROM. In my case the sequence led from stock back in time to a full install of the older KJ1 with CFROOT/CWM, and then forward again to Chris_X 4lph4 ROMs offering KL8 using the AbyssNote 2.6.35 kernel.
Stock to Chainfire KJ1 including the modified factoryfs (see posting 1+2). If your ROM doesn't offer CFROOT/CWM, you first need to install a version of odin on Windows that lists the Note as supported (v3 version 1.83 or 1.85 was the one I found - there doesn't seem to be any official home page or author, nor any sane version numbering; a list if provided e.g. in this overview). For Linux, heimdall might work (untested). To set the Note into download mode, use lower-vol + home + power and connect to USB. For more detail, check the XDA threads for your selected ROM.
Chainfire KJ1 to Chris_X 4lph4 (see XDA). As we now have CFROOT/CWM available, this step no longer requires Windows and Odin. I repeated this step once to go to a newer ROM from Chris, currently the v1.0.5 from early January, which is very stable (excluding Google's carefully prepared pitfalls concerning shared memory and system_server vs app2sd - given that Chris is using the AbyssNote kernel and source for older AbyssNote kernels can be found, we may be able to undo some of that damage and recompile the kernel).
As of mid February, Chris is at v1.7.
I will follow Chris once in a while until ICS is available as a CFROOT/CWM supported ROM. In the long run, I'll expect to end up on CM9 or newer.
Notes:
Finally
1. Nice to have market apps to improve on Android graphical offerings:
(voice recognition and more than word-level handwriting recognition aren't quite there yet in gingerbread; evernote features the last one, but the recognized text is inaccessible and only used for indexing/searching)
2. Required market apps for the rest of this article:
(there also are one or two apps that intend to simplify the creation of the chroot for sections 3 and 4 of this article - search for linux installer on the market; usually either Ubuntu or DEBIAN)
3. PC applications
The second article looks at twlauncher in more detail and includes some more categories and app suggestions sorted into those categories - if you want to call my personal app collection by that lofty title.
Summary: By now you've a rooted Android device, seen a few XDA support threads as well as some checklists for use when reflashing the ROM.
Do run and setup each of busybox installer, sshdroid, terminal ide, titanium backup and superuser. From now on, I assume that
Please unpack a copy of the example files somewhere (though not in / on your Android). You can check the Android directory for updated versions, as I will only very infrequently update the tarball. ./host.note is an extract of the Note's files, while ./host.anuurn contains some files from the Linux PC.
If you intend to use my INSTALL script to modify your Android shell setup, only files below /s (or /mnt/sdcard/COMPACT/setup) will be considered by INSTALL, so please start with empty /s/data, /s/system, and /s/root-tarballs directories and simply modify and add scripts, tarballs and configuration, as you proceed through the article and find interesting tidbits. In the example files, the location is host.note/mnt/sdcard/COMPACT/setup/INSTALL.
Now that we've taken care of the basics and have access to a rooted Note, let's get to the more interesting parts of this article.
libc and the most visible problem: Google reinvented an embedded small libc clone by the name of bionic, which together with the Android framework implements DNS access in a non-standard way. This leads to some confusion in porting as whether to look at the output of setprop and make quite a few code modifications or just look at the files in /etc and hope that something (the user...) keeps them in sync with the framework's properties (setprop/getprop shell command). For busybox, the stericson version you installed into /system/xbin can access DNS properly, while the versions from sshdroid and terminal ide currently do not and only permit raw IP addresses. In my setup, I have defined an alias iresolver to reorder PATH to enable DNS-related shell commands.
Google also reinvented the wheel with logcat. Again with interesting effects: they didn't mangle all parts of the kernel, so many kernel messages never make it to logcat at all. Worse, there is no syslog/klog, and messages at best can be seen for a very short period in a much too noisy dmesg output. The lost messages include uninteresting topics such as logged packets from user-created iptables rules. However, messages about filesystem data corruption are similarly mistreated. Which just happens to be a severe reliability issue. Note that there's nothing in a chroot that can work around this problem.
(logcat use may require insmod /lib/modules/logger.ko)
Other things that are not available on my Note (might be fixable by a kernel arm-toolchain rebuild if we get the current Source and config for the AbyssNote kernel, hoping that they're merely (badly) unconfigured rather than broken by bad patches):
The low-power portions of the Google patches neglect to properly fix sleep. So without active wake_locks to avoid system suspend-to-ram, even a one second sleep is completely unreliable and may have a duration of a few minutes. Large sleeps are affected way worse: Sleeps aren't ended when the system activates again and the real time advanced beyond the intended end of sleep. Thus a chroot cron becomes pretty much unusable without constantly maintaining active wakelocks. See the example system/xbin/WAKELOCK to set/clear/list wake locks.
(see also LWN: Wakelocks and the embedded problem - inconsiderate patches such as these with heavy collateral damage do have a hard time to make it upstream into the Linux kernel sources)
Editing text: a simple vi clone is part of busybox, and can be found in /system/xbin, and as part of sshdroid and terminal ide copies of busybox. A real port of vim is available with terminal ide. On the graphical side, consider e.g. jota.
I found a few working static armel binaries for bash, mc, rsync, strace, tcpdump, and wget. They are included in the example files in system/xbin. Bash and mc are also included with terminal ide, but the mc version lacks sub-shell support. A simplified wget is also part of busybox.
The shell: sh is part of both busybox and toolbox. As I've run into a slew of toolbox bugs, I'd suggest to use /system/xbin/sh. Or bash, as mentioned in the previous paragraph. (consider a symlink /bin/bash, but it's probably better to avoid /bin/sh). Don't mix GNU programs and busybox programs within a single pipe hosted by busybox sh (spurious output write errors between pipe stages).
To use perl in the shell, see the system/xbin/perl example and install sl4a and the perl apk from Scripting Languages for Android. Python and lua wrappers are also included, but I don't really use them. In case of sl4a crashes, uninstall and reinstall the crashing language. Consider creating the symlink /usr/bin/perl and adding /usr/bin to the PATH to permit normal invocation.
To use the Android GUI in say ssh scripts, you need the access data from a running "host environment" like sl4a: run the shell in sl4a and redirect the output of set to a file. Keep open sl4a and set suitable environment variables according to the captured output. Maybe also of interest/possibly working as a "host": tasker and taskbomb with its script support (all 3 available on market).
(in case you do not want sl4a: at the end of part 3, you will also be able to alias perl to x perl and use the chroot's perl installation)
Script examples:
# 1. output from TOP; the FLAGS are be M (mail), FS! (fs remounted-ro), and
# U! (errors in /data/system/uiderrors.txt)
BAT: 90(disc) CPU: 0(0.1) <FLAGS> 192.168.2.114 20120219-16:30:27
Mem: 762928K used, 48176K free, 0K shrd, 40272K buff, 165484K cached
CPU: 0.0% usr 10.0% sys 0.0% nic 90.0% idle 0.0% io 0.0% irq 0.0% sirq
Load average: 1.80 1.31 1.26 2/732 5270
PID PPID USER STAT VSZ %VSZ CPU %CPU COMMAND
3459 2590 1000 S 407m 51.2 0 0.0 system_server
8207 2590 10072 S 270m 34.0 1 0.0 {d.process.media} android.process.
27066 2590 10015 S 254m 32.0 0 0.0 {.app.twlauncher} com.sec.android.
16765 2590 10092 S 251m 31.6 0 0.0 {app.emailwidget} com.sec.android.
/dev/block/mmcblk0p9 858752 802860 55892 93% /system
2064208 1408044 656164 68% /data
11613376 2002848 9610528 17% /mnt/sdcard
4427304 576092 3851212 13% /mnt/sdcard/external_sd
/dev/block/mmcblk1p2 9289080 3410268 5690072 37% /
# 2. output from psg 2>&1 | cut -b-120
# the first block lists open shells on STDERR, the second programs of interest
# : is a normal shell command, R is running in the chroot,
# A is Android/Java, K is kernel
: pts/2 6641 6636 root S 800 0.1 0 0.0 -sh
: pts/2 6715 6712 root S 1800 0.2 0 0.0 /data/data/com.spartacusrex.spartacuside/files/system/bin/bash
: pts/0 16898 16889 root S 800 0.1 0 0.0 -sh
R pts/0 16914 16898 root S 4880 0.6 0 0.0 /bin/bash
R pts/2 24103 6715 root S 4940 0.6 0 0.0 /bin/bash
R - 26190 26162 root S 4864 0.6 0 0.0 bash
: pts/2 6712 6641 root S 800 0.1 0 0.0 {t} /system/bin/sh /data/data/berserker.android.apps.sshdroid/d
: - 23920 1 10122 S 15772 1.9 0 0.0 /data/data/com.theqvd.android.x/usr/X11R6/bin/Xvnc :0 -br -loca
R - 26143 1 root S 15388 1.8 0 0.0 icewm
R - 26162 1 root S 9356 1.1 0 0.0 xterm
R - 26202 1 root S 3528 0.4 0 0.0 /usr/bin/dbus-launch --exit-with-session x-session-manager
R - 26203 1 root S 2892 0.3 0 0.0 /usr/bin/dbus-daemon --fork --print-pid 5 --print-address 7 --s
# 3. short output of psg -m dropbear | grep 6636 | cut -b-120
: - 6636 root 0:01 /data/data/berserker.android.apps.sshdroid/dropbear/dropbear -H /data/data/berserker.an
# 4. long output of psg -l -m dropbear | grep 6636 | cut -b-120
: - 6636 11159 root S 1084 0.1 0 0.0 /data/data/berserker.android.apps.sshdroid/dropbear/dropbear -H
(there's no need to use java and the ide to really profit from installing and using terminal ide)
terminal ide aims to make standard Unix commands available to permit working with vim and java, thus the IDE in the terminal name. This provides the best working environment for the Android shell outside of a chroot, and it doesn't even require root. It offers e.g.
With some coaxing in the form of custom profiles and the script system/xbin/TERMIDE (aliased to t), the environment is also usable outside of the app, e.g. from ssh or for root. See both of data/data/com.spartacusrex.spartacuside/files/{root/,}. The example profiles will also clone two bash files from the chroot's root account into data/data/com.spartacusrex.spartacuside/files/root to maintain a single environment inside and outside of the chroot, inside the terminal ide app and outside, for both the terminal ide user account or for root. Running the app is not required for this.
Notes
Errata: Terminal IDE screenshot?
We can use the am shell command to invoke java entry point classes and send intents to the framework-side. As am invocations can be lengthy, a small wrapper for use in chroot or android shell is helpful: VIEW. See the script itself for more details. To get supported entry points and mime-types, check the manifest file included in the application apk. Note that most intents are intended only for app-internal use; furthermore, the am command is a debugging tool and doesn't return proper result values. Note also that intents from am start for already running apps/intents will fail and instead bring the running app/intent to front (unless --activity-multiple-task and app architecture permitting).
# wrapper invocation and corresponding am intent invocation
VIEW CALL tel:3311 # am start -a android.intent.action.CALL tel:3311 # actually do the call
VIEW tel:3311 # am start -a android.intent.action.VIEW tel:3311 # view entry (same as DIAL)
VIEW /long/path/file.mkv # am start -a android.intent.action.VIEW -d file:///long/path/file.mkv \
# -n com.clov4r.android.nil/com.clov4r.android.nil.MainActivity
VIEW 'geo:NewYork,USA' # am start geo:NewYork,USA / am start 'geo:0,0?q=NewYork,USA'
VIEW http://leo.org # am start http://leo.org / am start -a android.intent.action.VIEW \
# -n com.android.browser/.BrowserActivity -d http://leo.org
# display home screen # am start -a android.intent.action.MAIN -c android.intent.category.HOME
# view contact entry # 815 # am start -a android.intent.action.VIEW -d content://contacts/people/815 \
# -t vnd.android.cursor.item/person # required to resolve to contacts app (!?)
Further reading: Intents List: Invoking Google Applications, Intents, (Broadcast-)Receivers, Background Services, Instrumentation Testing with AM, ActivityManager.html, Helpful apps include Android Intent Playground, ManifestViewer, and Intent Intercept.
(I cannot find any content://com.android.contacts/ URI examples for contact search by name and display of result list in the contacts app (excluding maybe just content://com.android.contacts/phone_lookup/<NUMBER>) - every example encodes the query, possibly including some pseudo-SQL and then iterates using a database cursor, and at most only then triggering a display intent of an individual numeric contact <ID>. So this seems to require more than a simple am intent command: a bit of programming in Java or SL4A or just fully bypassing the framework and parsing the raw contacts database, similar to what I'll show in the second article for twlauncher. It's however quite annoying that some schemes like geo: URIs are expressive enough to include arbitrary queries, while on the other hand the contacts provider requires extra effort - besides crashing on most URIs. Search intents using the quicksearchbox also fail, as I don't seem to be able to successfully provide a query string. Summary: Intents do expose some internals for reuse, but Google's design is fragile and lacks standardization, making intents for ipc and application reuse needlessly difficult to unusable)
For the opposite direction, we need to start Unix commands from the framework. This can be as simple as Java invoking system() (e.g. tasker invoking shell scripts depending on location, or regularly running the "fake" cron abin/CRON.15min as used below). Or as indirect as using intents. Both apps and intents can be placed as icons on the home screen to trigger the action (in case of intents consider apps like xShortcut or Any Cut).
Intent example: Install Scripting Languages for Android/SL4A and SL4A Perl (if you see force-close, uninstall and reinstall using the SL4A Perl installer). We want to invoke a shell command using intents, more specifically a Perl script. We use SL4A's LAUNCH_BACKGROUND_SCRIPT intent with the ScriptingLayerServiceLauncher entry point to invoke the script. On the shell, we can create and invoke this intent as follows:
am start -a com.googlecode.android_scripting.action.LAUNCH_BACKGROUND_SCRIPT \ -n com.googlecode.android_scripting/.activity.ScriptingLayerServiceLauncher \ -e com.googlecode.android_scripting.extra.SCRIPT_PATH \ /x/tmp/test.pl
As we use SL4A already for the intent example, let's do something more interesting: add three lines to display a small framework widget. As a small lookahead to the chroot setup later on, add another line to have SL4A Perl invoke the chroot's Perl with a second script, which in turn does the same from inside the chroot. This works, as su/CHROOT.run retains the AP_HOST/AP_PORT environment variables and permit use of SL4A acting as the "framework-side graphical server". Here are the scripts used for this example:
# /x/tmp/test.pl - the first script as run from android shell or chroot via intent
# su is required for chroot to work
# (possibly due to SL4A id (e.g. app_247) being undefined inside the chroot?)
use Android;
my $droid = Android->new();
$droid->makeToast('Hello, Perl!');
system("su -c '/system/xbin/CHROOT.run perl /x/tmp/test2.pl' > /x/tmp/test2.out 2>&1");
my $rc=$?>>8;
$droid->makeToast("returns $rc");
# /x/tmp/test2.pl - interacting with android from within the chroot
BEGIN{push @INC, "/mnt/sdcard/com.googlecode.perlforandroid/extras/perl/site_perl"};
use Android;
my $droid = Android->new();
$droid->makeToast('Hello from the Chroot, Perl!');
Further reading: SL4A and remote control.
Browsing: some providers like to mangle HTML pages and provide different resolution images or even embed extra javascript in pages. In case of the German Telekom, most of this nonsense can be disabled by speed.telekom.de/laptop_confirmation.php?u=. This behaviour was questionable when it started and with increasing network speed and display sizes, it doesn't get more reasonable or acceptable.
Using ssh: An ssh-client can be found as part of sshdroid including sftp server and scp. Another copy is offered by terminal ide. A real openssh build is part of sshtunnel, also called the same name. For a graphical client, consider connectbot.
(note that I was unable to trivially port an old ssh-based tunnel trick of mine, as some ssh options are not supported by any of these. Work-around: use the openvpn installer and then run openvpn in the Android shell. See system/xbin/VPN*)
Using sshd: As ssh-server I use dropbear as provided by sshdroid (terminal ide again provides a second copy) and only rarely run a second sshd -p <PORT> on an alternate port in the chroot. See the profile in data/data/berserker.android.apps.sshdroid. Adding keys with the GUI didn't always work, so I just copied them into home/.ssh/authorized_keys in the above directory.
Using socks: proxydroid, sshtunnel and orbot (TOR) use iptables-based forwarding of http/https requests to a server-based based socks daemon. If you want to setup your own socks daemon, take a look at the hpsockd.conf in the examples (as tested once on Ubuntu 10.4; do not use the default port of 1080, as some broken botnet seems to love attempting email submission on that port).
the iptables command is hopefully part of your custom ROM. It is also available as part of proxydroid, sshtunnel and orbot. lbe privacy guard or any of the personal firewall tools should also contain or offer to install a copy.
rsync: I stumbled upon a static compile (included in the examples) which is as simple to use as rsync -a --delete root@note:/DIR /HOSTDIR. For a backup solution, check e.g. rsync-backup. Some of the rsync-related market apps should also provide suitable rsync binaries.
I think that for now I will want to fall short of providing the promised automated nightly backup solution (if you silently add a 'simple' in there). In the meantime, the files I actually use in my backup are part of the example files. My backup setup is bit older than rsync-backup and centers on filtering filelists. As invoked for the Note, it also uses rsync for copy and cleanup of removed files, and it also keeps numbered daily/weekly/... snapshots. Execution starts with home/jakobi/bin/xfer_backup.wrapper.anuurn.local, which is invoked by a nightly root cron job (ensure that titanium backup on the Note is finished first). Furthermore check the copied titanium backup location on the backup target after the first run - I initially used /mnt/sdcard/external_sd/BAK, however /BAK$ is automatically excluded in my setup...
Using a VPN: OpenVPN and tun.ko is hopefully already part of the custom ROM (or in case of tun.ko optionally also a part of the kernel - otherwise run insmod /lib/modules/tun.ko or similar; there also is a tun.ko installer on the market as well as an openvpn installer). The example files contain a shared-secret openvpn-pair example sans the secret:
Graphical apps using VNC (or RDP) to view remote computers: there are various VNC viewing apps available, including vncviewer, and pocketcloud. For the opposite direction, to view the Note's display using VNC and to remote control Android, consider androidVNC.
(the Note's resolution is great, but my eye isn't upto to the Note's 300dpi, so VNC use requires zooming. There's also a multi-head issue: no vnc app was up to deal with a multi-head display of two different sized monitors -> run a (smaller) vncserver on the remote host)
There is also Xvnc, which combines a VNC viewer hardwired to connect to a locally running VNC server, which in turn acts as X11 server (armel binaries included with Xvnc). If you have space issues when creating the chroot in part 4, Xvnc is very convenient to skip installing most of X11 inside the chroot - merely run aptitude install xterm icewm to install the client-side minimum of X11.
As we install a chroot in the next part, I'm not covering graphical apps that access remote files or offer access to remote files.
As for NFS, without kernel support, you cannot neither access NFS files nor offer them.
(unless you have access to a Linux box elsewhere and run fuse sshfs in the direction you want to: The Linux box can then either offer the Note's files per NFS or forward contents of NFS mounts to the Note. Chroot is required for this direction; see the example abin/SSHFS script later on)
To connect to a WLAN and or use the Note in AP mode (WLAN tethering), just use the settings menu on the Note. To connect to the Note in AP mode, use e.g. the Gnome network manager on the Linux box. For tethering via USB, connect the Note and select USB tethering. On the Linux box run modprobe usbnet, maybe also dhclient and have a look at usb0. I didn't test inverse tethering (making the Note use the Linux Box's connection to the internet), but on a short search I found 11 pretty dissimilar sketches from using ppp or ssh or adb with portforwarding to bridging.
Network security
Summary: By now we have a somewhat usable and moderately comfortable Android shell. We have basic networking commands to connect remote servers and we we can run perl scripts. An external host can be used to run shell commands on the Note or run rsync to create a file backup of the Note. We also have seen an automation example consisting of a remote triggered backup, and we implemented a real inbound firewall (system/xbin/FW).
I recently had a look at OPTWARE, which offers about 1300+ packages, with both busybox and real GNU binaries and a real libc. It is not sufficient to provide a mobile desktop, but it certainly goes way beyond terminal ide in providing missing binaries at the cost of a root prerequisite and some more diskspace. See article 01b Setup Alternatives for more information.
Let me start by pointing back to the warning against bad flash!
Let us create an SSD-style 4MB-aligned partitioning scheme by starting the first partition at 8192 and have all partitions start at sector numbers divisible by 8192 (this might offer a slight improvement also for sdcard flash(?); if you want to simplify the setup further, just create a two partition setup without alignment).
Sizing: For a basic desktop tasksel install plus build-essentials and some file serving, you need about 3.5GB plus maybe 2GB extra space during installation. A rule of thumb for a better guess may be to clean /var, /tmp and run aptitude clean, and double that size to add some reserve to permit upgrades. For me that results in using at least 7GB. Without any desktop task, you should still comfortably fit in a 4GB partition or in a maximum-size slightly-less-than-4GB file container on either the internal or external sdcard. However the container file setup requires a loop mount with some overhead, and I already had some data loss on the old external sdcards, so I strongly suggest to use a separate partition with ext4 for the chroot.
Put the sdcard to use in a card reader on your Linux box and umount any automatically mounted sdcard filesystems. Then run fdisk and create the filesystems.
(the commands are pasted from my notes; I shouldn't have changed much afterwards. Any interesting changes should be already reflected in the CHROOT.mount script)
# avoid problems on Ubuntu 10.4 with recent kernels echo 0 > /sys/block/sdd/queue/rotational echo noop > /sys/block/sdd/queue/scheduler fdisk DEVICE # /dev/sdd for me # press c and u, and define partitions that look like these #... #Units = sectors of 1 * 512 = 512 bytes #... # Device Boot Start End Blocks Id System #/dev/sdd1 8192 8880127 4435968 83 Linux # type c aka fat32 #Partition 1 does not end on cylinder boundary. # <--- ignore this #/dev/sdd2 8880128 27754495 9437184 83 Linux # chroot partition #/dev/sdd3 27754496 29433855 839680 83 Linux # reserved home/var #/dev/sdd4 29433856 31116287 841216 83 Linux # reserved, maybe also swap # type p to print and check; type w if everything is ok to write/quit fdisk. mkfs.vfat -n note2 /dev/sdd1 mke2fs -t ext4 -O extents -F -j -i 16384 -L ARMDEBCHROOT -m 2 /dev/sdd2 # 600K inodes, 2% root reserve mke2fs -t ext4 -O extents -F -j -i 8192 -L ARMDEBCHROOT_R1 -m 5 /dev/sdd3 # reserved for /var and/or /home mke2fs -t ext4 -O extents -F -j -i 8192 -L ARMDEBCHROOT_R2 -m 5 /dev/sdd4 # reserved for swap tune2fs -c 20 -i 2m -e remount-ro /dev/sdd2 # set fsck requirements nobody will honor tune2fs -c 20 -i 2m -e remount-ro /dev/sdd3 # and set the behaviour on errors tune2fs -c 20 -i 2m -e remount-ro /dev/sdd4 mkdir /mnt1; mount /dev/sdd2 /mnt1
(maybe consider to increase the journal size to better spread writes; this works in case the journal is rolling and doesn't reset to a fixed start of journal; not checked)
Run stage 1 on the Linux Box and copy the results to the chroot filesystem. Both stable (squeeze) and testing (wheezy) are suitable for use as chroot.
debootstrap --arch armel --foreign testing /chroots/sid-armel http://ftp.us.debian.org/debian (cd /chroots/sid-armel && cpio -pmdu /mnt1) sync; sleep 2; umount /mnt1 mount | grep ...
Umount and remove the sdcard from the card reader and reinsert it in the Note. Boot the Note. And further prepare the chroot, which will be mounted at /x.
umask 022 # (I had an umask of 011 from dropbear!?) cd / mkdir /x mount /dev/block/mmcblk1p2 /x mkdir /x/a cd /x/a ln -s mnt/sdcard i ln -s mnt/sdcard sdcard ln -s mnt/sdcard/external_sd e ln -s data/local/tmp t ln -s data/local/setup s ln -s system/etc . ln -s sys/kernel/debug d ln -s data/local/root . mkdir ./-p ../-p chmod 444 ./-p ../-p # mirror the outside files required to run commands (sys, dev, ...) # as well as make sdcards, system and data accessible from within the chroot # # (these commands are also the key part of system/xbin/CHROOT.mount and # required after each reboot of the Note to prepare the chroot for use) # mkdir -p proc sys acct system data dev efs cache mnt mkdir -p mnt/.lfs mnt/sdcard mnt/usb mnt/obb mnt/asec mnt/app-cache mkdir -p mnt/secure/asec # # I will skip some of the mounts like asec and lfs for now # # (it might be interesting to try to run app_launcher also within the # chroot to more directly interact with the Android framework. However # the chroot can also just run ssh localhost to reach the Android shell via sshdroid) # mount -orbind /sys sys mount -obind /acct acct mount -obind /data data mount -obind /system system mount -obind /cache cache mount -orbind /mnt/sdcard mnt/sdcard mount -orbind /dev dev mount -orbind /app-cache app-cache # we cannot bind anything from rootfs mkdir COPIED ( cd /; find bin config customkernel dbdata default.prop lib res sbin usr vendor | cpio -pmdu /x/a/COPIED ) ln -s COPIED/* . # cd /x mv dev dev.old ln -s a/* . 2>/dev/null # proc must be real and not a symlink mkdir proc mount -t proc proc proc
With the mounts and directories in place, enter the chroot for the first time, run the second stage of debootstrap to install the initial system, and update the system to the current package versions in the repositories.
cd /x chroot /x export PATH=/usr/bin:/usr/sbin:/bin:$PATH export TERM=linux export HOME=/root export USER=root # this requires a few minutes /debootstrap/debootstrap --second-stage # undo the damage done by debootstrap, and restore some of the symlinks cd / rmdir sys rm -r dev.old; mv dev dev.old ln -s a/* . cd /etc rm mtab; ln -s /proc/mounts mtab # android has no sane fstab, just vold.fstab # shall we combine resolv.conf? rm resolv.conf; ln -s /a/etc/resolv.conf . rm hosts; ln -s /a/etc/hosts . # shall we also use the android side passwd and groups? Not for now. cd / # permit using the /x prefix also within the chroot ln -s / x # some apps may honor this and do not startup during installation mv /x/usr/sbin/policy-rc.d /x/usr/sbin/policy-rc.d.ORIG # does not exist # policy taken from http://wiki.debian.org/ArmHardFloatChroot echo "echo 'All rc.d operations denied by policy' >&2; exit 101" > /x/usr/sbin/policy-rc.d mv /x/etc/apt/sources.list /x/etc/apt/sources.list.ORIG # debootstrap version is invalid echo "deb http://ftp.de.debian.org/debian testing main contrib non-free" >> /x/etc/apt/sources.list echo "deb-src http://ftp.de.debian.org/debian testing main contrib non-free" >> /x/etc/apt/sources.list # change the hostname away from name of the Linux box echo note > /etc/hostname # another few minutes apt-get update apt-get install debian-ports-archive-keyring apt-get update apt-get install build-essential # a first backup of the initial chroot cd / find . -xdev | sort > FIND.chroot cat FIND.chroot | cpio -o -H crc > /i/chroot-initial-debootstrap.cpio
Install required application packages, then you can optionally install one or more of the major meta-packages suggested by tasksel. In January, the Desktop-selection had issues with bluez and mono, as described in the notes below.
apt-get mc tmux vim vim-scripts m4 rsync sshfs elinks mutt # the bare X11 client-side mininum, with a lightweigth window manager apt-get icewm xterm # if you have enough space I also suggest these apt-get tightvncserver xtightvncviewer apt-get lsb-languages lsb-languages-armel lsb-languages-noarch lsb-core # check for issues and possible look at recommendations aptitude cd / find . -xdev | sort > FIND.chroot cat FIND.chroot | cpio -o -H crc > /i/chroot-initial-minimal.cpio # you may wish to install software for basic tasks like desktop or file serving; # requires more than just a few minutes, depending on your selection tasksel dpkg-reconfigure tzdata # all package lines should start with ii (properly installed) dpkg -l | grep -v '^ii' # has aptitude any issues or problems (press 'g') aptitude aptitude clean # backup the full chroot for the first time cd / find . -xdev | sort > FIND.chroot cat FIND.chroot | cpio -o -H crc > /i/chroot-initial-final.cpio
Notes
While the scripts in system/xbin are for the outside Android shell, the abin directory in the chroot is for in-chroot glue scripts. Thus the chroot-related scripts in system/xbin/CHROOT* often call scripts by the same name in abin once inside the chroot. The abin directory in also contains CHECK (monitoring), SSHFS (remote file access) and the CRON* files (enable some basic cron-style scripts that schedule somewhat sanely despite the unreliable sleep in Android kernels). The CHROOT scripts in detail:
(I still hold some hope of never having to fix CHROOT.umount due to never using it again :))
Root account: As I didn't want to maintain umpteen copies of profiles, I keep my own stuff in root/.bash{_aliases,rc}.pj and just add suitable sourcing or updating statements to the other files. That way, user accounts and terminal ide share this master copy when sourcing their own environment. The profiles set a useful prompt and window title providing port/pty/user/directory information, plus return code in case of PS1. They also contain aliases that shadow updatedb/locate and use the nightly generated filelists of the backup. Note that this abuse of /root requires a not-that-secure chmod 711 /root - the clean way would have been to put the config files to share into abin and just source them as everyone else does. As usual, you can find copies of all relevant files in the example tarball.
In setting up non-system user accounts, you do not want to trip over Android IDs. I changed /etc/login.def to use 20000 instead of 1000 for the values of UID_MIN, GID_MIN. In /etc/adduser.conf, change FIRST_UID and FISRT_GID similarly. echo android_network:x:3003:USERNAME >> /etc/group to /etc/group and fix the gshadow structure accordingly. The number 3003 is the public secret number of the group ID permitting network access to non-system(?) users. A search keyword for more detail on this is ANDROID_PARANOID_NETWORK.
Escaping the chroot can be done by ssh to localhost, as shown below in the example using the chroot's host key. If you use ssh-agent and passphrase-protected keys, invoke it as ssh-agent > ~/.ssh/env.$(hostname).
# The Great Escape (from chroot to Android shell) cd /root/.ssh ln -s /etc/ssh/ssh_host_rsa_key id_rsa ln -s /etc/ssh/ssh_host_rsa_key.pub id_rsa.pub cd /data/data/berserker.android.apps.sshdroid/dropbear ln -s /system/xbin/CHROOT.run x cd ../home/.ssh cat /root/.ssh/id_rsa.pub >> authorized_keys # check permission and owner ls -l authorized_keys # now you can run ssh localhost COMMAND from within the chroot
Umlauts and locale settings are a bit surprising, as I expected full UTF8 support everywhere on the Android by default. Umlauts seem properly displayed, on delete however they appear as multiple chars. I still need to really run DEBIAN as more than a server system to get a better grip on where and what packages offer locales, as I didn't find e.g. en_US.UTF8. For the time being, the sanest setting is LANG=C.UTF-8 (plus a paranoid LC_COLLATE=C), as this repairs those deletion issue e.g. for ssh logins to the chroot.
Some applications require a running dbus: /etc/init.d/dbus start. abin/CHROOT.start is intended to collect the startup commands for any lightweight daemon or service.
Terminal multiplexing: the screen command is broken and hangs on detach, however tmux is working properly. See /root/.tmux.conf for setting the command key to CTRL-a like with screen.
Virtualization, e.g. lxc: mostly unsupported (bridging may also be unsupported, depending on the kernel).
Basic cron services are problematic, as sleep is unreliable. abin/CRON* offers an example setup to to cope with this situation. The 15min job contains code to catch missed runs of other CRON* scripts and tries to resynchronize them with their original window if their execution was delayed. All you need to do is invoking x /abin/CRON.15min remotely or with tasker or from multiple invocation sources at once (the jobs maintain suitable locking). This example can be extended with anacron and more of the actual cron jobs configured in /etc/cron*. The abin/CRON.15min script is currently run by remote root cron in my home WLAN every 15 minutes.
Note that I've also added a repeated short period WAKELOCK invocation to the root cronjob on my Linux box. Thus sleep becomes reliable in my home WLAN and the ordinary cron (if started) can run normally during the night. However not-using-the-CPU-by-sleeping now costs extra battery, as the Note can no longer suspend to RAM.
Monitoring: abin/CHECK offers a trivial monitoring script, which can be run by ssh and evaluated remotely (see the root cronjob and watch_delta; run every 15 minutes).
Mail notification and forwarding: The sendmail service is provided by the exim package. Without a setup, exim places all local email in the /var/mail/mail spool file. abin/CHECK reports available mail, as does system/xbin/TOP. Again have a look at the root cronjob file for an example to forward this chroot-local mail to an account on the remote Linux box (again every 15 minutes). As I don't want a real mail configuration in a chroot, that's good enough for now.
Sending and receiving mail can still be done inside the chroot with mutt by using the remote imap/pop/sendmail servers without relying on a local sendmail. The example files contain a cut-down mutt setup as part of my normal user account in the chroot (home/jakobi/.mutt). There's also a wrapper in bin/mutt that I use to switch profiles and sender account according to folder contents, which you can ignore as long as you use only a single mail account or define macros to switch accounts.
Encryption: For now I only use apg on the Android side and gpg in the chroot for a couple of files. This is more than enough to use vim to decrypt a file with some web forum passwords. I would sugggest to only edit this file on your Linux box to avoid synchronization issues, as patch or unison aren't able to analyze in-file differences between encrypted files.
ERRATA: Demonstrate an encrypted loop mount?
For backup and synchronization, revisit the example file xfer_backup.wrapper.anuurn.local and the nightly backup job invoked by remote root cron. Besides running the filelist-based rsync, it also runs ct_note.update-copies and ct_note.update-unison (all on the Linux box in home/jakobi/bin). The first clones a subset of the my home directory to the Note, the second synchronizes a folder tree, with one layer also following symlinks; both only run when the chroot is mounted on the Note. As the Ubuntu 10.4 version of unison was too ancient, I installed a new build in /usr/local/bin on the Linux box (fast compile, no trouble at all with ocaml). Remote invocation is as simple as DISPLAY= unison /data/unison -batch -auto -servercmd "x unison" ssh://note//data/unison.
Using sshd: I'm still using sshdroid's dropbear as the primary ssh daemon, dealing with backup traffic, and everything. The chroot can be accessed from the Linux box by running ssh root@note x COMMAND (e.g. as done by unison; rsync dislikes the space and requires the chroot-rsync dummy file in the sshdroid/dropbear directory). If you need a real sshd within the chroot, just run sshd -p PORT & in the chroot, possibly stopping sshdroid. This can added to system/etc/init.d/S99zz_compact_jobs to start at boot of the Note.
Accessing remote filesystems: Have a look at the example abin/SSHFS containing simple mount/umount commands to permit access to files on remote computers. This requires the fuse sshfs package. Sshfs even offers access to anything mounted on the remote side. SSHFS is intended to also deal with SMB, and NFS (if the kernel permits), but this is untested.
File serving: Use the usual culprits of ftp, apache and samba as on any Linux box. Don't forget to edit and rerun FW on if you need to open ports in the firewall.
(as of 20120214, wiki.debian.org/ArmHardFloatChroot suggests the standard repositories for non-source packages, using 'sid' and 'unreleased'. If FPU support is indeed the only change, after the final backup and before removing the contents of the chroot to rebuild, it is tempting to just change the architecture in the apt/sources.list and see what happens. Maybe with first repeating the debootstrap steps, and then restoring the old /var and most of /etc. Do check the release notes first, as I might not be the only one dreaming about an armel to armhf migration path :))
Summary: By now we have a DEBIAN chroot and can access it from the Android shell interactively or as part of a pipe (system/xbin/CHROOT.run aka x). We are able to access remote files using everything but NFS (see the abin/SSHFS example). We also extended the remote automation with limited email checking, monitoring and folder synchronization.
The idea of VNC is to split X11 into the server component (e.g. tightvncserver) and a separate, remote display component (e.g. vncviewer), with just exchanging compressed graphic data between the two parts. Comparing local VNC to using a remote X11 display, transmitting graphic data is usually less efficient than other types of data, but eliminating remote call round trips to the X11 server removes all remote call latency, which in itself already makes VNC very interesting for non-LAN and higher latency situations.
For the chroot we ignore anything except 2d display, keyboard, mouse. If you want audio, consider a network-capable audio setup like pulseaudio, or have a look at sl4a, to send intents to the Android framework. If device access isn't blocked by something, you could also try accessing the device directly. I also ignore flash; if you need it w/o audio in the chroot, consider installing gnash - but even adobe itself nowadays seems pretty ambivalent towards use of flash. The idea behind RDP is similar to VNC, and some viewers support both protocols, but I didn't check for server components usable in a chroot.
A selection of VNC viewer components
A selection of VNC server components
(there are also some interesting ideas on VNC abuse for e.g. remote input devices, most of which seem to work fine at least for single head displays. Consider Android use as mouse-cum-touchpad or media remote. There is also - in opposite direction - an app called androidvnc that exports the Android display to vncviewers.)
My current setup reliably "remove-battery"-crashes the Note as soon as I use X. I think its a kernel issue, assuming that Google somehow messed up in crippling or deconfiguring shared memory. Short of a custom kernel rebuild, we can only enforce zero shared memory values as a work-around as shown below. This hack is part of the ivnc / ivncapp functions in the root bash setup. Applications that use shared memory only for speed-up will run a bit slower. New applications like chromium-browser that cannot work w/o real shared memory will fail.
# test if you crash or if you can get by without # this ugly trick, in which case don't forget to # remove it from the ivnc* functions in # /root/.bash_aliases.pj for i in /proc/sys/kernel/shm*; do echo 0 | sudo tee -a $i > /dev/null; done;
Initially, I suspected free memory issues, but neither watch -n1 cat /proc/meminfo nor strace indicated anything interesting. If you run into a real memory-issue, these are the values I played with: sysctl -w vm.min_free_kbytes=80000 or sync;sleep 2; /sbin/sysctl vm.drop_caches=3. If you have swap, also consider swappiness. You can also modify /proc/PID/oom_adj or set up limits by using cgroups (search wrapcgroup in my script-archive). Returning to my memory suspicion, I have about 300-350M free before X, which is enough for a vncserver, xterms, firefox 3.5 and LibreOffice with icewm, and still leaves enough head room to run extra commands and for buffers+caches. In testing, I never encountered an out-of-memory situation.
Still, being able to add swap space (the second reserved partition or just a swap file) would be nice, but my current kernel does permit swap. The fast internal flash would be tempting. But I'm a bit sceptical and thus would only suggest to use the removable external flash, with a swappiness setting to make swap a real last resort for the kernel just before invoking the out-of-memory-killer. I assume that swap write access patterns will have random, but very heavy hotspots, which would be the worst possible pattern for testing the implementation quality of a only possibly existing flash translation layer on sdcards. Given my experiences with bad flash and the frequent firmware updates as observed for better SSD devices from quality vendors in the last few years (indicating firmware fixes to also the FTL), I'll for now stick to my paranoia. Don't argue with database use of flash - as e.g. transaction logs are the most friendly sequential access pattern there is, that is, they do not stress nor need the flash FTL at all.
For starters, try running xterm on the Note and display the window on your Linux box: e.g. DISPLAY=<REMOTEDISPLAY> xterm. To work, the X-server must listen to tcp and permit connections (Ubuntu's doesn't by default). You may need to run xauth on the Linux box and copy the value of the current cookie to the chroot by running xauth add <REMOTEDISPLAY> MIT-MAGIC-COOKIE-1 <COOKIE>. For this quick test consider to temporarily turn off the firewall: FW off on the Android and maybe add an exception on the Linux box (e.g. man ufw on Ubuntu). Don't forget to reactivate both host-based firewalls when done.
If you've started a second sshd in the chroot (say port 0815), you can also run ssh -p 0815 root@note -Y ..., which should include the xauth invocation (in contrast to -X) and shouldn't AFAIR require the X server to listen to tcp. Ssh tunneling also avoids the FW issues from above.
Now invoke vncserver inside the chroot by running the ivnc function as provided in root/.bash_aliases.pj in the examples. Define some VNC password. Run echo $DISPLAY. Switch to the Linux Box and run vncviewer <DISPLAY_VALUE_FROM_ECHO> -via root@note. If you don't have the package tightvncviewer, you need to use ssh with port forwarding. Assuming that the echo returned localhost:4, you'd say ssh -L 5909:localhost:5904 -N -f root@note; vncviewer :9. Both examples use ssh for their connection.
As final test, run your vncviewer app on the Android and connect to the vncserver in the chroot. Using xvnc is identical, but skips configuring the connection in the app and explicitely starting the vncserver in the chroot. For xnvc use the ivncapp function in the chroot after starting X within xvnc.
You can use the file ~/.vnc/xstartup to add commands to X startup, such as starting the window manager automatically on starting the vncserver. In case of troubles, a chmod 755 ~/.vnc/xstartup might be required. A small example xstartup file:
#!/bin/sh xrdb $HOME/.Xresources icewm & sleep 2; xsetroot -solid black; xterm & /etc/X11/Xsession
In case of problems pasting between host-environment and VNC, consider running autocutsel to synchronize cutbuffer 0, selections and clipboard. My case of Ubuntu 10.4, Gnome and urxvt was made more difficult by something from Gnome partially messing up the selection within half a second, and urxvt being more choosy than xterm doesn't help. Workaround is modifying autocutsel by adding a system(sleep 0.6;xsel -op|xsel -ip) after copying to the buffer. If you have an idea to find the culprit, drop me a line.
Successfully tested applications
Fails: These applications either require shared memory or some daemons which I did not start or cannot easily start (hald/udevd/...)
The described setup with a small window manager like icewm permits normal LibreOffice text editing, programming or some light web browsing, even without swap: I've usually beyond 350MB free (free+buffers+cache) before starting X, which is enough for X11, some xterms, firefox and LibreOffice.
The speed with the current kernel/gingerbread is sufficient even if I only used a single CPU during tests. During heavy writes to flash, speed will suffer and I saw short hangs, especially in a fully configured vim with syntax highlighting, while the simple busybox vi outside the chroot is unaffected. Running huge resolutions like vncserver -geometry 1920x1600 is slow but works without issue.
When encountering the firefox problem, I switched the chroot from a pure testing aka wheezy setup to a mixed setup using apt pinning, see etc/apt/sources.list and etc/apt/preferences.d/personal in the examples. To install a package from a non-default repository in the mixed setup:
apt-get install <package>/unstable # prefer testing for resolution apt-get -t unstable install <package> # use unstable for resolution # afterwards consider pinning the package to a specific release in # /etc/apt/preferences.d/*
ERRATA: Add a screenshot of the Note using Xvnc + a shared vncviewer on Linux showing xterms and LibreOffice on the Note? Both here and at the top? Add it also to index.phtml.
Summary: This concludes the series and by now you can use an arbitrary PC's vncviewer or putty/ssh setup to connect to your NOTE and work locally on your Note in addition to using the Note to copy or serve files. The 1GB memory size of the Note permits also use of X11 apps including Firefox and LibreOffice/OpenOffice.
Excluding only the size of the display, the Note already has the capabilities of newer tablets as well as those of laptops, with at most a little bit of DEBIAN help.
And for the pc-less couch potato, not desiring to implement the above himself, there's still Canonical's take on chroot computing as seen e.g. on the Atrix: "In every dual-core phone, there is a PC trying to get out." - Well said, Canonical! (with the HDMI cable, the Note should permit already that kind of use with e.g. Xvnc; when it arrives, I'll test using a DVI display and some adapters).
ERRATA: another candidate for a photo.
Back to the Android Section Overview.
HOME | GIT Overview | Script-Archive: (docs) : (wiki) : (git) | Android: (articles) (files) | ...
jakobi(at)acm.org, 2012-01 - 2012-03