Subject: CCL on Android
Date: Tuesday 22nd November 2011 07:35:40 UTC (over 5 years ago)
Early in the year, I got CCL running (for at least some value of "running") on Android. It was of somewhat limited utility (there are only so many times that you can call (FACT 1000) on your phone before the thrill wears off), but anything of greater utility - something that could be used to develop Android applications - would have to start somewhere. I haven't had too much time to work on it since, and when I tried building it from current trunk sources a month or two ago I found that bitrot had set in. Once in a great while, someone asked about the (fairly old) binaries that were in svn, and the Android port was mentioned in passing in the iOS discussion a few weeks ago. The bitrot seems to have been relatively minor, and I was able to get it working (for some value of "working" ...) again today and updated the binaries in svn. If anyone has interest, some spare time, and a suitable Android device, I can explain how to install and run those binaries and how to build them from source; since (AFAIK) I'm the only person who's ever run it, I'm curious as to whether or not it works for anyone else. This is still a sort of spare-time project for me, but I think that it's potentially interesting and would like to put more spare time into it than I have over the last several months. If anyone's still interested, read on ... This all assumes that the reader is reasonably comfortable with the Unix command line and that (for instance) after installing the Android SDK you can make it convenient for yourself to invoke the SDK's "adb" program (by modifying your PATH/creating links/etc.) It also likely suffers from my own lack of knowledge of Android and of the world of Android devices out there: in some cases, I can only offer vague troubleshooting suggestions if things don't work for you as they do for me. I've used a Nexus One phone, which is getting to be a little long in the tooth but was originally intended as a sort of reference platform. Phone carriers all customize Android to some degree, and I don't know if or how a particular devices's software might differ from that reference platform. *** Prerequisites: 1) An Android version 2.2 or later device with an ARMv7 CPU. A lot of Android phones and tablets sold over the last year or two use ARMv7s, but this isn't universal. If you aren't sure, I can explain how to check. The Android SDK (see below) comes with an emulator; AFAIK, the emulator emulates an older ARM architecture version (v5), and CCL can't work on such a machine. 2) The Android SDK, which is available at <http://developer.android.com/sdk/index.html>. The SDK's a fairly large download, but at this point we're only really interested in one of the tools that it provides: the "Android Debug Bridge", which is a program that can be used to control and configure an Android device connected via USB (and by other means.) We're mostly just interested in using ADB to transfer files from a host machine to an Android device; there may be other ways to do this, but doing it via ADB is probably simplest. The SDK is available for MacOS, Windows, and Linux hosts; I've only used it on Macs. On Linux, being able to communicate over a USB cable may require some editing of device configuration files and on Windows it may require installation of device drivers. I don't know anything about the details of these issues, but there's a ton of information about them on the web. If you have "adb" installed on a host machine, you can verify that it works by: a) enabling "USB Debugging" on the device; AFAIK, this option is in Settings->Applications->Development. It's been present on the handful (3) of Android devices that I've used. I suppose that it's possible that a phone carrier may have disabled it, and I don't know what the next step would be if you can't do this. b) connecting the device to the host machine via a USB cable. When it senses the USB connection, the device may offer to expose some of its filesystem to the host device; although it sounds like this'd be a simple way to transfer files, the exported part of the filesystem doesn't allow execution of the files it contains. c) on the host, the command shell on host> adb devices will give output like: List of devices attached HT069P800445 device Google not only develops Android, but they offer a search engine that may help you find troubleshooting tips that may be helpful in this case. d) assuming that adb can see the device shell on host> adb shell cat /proc/cpuinfo will generate output that describes the CPU and its architecture; if the first line contains "ARMv7", you're in luck. If not, you went to a lot of trouble to find that out. 3) The CCL binaries for androidarm. If you'd rather build them yourself, I'll explain later how to do that. If you just want to try the ones that're in svn, the easiest way to get them to the host machine is: shell on host> svn export --ignore-externals http://svn.clozure.com/publicsvn/openmcl/trunk/androidarm/ccl androidccl You can use "svn co" instead of "svn export" if you prefer; the two commands behave in similar ways, but "svn export" doesn't populate the output directory with svn metadata (which you may not want to copy to the device.) The --ignore-externals option skips all of the source and interfaces directories. That'll create an "androidccl" directory and populate it with: aarmcl.image a CCL heap image aarmcl.so the CCL kernel, implemented as a shared library aarmcl a little executable file that loads aarmcl.so and calls into it 3) If you want to build your own copy of the kernel, you'll need to install the Android Native Development Kit (NDK) on the host machine. It's available from: <http://developer.android.com/sdk/ndk/index.html> Version 7 of the NDK was released a few weeks ago; the Makefile in ccl/lisp-kernel/androidarm is set up for the previous version (r6b). I don't know what's changed. The Makefile will probably be updated soon, and that may just involve changing the NDK version number in a few places. *** Installation Android's basically Linux with a few kernel extensions and with a stripped-down C library and minimal set of standard [f]utilities. The directory layout's different, and large portions of the filesystem aren't writable by whatever user id the adb program connects as. Small sample-size warnings apply, but I believe it to be true that the directory "/data/local/" exists and is writable and is usually mounted on a filesystem that has at least some free space. You can test this hypothesis by trying to create a "ccl" directory inside /data/local. On the host shell on host> adb shell mkdir /data/local/ccl If that works, it'll do so without generating any output. If you try to create that directory a second time: shell on host> adb shell mkdir /data/local/ccl mkdir failed for /data/local/ccl, File exists that output indicates that it worked the first time. If you get other output from the first attempt (indicating that /data/local doesn't exist or isn't writable), then the hypothesis is incorrect and we're up against the limits of my Android knowledge. You might try Googling to see if there's an equivalent directory on your device/OS release. Some people "root" (as a verb) their Android devices; that generally means modifying the bootloader so that they can install alternative firmware that lets them run some commands as root and change filesystem permissions etc. If you did that you could certainly create a /data/local directory with appropriate permissions, but it'd be much better if anyone who wanted to install CCL on their Android devices didn't have to root their devices to do so. Let's assume that you were able to create a "ccl" directory in /data/local; if you were able to create it elsewhere, substitute the actual location for "/data/local/ccl" below. Assuming that you checked out the binaries to an "androidccl" directory on the host (or copied them there after building them yourself), you can copy that directory's contents on the host to the /data/local/ccl directory on the device by doing: shell on host> cd androidccl shell on host> adb push . /data/local/ccl Note that there's a dot after the word "push" in the second command above. You ideally should see output like the following: push: ./aarmcl.so -> /data/local/ccl/aarmcl.so push: ./aarmcl.image -> /data/local/ccl/aarmcl.image push: ./aarmcl -> /data/local/ccl/aarmcl If you built FASLs on the host and copied them to the androidccl directory, you'd see a lot more output as those files are copied. At this point, the suspense may be killing you. If you do: shell on host> adb shell you should get a prompt from the shell on the device: $ and can type something like: $ cd /data/local/ccl $ ./aarmcl to (hopefully) get output like: Welcome to Clozure Common Lisp Version 1.8-dev-r15086M-trunk (AndroidARM32)! ? () NIL ? (defun foo (x) x) FOO ? (foo 3) 3 ? (values 1 2 3) 1 2 3 Like I said, the thrill wears off pretty quickly. "adb" seems to do some slightly funky things with I/O; I don't remember specifics, but it may terminate lines with CRLFs or do other slightly unpleasant things. I don't usually use SLIME, but would assume that (assuming that networking works) one could set things up to use SLIME/SWANK remotely. There are some terminal programs in Android Market (I don't remember whether they're free or how much they cost if not) that can get you a shell on the device. There are no svn clients that I know of that run natively on Android, so it's a little tedious to get CCL sources onto the device (you can use "adb push".) There's a basic set of interfaces in svn at http://svn.clozure.com/publicsvn/openmcl/trunk/android-headers and if you copy them to /data/local/ccl/android-headers you should be able to use #_/#$ and do ? (rebuild-ccl :clean t) to generate a natively-compiled image. *** Building from sources. It'd work best to use a 32-bit x86 CCL on the host machine. (It might also work to use the ARM Linux port, but i haven't tried that in a while.) When cross-compiling, there's sometimes confusion about whether (for instance) a declaration that something's a FIXNUM means "a fixnum on the host machine" or "a fixnum on the target". We've tried to be careful about that, but cross-compiling on a machine with the same word size as the target machine avoids that particular issue. You'd probably want to use an up-to-date trunk CCL to compile with. On the host, you'd need to get the android-headers directory out of svn; see above. I use the following little file to load the ARM and Android compiler stuff into a CCL running on the host: -------- (in-package "CCL") (defpackage "ARM-ANDROID" (:use)) (defpackage "ARM-LINUX" (:use)) (defpackage "ARM-DARWIN" (:use)) (defun load-android-backend () (update-modules '(arm-arch arm-asm arm-lap arm-backend arm-vinsns arm2) t) (setup-arm-ftd *androidarm-backend*) (update-modules '(arm-lapmacros arm-disassemble ffi-androidarm) t) (update-modules *arm-xload-modules* t)) (load-android-backend) -------- With that stuff loaded, doing: ? (ccl::cross-compile-ccl :androidarm t) will compile most of the CCL .lisp sources (except for the contents of ccl/level-0) into FASL files (.aafsl). The Android and ARM-specific sources will call functions that probably aren't defined on the host system; these calls will generate lots of (benign) warnings. ? (ccl::cross-xload-level-0 :androidarm :force) will compile the level-0 sources into .aafsl files, "load" those files into a simulated heap, and write that simulated heap's contents to "ccl:aarm-boot". To build the kernel, cd to ccl/lisp-kernel/androidarm and edit the Makefile. Near the top are 3 lines: NDK = /usr/local/android-ndk-r6b HOST = darwin-x86 ANDROIDVERSION = android-9 As mentioned above, I haven't tried the relatively new ndk-r7 yet and don't know if other things in the Makefile or kernel sources would need to change. If you have r7 installed (or have r6b installed in a different directory), you may need to change the definition of NDK; if you're using a different host machine, you'd need to change the HOST definition (I don't remember how Linux or Windows would be referenced here, but there can't be too many possibilities.) The definition of ANDROIDVERSION as android-9 selects an Android 2.3 (Gingerbread) set of C headers; at the fairly low level that the CCL kernel operates, there probably aren't too many differences between android-8 (2.2, Froyo) and later versions. Knock wood. Doing: shell on host> cd ccl/lisp-kernel/androidarm shell on host> make should produce the CCL kernel as two files: ccl/aarmcl.so is the CCL kernel as a shared library ccl/aarmcl is a tiny program that loads the shared library (from the same directory it's in) and calls into that library. The idea here is that (somewhere down the road) the little loader program could be replaced by a Java program which loads the kernel library (which loads the heap image) as part of a standard Android application package. (Like I said, that's somewhere down the road.) Once you have all of this stuff (a kernel, a bootstrapping image, and a set of .aafsl files) in the host's CCL directory, the next step is to copy it to the device. Assuming that you have an "androidccl" directory, one way to do that is: shell on host> cd ccl shell on host> cp aarm* /path/to/androidccl shell on host> tar cf - `find . -name '*.aafsl'` | tar cvf - -C /path/to/androidccl and then shell on host> cd /path/to/androidccl shell on host> adb push . /data/local/ccl and then shell on host> adb shell $ cd /data/local/ccl $ ./aarmcl aarm-boot ;Loading level-1.aafsl ;Loading ./l1-fasls/l1-cl-package.aafsl ;Loading ./l1-fasls/l1-utils.aafsl ;Loading ./l1-fasls/l1-init.aafsl [...] ;Loading /data/local/ccl/bin/jp-encode.aafsl ;Loading /data/local/ccl/bin/cn-encode.aafsl ;Loading /data/local/ccl/library/lispequ.aafsl ? (save-application "aarmcl.image") $ ./aarmcl Welcome to Clozure Common Lisp Version 1.8-dev-r15086M-trunk (AndroidARM32)! ? (quit) --------------- That's a lot of verbiage to describe something that I suspect most people would find a little anticlimactic. I suspect that some people aren't at all interested in this and that others might be if things were further along. If anyone else was interested enough to read this I hope that it was understandable and if anyone had a chance to try it I'd be interested in knowing what worked and what didn't.