Ecosyste.ms: Awesome

An open API service indexing awesome lists of open source software.

Awesome Lists | Featured Topics | Projects

https://github.com/mid1221213/rakudroid

Port of Rakudo on Android - PoC - WIP
https://github.com/mid1221213/rakudroid

android cross-compilation moarvm perl6 port proof-of-concept raku rakudo work-in-progress

Last synced: 4 months ago
JSON representation

Port of Rakudo on Android - PoC - WIP

Awesome Lists containing this project

README 

        
# RakuDroid



RakuDroid is a PoC.



RakuDroid is a WIP. Please read this file entirely before doing anything.



RakuDroid's goal is to port Raku (a.k.a. Perl6) on Android platform.



The first work has been to cross compile Rakudo / MoarVM for Android. It uses the latest release of both at the time of writing, i.e. `2019.11`. This has been possible with just 2 patches to MoarVM. The first makes the cross-compilation possible with Android SDK/NDK (see `src/librakudroid/0001-Make-MoarVM-cross-compile-nicely-for-Android.patch`), the second allows jit to work but, of course, only on x86_64 (see `src/librakudroid/0001-Make-MoarVM-cross-compile-w-jit-nicely-for-Android.patch`).



Next, work has been done to write JNI stuff, Raku helpers and to automatically generate bindings to interact with the Java side of the Android system.



## Supported platforms



* Build (cross compilation host): Linux x86_64 (I personally use Ubuntu, not tested on other distros but should work)

* Targets:

  * Architectures:

	* `x86_64` (typically the Android emulator for testing)

	* `aarch64` (arm 64-bit, most new smartphones)

  * Android versions: 8.0 (Oreo) and above (API 26+)



## Requirements for building



* Internet access (to download Rakudo and MoarVM)

* `git` and standard build tools (`sudo apt install build-essential`)

* An Android device **connected** (can be the emulator) and reachable through `adb` (only one at a time), because cross-compilation tries to guess things by executing code on the device

* Android Studio + Android SDK / NDK (default Makefile config is for installation of these in default paths, install SDK & NDK from Android Studio) - please refer to Android documentation



## Building



To first check if everything seems to be OK, and look at customizable variables:



	make check



To build:



	make install-precomp



To specify a different arch than the default (which is x86_64):



	make ARCH=aarch64 check



### Customizable variables



The configuration variables can be passed as in the example above to the make command (no need to hack the `Makefile`). The list of these variables and their current default value is:



```make

ARCH           ?= x86_64

API_VERSION    ?= 26

ANDROID_SDK    ?= ${HOME}/Android/Sdk

RELEASE        ?= 2019.11

DBG_CFLAGS     ?= -g -O0

PROJ_JAVA_PATH ?= com/example/myapplication

```



### Makefile targets



These targets usable with `make`:

* `help` (default): displays the file you're currently reading (`README.md`)

* `check`: makes somes sanity checks about configuration and variables. Tests `adb` availability, NDK's cross compiler availability, presence and arch of connected device, etc… It also prints the variables actual values (so you can customize them when calling `make check`)

* `clean`: cleans up RakuDroid target directory (`app`) and generated files (`gen` & `gen.touch`)

* `clean-all`: cleans up everything and the resulting directory structure should become the one you freshly installed

* `clean-arch`: cleans up what needs to be rebuilt for a different arch (to switch between them)

* `all`: builds everything

* `install`: make a `gzip`'ed tarball of the complete `app` directory with all needed files in it. You can actually use this target directly after `make check` to build everything

* `install-precomp`: like `make install` but does the precompilation phase. **You must use this target the first time, and only once**. Unless you modify the files precompiled, you should only need to `make install` afterwards, e.g. when changing arch



## Building process explained



MoarVM is arch dependant.



Rakudo is not completely arch independant. There is (for what we need) one shared library that is built with it: `dynext/libperl6_ops_moar.so`. That `dynext/` part is a bit annoying because it prevents the library to be installed in `jniLibs/$(JNI_ARCH)/` like the others because the Android system does not support subdirectories in that directory. We are then obliged to install it elsewhere. It also prevents the use of multi-arch building in Android Studio. This is why you must, when changing the arch, issue a `make clean-arch` then, after having switched the connected device, rebuild with `make install`.



When building the following occurs (in an order decided by `make`):

* Rakudo is "`git clone`'d" from github at the release specified in the `RELEASE` variable. It is then built with no special processing, as it would be built for the host (it is)

* MoarVM is "`git-clone`'d" from github at the release specified in the same `RELEASE` variable. It is then patched to allow cross compiling for Android. It is at this very moment that the device (be it a real device or an emulator) is used. Only **one** device should be connected and reachable with `adb`

* `libperl6_ops_moar.so` is cross compiled using the source in Rakudo

* `librakudroid.so` is cross compiled from the source in `src/librakudroid/`

* The file `android.jar` is extracted, then `javap` is used on it to extract the class definitions and signatures. The resulting file is then parsed (using an ugly Perl 5 script that should be rewritten in Raku, especially for the regexes ;-)) to generate the bindings

* If you typed a `make install-precomp`, there is a precompilation phase. Most RakuDroid modules are precompiled, and some generated bindings (the ones needed for startup, i.e. `Activity.pm6` and its dependencies), along with the big sized `RakuDroidRoles.pm6`

* As a final step, the `app` directory is created and populated, then `tar.gz`-ipped



## Installation and test



As said before, this is a PoC. It is actually the second one.



The first goal was to have an Android application that, when launched, displays a kind or REPL. You entered the expression to eval, then clicked on "Eval", and the result (returned value) was displayed. The button "Extract Assets" was here in case the asset extraction has been interrupted at the beginning (should not happen, and this button is actually useless). Be careful, the evaluation step of REPL is done independantly each time. That means that a "`my`" variable is lost between evaluations. An "`our`" variable is kept, but you need to use it with the special package name `RakuDroidHelper`, e.g. `$RakuDroidHelper::my-var` at 2nd and forth times.



This is the second version of the PoC. It brings the Android system bindings. The variable `$RakuDroidHelper::main-activity` holds the activity object displayed and you can use the pre-filled text just by clicking "Eval" to display an Android Toast. More info will be given later in a separate file on how bindings are done and their use.



Once you have successfully `make install-precomp`'ed (that takes about 7mn on my machine), you should end up with a file `MyApplication.tgz`. Launch Android Studio, select « New Project » and use the `Native C++` activity template. Keep the default settings (especially the ID of the application `com.example.myapplication`, or else overwrite the `PROJ_JAVA_PATH` variable), except for Android version to use, it should be **at least** Oreo (8.0).



Once this is done, in a shell, `cd` to the new application directory (which should be, using all defaults, `${HOME}/AndroidStudioProjects/MyApplication/`), then issue a `tar -xzvf /path/to/RakuDroid/MyApplication.tgz`.



Last step, in Android Studio (which should have detected and refreshed all changed files), click on "run" (the green triangle) and voilà…



The application startup time is, on the emulator on my PC, about 10 seconds. On my smartphone it is about 7 seconds. This may seem long, but in my first tests it was about 13 minutes (yes, minutes !) on the emulator. Maybe this can be improved more.



## Additional details / FAQ



### What does the patch do?



The current MoarVM cross-compilation system:

* makes assumtions on the target instead of trying to compile / execute tests executables

* does not support the Android architecture at all

  * Android uses "bionic" as libc

  * first tests using gcc cross compiler on command line failed. Using Android Studio-installed SDK / NDK was the solution. These could have been installed from the command line too, although I haven't tested because I needed Android Studio anyway.



The 1st patch modifies the following 4 files in the MoarVM source tree:

* `build/probe.pm`:

  * add a `sub run` which will be used instead of `system()` (or the backtick operator) calls when trying to run a test result. The sub replaces seamlessly (I hope so, not tested at the moment) the normal test runs when no `try_run` is defined. When there is one, it replaces it with a `try_cp` (to copy the executable to test on the target tempdir) followed by an effective call to `try_run` to run the test, and captures and returns the result back to the caller

  * replace all calls to `system()` and backticks with the appropriate call to `run()`

* `build/setup.pm`:

  * add a `%TC_ANDROID` into `%TOOLCHAINS` list. It declares the compiler family as 'android' (actually that means `clang` with "bionic", that is set elsewhere) and adds a rpath for finding libs in relocatable versions

  * add a `android` key in `%COMPILERS`, leading to parameters suited for it, including the `try_cp` and `try_run` mentioned above

  * add a `android` key in `%SYSTEMS`, leading to an `%OS_ANDROID` containing a `syslibs` overwrite and that changes a `-thirdparty`/`uv` variable (`UV_ANDROID`, added in `Makefile.in`, see below, instead of `UV_LINUX`)

* `build/Makefile.in`: adds the `UV_ANDROID` variable, adding `3rdparty/libuv/src/unix/pthread-fixes@obj@` to the list of objects needed by the build. It helps with some missing thread functions in bionic (no `libpthread.so` on Android)

* `Configure.pl`:

  * bypass the "guessed" configuration based on crossconf detection if `try_run` is set

  * patch the `sub setup_cross` (this is where I don't know if the patch is good in all situations) to:

	* detect system triplets ending with `-android(\d+)`, and remove the API version (the number) from it. That number is not used elsewhere, it is just here to detect and set the Android SDK compilation environment

	* do some trickery to get the configuration work correctly (need to test that on another cross-compiled thing to check if nothing is broken)



The 2nd patch modifies the following 3 files in the MoarVM source tree:

* `Configure.pl`: add a variable `cc_host` (containing the cross-compilation build system compiler)

* `build/Makefile.in`: set a Make variable from the above, and use that for (not cross) compiling `3rdparty/dynasm/minilua.c`

* `build/setup.pm`: set the above variable to `gcc` when compiling for Android



### Why Oreo?



During my tests I've been stuck a while trying to generate a shared library you can also call as an (almost) regular executable. It seems that before Oreo, the linker was not sending `argc` and `argv` to the functions declared in the section `.init_array`, which I use. I could have used some nasty trickery with the stack there, but I prefered not to. So the dependancy to at least Android API 26 a.k.a. Oreo a.k.a 8.0.