Home > Android > Android Source Tree building environment

Android Source Tree building environment

This entry tries to cover howto compile the complete Android environment from scratch. It includes how to download the source code of all packages, building and images generation. For further information you can follow the official guide.

Fullfill the requirements

Install Java (Sun version).

$ sudo add-apt-repository ppa:sun-java-community-team/sun-java6 
$ sudo apt-get update 
$ sudo apt-get install sun-java6-jdk

Install the required libraries.

 $ sudo apt-get install git-core gnupg flex bison gperf build-essential \ 
zip curl zlib1g-dev libc6-dev lib32ncurses5-dev ia32-libs \ 
x11proto-core-dev libx11-dev lib32readline5-dev lib32z-dev \ 
libgl1-mesa-dev g++-multilib mingw32 tofrodos 

It is recommend you enable multiuniverse Ubuntu repository, which can be done using Synaptic or modifying file /etc/apt/sources.list

Install valgrind. You’ll never know when you want to have a good tool for debugging 😉

$ sudo apt-get install valgrind

Some of the maybe are not necessary but this is what is said on [1].

Android Source Tree download

Now it is time to download the Android source tree from repository.

You will need the repo application.

$ cd ~ $ mkdir bin 
$ curl http://android.git.kernel.org/repo > ~/bin/repo 
$ chmod a+x ~/bin/repo 

In Ubuntu directory ~/bin is automatically included in PATH.

Download the source code using the repo application.

$ mkdir mydroid 
$ cd mydroid 
$ repo init -u git://android.git.kernel.org/platform/manifest.git 

You can use both the the git:// or the http:// and even there is a possibility to set the $GIT_SSH (man git for more information).

In order to download an specific branch or tag:

$ repo init -u git://android.git.kernel.org/platform/manifest.git -b android-2.3.4_r1 

Although there is a 2.3.5_r1 at the repository that is running on a Nexus S 4G for the moment let’s try with 2.3.4_r1 till for Nexus S the 2.3.5 is officially released.

$ repo sync 

When executing this command download starts, but you’ll never know when it ends. Be patient as it takes quite a long time.

Note: On August-September 2011, kernel.org was hacked and the repo links were down for some days. If the problem still persists follow the instructions on [12]

In order to get some tips on repo (change branch, increase sync speed), just click.

Load the following public key into your GnuPG key database. The key is used to sign annotated tags that represent releases.

 $ gpg --import 

Copy and paste the key(s) below, then enter EOF (Ctrl-D) to end the input and process the keys.

-----BEGIN PGP PUBLIC KEY BLOCK----- 
Version: GnuPG v1.4.2.2 (GNU/Linux)
mQGiBEnnWD4RBACt9/h4v9xnnGDou13y3dvOx6/t43LPPIxeJ8eX9WB+8LLuROSV
lFhpHawsVAcFlmi7f7jdSRF+OvtZL9ShPKdLfwBJMNkU66/TZmPewS4m782ndtw7
8tR1cXb197Ob8kOfQB3A9yk2XZ4ei4ZC3i6wVdqHLRxABdncwu5hOF9KXwCgkxMD
u4PVgChaAJzTYJ1EG+UYBIUEAJmfearb0qRAN7dEoff0FeXsEaUA6U90sEoVks0Z 
wNj96SA8BL+a1OoEUUfpMhiHyLuQSftxisJxTh+2QclzDviDyaTrkANjdYY7p2cq
/HMdOY7LJlHaqtXmZxXjjtw5Uc2QG8UY8aziU3IE9nTjSwCXeJnuyvoizl9/I1S5
jU5SA/9WwIps4SC84ielIXiGWEqq6i6/sk4I9q1YemZF2XVVKnmI1F4iCMtNKsR4
MGSa1gA8s4iQbsKNWPgp7M3a51JCVCu6l/8zTpA+uUGapw4tWCp4o0dpIvDPBEa9
b/aF/ygcR8mh5hgUfpF9IpXdknOsbKCvM9lSSfRciETykZc4wrRCVGhlIEFuZHJv
aWQgT3BlbiBTb3VyY2UgUHJvamVjdCA8aW5pdGlhbC1jb250cmlidXRpb25AYW5k
cm9pZC5jb20+iGAEExECACAFAknnWD4CGwMGCwkIBwMCBBUCCAMEFgIDAQIeAQIX
gAAKCRDorT+BmrEOeNr+AJ42Xy6tEW7r3KzrJxnRX8mij9z8tgCdFfQYiHpYngkI
2t09Ed+9Bm4gmEO5Ag0ESedYRBAIAKVW1JcMBWvV/0Bo9WiByJ9WJ5swMN36/vAl
QN4mWRhfzDOk/Rosdb0csAO/l8Kz0gKQPOfObtyYjvI8JMC3rmi+LIvSUT9806Up
hisyEmmHv6U8gUb/xHLIanXGxwhYzjgeuAXVCsv+EvoPIHbY4L/KvP5x+oCJIDbk
C2b1TvVk9PryzmE4BPIQL/NtgR1oLWm/uWR9zRUFtBnE411aMAN3qnAHBBMZzKMX
LWBGWE0znfRrnczI5p49i2YZJAjyX1P2WzmScK49CV82dzLo71MnrF6fj+Udtb5+
OgTg7Cow+8PRaTkJEW5Y2JIZpnRUq0CYxAmHYX79EMKHDSThf/8AAwUIAJPWsB/M
pK+KMs/s3r6nJrnYLTfdZhtmQXimpoDMJg1zxmL8UfNUKiQZ6esoAWtDgpqt7Y7s
KZ8laHRARonte394hidZzM5nb6hQvpPjt2OlPRsyqVxw4c/KsjADtAuKW9/d8phb
N8bTyOJo856qg4oOEzKG9eeF7oaZTYBy33BTL0408sEBxiMior6b8LrZrAhkqDjA
vUXRwm/fFKgpsOysxC6xi553CxBUCH2omNV6Ka1LNMwzSp9ILz8jEGqmUtkBszwo
G1S8fXgE0Lq3cdDM/GJ4QXP/p6LiwNF99faDMTV3+2SAOGvytOX6KjKVzKOSsfJQ
hN0DlsIw8hqJc0WISQQYEQIACQUCSedYRAIbDAAKCRDorT+BmrEOeCUOAJ9qmR0l
EXzeoxcdoafxqf6gZlJZlACgkWF7wi2YLW3Oa+jv2QSTlrx4KLM= =Wi5D
-----END PGP PUBLIC KEY BLOCK----- 

To check the tag list or the available branches enter a project directory and run

$ git tag list 

Repo Tips

repo init

If you want to download and specific branch or tag, you can. For the Android 2.3.4 use

$ repo init -u git://android.git.kernel.org/platform/manifest.git -b android-2.3.4_r1 

If once you have downloaded the repository you want to change from git protocol to http or the other way round, you can edit the file ./repo/manifest.xml.

<?xml version="1.0" encoding="UTF-8"?>
<manifest>
  <remote  name="korg"
           fetch="http://android.git.kernel.org/"
           review="review.source.android.com" />
  <default revision="master"
           remote="korg" />

  <project path="build" name="platform/build">
    <copyfile src="core/root.mk" dest="Makefile" />
  </project>
  <project path="bionic" name="platform/bionic" />
  <project path="bootable/bootloader/legacy" name="platform/bootable/bootloader/legacy" />
  <project path="bootable/diskinstaller" name="platform/bootable/diskinstaller" />
  <project path="bootable/recovery" name="platform/bootable/recovery" />
....
</manifest>

In this file you can also find which revision branch you are using and the projects that are being downloaded.

For more information:

$ repo init --help 

repo sync

Thanks to Koba for his nice post entry on repo tips.

In order to reduce the downloading time, similar to make -jN option to run muliple jobs in parallel, repo sync has the same utilites.

Let’s examine the command help

repo sync --help
Usage: repo sync [...]

Options:
  -h, --help            show this help message and exit
  -f, --force-broken    continue sync even if a project fails to sync
  -l, --local-only      only update working tree, don't fetch
  -n, --network-only    fetch only, don't update working tree
  -d, --detach          detach projects back to manifest revision
  -q, --quiet           be more quiet
  -j JOBS, --jobs=JOBS  number of projects to fetch simultaneously
  -s, --smart-sync      smart sync using manifest from a known good build

  repo Version options:
    --no-repo-verify    do not verify repo source code

So, to fetch the repository using 4 jobs in parallel:

$ repo sync -j 4 

You will notice an speed increase, but be aware not to jeopardize your connection bandwidth.

Compile the source

Initilize the environment

$ cd ~/mydroid 
$ source build/envsetup.sh 
including device/htc/passion/vendorsetup.sh 
including device/samsung/crespo4g/vendorsetup.sh 
including device/samsung/crespo/vendorsetup.sh 
including sdk/bash_completion/adb.bash 

It is time to try to compile stuff. Let’s try something quick like adb and fastboot

$ make adb fastboot 

Output files are in

 out/host/linux-x86/bin/adb 
 out/host/linux-x86/bin/fastboot 

Choose the target

There are many possible build options:

  • The user build is meant to provide the appropriate security safeguards.
  • The userdebug build tries to be as close as possible to user builds, with just the minimal changes to allow debugging (e.g. logging, attaching a debugger to any process, root access). It’s a good build for dogfooding and beta-testing.
  • The eng build is allowed to diverge further from the user builds, with additional tools and settings to help the development tasks, with the understanding that the differences can and do create behaviors that aren’t the same as user builds.

Users should get user builds, not userdebug or eng (and you should be sure to sign the official releases with a secret key). In our case, the eng build is used as to get the most information for this very first steps.

Other building options refer to the destination of the build, the target, emulator or phone:

Build name Device Notes
generic emulator lowest-common denominator
full emulator fully configured with all languages, apps, input methods
full_crespo crespo full build running on Nexus S (“crespo”)

and the BUILDTYPE is one of the following:

Buildtype Use
user limited access; suited for production
userdebug like “user” but with root access and debuggability; preferred for debugging
eng development configuration with additional debugging tools

When the build options are selected run:

$ lunch full_crespo-eng 
        BUILD-BUILDTYPE
============================================
PLATFORM_VERSION_CODENAME=AOSP
PLATFORM_VERSION=3.1.4.1.5.9.2.6.5
TARGET_PRODUCT=full_crespo
TARGET_BUILD_VARIANT=eng
TARGET_SIMULATOR=false
TARGET_BUILD_TYPE=release
TARGET_BUILD_APPS=
TARGET_ARCH=arm
TARGET_ARCH_VARIANT=armv7-a-neon
HOST_ARCH=x86
HOST_OS=linux
HOST_BUILD_TYPE=release
BUILD_ID=OPENMASTER
============================================

In case using a specific tag or branch the output would indicate so

$ lunch full_crespo-eng 
============================================
PLATFORM_VERSION_CODENAME=REL
PLATFORM_VERSION=2.3.4
TARGET_PRODUCT=full_crespo
TARGET_BUILD_VARIANT=eng
TARGET_SIMULATOR=false
TARGET_BUILD_TYPE=release
TARGET_BUILD_APPS=
TARGET_ARCH=arm
HOST_ARCH=x86
HOST_OS=linux
HOST_BUILD_TYPE=release
BUILD_ID=GRJ22
============================================

Propietary drivers’ binaries

While it’s possible to build and run a system using exclusively source code from Android Open-Source Project, such a system can only use the devices’ hardware capabilities for which Open-Source support exists.

Official binaries for Nexus S and Nexus S 4G can be downloaded from Google’s Nexus driver page, which add access to additional hardware capabilities with non-Open-Source code. Follow the instructions on that page, which are:

  • Download the apropiate drivers
  • Uncompress them in your droid root directory
  • Execute extract-*.sh commands
  • Your files will be available at the vendor directory.

If you want to get the ones running in your phone, plug your phone to the computer and

$ cd ~/mydroid/device/samsung/crespo
$ ./extract-files.sh

Compilation

Run make to compile. It is recommended you use the -j option to speed up the process using the full power of your computer.

$ make -j2

Another option could be in case you haven’t run the lunch utility is to directly set the make variables in the command line.

TARGET_PRODUCT=full_crespo TARGET_BUILD_VARIANT=user TARGET_BUILD_TYPE=debug make

Now be patience and cross-fingers everything goes well. At the beginning there are several warning, specially when compiling some Java code (I guess Dalvik VM).
In my Core Duo 1.2GHz it took something like 22 hours.

If at the end of the process you want to know how long compilation took, just run

$ time make -j8 2>&1 | tee make.log

Errors during compilation

Everything can go smothly or as it always happens something goes wrong.

If you get a compilation error due to source code, you are on your own with your favourite search engine.

You can also get an error on the Android.mk files, like the one following.


build/core/base_rules.mk:78: *** Module name: jsilver
build/core/base_rules.mk:79: *** Makefile location: external/jsilver
build/core/base_rules.mk:80: * 
build/core/base_rules.mk:81: * Each module must use a LOCAL_MODULE_TAGS in its
build/core/base_rules.mk:82: * Android.mk. Possible tags declared by a module:
build/core/base_rules.mk:83: * 
build/core/base_rules.mk:84: *     optional, debug, eng, tests, samples
build/core/base_rules.mk:85: * 
build/core/base_rules.mk:86: * If the module is expected to be in all builds
build/core/base_rules.mk:87: * of a product, then it should use the
build/core/base_rules.mk:88: * "optional" tag: 
build/core/base_rules.mk:89: * 
build/core/base_rules.mk:90: *    Add "LOCAL_MODULE_TAGS := optional" in the
build/core/base_rules.mk:91: *    Android.mk for the affected module, and add
build/core/base_rules.mk:92: *    the LOCAL_MODULE value for that component
build/core/base_rules.mk:93: *    into the PRODUCT_PACKAGES section of product
build/core/base_rules.mk:94: *    makefile(s) where it's necessary, if
build/core/base_rules.mk:95: *    appropriate.
build/core/base_rules.mk:96: * 
build/core/base_rules.mk:97: * If the component should be in EVERY build of ALL
build/core/base_rules.mk:98: * products, then add its LOCAL_MODULE value to the
build/core/base_rules.mk:99: * PRODUCT_PACKAGES section of
build/core/base_rules.mk:100: * build/target/product/core.mk
build/core/base_rules.mk:101: * 
build/core/base_rules.mk:102: *** user tag detected on new module - user tags are only supported on legacy modules.  Stop.

This is due to the declaration of LOCAL_MODULE_TAGS. To fix it just follow the instructions that suggest including the optional tag. Thanks to [10] for the command line that replace user tag by optional tag on every affected file.

$ find ./ -exec grep -l "LOCAL_MODULE_TAGS := user" {} \; -exec sed -i.bak s/"LOCAL_MODULE_TAGS := user"/"LOCAL_MODULE_TAGS := optional"/g {} \;

Flash and test the new images

To flash the phone, execute:

$ ./out/host/linux-x86/bin/adb reboot bootloader
$ ANDROID_PRODUCT_OUT=out/debug/target/product/crespo ./out/host/linux-x86/bin/fastboot -w flashall

If flashall fails:

$ ./out/host/linux-x86/bin/fastboot flash boot out/debug/target/product/crespo/boot.img
$ ./out/host/linux-x86/bin/fastboot flash system out/debug/target/product/crespo/system.img
$ ./out/host/linux-x86/bin/fastboot flash userdata out/debug/target/product/crespo/data.img

Note: Skip flashing the boot partition if you don’t want to replace the existing kernel. The new boot partition has to be linked with the kernel image. This will be part of another post where how to generate a Nexus S kernel image will be done.

Reboot the system and enjoy the just created version

$ ./out/host/linux-x86/bin/fastboot reboot

If there are problems while booting it’s worth to try

$ ./out/host/linux-x86/bin/fastboot erase userdata

instead of flashing the user partition.

References

[1] Initializing a Build Environment | Android Open Source
[2] [How-To] compile the Nexus S kernel from source – Android Forums
[3] Download Sourcery CodeBench Lite Edition for ARM
[4] Building Kernel from source – CyanogenMod Wiki
[5] HOWTO: Unpack, Edit, and Re-Pack Boot Images – Android Wiki
[6] Toolbox – Android Development
[7] Complete How-To Root, Busybox, Flash/Recovery, Framework-res.apk, custom boot ani…
[8] rooting – Is there a way for me to run Adb shell as root without typing in `su`? – Android – Stack Exchange
[9] Cyanogenmod on Nexus S | Adriel Blog
[10] Compile Android Gingerbread on Ubuntu 10.10 Journal
[11] Tips of “repo sync” of Android « Koba’s blog
[12] Kernel.org hacked – how to get Android repo? « mindexception: out of context

Advertisements
Categories: Android Tags: ,
  1. May 7, 2013 at 13:40

    Thank you for this super article. Will now more stop by. Greetings from Cologne

  1. No trackbacks yet.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: