Building/Debugging android native C applications

In this post I will explain how to compile, install and debug an Android native "C" application.
If you are reading this post just because you have googled the magic keywords ("android" + "native code") then you should know that there is an easier way to build native applications using android makefiles ("Android.mk" and "Application.mk").
The method I'm describing here is only useful if you want to understand how things work in order to create more complex standard GNU makefiles. This is also useful if you would like to create your own GNU autotools wrappers to compile projects using GNU configure.
I'm using Windows Vista as host machine but any other supported platforms (e.g. linux-x86 or darwin-x86) should work.

I have tested both the NDK (1.6) and SDK (2.1) on:

• Windows XP (32-bit) and Vista (64-bit)
  
• Mac OS X Snow Leopard
  
• Ubuntu Intrepid
  

Installing Android SDK

To download the latest Android SDK, visit this address http://developer.android.com/sdk/index.html.
If you need information on how to install the SDK, visit this address http://developer.android.com/sdk/installing.html.
If the "SDK setup" fail to update the installed packages you can change the remote site URL from https://dl-ssl.google.com/android/repository/repository.xml to http://dl-ssl.google.com/android/repository/repository.xml (change the URL scheme from HTTPS to HTTP) or try to disable your anti-virus or firewall.

I have installed the SDK version 2.1 under c:/android-sdk (a.r.a /cygdrive/c/android-sdk).
Add an environment variable named ANDROID_SDK_ROOT pointing to the SDK root directory.

Important: You should add "$ANDROID_SDK_ROOT/tools" directory to the $PATH environment variable.
Under *nix:

export PATH=$ANDROID_SDK_ROOT/tools:$PATH

Under Cygwin: Open C:\Cygwin\Cygwin.bat and add:

set PATH=%ANDROID_SDK_TOOLS%;%PATH%

Installing Cygwin

If you are using Windows XP or Vista as host machine then you MUST install Cygwin Devel package with GNU Make (3.81 or later) before installing the NDK.
It should also work with MinGW.

Installing the Android NDK

To download the latest Android NDK, visit this address http://developer.android.com/sdk/ndk/1.6_r1/index.html.
I have uncompressed the NDK version 1.6 under c:/android-ndk (a.r.a /cygdrive/c/android-ndk).
Add an environment variable named ANDROID_NDK_ROOT pointing to the NDK root directory.
To install the NDK:

cd $ANDROID_NDK_ROOT
build/host-setup.sh

If all is OK then the console will print Host setup complete.

To test that the toolchain has been correctly installed you can try to build the hello-jni sample which comes with the NDK by doing this:

cd $ANDROID_NDK_ROOT
make -APP=hello-jni

If all is OK then the console will print:

Android NDK: Building for application 'hello-jni'
Compile thumb : hello-jni <= sources/samples/hello-jni/hello-jni.c SharedLibrary : libhello-jni.so Install : libhello-jni.so => apps/hello-jni/project/libs/armeabi

This mean that your native shared library (libhello-jni.so) have been successfully generated under $ANDROID_NDK_ROOT/apps/hello-jni/project/libs/armeabi folder.

Creating an AVD

AVD stands for Android Virtual Device and can be seen as a device profile (keyboard, dialing pad, skin, screen dimensions, appearance ...) to load into your emulator. You can create as many AVDs as you need.
To create an AVD named "avdtest" targeting platform 2.1 (targetID=android-7):

android create avd -n avdtest -t android-7

If all is OK the console will print:

Created AVD 'avdtest' based on Android 2.1, with the following hardware config: hw.lcd.density=160

Create test.c

Here I will create a basic test.c file under C:\tmp with the following content:

#include <stdio.h>// printf

int main(int argc, char **argv)
{
int i = 1;
i+=2;

printf("Hello, world (i=%d)!\n", i);

return 0;
}

Create makefile

Just create an empty file named makefile (without any extension) under C:\tmp (which is the same directory as test.c).
Now We will fill the makefile step by step.

Add application name, $ROOT directory, install directory and the NDK platform version:

APP := test
ROOT:=/cygdrive/c
NDK_PLATFORM_VER := 1.5
INSTALL_DIR := /data/tmp

Add useful environment vars:

ANDROID_NDK_ROOT:=$(ROOT)/android-ndk
ANDROID_NDK_HOST:=windows
ANDROID_SDK_ROOT:=$(ROOT)/android-sdk
PREBUILD:=$(ANDROID_NDK_ROOT)/build/prebuilt/$(ANDROID_NDK_HOST)/arm-eabi-4.2.1
BIN := $(PREBUILD)/bin

You MUST change ANDROID_NDK_HOST value from windows to linux-x86 if you are under *nix or darwin-x86 on MAC OS X.

Add GCC options:

CPP := $(BIN)/arm-eabi-g++
CC := $(BIN)/arm-eabi-gcc
CFLAGS :=
LDFLAGS := -Wl

Add targets

all: $(APP)

OBJS += $(APP).o

$(APP): $(OBJS)

$(CPP) $(LDFLAGS) -o $@ $^

%.o: %.c

$(CC) -c $(INCLUDE) $(CFLAGS) $< -o $@

install: $(APP)

$(ANDROID_SDK_ROOT)/tools/adb push $(APP) $(INSTALL_DIR)/$(APP)

$(ANDROID_SDK_ROOT)/tools/adb shell chmod 777 $(INSTALL_DIR)/$(APP)

shell:

$(ANDROID_SDK_ROOT)/tools/adb shell

run:

$(ANDROID_SDK_ROOT)/tools/adb shell $(INSTALL_DIR)/$(APP)

clean:

@rm -f $(APP).o $(APP)

Building the application

To build the application, switch to the directory where you have created both files and then:

make

At the output of the console you will get many errors saying that it's impossible to find stdlib.h, stdio.h etc etc.
To resolve this issue, add the Bionic header files to $CFLAGS variable like this:

CFLAGS := -I$(ANDROID_NDK_ROOT)/build/platforms/android-$(NDK_PLATFORM_VER)/arch-arm/usr/include

If you retry (make) you will now get this link error:

crt0.o: No such file: No such file or directory

To avoid directly linking against the "C runtime" you must add "-nostdlib" flag to the link options like this:

LDFLAGS := -Wl -nostdlib

If you retry (make) you will now get these link errors:

test.c:(.text+0x34): undefined reference to `printf'
test.c:(.text+0x3c): undefined reference to `exit'

You get these errors because Bionic libc is missing. To add libc you MUST change $LDFLAGS like this:

LDFLAGS := -Wl -L$(ANDROID_NDK_ROOT)/build/platforms/android-$(NDK_PLATFORM_VER)/arch-arm/usr/lib
LDFLAGS += -nostdlib -lc

If you retry (make) you will now get this link error:

/cygdrive/c/android-ndk/build/platforms/android-1.5/arch-arm/usr/lib/libc.so: undefined reference to `dl_unwind_find_exidx'

To resolve this issue you MUST specify the first set of directories into which to search the system shared libraries (*.so) . This is done by adding the "-rpath-link" option to the link options like this:

LDFLAGS := -Wl,-rpath-link=$(ANDROID_NDK_ROOT)/build/platforms/android-$(NDK_PLATFORM_VER)/arch-arm/usr/lib -L$(ANDROID_NDK_ROOT)/build/platforms/android-$(NDK_PLATFORM_VER)/arch-arm/usr/lib

If you retry (make) you will now get this warning:

/cygdrive/c/android-ndk/build/prebuilt/windows/arm-eabi-4.2.1/bin/../lib/gcc/arm
-eabi/4.2.1/../../../../arm-eabi/bin/ld: warning: cannot find entry symbol _start; defaulting to 000082c8

This is an Android known issue. You have this warning because the linker search "_start" as entry point. You can resolve this issue by renaming your main function. But the elegant way to resolve this issue is to specify the entry point in the link options like this:

LDFLAGS := -Wl,--entry=main,-rpath-link=$(ANDROID_NDK_ROOT)/build/platforms/android-$(NDK_PLATFORM_VER)/arch-arm/usr/lib -L$(ANDROID_NDK_ROOT)/build/platforms/android-$(NDK_PLATFORM_VER)/arch-arm/usr/lib
LDFLAGS += -nostdlib -lc

Now When you retry (make) your application will successfully build without any errors or warnings.

Testing your application

Before testing your application you MUST run the emulator like this:

emulator -avd avdtest

where "avdtest" is the name of the previously created avd (see "creating an avd" section).
To install the application on the emulator, open a new console and go to to directory where you have created test.c and makefile. Install your application on the emulator like this:

make install

If all is OK the console will print:

/cygdrive/c/android-sdk/tools/adb push test /data/tmp/test
304 KB/s (2493 bytes in 0.008s)
/cygdrive/c/android-sdk/tools/adb shell chmod 777 /data/tmp/test

To run the application type:

make run

You will probably get an error message saying:

/cygdrive/c/android-sdk/tools/adb shell /data/tmp/test
/data/tmp/test: not found

This error message is a bit confusing because if you browse the /data/tmp directory you will notice that the executable is here. The question is why?
I spent hours searching and I found that this error happens because the loader fails to load the application because it cannot found a proper linker.
To specify a search directory for the dynamic linker (at run time) you MUST change the link options like this:

LDFLAGS := -Wl,--entry=main,-rpath-link=$(ANDROID_NDK_ROOT)/build/platforms/android-$(NDK_PLATFORM_VER)/arch-arm/usr/lib,-dynamic-linker=/system/bin/linker -L$(ANDROID_NDK_ROOT)/build/platforms/android-$(NDK_PLATFORM_VER)/arch-arm/usr/lib
LDFLAGS += -nostdlib -lc

Now rebuild and install your application (make clean && make && make install) then run it again (make run).
The console will print the expected result ("hello, world (i=3)!") but just after we have an segmentation fault error ("[1] Segmentation fault /data/tmp/test").
To resolve this issue you can exit the program (exit(0);) just before the main function returns (return 0;). You should also include <stdlib.h>.
If you retry the build&&run process (make clean && make && make install && make run) then you should have:

/cygdrive/c/android-sdk/tools/adb shell /data/tmp/test
Hello, world (i=3)!

which is the expected result.

Debugging your application
Before doing anything you MUST copy the gdbserver file to the emultor.
This file is under $BIN ($ANDROID_NDK_ROOT/build/prebuilt/$ANDROID_NDK_HOST/arm-eabi-4.2.1/bin).
Copy gdbserver to the emulator like this:

adb push gdbserver $INSTALL_DIR/gdbserver
adb shell chmod 777 $INSTALL_DIR/gdbserver

where $INSTALL_DIR is the directory where you have installed your application (it's not mandatory to copy it in this directory).

Before running the server on port 1234 you MUST redirect all tcp connection to this port like this:

adb forward tcp:1234: tcp:1234

it's not mandatory to forward connections to the same port number.
Now it's time to run the server:

adb shell $INSTALL_DIR/gdbserver :1234 $INSTALL_DIR/$APP

note that only the server port is specified (no host).
If all is OK the the server will print something like this:

Process /data/tmp/test created; pid = 246
Listening on port 1234

Now to debug our application we will change the makefile by adding a new debug target like this.

GDB_CLIENT := $(BIN)/arm-eabi-gdb

debug:

$(GDB_CLIENT) $(APP)

To launch the application in debug mode type "make debug" (after make clean && make && make install of course). If you do this, you will see a warning message saying that "no debugging symbols found". No symbols ==> no debug.
To generate debug symbols you MUST change the makefile like this (should not be hard coded like this):

DEBUG = -g
CFLAGS := $(DEBUG) -I$(ANDROID_NDK_ROOT)/build/platforms/android-$(NDK_PLATFORM_VER)/arch-arm/usr/include

Now rebuild and install your application (make clean && make && make install) then run it again (make debug). This (make debug) should open gdb invite command((gdb)) on the same console.
Connect to the server (from the same console) like this:

target remote :1234

Set a breakpoint on the main function and execute step by step (commands above are informational and you can use any gdb commands):

b main
c
n
p i
#$1 = 1
n
#9 printf("Hello, world (i=%d)!\n", i);
p i
#$2 = 3
c
#Program exited normally.

The final makefile and test.c files are shown below:

makefile

APP := test
ROOT:=/cygdrive/c
INSTALL_DIR := /data/tmp
NDK_PLATFORM_VER := 1.5

ANDROID_NDK_ROOT:=$(ROOT)/android-ndk
ANDROID_NDK_HOST:=windows
ANDROID_SDK_ROOT:=$(ROOT)/android-sdk
PREBUILD:=$(ANDROID_NDK_ROOT)/build/prebuilt/$(ANDROID_NDK_HOST)/arm-eabi-4.2.1
BIN := $(PREBUILD)/bin
GDB_CLIENT := $(BIN)/arm-eabi-gdb

DEBUG = -g

CPP := $(BIN)/arm-eabi-g++
CC := $(BIN)/arm-eabi-gcc
CFLAGS := $(DEBUG) -I$(ANDROID_NDK_ROOT)/build/platforms/android-$(NDK_PLATFORM_VER)/arch-arm/usr/include
LDFLAGS := -Wl,--entry=main,-rpath-link=$(ANDROID_NDK_ROOT)/build/platforms/android-$(NDK_PLATFORM_VER)/arch-arm/usr/lib,-dynamic-linker=/system/bin/linker -L$(ANDROID_NDK_ROOT)/build/platforms/android-$(NDK_PLATFORM_VER)/arch-arm/usr/lib
LDFLAGS += -nostdlib -lc

all: $(APP)

OBJS += $(APP).o

$(APP): $(OBJS)

$(CPP) $(LDFLAGS) -o $@ $^

%.o: %.c

$(CC) -c $(INCLUDE) $(CFLAGS) $< -o $@

install: $(APP)

$(ANDROID_SDK_ROOT)/tools/adb push $(APP) $(INSTALL_DIR)/$(APP)

$(ANDROID_SDK_ROOT)/tools/adb shell chmod 777 $(INSTALL_DIR)/$(APP)

shell:

$(ANDROID_SDK_ROOT)/tools/adb shell

run:

$(ANDROID_SDK_ROOT)/tools/adb shell $(INSTALL_DIR)/$(APP)

debug:

$(GDB_CLIENT) $(APP)

clean:

@rm -f $(APP).o $(APP)

test.c

#include <stdio.h> // printf
#include <stdlib.h> //exit

int main(int argc, char **argv)
{
int i = 1;
i+=2;

printf("Hello, world (i=%d)!\n", i);

exit(0);
return 0;
}

SIP/IMS and NAT traversal (Part 1)

This is the first part of a three-part series about NAT traversal solutions for SIP/IMS. I will begin from simple solutions (e.g. "rport" extension) and end with some more complex one (e.g ICE).

Today most Internet users are in a private network behind one or several NATs. This is not a problem for your browser (HTTP, FTP...) or email client (SMTP, POP...) because they use protocols that can operate behind NATs. The problem comes when you try to use your SIP phone to REGISTER or place a call to another softphone in different private network through Internet (Public network).
The problem with SIP is that IP addresses and Ports where to contact/respond an agent (e.g. PC, Box or mobile phone) are embedded in the SIP message itself.

For example, when you REGISTER to a registrar (e.g Serving-CSCF through Proxy-CSCF) you add a contact header (IP address and Port - Contact: <sip:alice@192.168.16.108:27208;transport=udp>) where the registrar should send all your incoming calls(INVITEs). This address is called Address-Of-Record (a.k.a AOR).
If you are in a private network you will add your private IP address and port in this header (Contact) and the problem is that this AOR is not visible to the elements outside your private network.
This mean that INVITE requests will never reach your agent.
You will have the same problem when you are the caller, as you will add in your SDP the IP addresses and ports where RTP packets shall be sent to you.

So, NAT problem concern both signaling (SIP/SDP) and media (RTP/RTCP) plans.

rport

This is the simplest way to deal with NATs problem at signalling (SIP only) plan for connectionless protocols (e.g. UDP). rport has been defined in RFC 3581 and apply to both connectionless (e.g. UDP) and connection-oriented (e.g. TCP, TLS, SCTP) protocols.

Because connection-oriented protocols such as TCP are bidirectional it is easier to deal with NAT traversal (all responses will be sent back using the same connection from which the request has been received from).

The philosophy (of rport) is that when you add this attribute (rport) in your requests (Via: SIP/2.0/UDP 192.168.16.108:27208;branch=z9hG4bK1234;rport) then all responses will be sent back to the ip:port from which the request has been received from. This parameter MUST be added in the top most Via header.

The response will contain a new parameter (« received ») containing the IP address from which the request has been received and the rport value will be filled with the mapped (public?) source port. In this way the client/caller (the sender of the request) can learn it's public/reflexive IP address and port (Via: SIP/2.0/UDP 192.168.16.108:27208;branch=z9hG4bK1234;rport=1234;received=10.1.1.1).

By examining the response the client can know if it's behind a NAT or not.

The problem with this solution is that it only deals with SIP messages and cannot be used for RTP/RTCP packets as the SDP will contain wrong IP addresses and ports. A solution to this issue could be using Symmetric RTP/RTCP (see above).

Symmetric RTP / RTP Control Protocol (RTCP)

As we have seen above (rport), we cannot use « rport » to deal with RTP/RTCP packets which are almost always transported using UDP. A solution to this problem could be using « Symetric RTP/RTCP » as per RFC 4961. This solution is a bit like using « rport ».

In this case the caller (INVITE originator) will create the request as per RFC 3261 as usual. When the 2xx (with SDP) or 1xx (with SDP) is received, the caller will ignore the IP addresses and ports defined in the response and send RTP/RTCP packets to IPs/Ports from which the RTP/RTCP packets have been sent (like rport).

To summarize:

1. Bob sends an INVITE to Alice
2. Alice sends back a 2xx (with SDP) or 1xx (with SDP) to Bob
3. Bob waits for first RTP/RTCP packets to come from Alice
4. Alice send first RTP/RTCP packest from IP-a and Port-a to IP-b and Port-b (Bob address and port defined in the SDP).
5. Instead of sending RTP/RTCP packets to IP-asdp and Port-asdp as sepecified in the Alice's SDP, Bob will begin sending media stream to IP-a and Port-a.

As you can imagine this solution only work for only some NATs.

Session border controller

As its name says, it controls (both media and SIP packets) sessions (SIP calls) and is in the border (between two networks) of the networks.

In our case (NAT traversal) the role of SBC will be to inspect/control all outgoing and incoming SIP/RTP/RTCP packets. All SIP packets will be inspected and IP addresses and ports within the packet will be rewritten (e.g from private IP to public IP).

This solution has several problems:

• Very expensive (€€$$££)
  

• As the SBC operates on the SIP packets then it MUST be aware of all headers (or functions) in order to know what should be changed and what should not be changed. This mean that the SBC MUST always be up to date in order to efficiently handle SIP packets

• Most SBCs can only handle well-know protocols such as SIP, RTP or RTCP and will drop all unknown protocols
  

• There is also many problems when End-to-End encryption (e.g. TLS, SRTP or IPSec) is used unless the SBC has the key (which is a bad idea)

• As the SBC will be used to relay all RTP/RTCP packets then this will introduce additional delay (dad sound quality)
  

• When unreliable transport is used this (relaying packets) could also increase packet loss (QoS problem)
  

In the 3GPP IMS context the SBC is in most case bundled with the Proxy-CSCF (P-CSCF plus IMS-ALG) and this could resolve the security issue (SIP-IPSec).

STUN

STUN was previously defined in RFC 3489 and updated by RFC 5389. STUN stands for « Session Traversal Utilities for NAT » and is a client-server protocol (request <-> response) as SIP.

There is also « indication requests » that don't generate responses like « binding requests ».

Both reliable (e.g. UDP) and unreliable (e.g TCP, TLS or SCTP) are supported.

Like SIP, when unreliable transoport is used there is the notion of transctions and retransmissions.

When STUN is used the client learn its public IP address and port (a.k.a reflexive transport address) by sending a binding request to the STUN server (default UDP/TCP port:3478 and default TLS port: 5349). The server will in some case challenge (401) the client which should resend its request with all credentials (HTTP digest authentication).

If the request is suceessfuly autheticated by the server, then a success binding response is sent back to the client. This response contains a STUN attribute (XOR-MAPPED-ADDRESS)

with the client's public IP address, family and port. This is also called the « reflexive transport address ».

Once this reflexive transport address is know then the client can for example begin REGISTERing using this address and port as AOR.

====== Step 1: Sending binding request ======

In this request I send my STUN request from "192.168.16.108:1115" to the server in order to get the public IP address and port associated to this local socket/Private address (file descriptor).

====== Step 2 Success binding response ======

In step 2 the STUN client receive an response from the server with its public IP address and port. To match the response with the request we compare the transaction IDs.
As some routers rewrite the content of the packets the IP address and port are not sent "as is" but in XOR format (into the XOR-MAPPED-ADDRESS attribute).
To retrieve the port:
uint16_t port = ntohs(*((uint16_t*)payloadPtr));
port ^= 0x2112; /* First two bytes of the STUN2 magic cookie. */
To retrive the IPv4 address:
uint32_t addr = ntohl(*((uint32_t*)payloadPtr));
addr ^= 0x2112A442; /* The STUN2 magic cookie */

====== Step 3 Sending first SIP request ======


From step 1 and 2 the sip agent can assert that [192.168.16.108:1159] is mapped to [89.127.73.39:1115] (take care to the port mapping).
In step 3 when sending it's first REGISTER request it will use this pulic IP address and port to build its AOR. As rport option is used then you could keep the Via IP address and port inchanged (or not).

The major problem with STUN is that it could not be used behind bi-directional NATs. In the next parts I will explain how to overcome this problem by using TURN and ICE.