Building/Debugging android native C applications

In this post I will explain how to compile, install and debug an Android native "C" application.
If you are reading this post just because you have googled the magic keywords ("android" + "native code") then you should know that there is an easier way to build native applications using android makefiles ("Android.mk" and "Application.mk").
The method I'm describing here is only useful if you want to understand how things work in order to create more complex standard GNU makefiles. This is also useful if you would like to create your own GNU autotools wrappers to compile projects using GNU configure.
I'm using Windows Vista as host machine but any other supported platforms (e.g. linux-x86 or darwin-x86) should work.

I have tested both the NDK (1.6) and SDK (2.1) on:

• Windows XP (32-bit) and Vista (64-bit)
  
• Mac OS X Snow Leopard
  
• Ubuntu Intrepid
  

Installing Android SDK

To download the latest Android SDK, visit this address http://developer.android.com/sdk/index.html.
If you need information on how to install the SDK, visit this address http://developer.android.com/sdk/installing.html.
If the "SDK setup" fail to update the installed packages you can change the remote site URL from https://dl-ssl.google.com/android/repository/repository.xml to http://dl-ssl.google.com/android/repository/repository.xml (change the URL scheme from HTTPS to HTTP) or try to disable your anti-virus or firewall.

I have installed the SDK version 2.1 under c:/android-sdk (a.r.a /cygdrive/c/android-sdk).
Add an environment variable named ANDROID_SDK_ROOT pointing to the SDK root directory.

Important: You should add "$ANDROID_SDK_ROOT/tools" directory to the $PATH environment variable.
Under *nix:

export PATH=$ANDROID_SDK_ROOT/tools:$PATH

Under Cygwin: Open C:\Cygwin\Cygwin.bat and add:

set PATH=%ANDROID_SDK_TOOLS%;%PATH%

Installing Cygwin

If you are using Windows XP or Vista as host machine then you MUST install Cygwin Devel package with GNU Make (3.81 or later) before installing the NDK.
It should also work with MinGW.

Installing the Android NDK

To download the latest Android NDK, visit this address http://developer.android.com/sdk/ndk/1.6_r1/index.html.
I have uncompressed the NDK version 1.6 under c:/android-ndk (a.r.a /cygdrive/c/android-ndk).
Add an environment variable named ANDROID_NDK_ROOT pointing to the NDK root directory.
To install the NDK:

cd $ANDROID_NDK_ROOT
build/host-setup.sh

If all is OK then the console will print Host setup complete.

To test that the toolchain has been correctly installed you can try to build the hello-jni sample which comes with the NDK by doing this:

cd $ANDROID_NDK_ROOT
make -APP=hello-jni

If all is OK then the console will print:

Android NDK: Building for application 'hello-jni'
Compile thumb : hello-jni <= sources/samples/hello-jni/hello-jni.c SharedLibrary : libhello-jni.so Install : libhello-jni.so => apps/hello-jni/project/libs/armeabi

This mean that your native shared library (libhello-jni.so) have been successfully generated under $ANDROID_NDK_ROOT/apps/hello-jni/project/libs/armeabi folder.

Creating an AVD

AVD stands for Android Virtual Device and can be seen as a device profile (keyboard, dialing pad, skin, screen dimensions, appearance ...) to load into your emulator. You can create as many AVDs as you need.
To create an AVD named "avdtest" targeting platform 2.1 (targetID=android-7):

android create avd -n avdtest -t android-7

If all is OK the console will print:

Created AVD 'avdtest' based on Android 2.1, with the following hardware config: hw.lcd.density=160

Create test.c

Here I will create a basic test.c file under C:\tmp with the following content:

#include <stdio.h>// printf

int main(int argc, char **argv)
{
int i = 1;
i+=2;

printf("Hello, world (i=%d)!\n", i);

return 0;
}

Create makefile

Just create an empty file named makefile (without any extension) under C:\tmp (which is the same directory as test.c).
Now We will fill the makefile step by step.

Add application name, $ROOT directory, install directory and the NDK platform version:

APP := test
ROOT:=/cygdrive/c
NDK_PLATFORM_VER := 1.5
INSTALL_DIR := /data/tm

Add useful environment vars:

ANDROID_NDK_ROOT:=$(ROOT)/android-ndk
ANDROID_NDK_HOST:=windows
ANDROID_SDK_ROOT:=$(ROOT)/android-sdk
PREBUILD:=$(ANDROID_NDK_ROOT)/build/prebuilt/$(ANDROID_NDK_HOST)/arm-eabi-4.2.1
BIN := $(PREBUILD)/bin

You MUST change ANDROID_NDK_HOST value from windows to linux-x86 if you are under *nix or darwin-x86 on MAC OS X.

Add GCC options:

CPP := $(BIN)/arm-eabi-g++
CC := $(BIN)/arm-eabi-gcc
CFLAGS :=
LDFLAGS := -Wl

Add targets

all: $(APP)

OBJS += $(APP).o

$(APP): $(OBJS)

$(CPP) $(LDFLAGS) -o $@ $^

%.o: %.c

$(CC) -c $(INCLUDE) $(CFLAGS) $< -o $@

install: $(APP)

$(ANDROID_SDK_ROOT)/tools/adb push $(APP) $(INSTALL_DIR)/$(APP)

$(ANDROID_SDK_ROOT)/tools/adb shell chmod 777 $(INSTALL_DIR)/$(APP)

shell:

$(ANDROID_SDK_ROOT)/tools/adb shell

run:

$(ANDROID_SDK_ROOT)/tools/adb shell $(INSTALL_DIR)/$(APP)

clean:

@rm -f $(APP).o $(APP)

Building the application

To build the application, switch to the directory where you have created both files and then:

make

At the output of the console you will get many errors saying that it's impossible to find stdlib.h, stdio.h etc etc.
To resolve this issue, add the Bionic header files to $CFLAGS variable like this:

CFLAGS := -I$(ANDROID_NDK_ROOT)/build/platforms/android-$(NDK_PLATFORM_VER)/arch-arm/usr/include

If you retry (make) you will now get this link error:

rt0.o: No such file: No such file or directory

To avoid directly linking against the "C runtime" you must add "-nostdlib" flag to the link options like this:

LDFLAGS := -Wl -nostdlib

If you retry (make) you will now get these link errors:

test.c:(.text+0x34): undefined reference to `printf'
test.c:(.text+0x3c): undefined reference to `exit'

You get these errors because Bionic libc is missing. To add libc you MUST change $LDFLAGS like this:

LDFLAGS := -Wl -L$(ANDROID_NDK_ROOT)/build/platforms/android-$(NDK_PLATFORM_VER)/arch-arm/usr/lib
LDFLAGS += -nostdlib -lc

If you retry (make) you will now get this link error:

/cygdrive/c/android-ndk/build/platforms/android-1.5/arch-arm/usr/lib/libc.so: undefined reference to `dl_unwind_find_exidx'

To resolve this issue you MUST specify the first set of directories into which to search the system shared libraries (*.so) . This is done by adding the "-rpath-link" option to the link options like this:

LDFLAGS := -Wl,-rpath-link=$(ANDROID_NDK_ROOT)/build/platforms/android-$(NDK_PLATFORM_VER)/arch-arm/usr/lib -L$(ANDROID_NDK_ROOT)/build/platforms/android-$(NDK_PLATFORM_VER)/arch-arm/usr/lib

If you retry (make) you will now get this warning:

/cygdrive/c/android-ndk/build/prebuilt/windows/arm-eabi-4.2.1/bin/../lib/gcc/arm
-eabi/4.2.1/../../../../arm-eabi/bin/ld: warning: cannot find entry symbol _start; defaulting to 000082c8

This is an Android known issue. You have this warning because the linker search "_start" as entry point. You can resolve this issue by renaming your main function. But the elegant way to resolve this issue is to specify the entry point in the link options like this:

LDFLAGS := -Wl,--entry=main,-rpath-link=$(ANDROID_NDK_ROOT)/build/platforms/android-$(NDK_PLATFORM_VER)/arch-arm/usr/lib -L$(ANDROID_NDK_ROOT)/build/platforms/android-$(NDK_PLATFORM_VER)/arch-arm/usr/lib
LDFLAGS += -nostdlib -lc

Now When you retry (make) your application will successfully build without any errors or warnings.

Testing your application

Before testing your application you MUST run the emulator like this:

emulator -avd avdtest

where "avdtest" is the name of the previously created avd (see "creating an avd" section).
To install the application on the emulator, open a new console and go to to directory where you have created test.c and makefile. Install your application on the emulator like this:

make install

If all is OK the console will print:

/cygdrive/c/android-sdk/tools/adb push test /data/tmp/test
304 KB/s (2493 bytes in 0.008s)
/cygdrive/c/android-sdk/tools/adb shell chmod 777 /data/tmp/test

To run the application type:

make run

You will probably get an error message saying:

/cygdrive/c/android-sdk/tools/adb shell /data/tmp/test
/data/tmp/test: not found

This error message is a bit confusing because if you browse the /data/tmp directory you will notice that the executable is here. The question is why?
I spent hours searching and I found that this error happens because the loader fails to load the application because it cannot found a proper linker.
To specify a search directory for the dynamic linker (at run time) you MUST change the link options like this:

LDFLAGS := -Wl,--entry=main,-rpath-link=$(ANDROID_NDK_ROOT)/build/platforms/android-$(NDK_PLATFORM_VER)/arch-arm/usr/lib,-dynamic-linker=/system/bin/linker -L$(ANDROID_NDK_ROOT)/build/platforms/android-$(NDK_PLATFORM_VER)/arch-arm/usr/lib
LDFLAGS += -nostdlib -lc

Now rebuild and install your application (make clean && make && make install) then run it again (make run).
The console will print the expected result ("hello, world (i=3)!") but just after we have an segmentation fault error ("[1] Segmentation fault /data/tmp/test").
To resolve this issue you can exit the program (exit(0);) just before the main function returns (return 0;). You should also include <stdlib.h>.
If you retry the build&&run process (make clean && make && make install && make run) then you should have:

/cygdrive/c/android-sdk/tools/adb shell /data/tmp/test
Hello, world (i=3)!

which is the expected result.

Debugging your application
Before doing anything you MUST copy the gdbserver file to the emultor.
This file is under $BIN ($ANDROID_NDK_ROOT/build/prebuilt/$ANDROID_NDK_HOST/arm-eabi-4.2.1/bin).
Copy gdbserver to the emulator like this:

adb push gdbserver $INSTALL_DIR/gdbserver
adb shell chmod 777 $INSTALL_DIR/gdbserver

where $INSTALL_DIR is the directory where you have installed your application (it's not mandatory to copy it in this directory).

Before running the server on port 1234 you MUST redirect all tcp connection to this port like this:

adb forward tcp:1234: tcp:1234

it's not mandatory to forward connections to the same port number.
Now it's time to run the server:

adb shell $INSTALL_DIR/gdbserver :1234 $INSTALL_DIR/$APP

note that only the server port is specified (no host).
If all is OK the the server will print something like this:

Process /data/tmp/test created; pid = 246
Listening on port 1234

Now to debug our application we will change the makefile by adding a new debug target like this.

GDB_CLIENT := $(BIN)/arm-eabi-gdb

debug:

$(GDB_CLIENT) $(APP)

To launch the application in debug mode type "make debug" (after make clean && make && make install of course). If you do this, you will see a warning message saying that "no debugging symbols found". No symbols ==> no debug.
To generate debug symbols you MUST change the makefile like this (should not be hard coded like this):

DEBUG = -g
CFLAGS := $(DEBUG) -I$(ANDROID_NDK_ROOT)/build/platforms/android-$(NDK_PLATFORM_VER)/arch-arm/usr/include

Now rebuild and install your application (make clean && make && make install) then run it again (make debug). This (make debug) should open gdb invite command((gdb)) on the same console.
Connect to the server (from the same console) like this:

target remote :1234

Set a breakpoint on the main function and execute step by step (commands above are informational and you can use any gdb commands):

b main
c
n
p i
#$1 = 1
n
#9 printf("Hello, world (i=%d)!\n", i);
p i
#$2 = 3
c
#Program exited normally.

The final makefile and test.c files are shown below:

makefile

APP := test
ROOT:=/cygdrive/c
INSTALL_DIR := /data/tmp
NDK_PLATFORM_VER := 1.5

ANDROID_NDK_ROOT:=$(ROOT)/android-ndk
ANDROID_NDK_HOST:=windows
ANDROID_SDK_ROOT:=$(ROOT)/android-sdk
PREBUILD:=$(ANDROID_NDK_ROOT)/build/prebuilt/$(ANDROID_NDK_HOST)/arm-eabi-4.2.1
BIN := $(PREBUILD)/bin
GDB_CLIENT := $(BIN)/arm-eabi-gdb

DEBUG = -g

CPP := $(BIN)/arm-eabi-g++
CC := $(BIN)/arm-eabi-gcc
CFLAGS := $(DEBUG) -I$(ANDROID_NDK_ROOT)/build/platforms/android-$(NDK_PLATFORM_VER)/arch-arm/usr/include
LDFLAGS := -Wl,--entry=main,-rpath-link=$(ANDROID_NDK_ROOT)/build/platforms/android-$(NDK_PLATFORM_VER)/arch-arm/usr/lib,-dynamic-linker=/system/bin/linker -L$(ANDROID_NDK_ROOT)/build/platforms/android-$(NDK_PLATFORM_VER)/arch-arm/usr/lib
LDFLAGS += -nostdlib -lc

all: $(APP)

OBJS += $(APP).o

$(APP): $(OBJS)

$(CPP) $(LDFLAGS) -o $@ $^

%.o: %.c

$(CC) -c $(INCLUDE) $(CFLAGS) $< -o $@

install: $(APP)

$(ANDROID_SDK_ROOT)/tools/adb push $(APP) $(INSTALL_DIR)/$(APP)

$(ANDROID_SDK_ROOT)/tools/adb shell chmod 777 $(INSTALL_DIR)/$(APP)

shell:

$(ANDROID_SDK_ROOT)/tools/adb shell

run:

$(ANDROID_SDK_ROOT)/tools/adb shell $(INSTALL_DIR)/$(APP)

debug:

$(GDB_CLIENT) $(APP)

clean:

@rm -f $(APP).o $(APP)

test.c

#include <stdio.h> // printf
#include <stdlib.h> //exit

int main(int argc, char **argv)
{
int i = 1;
i+=2;

printf("Hello, world (i=%d)!\n", i);

exit(0);
return 0;
}

SIP/IMS and NAT traversal (Part 1)

This is the first part of a three-part series about NAT traversal solutions for SIP/IMS. I will begin from simple solutions (e.g. "rport" extension) and end with some more complex one (e.g ICE).

Today most Internet users are in a private network behind one or several NATs. This is not a problem for your browser (HTTP, FTP...) or email client (SMTP, POP...) because they use protocols that can operate behind NATs. The problem comes when you try to use your SIP phone to REGISTER or place a call to another softphone in different private network through Internet (Public network).
The problem with SIP is that IP addresses and Ports where to contact/respond an agent (e.g. PC, Box or mobile phone) are embedded in the SIP message itself.

For example, when you REGISTER to a registrar (e.g Serving-CSCF through Proxy-CSCF) you add a contact header (IP address and Port - Contact: <sip:alice@192.168.16.108:27208;transport=udp>) where the registrar should send all your incoming calls(INVITEs). This address is called Address-Of-Record (a.k.a AOR).
If you are in a private network you will add your private IP address and port in this header (Contact) and the problem is that this AOR is not visible to the elements outside your private network.
This mean that INVITE requests will never reach your agent.
You will have the same problem when you are the caller, as you will add in your SDP the IP addresses and ports where RTP packets shall be sent to you.

So, NAT problem concern both signaling (SIP/SDP) and media (RTP/RTCP) plans.

rport

This is the most simple way to deal with NATs problem at signaling (SIP only) plan for connectionless protocols (e.g. UDP). rport has been defined in RFC 3581 and apply to both connectionless (e.g. UDP) and connection-oriented (e.g. TCP, TLS, SCTP) protocols.

Because connection-oriented protocols such as TCP are bidirectional it is easier to deal with NAT traversal (all responses will be sent back using the same connection from which the request has been received from).

The philosophy (of rport) is that when you add this attribute (rport) in your requests (Via: SIP/2.0/UDP 192.168.16.108:27208;branch=z9hG4bK1234;rport) then all responses will be sent back to the ip:port from which the request has been received from. This parameter MUST be added in the top most Via header.

The response will contain a new parameter (« received ») containing the IP address from which the request has been received and the rport value will be filled with the mapped (public?) source port. In this way the client/caller (the sender of the request) can learn it's public/reflexive IP address and port (Via: SIP/2.0/UDP 192.168.16.108:27208;branch=z9hG4bK1234;rport=1234;received=10.1.1.1).

By examining the response the client can know if it's behind a NAT or not.

The problem with this solution is that it only deal with SIP messages and cannot be used for RTP/RTCP packets as the SDP will contain wrong IP addresses and ports. A solution to this issue could be using Symmetric RTP/RTCP (see above).

Symmetric RTP / RTP Control Protocol (RTCP)

As we have seen above (rport), we cannot use « rport » to deal with RTP/RTCP packets which are almost always transported using UDP. A solution to this problem could be using « Symetric RTP/RTCP » as per RFC 4961. This solution is a bit like using « rport ».

In this case the caller (INVITE originator) will create the request as per RFC 3261 as usual. When the 2xx (with SDP) or 1xx (with SDP) is received, the caller will ignore the IP addresses and ports defined in the response and send RTP/RTCP packets to IPs/Ports from which the RTP/RTCP packets have been sent (like rport).

To summarize:

1. Bob sends an INVITE to Alice
2. Alice sends back a 2xx (with SDP) or 1xx (with SDP) to Bob
3. Bob waits for first RTP/RTCP packets to come from Alice
4. Alice send first RTP/RTCP packest from IP-a and Port-a to IP-b and Port-b (Bob address and port defined in the SDP).
5. Instead of sending RTP/RTCP packets to IP-asdp and Port-asdp as sepecified in the Alice's SDP, Bob will begin sending media stream to IP-a and Port-a.

As you can imagine this solution only work for only some NATs.

Session border controller

As its name says, it controls (both media and SIP packets) sessions (SIP calls) and is in the border (between two networks) of the networks.

In our case (NAT traversal) the role of SBC will be to inspect/control all outgoing and incoming SIP/RTP/RTCP packets. All SIP packets will be inspected and IP addresses and ports within the packet will be rewritten (e.g from private IP to public IP).

This solution has several problems:

• Very expensive (€€$$££)
  

• As the SBC operates on the SIP packets then it MUST be aware of all headers (or functions) in order to know what should be changed and what should not be changed. This mean that the SBC MUST always be up to date in order to efficiently handle SIP packets

• Most SBCs can only handle well-know protocols such as SIP, RTP or RTCP and will drop all unknown protocols
  

• There is also many problems when End-to-End encryption (e.g. TLS, SRTP or IPSec) is used unless the SBC has the key (which is a bad idea)

• As the SBC will be used to relay all RTP/RTCP packets then this will introduce additional delay (dad sound quality)
  

• When unreliable transport is used this (relaying packets) could also increase packet loss (QoS problem)
  

In the 3GPP IMS context the SBC is in most case bundled with the Proxy-CSCF (P-CSCF plus IMS-ALG) and this could resolve the security issue (SIP-IPSec).

STUN

STUN was previously defined in RFC 3489 and updated by RFC 5389. STUN stands for « Session Traversal Utilities for NAT » and is a client-server protocol (request <-> response) as SIP.

There is also « indication requests » that don't generate responses like « binding requests ».

Both reliable (e.g. UDP) and unreliable (e.g TCP, TLS or SCTP) are supported.

Like SIP, when unreliable transoport is used there is the notion of transctions and retransmissions.

When STUN is used the client learn its public IP address and port (a.k.a reflexive transport address) by sending a binding request to the STUN server (default UDP/TCP port:3478 and default TLS port: 5349). The server will in some case challenge (401) the client which should resend its request with all credentials (HTTP digest authentication).

If the request is suceessfuly autheticated by the server, then a success binding response is sent back to the client. This response contains a STUN attribute (XOR-MAPPED-ADDRESS)

with the client's public IP address, family and port. This is also called the « reflexive transport address ».

Once this reflexive transport address is know then the client can for example begin REGISTERing using this address and port as AOR.

====== Step 1: Sending binding request ======

In this request I send my STUN request from "192.168.16.108:1115" to the server in order to get the public IP address and port associated to this local socket/Private address (file descriptor).

====== Step 2 Success binding response ======

In step 2 the STUN client receive an response from the server with its public IP address and port. To match the response with the request we compare the transaction IDs.
As some routers rewrite the content of the packets the IP address and port are not sent "as is" but in XOR format (into the XOR-MAPPED-ADDRESS attribute).
To retrieve the port:
uint16_t port = ntohs(*((uint16_t*)payloadPtr));
port ^= 0x2112; /* First two bytes of the STUN2 magic cookie. */
To retrive the IPv4 address:
uint32_t addr = ntohl(*((uint32_t*)payloadPtr));
addr ^= 0x2112A442; /* The STUN2 magic cookie */

====== Step 3 Sending first SIP request ======


From step 1 and 2 the sip agent can assert that [192.168.16.108:1159] is mapped to [89.127.73.39:1115] (take care to the port mapping).
In step 3 when sending it's first REGISTER request it will use this pulic IP address and port to build its AOR. As rport option is used then you could keep the Via IP address and port inchanged (or not).

The major problem with STUN is that it could not be used behind bi-directional NATs. In the next parts I will explain how to overcome this problem by using TURN and ICE.

SMS over 3GPP IMS Network

SMS stands for Short Message Service or Silent Messaging Service and is a communication service standardized in the GSM mobile communication system, using standardized communications protocols allowing the interchange of short text messages between mobile telephone devices.

SMS technology has been specified by the ETSI in GSM 03.40 and 03.38 documents (3GPP TS 23.040 and 3GPP TS 23.038 respectively). These documents only describe how to use SMS over mobile networks (nothing for IP networks).
In real world there are two way to receive SMS messages over mobile networks: Binary (PDU) and Text mode.
In this post I will explain how to use SMS technology over IP within the IP Multimedia (IM) Core Network (CN) subsystem based on the Session Initiation Protocol (SIP) and SIP Events as defined in 3GPP TS 24.229.
Note: In the coming chapters, « SMS over IMS Network » and « SMS over IP Network » have the same meaning.

Text mode

The message payload is transfered « as is ». This mode is out of scope.

Binary mode

Also know as PDU (Protocol Data Unit) mode, it is use to encode the payload of the SMS sent over IMS networks. In this case the payload only contains a sequence of hexa-decimal octets or decimal semi-octets strings. The overall PDU string contains some useful information (SMSC address, Service center time stamp, sender number, message reference ...) plus the actual message.

The message length can be up to 160 characters where each character represent 7bits [160/7bits], [140/8bits] or [70/16bits]. By default, each character represent 7bits encoded as per GSM 03.38.

For IMS Networks, SMS message shall be encapsulated in RPDUs (Relay Protocol Data Unit) data string as defined in 3GPP TS 24.011, subclause 7.3. The RPDU data is transfered from SM entity to SM entity using SIP MESSAGE requests. These SIP requests shall use the MIME type "application/vnd.3gpp.sms" for this purpose.

GSM 7 bit Default Alphabet

Each character represent 7bits. For more information see 3GPP TS 23.038 or ETSI GSM 03.38.

Figure 1: GSM 7bit alphabet

3GPP IMS Registration

On sending a REGISTER request, the IMS Client shall indicate its capability to receive traditional short messages over IMS network by including a "+g.3gpp.smsip" parameter into the Contact header according to RFC 3840.

Figure 2: SIP/IMS registration request

Mobile-Originated SMS (MO-SMS)

The IMS Client sender shall build and populate RP-DATA message (RPDU encapsulating TPDU data string), containing all the information that a mobile station submitting an SMS message according to 3GPP TS 24.011 would place, for successful delivery. The sender shall parse and interpret RP- DATA, RP-ACK and RP-ERROR messages, containing all the information that a mobile station receiving an SM according to 3GPP TS 24.011 would see, in a SM submission or status report.

Figure 3: Signaling flows demonstrating successful UE originated SM submit procedure over IP as per 3GPP TS 24.341

** [1,2,4] RP-DATA (SMS-SUBMIT)

Below is represented the first request (RP-DATA(SMS-SUBMIT)) sent from the IMS Client to the P-CSCF.

Figure 4: RP-DATA(SMS-SUBMIT) from Mercuro to the P-CSCF

The SIP MESSAGE (Figure 4) request sent from the IMS client to the P-SCSF carry the RP-DATA (Mobile station to IMS Network) message encoded as per 3GPP TS 24.011 chapter 7.3.1.2.

Figure 5: RP-DATA (Mobile Station to Network) as per 3GPP TS 24.011 subclause 7.3.1.2

The SIP MESSAGE content (Figure 4) can be dissected as following:

RP-MESSAGE: 0000000691330100000F320691330100000F11000A9133163254760000AA05F330BB4E07
RPDU:0000000691330100000F32

TPDU (SMS-SUBMIT):0691330100000F11000A9133163254760000AA05F330BB4E07

The RP-MESSAGE can be dissected as follow (Figure 6):

Figure 6: RP-DATA (Mobile Station to Network) content dissection

** [9,10,11] RP-ACK(SMS-SUBMIT-REPORT)

The SIP MESSAGE payload includes the RP-ACK message. Its RP-User-Data information element includes a TPDU of type SMS-SUBMIT-REPORT.

Figure 7: RP-ACK as per 3GPP TS 24.011 subclause 7.3.3

RP-MESSAGE:020012069133163254760080

For more information, see 3GPP TS 24.011 subclause 7.3.3 for RPDU (RP-ACK) data and 3GPP TS 23.040 subclause 9.2.2.2a for TPDU (SMS-SUBMIT-REPORT) encoding. If error occurs when decoding the message, RP-ERROR (3GPP TS 24.011 subclause 7.3.4) message is sent instead of RP-ACK.

Mobile-Terminated SMS (MT-SMS)

Figure 8: UE originated SM deliver procedure over IP signaling as per 3GPP TS 24.341

** [2,3,4] RP-DATA(SMS-DELIVER)

This request sent from the IP-SM-GW includes a vnd.3gpp.sms payload that includes the short message and routing information for the S-CSCF to forward the short message.

The payload includes the RP-DATA message. Its RP-User-Data information element includes a TPDU of type SMS-DELIVER.

Figure 9: RP-DATA (Network to Mobile Station ) as per 3GPP TS 24.011 subclause 7.3.1.1

RP-MESSAGE: 01000691330100000F003C0691330100000F040A91331632547600000000000000000005F330BB4E07

For more information, see 3GPP TS 24.011 subclause 7.3.1.1 for RPDU (RP-DATA) data and 3GPP TS 23.040 subclause 9.2.2.1 for TPDU (SMS-DELIVER) encoding.

** [8,9,11] RP-ACK(SMS-DELIVER-REPORT)

This request includes a vnd.3gpp.sms payload that includes the SMS-DELIVER-REPORT and routing information for the IP-SM-GW to forward the delivery report.
The payload includes an RP-ACK message. Its RP-User-Data information element includes a TPDU of type SMS-DELIVER-REPORT.

RP-MESSAGE:020112069133163254760080

For more information, see 3GPP TS 24.011 subclause 7.3.3 for RPDU (RP-ACK) data and 3GPP TS 23.040 subclause 9.2.2.1a for TPDU (SMS-DELIVER-REPORT) encoding.
If error occurs when decoding the message, RP-ERROR (3GPP TS 24.011 subclause 7.3.4) message is sent instead of RP-ACK.

References:

• http://en.wikipedia.org/wiki/SMS

• 3GPP TS 23.038 - Alphabets and language-specific information
• 3GPP TS 23.040 - Technical realization of Short Message Service (SMS)
• 3GPP TS 24.011 - Point-to-Point (PP) Short Message Service (SMS) support on mobile radio interface.
• 3GPP TS 24.341 - Support of SMS over IP networks; Stage 3
• 3GPP TS 24.451 - Support of SMS and MMS over NGN IMS subsystem; Stage 3 [Endorsement of 3GPP TS 24.341 Release 7]

OpenIPSec: Introduction (Part 1)

This is the first part of a series of articles in which I will explain how I will develop an open source IPSec Framework (Ndis Driver, Crypto API and Command Line Tools) for Windows OS.
This Framework will be available under GPLv3 license and all the source code will be available for download.

This first part is an introduction to IPSec and a kick overview of OpenIPSec project.

1. IPSec

IPSec stands for Internet Protocol Security and has been specified by the Internet Engineering Task Force (IETF).
IPSec operates at at the Network Layer (OSI Layer 3). This imply that it can protect all upper protocols such as UDP, TCP or ICMP.
IPSec is a successor of the ISO standard Network Layer Security Protocol (NLSP). NLSP was based on the SP3 protocol that was published by the NIST but designed by the Secure Data Network System project of the National Security Administration (NSA). Because it operates at OSI layer 3 it must be integrated at the Kernel/OS Layer. On Windows OS all Network APIs (e.g. Windows Sockets 2 or Layered Service Provider) operate at the upper layers (user/application Layer).
An example of security protocols working at user/application layer are Secure Sockets Layer (SSL), Transport Layer Security (TLS) or Secure Shell (SSH).

1.1 Protocols

IPSec have three main protocols: AH, ESP and IKE. AH and ESP are commonly called Security Protocols.

=>AH: Stands for Authentication Header and has been defined in RFC4302. It is used to provide connectionless integrity and data origin authentication for IP datagrams and to provide protection against replays.


=>ESP: Stands for Encapsulating Security Payload and has been defined in RFC4303. It is used to provide confidentiality, data origin authentication, connectionless integrity, an anti-replay service (a form of partial sequence integrity), and (limited) traffic flow confidentiality. It may be applied alone or in combination with AH.

=>IKEv1/IKEv2: Stands for Internet Key Exchange and have been defined in RFC4109 and RFC4306 respectively. Both are used for mutual authentication between two parties and establishes an IKE security association (SA) that includes shared secret information that can be used to efficiently establish SAs for Encapsulating Security Payload (ESP) and/or Authentication Header (AH) and a set of cryptographic algorithms to be used by the SAs to protect the traffic that they carry.

1.2 Security Association (SA)

IPSec Security Association (SA) is a virtual unidirectional association between two or more peers/entities (IP:port) . This mean that you should create two SAs for bidirectional communications. Each SA has its own ID called SAID stored in the Security Association Database (SAD) as per RFC4301. The SA defines:

• A unique security parameter index (SPI)
• Shared security Keys
• Lifetime
• Encryption algorithm to use : DES, 3DES or AES (all supported by OpenSec)
• Authentication algorithm to use: HMAC-MD5-96, AES-XCBC-MAC-96 or HMAC-SHA-1-96 (all supported by OpenSec)
• Which Protocol to use: ESP or AH

1.3 Key Management

Key Management mechanism is used to exchange mandatory security keys to setup SAs. Can be manual or automated using IKE.
For Example, 3GPP IMS use SIP to exchange security information and manually setup SAs (e.g. using ipsec command line tools).

1.4 Mode of operation (transport and tunnel)

Both ESP and AH may be applied individually or in combination with each other to provide IPv4 and IPv6 security services. Both supports two modes of use: transport mode and tunnel mode. In transport mode, AH and ESP provide protection primarily for next layer protocols; in tunnel mode, AH and ESP are applied to tunneled IP packets.

==>Transport: only the payload (without IP header) of the IP packet is encrypted (ESP) and/or authenticated (ESP or AH) before (re)transmission. This is the default mode in 3GPP IMS context (Both UE and P-CSCF).

=>Tunnel: both the IP header and the payload are encrypted (ESP) and/or authenticated (ESP or AH) before (re)transmission. It is used to create Virtual Private Networks (VPN).

2. OpenIPSec Framework

OpenIPSec is an open source IPSec Framework for Windows XP/Vista/Blackcomb/CE. The Framework will include an API for software development, command line tools, open source code (GPLv3) and documentation.
Development is done from scratch and all dependent projects are my own projects (To ease bug fix; release a soft 100% free and open source). This project depends on OpenSec and libonid. The first, contains IPSec algorithms implementation and the second allows monitoring Network cards. These two projects are under development (OpenSec: 90% done - libonid: 40% done) and the source code is freely available for download. To ease adoption, command line tools (setkey, spdadd, add ...) will have same name and options than those provided on Linux.

2.1 Standards

* RFC1825 - Security Architecture for the Internet Protocol (Obsoleted by: 2401)
* RFC2401 - Security Architecture for the Internet Protocol (Obsoletes: 1825 and Obsoleted by: 4301 and Updated by: 3168)
* RFC2402 - IP Authentication Header (Obsoleted by: 4302, 4305 and Obsoletes: 1826)
* RFC2410 - The NULL Encryption Algorithm and Its Use With IPsec
* RFC2405 - The ESP DES-CBC Cipher Algorithm With Explicit IV
* RFC2406 - IP Encapsulating Security Payload (ESP) (Obsoleted by: 4303, 4305 and Obsoletes: 1827)
* RFC2407 - The Internet IP Security Domain of Interpretation for ISAKMP (Obsoleted by: 4306)
* RFC2408 - Internet Security Association and Key Management Protocol (ISAKMP) (Obsoleted by: 4306)
* RFC2409 - The Internet Key Exchange (IKE) (Obsoleted by: 4306 and Updated by: 4109)
* RFC2481 - A Proposal to add Explicit Congestion Notification (ECN) to IP (Obsoleted by: 3168)
* RFC3168 - The Addition of Explicit Congestion Notification (ECN) to IP (Updates: 2474, 2401, 793 and Obsoletes: 2481)
* RFC3602 - The AES-CBC Cipher Algorithm and Its Use with IPsec
* RFC4109 - Algorithms for Internet Key Exchange version 1 (IKEv1) (Updates: 2409)
* RFC4301 - Security Architecture for the Internet Protocol
* RFC4302 - IP Authentication Header
* RFC4303 - IP Encapsulating Security Payload (ESP) (Obsoletes: 2406)
* RFC4305 - Cryptographic Algorithm Implementation Requirements for Encapsulating Security Payload (ESP) and Authentication Header (AH) (Obsoleted by: 4835 and Obsoletes: 2404, 2406 )
* RFC4306 - Internet Key Exchange (IKEv2) Protocol
* RFC4307 - Cryptographic Algorithms for Use in the Internet Key Exchange Version 2 (IKEv2)
* RFC4835 - Cryptographic Algorithm Implementation Requirements for Encapsulating Security Payload (ESP) and Authentication Header (AH) (Obsoletes: 4305)

/!\ OpenSec is now available for download at http://code.google.com/p/opensec/

References:
http://en.wikipedia.org/wiki/IPsec

OpenSec Beta 0.3

First beta version of OpenSec is now available for download.
To download source code:
- go to http://code.google.com/p/opensec/ or
- use command line "svn checkout http://opensec.googlecode.com/svn/trunk/ opensec-read-only"

OpenSec is an Open-Source Security API. This project is part of OpenIPSec project.
This API contains all mandatory or recommanded security algorithms needed to implement an IPSec Framework.

Already supported algorithms: MD5, SHA-1, HMAC-MD5-96, HMAC-SHA-1-96, DES-ECB, DES-CBC, 3DES-ECB, 3DES-CBC, AES[128,192,256]-ECB, AES[128,192,256]-CBC and AES[128]-XCBC-MAC-96.

The source code includes a test application to see how these algorithms can be used.

Standards

* RFC1321 - The MD5 Message-Digest Algorithm
* RFC1829 - The ESP DES-CBC Transform
* RFC1853 - IP in IP Tunneling
* RFC2104 - HMAC: Keyed-Hashing for Message Authentication
* RFC2402 - IP Authentication Header
* RFC2403 - The Use of HMAC-MD5-96 within ESP and AH
* RFC2404 - The Use of HMAC-SHA-1-96 within ESP and AH (Obsoleted by RFC4305)
* RFC2405 - The ESP DES-CBC Cipher Algorithm With Explicit IV
* RFC2406 - IP Encapsulating Security Payload (ESP) (Obsoleted by RFC4305)
* RFC2451 - The ESP CBC-Mode Cipher Algorithms
* RFC3174 - US Secure Hash Algorithm 1 (SHA1)
* RFC3566 - The AES-XCBC-MAC-96 Algorithm and Its Use With IPsec
* RFC3602 - The AES-CBC Cipher Algorithm and Its Use with IPsec
* RFC4305 - Cryptographic Algorithm Implementation Requirements for Encapsulating Security Payload (ESP) and Authentication Header (AH)

* FIPS 197 - AES (Advanced Encryption Standard)
* FIPS-46-1/3 - DES and Triple DES (Data Encryption Standard)
* FIPS 81 - DES MODES OF OPERATION

* RFC2144 - The CAST-128 Encryption Algorithm

* RFC2202 - Test Cases for HMAC-MD5 and HMAC-SHA-1

Google Android NDK

Weeks ago, Google announced the first public release of Android NDK.

I was so excited because the IMS Client we are developing is written in pure C/C++ (Only Java was supported).
Before Android NDK, I have looked at Google's Android SDK and I found that they are using non-standard JVM. This mean that applications developed using J2ME can't run on Android devices. Thanks to Dalvik virtual machine.
It was clear that I won't rewrite xxx thousand lines of code for Android mobile devices which represent less than 1 percent of all mobile devices.
It's amazing to think that Android OS will gain momentum without compliant embedded JVM. First deception.

Because performance is critical for us and we don't have time and money to rewrite an application which will only run on Android devices, we decided to wait for the Android NDK.

The first release of Android NDK only comes with these native libraries:
# libc (C library)
# libm (math library)
# libz (Zlib compression)
# liblog (Android logging)

This mean that you don't have:
# STL support (string, list, map, queue, ....)
# C++ exceptions handling
# Threading support
# Cryptography support
# Secure sockets/transport (SSL, TLS, S/MIME, ...)
# Classes to access the system resources (webcam, soundcard, drivers, ...)
# .... to be continued

Of course, you must use JNI if you want to have access to the native libraries. I let you imagine about performance ...
Second deception.

Mercuro with MMS

Support for MMS (Multimedia Messaging Service) messaging has been added in the last release of Mercuro IMS Client.
MMS messages are sent over the network using MSRP/SMIL to one or multiple contacts (using conference factory).