Hand detection through Haartraining: A hands on approach

Detection of objects from images or from video is no trivial task. We need to use some type of machine learning algorithm and train it to detect features and also identify misses and false positives.  The haartraining algorithm does just this. It creates a series of haarclassifiers which ensure that non-features are quickly rejected as the object is identified.

This post will highlight the necessary steps required to build a haarclassifier for detection a hand or any object of interest. This post is sequel to my earlier post (OpenCV: Haartraining and all that jazz!) and has a lot more detail. In order to train the haarclassifier, it is suggested, that at least 1000 positive samples (images with the object of interest- hand in this case) and 2000 negative samples (any other image) is required.

  

As before for performing haartraining the following 3 steps have to be performed

1)      Create samples (createsamples.cpp)

2)      Haar Training (haartraining.cpp)

3)      Performance testing of the classifier (performance.cpp)

In order to build the above 3 utilities please refer to my earlier post OpenCV: Haartraining and all that jazz!

The steps required for training a haarcascade to recognize on normal open palm is given below

1) Create Samples: This step can be further broken down into the following 3 steps

a)      Creation of positive samples

b)      Superimposing the positive sample on the negative sample

c)      Merging of vector files of samples.

a)      Creation of positive samples:

Get a series of images with objects of interest (positive samples):

For this step take photos using the webcam (or camera) of the objects that you are interested. I had taken several snapshots of my hand ( I later simply downloaded images from Google images for because of the excessive clutter in the snaps taken).

b) Crop Images:

In this step you need to crop the images such that it only contains the object of interest. You can use any photo editing tool of your choice and save all the positive images in a directory for e.g. ./hands

c) Mark the object

Now the image with the positive sample has to be marked. This will be used for creating the samples.

The tool that is to be used for marking is objectmarker.cpp. You can downloaded the source code for this from Achu Wilson's blog. Now build objectmarker.cpp with the include directories and libraries of OpenCV. Once you have successfully built objectmarker you are ready to mark the positive samples. Samples have to be marked  because the description file for the positive samples file must be in the following format

[filename] [# of objects] [[x y width height] [... 2nd object] ...]

This will be used while generating the positive training samples.

This file is will be used with the createsamples utility to create positive training samples.

The command to use to run objectmarker is as follows

$objectmarker pos_desc.txt ./images

where dir is the directory containing the positive images

and output file will contain the positions of the objects marked as follows

pos_desc.txt

/images/hand1.bmp 1 0 0 246 50

/images/hand2.bmp 1 187 26 333 400

....

The use of the utility objectmarker is an art ;-) and you need to be trained to get the proper data. Make sure you get sensible widths and heights. There are times when you get -ve widths and -ve heights which are clearly wrong.

An alternative easy way is to open the jpg/bmp in a photo editor and check the width and height in pixels (188 x 200 say) and create the description file as

/images/hand1.bmp 1 0 0 188 200

Ideally the objectmarker should give you values close to this if the sample image file has only the object of interest (hand)

d) Create positive training samples

The createsamples utility can now be used for creating positive training sample files. The command is

createsamples -info pos_desc.txt -vec pos.vec -w 20 -h 20

This will create positive training samples with the object of interest, in our case the “hand”.

 Now, you should verify that the tool has done something sensible by checking the training samples generated.

 The command to use to display the positive training samples captured in the pos.vec is to run

 createsamples -vec pos.vec -w 20 -h 20

If the objects have been marked accurately you should see a series of hands (positive samples). If the samples have not been marked correctly the positive training file will give incorrect results.

Create negative background training samples

This step is used to create training samples with one positive image superimposed against a set of negative background samples. The command to use is

createsamples -img ./image/hand_1.BMP -num 9 -bg bg.txt -vec neg1.vec -maxxangle 0.6 -maxyangle 0 -maxzangle 0.3 -maxidev 100 -bgcolor 0 -bgthresh 0 -w 20 -h 20

where bg.txt will contain the list of negative samples in the following format

./negative/airplane.jpg

./negative/baboon.jpg

./negative/cat.jpg

...

The positive hand images will be superimposed against the negative background in various angles of distortion.

All the negative training samples will be collected in the training file neg1.vec as above.

As before you can verify that the createsamples utility has done something reasonable by executing

createsamples -vec neg1.vec -w 20 -h 20.

This should show a series of images of hands in various angles of distortion against the negative image background.

Creating several negative training samples with all positive samples

The createsamples utility takes one positive sample and superimposes it against the negative samples in bg.txt file. However we need to repeat this process for each of the positive sample (hand) that we have. Since this can be laborious process you can use the following shell script, I wrote,  to repeat the negative training sample with every positive sample with create_neg_training.sh

create_neg_training.sh

#!/bin/bash

let j=1

for  i in `cat hands.txt`

do

        createsamples -img ./image/$i -num 9 -bg bg.txt -vec "neg_training$j.vec" -maxxangle 0.6 -maxyangle 0 -maxzangle 0.3 -maxidev 100 -bgcolor 0 -bgthresh 0 -w 20 -h 20

       let j=j+1

done

where the positive samples are under ./image directory. For each positive hand sample a negative training sample file is create namely neg_training1.vec, neg_training2.vec etc.

Merging samples:

As seen above the createsamples utility superimposes only positive sample (hand) against a series of negative samples. Since we would like to repeat the utility for multiple positive samples there needs to be a way for merging all the training samples. A useful utility (mergevec.cpp) has been created  and is available for download in Natoshi Seo'sblog 

Download mergevec.cpp and build it along with (cvboost.cpp, cvhaarclassifier.cpp, cvhaartarining.cpp, cvsamples.cpp, cvcommon.cpp) with usual include files and the link libraries.

Once built it can be executed by executing

mergevec pos_neg.txt pos_neg.vec – w 20 -h 20

where pos_neg.txt will contain both the positive and negative training sample files as follows

pos_neg.txt

./vec/pos.vec

./vec/neg_training1.vec

./vec/neg_training2.vec

....

As before you can verify that the entire training file pos_neg.vec is sensible by executing

createsamples -vec pos_neg.vec -w 20 -h 20

Now the pos_neg.vec will contain all the training samples that are required for the haartraining process.

HaarTraining

The haartraining can be run with the training samples generated from the mergevec utility described above.

The command is

./haartrainer -data haarcascade -vec pos_neg.vec -bg bg.txt -nstages 20 -nsplits 2 -minhitrate 0.999 -maxfalsealarm 0.5 -npos 7 -nneg 9 -w 20 -h 20 -nonsym -mem 512 -mode ALL

Several posts have suggested that nstages should be ideally 20 and splits should be 2.

npos indicates the number of positive samples and -nneg the number of negative samples. In my case I had just used 7 positive and 9 negative samples.

This step is extremely CPU intensive and can take several hours/days to complete. I had reduced the number of stages to 14.

The haartraining utility will create a haarcascade directory and a haarcascade.xml.

Performance testing : The first way to test the integrity of the haarcascades is to run the performance utility described in my earlier post OpenCV:Haartraining and all that jazz! If you want to use the performance utility you should also create test samples which can be used for testing with the command

createsamples -img ./image/hand-1.bmp -num 10 -bg bg.txt -info test.dat -maxxangle 0.6 -maxyangle 0 -maxzangle 0.3 -maxidev 100 -bgcolor 0 -bgthresh 0

The test.dat can be used with the performance utility as follows

./performance -data haarcascade.xml -info test.dat -w 20 -h 20

I wanted something that would be more visually satisfying that seeing the output of the  performance utility. This utility just spews out textual information of hits, misses.

So I decided to use the facedetect.c with the haarcascade trained by my positive and negative samples to check whether it was working.  The test below describes using the facedetect.c for detecting the hand.

The code for facedetect.c  can be downloaded from Willow GarageWiki.

Compile and link the facedetect.c as handetect with the usual suspects. Once this builds successfully you can use

handdetect –-cascade=haarcascade.xml hands.jpg

where hands.jpg was my test image. If the handdetect does indeed detect the hand in the image it will l enclose the object detected with a rectangle. See the output below

While my haarcascade does detect 5 hands it does appear shifted. This could be partly because the number of positive and negative training samples used were 7 & 9 respectively which is very low. Also I used only 14 stages in the haartraining.

As mentioned in the beginning there is a need for at least 1000 positive and 2000 negative samples which has to be used. Also haartraining should have 20 stages and 2 nsplits. Hopefully if you follow this you would have developed a fairly true haarcascade.

If you are adventurous enough you could run the above with the webcam as

handdetect –cascade=haarcascade.xml 0

where 0 indicates the webcam. Assuming that your haarcascade is perfect you should be able to track your hand in real time.

Haarpy training!


INWARDi Technologies

OpenCV: Haartraining and all that jazz

Object detection in OpenCV can be done through Haartraining. OpenCV haartraining applications provide the ability to detect objects of interest to us like faces, eyes, moving cars etc. What is required is that we need to identify positive and negative samples and use the utilities that OpenCV provides us to train the application to be able to recognize the objects we want. The positive and negative samples are used to create classifiers which are then utilized to detect the objects that we intend to.

Fortunately the tar file (OpenCV-2.3.1a.tar.bz2) which can be downloaded from the OpenCV site already comes bundled with all the necessary utilities to create a fully trained haar classifier. We just to have to build and run the commands to create our our classifiers.

This post will look at the necessary steps for creating a haarclassifier. To get started with OpenCV please look at my earlier post Computer Vision:Getting started with OpenCV.

Once you have installed OpenCV look under modules/haartraining. All the necessary files are included.

There are 3 steps in this process
1) Create samples 2) Train and create a classified haar 3) Performance testing

Create Samples (createsamples.cpp) : This is the first utility that has to be executed. This utility takes as input positive samples (images of the object that we are interested in) and negative samples (images that do not include the object that we are interested in). The createsamples utility superimposes the objects that we want to recognize in various degrees of rotation against the background of the negative samples. The composite images file is then used to train the haar application.

The first step is to build createsamples.cpp. Make sure you include ../OpenCV-2.3./modules/haartraining. The build should include the following files
(createsamples.cpp, cvboost.cpp, cvhaarclassifier.cpp, cvhaartarining.cpp, cvsamples.cpp, cvcommon.cpp)

Once createsamples.cpp successfully builds we can create samples required for the training.
The command is
$./myhaartraining -img logo.png -vec samples1.jpg -bg bg.txt -w 20 -h 20

Where I have chosen the OpenCV logo as my object of interest.

The samples are created in samples1.jpg
The bg.txt contains a list of the negative samples included as below

./img/airplane.jpg
./img/baboon.jpg
./img/kid.jpg
./img/lena.jpg
-w stands for the width and -h stands for the height of the samples.


This will create superimposed positive samples in the negative sample background. A few of these a are shown below.

Make sure that the width and height are small i.e < 50 otherwise the haartraining application core dumps because of lack of memory. To check if the samples are created properly run a test round as follows run the command with the following options $./myhaartraining -img logo.png -vec samples1.jpg -bg bg.txt -n 10 - show -w 20 -h 20 where -n is the number of samples to be generated (default is 1000) -show will show a series of images with the positive samples superimposed on the negative samples. This can be used to check if createsamples utility is working properly. For a more thorough and detailed explanation see my post  Hand detection through haartraining: A Hands on approach

Haartrainer (haartraining.cpp): This utility takes as input the samples from the createsamples utility and creates a trained haar classifier. To build the haartrained files, build the haartraining.cpp. As before make sure you include the appropriate files along with the all the opencv libraries. The build files are
(haartraining.cpp, cvboost.cpp, cvhaarclassifier.cpp, cvhaartarining.cpp, cvsamples.cpp, cvcommon.cpp)

The command to use is
./haartrainer -data test2 -vec samples1.jpg -bg bg.txt -npos 1 -nneg 4 -nstages 20 -mem 500 -w 20 -h 20

In the above command test2 is the directory name in which the trained classifier is stored. The -vec option denotes the samples that were captured by the createsamples utility above. The bg.txt contains the negative samples file. The width and height have to be the same as used in the create samples utility. As mentioned before if the width and height are too large you the haartrainer will bail out with “Insufficient memory”

If the haartrainer executes successfully the test2 directory will have all the trained files and the directory will also have the haarclassifier as a test2.xml.

Once these two steps go through successfully we have to run the performance step

Performance (performance.cpp: This step is run to ensure that we have trained our application to properly recognize the object we intended to. As before the necessary file to build is performance.cpp. The files to include in the build are
(performance.cpp, cvboost.cpp, cvhaarclassifier.cpp, cvhaartarining.cpp, cvsamples.cpp, cvcommon.cpp)

Once this built successfuly you can now run using the command

./performance -data test2.xml -info bg.txt -w 20 -h 20

Make sure that the bg.txt contains the images from which the object has to be detected in the following
format
[positive filename] [# of objects] [[x y width height]

bg.txt
/img/logo5.jpg 1 145 100 20 20
./img/logo6.jpg 1 145 100 20 20
./img/logo3.jpg 1 145 100 45 45
./img/logo4.jpg 1 145 100 45 45
./img/airplane.jpg 1 145 100 45 45
./img/baboon.jpg 1 145 100 45 45
./img/kid.jpg 1 145 100 45 45
./img/lena.jpg 1 145 100 45 45
./img/opencv-logo2.png 1 145 100 35 35
./img/logo.png 1 145 100 45 45

When this is run we get


ganesh@localhost Debug]$ ./performance -data test3.xml -info bg.txt -w 20 -h 20
+================================+======+======+======+
| File Name | Hits |Missed| False|
+================================+======+======+======+
| ./img/logo5.jpg| 0| 1| 43|
+--------------------------------+------+------+------+
| ./img/logo6.jpg| 0| 1| 51|
+--------------------------------+------+------+------+
| ./img/logo3.jpg| 1| 0| 37|
+--------------------------------+------+------+------+
| ./img/logo4.jpg| 0| 1| 7|
+--------------------------------+------+------+------+
| ./img/airplane.jpg| 1| 0| 226|
+--------------------------------+------+------+------+
| ./img/baboon.jpg| 0| 1| 236|
+--------------------------------+------+------+------+
| ./img/kid.jpg| 0| 1| 1291|
+--------------------------------+------+------+------+
| ./img/lena.jpg| 0| 1| 188|
+--------------------------------+------+------+------+
| ./img/opencv-logo2.png| 0| 1| 3|
+--------------------------------+------+------+------+
| ./img/logo.png| 0| 1| 33|
+--------------------------------+------+------+------+
| Total| 2| 8| 2115|
+================================+======+======+======+

As can be seen the training has been too accurate. There are hits and misses along with a false
positive.

The number of positive and negative samples, the co-ordinates and the number of stages
all have to fine tuned to get the correct result.

Watch this space!

I will be back! Hasta la vista!

Please do take a look at my sequel to this post Hand detection through Haartraining: A Hands-on approach for a more robust haartraining method

INWARDi Technologies

Adding the OpenFlow variable in the IMS equation

IMS a non-starter: IP Multimedia Systems (IMS) has been the grand vision of this decade. Unfortunately it has remained just that, a vision, with sporadic deployments. IMS has been a non-starter in many respects. Operators and Network Providers somehow don’t find any compelling reason to re-architect the network with IMS network elements. There have been no killer applications too. But IP Multimedia Systems definitely hold enormous potential and a couple of breakthroughs in key technologies can result in the ‘tipping point’ of this great technology which promises access agnostic services including applications like video-conferencing, multi-player gaming, white boarding all using an all-IP backbone. In this context please do read my post "The Case for a Cloud based IMS solution"

SDNs, truly revolutionary: In this scenario a radically new, emerging concept is the Software Define Networks (SDNs). SDN is the result of pioneering effort by Stanford University and University of California, Berkeley and is based on the Open Flow Protocol and represents a paradigm shift to the way networking elements operate. SDNs consist of the OpenFlow Controller which is able to control network resources in a programmatic manner. These network resources can span routers, hubs and switches, known as a slice, and can be controlled through the OpenFlow Controller. The key aspect of the OpenFlow protocol is the ability to manage slices of virtualized resources from end-to-end. It is this particular aspect of Software Defined Networks (SDNs) and the OpenFlow protocol which can be leveraged in IMS networks.Do read my post Software Defined Networks: A glimpse of tomorrow for more details

This article looks at a way in which OpenFlow protocol can be included in the IMS fabric and provide for QoS in the IMS network. However, please note that this post is exploratory in nature and does not purport to be a well researched article. Nevertheless the idea is well worth mulling over.

QoS in IMS: The current method of ensuring differentiated QoS in IMS networks is through two key network elements, namely the Policy Decision Point (PDP) and the Policy Enforcement Point (PEP). The PDP retrieves the necessary policy rules (flow parameters), in response to a RSVP message, which it then sends to the PEP. The PEP then executes these instructions. In the IMS the Policy Control Function (PCF), in the P-CSCF, plays the role of the PDP. The PEP resides in the GGSN. In an IMS call flow, the SDP message is encapsulated within SIP and carries the QoS parameters. The PCF examines the parameters, retrieves appropriate policies and informs the PEP in the GGSN for that traffic flow.

OpenFlow Controller in P-CSCF: This post looks at a technique in which the OpenFlow Controller can be a part of the P-CSCF. The QoS parameters which come in the SDP message can be similarly examined. Instead of retrieving fixed policy rules, the OpenFlow Controller in the P-CSCF can be made to programmatically identify bandwidth speeds, the routers and the network slice through which the flow should flow. It would then inform the equivalent of the OpenFlow Switch in the GGSN which would control the necessary network resources end-to-end. The advantage of using OpenFlow Controller/OpenFlow switch to the PDP/PEP combination would be the ability to adapt the network flow according to bandwidth changes and traffic. The OpenFlow Controller will be able to dynamically re-route the traffic to alternate network resources, or a different ‘network slice’ in cases of congestion.

Conclusion: In my opinion, adding the OpenFlow Protocol to the IP Multimedia Switching fabric can provide for a much more control and better QoS in the network. It may also be able to provide for a lot more interesting applications to the already existing set of powerful applications

Assistive Technology and OpenCV

Assistive technology (AT) or Adaptive technology refers to technology or devices that assist people with disabilities. AT provides for a greater degree of freedom and independence by enabling disabled people to perform tasks that they were formerly unable to perform.
Examples of assistive technology include large computer keyboards for visually impaired, Braille buttons in elevators, hearing aids etc. People with learning abilities, for e.g dyslexics find text –to –speech (TTS) technology is useful.

In the context of Assistive technology, OpenCV can be used in multiple ways. OpenCV (Open Source Computer Vision) is a set of powerful APIs that can perform real time computer image processing. OpenCV is being used by many organizations in complex application like biometrics for recognizing fingerprints, face, to medical imaging for detection of tumors and cancerous cells. Applications of OpenCV have also been developed in office security for the detection of intruders to terrain mapping by spy planes and drones. OpenCV is truly a powerful tool for performing complex image processing operation.

One such application of OpenCV is the area of gesture detection and recognition. A successful implementation of gesture detection and recognition can have significant implications. It can be used to interpret sign language, the language of the deaf, and those with motor and speech disabilities.

Clearly the ability to recognize gestures is no easy task. The software has to be trained to initially recognize different gestures of sign language. An image processing tool that can recognize the symbols of sign language will be a boon to those with severe motor disabilities that they can use the keyboard for typing.

Imagine somebody who has motor disabilities being able to use the PC for browsing, accessing social networks all through sign language. Such a tool will really allow such people to live life in a much more regular way. Assuming the software is powerful enough the disable person will also be able to create documents through sign language.

If the power of OpenCV can be harnessed for gesture detection and interpretation the applications can be manifold. One such application would be sign language which would be a boon for many disabled people.

Added as an afterthought
While the post is about using OpenCV for recognition of sign language to empower disabled people, I think gesture recognition would be really welcome for everybody. Imagine being able to browse websites using gestures, scrolling up/down, going back/forward and pointing and clicking on links by simply waving the hand appropriately. If we could use gestures to flip through TV channels by snapping our fingers or waving left to right or right to left, that would be really cool. The possibilities are endless...

Do read my futuristic short story – The Anomaly for an interesting application of the above technology

INWARDi Technologies

Computer Vision: Getting started with OpenCV

OpenCV (Open Source Computer Vision) is a library of APIs aimed at enabling real time computer vision. OpenCV had Intel as the champion during its early developmental stages from 1999 to its first formal release in 2006. This post looks at the steps involved in installing OpenCV on your Linux machine and getting started with some simple programs. For the steps for installing OpenCV in Windows look at my post "Installing and using OpenCV with Visual Studio 2010 express"

As a first step download the tarball OpenCV-2.3.1a.tar.bz2 from the link below to directory
$HOME/opencv
http://sourceforge.net/projects/opencvlibrary/files/opencv-unix/2.3.1/

Unzip and and untar the bzipped file using

$ tar jxvf OpenCV-2.3.1a.tar.bz2
Now
$ cd OpenCV-2.3.1

You have to run cmake to configure the directories before running make. I personally found it
easier to run the cmake wizard using
$ cmake -i

Follow through all the prompts that the cmake wizard gives and make appropriate choices.

Once this complete

Run
$ make

Login as root

$ su – root
password: *******

Run
$ make install

This should install all the appropriate files and libraries in /usr/local/lib

Now assuming that everything is fine you should be good to go.

Start Eclipse, open a new C project.

Under Project->Properties->Settings->GCC Compiler ->Directories include the following 2 include paths

../opencv/OpenCV-2.3.1/include/opencv
../opencv/OpenCV-2.3.1/include/opencv2

Under
Project->Properties->Settings->GCC Linker ->Libraries in the library search path
include /usr/local/lib

Under libraries include the following
opencv_highgui , opencv_core , opencv_imgproc , opencv_highgui , opencv_ml , opencv_video, opencv_features2d , opencv_calib3d , opencv_objdetect , opencv_contrib , opencv_legacy , opencv_flann

(To get the list of libraries you could also run the following command)
pkg-config --libs /usr/local/lib/pkgconfig/opencv.pc
-L/usr/local/lib -lopencv_core -lopencv_imgproc -lopencv_highgui -lopencv_ml -lopencv_video -lopencv_features2d -lopencv_calib3d -lopencv_objdetect -lopencv_contrib -lopencv_legacy -lopencv_flann

Now you are ready to create your first OpenCV program

The one below will convert an image to test.png
#include "highgui.h"

int main( int argc, char** argv ) {
IplImage* img = cvLoadImage( argv[1],1);

cvSaveImage( "test.png", img, 0);
cvReleaseImage( &img );
return 0;
}

If you get a runtime error cannot find shared library
“ibopencv_core.so.2.3: cannot open shared object file: No such file or directory”
then you need to ensure that the linker knows the paths of the libraries.

The commands are as follows

$vi /etc/ld.so.conf.d/opencv.conf
Enter
/usr/local/lib
and save file

Now execute
$ldconfig /etc/ld.so.conf

You can check if everything is fine by running
$[root@localhost mycode]# ldconfig -v | grep open
ldconfig: /etc/ld.so.conf.d/kernel-2.6.32.26-175.fc12.i686.PAE.conf:6: duplicate hwcap 0 nosegneg
libopencv_gpu.so.2.3 -> libopencv_gpu.so.2.3.1
libopencv_ml.so.2.3 -> libopencv_ml.so.2.3.1
libopencv_legacy.so.2.3 -> libopencv_legacy.so.2.3.1
libopencv_objdetect.so.2.3 -> libopencv_objdetect.so.2.3.1
libopencv_video.so.2.3 -> libopencv_video.so.2.3.1
.....

Now re-build the code and everything should be fine.

Here's a second program to run various transformations to an image
IplImage* img = cvLoadImage( argv[1],1);
// create a window. Window name is determined by a supplied argument
cvNamedWindow( argv[1], CV_WINDOW_AUTOSIZE );
// Apply Gaussian smooth
//cvSmooth( img, img, CV_GAUSSIAN, 9, 9, 0, 0 );

cvErode (img,img,NULL,2);
// Display an image inside and window.
cvShowImage( argv[1], img );
//Save image
cvSaveImage( "/home/ganesh/Desktop/baby2.png", img, 0);

....

There are many samples also downloaded along with the installation. You can try them out. I found the facedetect.cpp sample interesting. It is based on Haar cacades and works really well. Compile facedetect.cpp under samples/c

Check it out. Including facedetect.cpp detecting my face in real time ...

Have fun with OpenCV.

Get going! Get hooked!

INWARDi Technologies

Technologies to watch: 2012 and beyond

Published in Telecom Asia - Technologies to watch:2012 and beyond
Published in Telecoms Europe - Hot technologies for 2012 and beyond

A keen observer of the technological firmament, today, will observe a grand spectacle of diverse technological events. Some technological trends will blaze a trail and will become trend setters while others will vanish without a trace. The factors that make certain technologies to endure in comparison to others could be many, ranging from pure necessity to a coolness factor, from innovativeness to a cost factor. This article looks at some of the technologies that are certain to be trail blazers in the years to come.

Software Defined Networks (SDNs): Software Defined Networks (SDNs) are based on the path breaking paradigm of separating the control of a network flow from the actual flow of data. SDN is the result of pioneering effort by Stanford University and University of California, Berkeley and is based on the Open Flow Protocol and represents a paradigm shift to the way networking elements operate. Software Defined Networks (SDN) decouples the routing and switching of the data flows and moves the control of the flow to a separate network element namely, the Flow controller. The motivation for this is that the flow of data packets through the network can be controlled in a programmatic manner. The OpenFlow Protocol has 3 components to it. The Flow Controller that controls the flows, the OpenFlow switch and the Flow Table and a secure connection between the Flow Controller and the OpenFlow switch. Software Define Networks (SDNs) also include the ability to virtualize the network resources. Virtualized network resources are known as a “network slice”. A slice can span several network elements including the network backbone, routers and hosts. The ability to control multiple traffic flows programmatically provides enormous flexibility and power in the hands of users. SDNs are bound to be the networks elements of the future.

Smart Grids: The energy industry is delicately poised for a complete transformation with the evolution of the smart grid concept. There is now an imminent need for an increased efficiency in power generation, transmission and distribution coupled with a reduction of energy losses. In this context many leading players in the energy industry are coming up with a connected end-to-end digital grid to smartly manage energy transmission and distribution. The digital grid will have smart meters, sensors and other devices distributed throughout the grid capable of sensing, collecting, analyzing and distributing the data to devices that can take action on them. The huge volume of collected data will be sent to intelligent device which will use the wireless 3G networks to transmit the data. Appropriate action like alternate routing and optimal energy distribution would then happen. Smart Grids are a certainty given that this technology addresses the dire need of efficient energy management. Smart Grids besides managing energy efficiently also save costs by preventing inefficiency and energy losses.

The NoSQL Paradigm: In large web applications where performance and scalability are key concerns a non –relational database like NoSQL is a better choice to the more traditional relational databases. There are several examples of such databases – the more reputed are Google’s BigTable, HBase, Amazon’s Dynamo, CouchDB & MongoDB. These databases partition the data horizontally and distribute it among many regular commodity servers. Accesses to the data are based on get(key) or set(key, value) type of APIs. Accesses to the data are based on a consistent hashing scheme for example the Distributed Hash Table (DHT) method. The ability to distribute data and the queries to one of several servers provides the key benefit of scalability. Clearly having a single database handling an enormous amount of transactions will result in performance degradation as the number of transaction increases. Applications that have to frequently access and manage petabytes of data will clearly have to move to the NoSQL paradigm of databases.

Near Field Communications (NFC): Near Field Communications (NFC) is a technology whose time has come. Mobile phones enabled with NFC technology can be used for a variety of purposes. One such purpose is integrating credit card functionality into mobile phones using NFC. Already the major players in mobile are integrating NFC into their newer versions of mobile phones including Apple’s iPhone, Google’s Android, and Nokia. We will never again have to carry in our wallets with a stack of credit cards. Our mobile phone will double up as a Visa, MasterCard, etc. NFC also allows retail stores to send promotional coupons to subscribers who are in the vicinity of the shopping mall. Posters or trailers of movies running in a theatre can be sent as multi-media clips when travelling near a movie hall. NFC also allows retail stores to send promotional coupons to subscribers who are in the vicinity of the shopping mall besides allowing exchanging contact lists with friends when they are close proximity.

The Other Suspects: Besides the above we have other usual suspects

Long Term Evolution (LTE): LTE enables is latest wireless technology that enables wireless access speeds of up to 56 Mbps. With the burgeoning interest in tablets, smartphones with the countless apps LTE will be used heavily as we move along.For a vision of where telecom is headed, do read my post 'The Future of Telecom".

Cloud Computing: Cloud Computing is the other technology that is bound to gain momentum in the years ahead. Besides obviating the need for upfront capital expenditure the cloud enables quick and easy deployment of applications. Moreover the elasticity of the cloud will make it irresistible to large enterprises and corporations.

The above is a list of technologies to watch as create new paths and blaze new trails. All these technologies are bound to transform the world as we know it and make our lives easier, better and more comfortable. These are the technologies that we need to focus on as we move bravely into our future. Do read my post for the year 2011 "Technology Trends - 2011 and beyond"

INWARDi Technologies

Tomorrow's wireless ecosystem

The wireless networks of today had its humble beginnings in 1924 when the first mobile radio was demonstrated. It was many years since this beginning, that a completely functional cellular network was established. The earliest systems were the analog 1G system that was demonstrated in 1978 in US with great success. The initial mobile systems were primarily used for making mobile voice calls. This continued for the next 2 decades as the network evolved to digital based 2G systems.

It was around 1999-2000 that ETSI standardized GPRS or 2.5G technology to use the cellular network for data. Though the early data rates, of 144 kbps, were modest, the entry of GPRS proved to be a turning point in technological history. GPRS provided the triple benefits of wireless connectivity, mobility and internet access. Technological advancement enabled faster and higher speeds of wireless, mobile access to the internet. The deployments of 3G enabled speeds of up to 2 Mbps for fixed access while LTE promised speeds of almost 56 Mbps per second coupled with excellent spectral efficiency.

The large increase of bandwidth along with mobility has allowed different technologies to take advantage of the wireless infrastructure for their purposes. While Wi-Fi networks based on 802.11 and WiMAX based on 802.16 will play a part in the wireless ecosystem this post looks at the role that will be played by cellular networks from 2G to 4G.

The cellular network with its feature of wireless access, mobility and the ability to handle voice, video and data calls will be the host of multiple disparate technologies as we move forward into the future. Below are listed some of the major users of the wireless network in the future

Mobile Phones: The cellular network was created to handle voice calls originating from mobile phones. A large part of mobile traffic will still be for mobile to mobile calls. As the penetration of the cellular networks occurs in emerging economies we can expect that there will be considerable traffic from voice calls. It is likely that as the concept of IP Multimedia System (IMS) finds widespread acceptance the mobile phone will also be used for making video calls. With the advent of the Smartphone this is a distinct possibility in the future.

Smartphones, tablets and Laptops: These devices will be the next major users of the cellular network. Smartphones, besides being able to make calls, also allow for many new compelling data applications. Exciting apps on tablets like the iPad and laptops consume a lot of bandwidth and use the GPRS, 3G or LTE network for data transfer. In fact in a recent report it has been found that a majority of data traffic in the wireless network are video. Consumers use the iPad and the laptop for watching videos on Youtube and for browsing using the wireless network.

Internet of Things (IoT): The internet of things, also known as M2M, envisages a network in which passive or intelligent devices are spread throughout the network and collect and transmit data to back end database. RFIDs were the early enablers of this technology. These sensors and intelligent devices will collect data and transmit the data using the wireless network. Applications for the Internet of Things range from devices that monitor and transmit data about the health of cardiac patients to being able to monitor the structural integrity of bridges.

Smart Grid: The energy industry is delicately poised for a complete transformation with the evolution of the smart grid concept. There is now an imminent need for an increased efficiency in power generation, transmission and distribution coupled with a reduction of energy losses. In this context many leading players in the energy industry are coming up with a connected end-to-end digital grid to smartly manage energy transmission and distribution. The digital grid will have smart meters, sensors and other devices distributed throughout the grid capable of sensing, collecting, analyzing and distributing the data to devices that can take action on them. The huge volume of collected data will be sent to intelligent device which will use the wireless 3G networks to transmit the data. Appropriate action like alternate routing and optimal energy distribution would then happen. The Smart Grid will be a major user of the cellular wireless network in the future.

Hence it can be seen the users of the wireless network will increase dramatically as we move forward into the future. Multiple technologies will compete for the available bandwidth. For handling this exponential growth in traffic we not only need faster speeds for the traffic but also sufficient spectrum available for use and it is necessary that ITU addresses the spectrum needs on a war footing.

It is thus clear that the telecom network will have to become more sophisticated and more technologically advanced as we move forward into the future.

Profiting from a cloud deployment

Cloud computing does offer enterprises and organizations a mixed bag of goodies. For one it provides for a utility style computing, the ability to grow and shrink with changing loads, zero upfront costs etc. The benefits of cloud computing are many but does it all add up to profit for an enterprise? That is the critical question that needs to be answered.

This post will take a look on what it takes for a cloud deployment to be profitable for an organization.

The critical parameters for any web application are latency and throughput. A well designed web application whether it is an e-retail site or an ad serving application will try to minimize the latency or response time while at the same time maximizing the throughput of the application. For any application while the latency can be kept within specified limits the throughput will tend to plateau at a certain level and will not increase with increasing traffic. Utilizing a larger instance can improve the throughput plateau slightly. In any case the reality is that throughput tends to flatten as the traffic is increased.

A typical cloud application will be made of several compute instances, database instances, DNS services etc. Cloud usage is billed by the hour. Hence we can represent the cost of a cloud deployment as follows

Cost (cloud deployment) = m * compute instance + n * database instance + o * network bytes + P
Where P = cost of DNS + Elastic IPs + other costs.

This can be represented by the formula
C = a * D * t
where C = cost of cloud deployment
D = costs per hour of the deployment
and ‘a’ is some arbitrary constant and ‘t’ is the time

Let us assume that for the cloud deployment we get a throughput of T.
The revenue for a web application whether it is an e-commerce site, an e-ticketing site or an ad serving engine will all depend on the throughput i.e. larger the throughput, larger the revenue and hence profit. We can then say that ‘R’ the revenue is

R (revenue) α k * T * t

In others words the revenue is proportional to the throughput.
Hence to determine the profitability of a particular cloud deployment we need to compare the cost of the deployment for a given throughput versus a projected profit margin. As long the cost of the deployment is less than the revenue arising from the throughput, the deployment will be profitable. This can be represented pictorially as below.

The graph clearly shows that for a profitable deployment
d/dt (k * T *t) > d/dt (a * D * t) or
k * T > a * D

Hence as can seen from the picture as long as the slope of the cumulative deployment costs are less that the slope of the revenue the deployment will be profitable.

INWARDi Technologies

Software Defined Networks (SDNs); A glimpse of tomorrow

Published in Telecom Asia, Jul 28,2011 - A glimpse into the future of networking
Published in Telecoms Europe, Jul 28,2011 - SDNs are new era for networking

Networks and networking, as we know it, is on the verge of a momentous change, thanks to a path breaking technological concept known as Software Defined Networks (SDN). SDN is the result of pioneering effort by Stanford University and University of California, Berkeley and is based on the Open Flow Protocol and represents a paradigm shift to the way networking elements operate.

Networks and network elements, of today, have been largely closed and have been based on proprietary architecture. In today’s network and switching and routing of data packets happen in the same network elements for e.g. the router.

Software Defined Networks (SDN) decouples the routing and switching of the data flows and moves the control of the flow to a separate network element namely, the flow controller. The motivation for this is that the flow of data packets through the network can be controlled in a programmatic manner. A Flow Controller can be typically implemented in a standard PC. In some ways this is reminiscent of Intelligent Networks and Intelligent Network Protocol which delinked the service logic from the switching and moved it a network element known as the Service Control Point.

The OpenFlow Protocol has 3 components to it. The Flow Controller that controls the flows, the OpenFlow switch and the Flow Table and a secure connection between the Flow Controller and the OpenFlow switch. The OpenFlow Protocol is an open source API specification for modifying the flow table that exists in all routers, Ethernet switches and hubs. The ability to securely control the flow of traffic programmatically opens ups amazing possibilities.

Alternatively, existing branded routers can implement the OpenFlow Protocol as an added feature to their existing routers and Ethernet switches. This will enable these routers and Ethernet switches to support both production traffic and research based traffic using the same set of network resources.

The single greatest advantage of separating the control and data plane of network routers and Ethernet switches is the ability to modify and control different traffic flows through a set of network resources. In addition to this benefit Software Define Networks (SDNs) also include the ability to virtualize the network resources. Virtualized network resources are known as a “network slice”. A slice can span several network elements including the network backbone, routers and hosts.

Computing resources can be virtualized through the use of the Hypervisor which abstracts the hardware and enables several guest OS to run in complete isolation. Similarly when a network element a FlowVisor, experimentally demonstrated, is used along with the OpenFlow Controller it is possible to virtualize the network resources. Hence each traffic flow gets a combination of bandwidth, routers, traffic flows and computing resources. Hence Software Defined Networks (SDNs) are also known as Virtualized Programmable Networks owing to the ability of different traffic flows being able to co-exist in perfect isolation of one another allowing for traffic flows through the resources to be controlled by programs in the Flow Controller.

The ability to manage different types of traffic flows across network resources opens up endless possibilities. SDNs have been successfully demonstrated in wireless handoffs between networks and in running multiple different flows through a common set of resources. SDNs in public and private clouds allow appropriate resources to be pooled during different times of the day based on the geographical location of the requests. Telcos could optimize the usage of their backbone network based on peak and lean traffic periods through the Core Network.

The OpenFlow Protocol has already gained widespread support in the industry and has resulted in the formation of the Open Networking Foundation (ONF). The members of ONF include behemoths like Google, Facebook, Yahoo, and Deutsche Telekom to networking giants like Cisco, Juniper, IBM and Brocade etc. Currently the ONF has around 43 member companies.

Software Define Networks is a tectonic shift in the way networks operate and truly represent the dawn of a new networking era. A related post of interest is "Adding the OpenFlow variable in the IMS equation"

INWARDi Technologies

The Case for Cloud Based IMS solution

IP Multimedia Systems (IMS) has been in the wings for some time. There have been several deployments by the major equipment manufacturers, but IMS is simply not happening. The vision of IMS is truly grandiose. IMS envisages an all-IP core with several servers known as Call Session Control Function (CSCF) participating to setup, maintain and release call sessions.

In the 3GPP Release 5 Architecture IMS draws an architecture of Proxy CSCF (P-CSCF), Serving CSCF(S-CSCF), Interrogating CSCF(I-CSCF), Breakout CSCF(B-CSCF), Home Subscriber Server(HSS) and Application Servers (AS) acting in concert in setting up, maintaining and release media sessions. The main protocols used in IMS are SIP/SDP for managing media sessions which could be voice, data or video and DIAMETER for connecting to the HSS and the Application Servers.

IMS is also access agnostic and is capable of handling landline or wireless calls over multiple devices from the mobile, laptop, PDA, smartphones or tablet PCs. The application possibilities of IMS are endless from video calling, live multi-player games to video chatting and mobile handoffs of calls from mobile phones to laptop. Despite the numerous possibilities IMS has not made prime time. While IMS technology paints a grand picture it has somehow not caught on. IMS as a technology, holds a lot of promise but has remained just that – promising technology.

The technology has not made the inroads into people’s imaginations or turned into a money spinner for Operators. One of the reasons may be that Operators are averse to investing enormous amounts into new technology and turning their network upside down.

This article provides an innovative approach to introducing IMS in the network by taking advantage of the public cloud!

Since IMS is an all-IP network and the protocol between the CSCF servers is SIP/SDP over TCP IP it can be readily seen that IMS is a prime candidate for the public cloud. An IMS architecture that has to be deployed on the cloud would have several instances of P-CSCFs, S-CSCFs, B-CSCFs, HSS and ASes all sitting on the cloud. An architectural diagram is shown below.

Deploying the CSCFs on the public cloud has multiple benefits. For one it a cloud deployment will eliminate the upfront CAPEX costs for the Operator. The cost savings can be passed on to the consumers whose video, data or voice calls will be cheaper. Besides, the absence of CAPEX will provide better margins to the operator. Lower costs to the consumer and better margins for the Operator is truly an unbeatable combination.

Also the elasticity of the cloud can be taken advantage of by the operator who can start small and automatically scale as the user base grows.

Thus a cloud based IMS deployment is truly a great combination both for the subscriber, the operator and the equipment manufactures. The cloud’s elasticity will automatically provide for growth as the irresistibility of IMSes high speed video applications catches public imagination.

If IMS as a technology needs to become common place then Operators should plan on deploying their IMS on the public cloud and reap the manifold benefits.A related post of relevance is "Adding the OpenFlow variable to the IMS equation".

INWARDi Technologies