Tuesday, September 25, 2012

The dark side of the Internet

Imagine a life without the internet. You can’t! That’s how inextricably enmeshed the internet is in our lives. Kids learn to play “angry birds” on the PC before they learn to say “duh”, school children hobnob on Facebook and many of us regularly browse, upload photos, watch videos and do a dozen other things on the internet.

So on one side of the internet is the user with his laptop, smartphones or iPad. So what’s on the other side of the Internet and what is the Internet? The Internet is a global system of interconnected computer networks that uses the TCP/IP protocol.  The Internet or more generally the internet is network of networks made of hundreds of millions of computers.

During the early days the internet was most probably used for document retrieval, email and browsing. But with the passage of time the internet and the uses of the internet have assumed gigantic proportions. Nowadays we use the internet to search billions of documents, share photographs with our online community, blog and stream video. So, while the early internet was populated with large computers to perform the tasks, the computations of the internet of today require a substantially larger infrastructure. The internet is now powered by datacenters. Datacenters contain anywhere between 100s to 100,000s servers. A server is a more beefed up computer that is designed for high performance sans a screen and a keyboard. Datacenters contain servers stacked over one another on a rack.

These datacenters are capable of handling thousands of simultaneous users and delivering results in split second. In this age of exploding data and information overload where split second responses and blazing throughputs are the need of the hour, datacenters really fill the need. But there is a dark side to these data centers. The issue is that these datacenters consume a lot of energy and are extremely power hungry besides. In fact out of a 100% of utility power supplied to datacenter only 6 – 12 % is used for actual computation. The rest of the power is either used for air conditioning or is lost through power distribution.

In fact a recent article “Power, pollution and the Internet” in the New York Times claims that “Worldwide, the digital warehouses use about 30 billion watts of electricity, roughly equivalent to the output of 30 nuclear power plants.”  Further the article states that “it is estimated that Google’s data centers consume nearly 300 million watts and Facebook’s about 60 million watts or 60 MW”

For e.g. It is claimed that Facebook annually draws 509 million kilowatt hours  of power for its  data centers  (see Estimate: Facebook running 180,000 servers). This article further concludes “that the social network is delivering 54.27 megawatts (MW) to servers” or approximately 60 MW to its datacenter.  The other behemoths in this domain including Google, Yahoo, Twitter, Amazon, Microsoft, and Apple all have equally large or larger data centers consuming similar amounts of energy.  Recent guesstimates have placed Google’s server count at more than 1 million and consuming approximately 220 MW. Taking a look at the power generation capacities of power plants in India we can see that 60 MW is between to 20%-50% of the power generation capacity of  power plants  while 220 MW is entire capacity of medium sized power plants (see List of power stations in India”)

One of the challenges that these organizations face is the need to make the datacenter efficient. New techniques are constantly being used in the ongoing battle to reduce energy consumption in a data center. These tools are also designed to boost a data center's Power Usage Effectiveness (PUE) rating. Google, Facebook, Yahoo, and Microsoft compete to get to the lowest possible PUE measure in their newest data centers. The earlier datacenters used to average 2.0 PUE while advanced data centers these days aim for lower ratings of the order of 1.22 or 1.16 or lower.

In the early days of datacenter technology the air-conditioning systems used to cool by brute force. Later designs segregated the aisles as hot & cold aisle to improve efficiency. Other technique use water as a coolant along with heat exchangers. A novel technique was used by Intel recently in which servers were dipped in oil. While Intel claimed that this improved the PUE rating there are questions about the viability of this method considering the messiness of removing or inserting new circuit board from the servers.

Datacenters are going to proliferate in the coming days as information continues to explode. The hot new technology “Cloud Computing” is nothing more that datacenters which uses virtualization technique or the ability to run different OS on the hardware improving server utilization.

Clearly the thrust of technology in the days to come will be on identifying renewable sources of energy and making datacenters more efficient.

Datacenters will become more and more prevalent in the internet and technologies to make them efficient as we move to a more data driven world

Tuesday, September 18, 2012

The moving edge of computing




“The moving edge of computing computes and having computed moves on…” We could thus rephrase the Rubaiyat of Omar Khayyam’s “The moving hand…” Computing technology has really advanced by leaps and bounds. We are now in a new era of computing. We are in the midst of “intelligent and cognitive” computing.

From the initial days of number crunching by languages of FORTRAN, to the procedural methodology of Pascal or C and later the object oriented paradigm of C++ and Java we have now come a long way.  In this age of information overload technologies that can just solve problems through steps & procedures are no longer adequate. We need technology to detect complex patterns, trends, understand nuances in human language and to automatically resolve problems. In this new era of computing the following 3 technologies are furthering the frontiers of computing technology.

Predictive Analytics
By 2016 130 Exabyte’s (130 * 2 ^ 60) will rip through the internet. The number of mobile devices will exceed the human population this year, 2012 and by 2016 the number of connected devices will touch almost 10 billion. The devices connected to the net will range from mobiles, laptops, tablets, sensors and the millions of devices based on the “internet of things”. All these devices will constantly spew data on the internet. A hot and happening trend in computing is the ability to make business and strategic decisions by determining patterns, trends and outliers among mountains of data. Predictive analytics will be a key discipline in our future and experts will be much sought after. Predictive analytics uses statistical methods to mine intelligence, information and patterns in structured, unstructured and streams of data. Predictive analytics will be applied across many domains from banking, insurance, retail, telecom, energy. There are also applications for energy grids, water management, besides determining user sentiment by mining data from social networks etc.
Cognitive Computing
The most famous technological product in the domain of cognitive computing is IBM’s supercomputer Watson. IBM’s Watson is an artificial intelligence computer system capable of answering questions posed in natural language. IBM’s supercomputer Watson is best known for successfully trouncing a national champion in the popular US TV quiz competition, Jeopardy. What makes this victory more astonishing is that IBM’s Watson had to successfully decipher the nuances of natural language and pick the correct answer.  Following the success at Jeopardy, IBM’s Watson supercomputer has now  been employed by a leading medical insurance firm in US to diagnose medical illnesses and to recommend treatment options for patients. Watson will be able to analyze 1 million books, or roughly 200 million pages of information. The other equally well known mobile app is Siri the voice recognition app on the iPhone. The earlier avatar of cognitive computing was expert systems based on Artificial Intelligence. These expert systems were inference engines that were based on knowledge rules. The most famous among the expert systems were “Dendral” and “Mycin”. We appear to be on the cusp of tremendous advancement in cognitive computing based on the success of IBM’s Watson.
 Autonomic Computing
This is another computing trend that will become prevalent in the networks of tomorrow. Autonomic computing refers to the self-managing characteristics of a network. Typically it signifies the ability of a network to self-heal in the event of failures or faults. Autonomic network can quickly localize and isolate faults in the network while keeping other parts of the network unaffected. Besides these networks can quickly correct and heal the faulty hardware without human intervention. Autonomic networks are typical in smart grids where a fault can be quickly isolated and the network healed without resulting in a major outage in the electrical grid.

These are truly exciting times in computing as we move towards true intelligence!

Tuesday, September 4, 2012

Tips for building a decent HTA/HTML application


This post includes some basic tips to consider while building a HTA/HTML application. Clearly I am no Web guru knowledgeable of the various technologies that make the Web zip and zing. Here are my 2 cents

Any HTML page will contain tags for e.g
etc.
Any self-respecting Web page will include a vertical or a horizontal menu. For this you would need to use a Cascade Style Sheet (CSS). Rather than re-inventing the wheel do re-use the cool menu items & CSS style sheets at the following site http://www.dynamicdrive.com/style/layouts/

You will also be adding textboxes, radio buttons, submit  and reset buttons. Assuming you have all the form controls in a single form you need to use the following

Consider a form
Enter the file name containing the server list (srvr_list.txt) :

For this to get the input from the text box use
file = form1.servers.value

Using a radio button is a little tricky. Radio buttons belong to a group and the way to handle this in VBscript is as follows

            Include subdirectories
            Yes   
            No

This is done by iterating through the radio button list
for each radio in form1.Radio1

    If radio.checked = true Then
        
         If radio.value = "Yes" then
                  objInputFormat.recurse = -1
         ElseIf radio.value = "No" Then                 
                   objInputFormat.recurse = 0
         End If
    End if

Next

Handling a Submit requires that you handle the onClick action as below

This indicates that when you click the Submit button you will invoke the subroutine fileSize

A Reset button is of type “reset”. As long as you enclose all the controls in a single form hitting the reset button will clear all the controls.


Populating output in a table: To populate output dynamically in a table I used MS Tabular Data Control (TDC) when the output is written to a CSV file. The cool thing about this is that it automatically paginates output and display “n” entries per page instantaneously. I had tried various thing like populating using DOM, InnerHTML but all of those failed miserably when the o/p was very large. With TDC I can display ~30K entries instantaneously from a CSV file.

This is done as below

           

           

           

           


The CLASSID is the TDC control. Make sure you choose the correct delimiters and text qualifiers. Populate your CSV file.

You then need to connect this with Datasrc as below
                   
Number
                   
Path
                   
File Size
                   
Last Write Time
                                                           
                                                           
           

           

           

           

           

Add cool first,previous,next and last buttons as follows
 
       
       
       
       
       


Finally in your sub routine which you invoke to populate the table make sure you rebind your table with the new data as below

       data2.DataURL = ""
         data2.DataURL = "c:\\files.csv"
         data2.reset()

This is shown below

Thursday, July 26, 2012

Slicing and dicing with LogParser & VBA


LogParser  is probably one of the most understated utilities from Microsoft. LogParser 2.2. Log parser is a powerful and a versatile tool that provides a SQL like query access to text-based data such as log files, XML files and CSV files, as well as key data sources on the Windows such as the Event Log, the Registry, the file system, and Active Directory. There are so many things that you can do to search and collate data on Windows with LogParser. The nice thing about it is that the query is SQL like and fairly intuitive.
On a UNIX/Linux system we would have to run a shell command for e.g. (ls –lrt) and pipe it to “sort”,”awk” or a “sed” utility. LogParser is all this rolled into one. If you have been reading my earlier post “Building a respectable VBA with Excel Application” will realize that using LogParser with VBA is a fairly potent combination and you can build nifty applications quickly.
Display files in directories
For e.g. to display file sizes in specific directories or under all directories we can create VBA GUI as below
The logParser commands are to display all files in the directory with Size, LastWriteTime & name is below
Set objLogParser = CreateObject(“MSUtil.LogQuery”)
Set objInputFormat = _
CreateObject(“MSUtil.LogQuery.FileSystemInputFormat”)
If OptionButton1.value = True Then
objInputFormat.recurse = -1
End If
If OptionButton2.value = True Then
objInputFormat.recurse = 0
End If
strQuery = “SELECT TO_LOWERCASE (Name) AS NewName,  Size, Path, LastWriteTime FROM ‘” &   files & “‘ ORDER BY LastWriteTime ASC”
Set objRecordSet = objLogParser.Execute(strQuery, objInputFormat)
Do While Not objRecordSet.AtEnd
Set objRecord = objRecordSet.GetRecord
strPath = objRecord.GetValue(“Path”)
fileSize = objRecord.GetValue(“Size”)
lastWriteTime = objRecord.GetValue(“LastWriteTime”)
objRecordSet.MoveNext
Loop
Top N files
You can display the Top N files by size, lastWriteTime or name with a suitable VBA GUI as below
This can be written in logParser as
Set objLogParser = CreateObject(“MSUtil.LogQuery”)
Set objInputFormat = _
CreateObject(“MSUtil.LogQuery.FileSystemInputFormat”)
If OptionButton7.value = True Then
strQuery = “SELECT TOP ” & topN & ” TO_LOWERCASE (Name) AS NewName, Size, Path, LastWriteTime FROM ‘” & _
files & “‘ WHERE NOT Attributes LIKE ‘%D%’ ORDER BY Size DESC”
ElseIf OptionButton8.value = True Then
strQuery = “SELECT TOP ” & topN & ” TO_LOWERCASE (Name) AS NewName, Size, Path, LastWriteTime FROM ‘” & _
files & “‘ WHERE NOT Attributes LIKE ‘%D%’ ORDER BY lastWriteTime ASC”
ElseIf OptionButton9.value = True Then
strQuery = “SELECT TOP ” & topN & ” TO_LOWERCASE (Name) AS NewName, Size, Path, LastWriteTime FROM ‘” & _
files & “‘ WHERE NOT Attributes LIKE ‘%D%’ ORDER BY NewName ASC”
End If
The SpinButtons subroutines can be updates as follows
Private Sub SpinButton1_SpinDown()
If TextBox7.value <= 1 Then
MsgBox (“Cannot decrement below 1″)
TextBox7.value = 1
Exit Sub
Else
TextBox7.value = TextBox7.value – 1
End If
End Sub
Private Sub SpinButton1_SpinUp()
TextBox7.value = TextBox7.value + 1
End Sub
Extension based disk management
logParser can also be used to select all files with specified extensions for e.g. tmp,.log etc
Set objLogParser = CreateObject(“MSUtil.LogQuery”)
Set objInputFormat = _
CreateObject(“MSUtil.LogQuery.FileSystemInputFormat”)
files = baseDirectory & “\” & fileExt
strQuery = “SELECT TO_LOWERCASE (Name) AS NewName,  Size ,Path, LastWriteTime FROM ‘” & _   files & “‘ ORDER BY LastWriteTime ASC”
Set objRecordSet = objLogParser.Execute(strQuery, objInputFormat)
You can also search Event Logs with LogParser
Event Logs
Set objLogParser = CreateObject(“MSUtil.LogQuery”)
Set objInputFormat = _
CreateObject(“MSUtil.LogQuery.EventLogInputFormat”)
If OptionButton1.Value = True Then
strQuery = “SELECT TimeGenerated, EventID, EventTypeName,Message, Strings, SourceName FROM Application WHERE EventID IN ” & str
ElseIf OptionButton2.Value = True Then
strQuery = “SELECT TimeGenerated, EventID, EventTypeName, Message, Strings, SourceName FROM System WHERE EventID IN ” & str
ElseIf OptionButton3.Value = True Then
strQuery = “SELECT TimeGenerated, EventID, EventTypeName, Message, Strings, SourceName FROM Security WHERE EventID IN ” & str
ElseIf OptionButton4.Value = True Then
strQuery = “SELECT TimeGenerated, EventID, EventTypeName, Message, Strings, SourceName FROM Setup WHERE EventID IN ” & str
End If
Set objRecordSet = objLogParser.Execute(strQuery, objInputFormat)
LogParser is really a cool utility and when combined with VBA can really help in developing nifty applications.

Monday, June 18, 2012

Working with binary trees in Lisp


Success finally! I have been trying to create and work with binary trees in Lisp for sometime. It is really difficult in Lisp in which there are no real data structures. Trees are so obvious in a language like C where one can visualize the left & right branches with pointers. The structure of a node in C is
struct node tree{
int value
struct node *left;
struct node *right;
}
Lisp has no such structures. Lisp is a list or at best a list of lists. It is really how the program interprets the nested list of lists that makes the list a binary or a n-ary tree. As I mentioned before I have not had a whole lot of success in creating the binary tree in Lisp for quite sometime. Finally I happened to come across a small version of adding an element to a set in “Structure and Interpretation of Computer Programs (SICP)” by Harold Abelson, Gerald and Julie Sussman. This is probably one of the best books I have read in a long time and contains some truly profound insights into computer programming.
I adapted the Scheme code into my version of a adding a node. Finally I was able to code inorder, pre-order and post order traversals.
In this version the a node of a binary tree is represented as
(node left right) so
node -> (car tree)
left_branch -> (cadr tree)
right_branch is -> (caddr tree)
Here is the code
(defun entry (tree)
(car tree))
(defun left-branch (tree)
(cadr tree))
(defun right-branch (tree)
(caddr tree))
//Create node in a binary tree
(defun make-tree (entry left right)
(list entry left righta))
// Insert element into tree
add (x tree)
(cond ((null tree) (make-tree x nil nil))
((= x (entry tree)) tree)
((< x (entry tree)) (make-tree (entry tree) (add x
(left-branch tree)) (right-branch tree)))
((> x (entry tree)) (make-tree (entry tree)
(left-branch tree) (add x (right-branch tree))))))
So I can now create a tree with a create-tree function

(defun create-tree(elmnts)
(dolist (x elmnts)
(setf tree (add x tree)))
)
(setf tree nil)
NIL
(setf lst (list 23 12 1 4 5 28 4 9 10 45 89))
(23 12 1 4 5 28 4 9 10 45 89)
(create-tree lst)
NIL

Now I display the tree
tree
(23 (12 (1 NIL (4 NIL (5 NIL (9 NIL (10 NIL NIL))))) NIL) (28 NIL (45 NIL (89 NIL NIL))))

This can be represented pictorially as


Now I created the 3 types of traversals
(defun inorder (tree)
(cond ((null tree))
(t (inorder (left-branch tree))
(print (entry tree))
(inorder (right-branch tree))))))
(defun preorder (tree)
(cond ((null tree))
(t (print (entry tree))
(preorder (left-branch tree))
(preorder (right-branch tree))))))
(defun postorder (tree)
(cond ((null tree))
(t (postorder (left-branch tree))
(postorder (right-branch tree))
(print (entry tree)))))
[89]> (inorder tree)
1
4
5
9
10
12
23
28
45
89
[90]> (preorder tree)
23
12
1
4
5
9
10
28
45
89
T
[91]> (postorder tree)
10
9
5
4
1
12
89
45
28
23
23

Sunday, June 10, 2012

Test driving Apache Hadoop: Standalone & pseudo-distributed mode


The Hadoop paradigm originated from Google and is used in crunching large data sets. It is ideally suited for applications like Big Data, creating an inverted index used by search engines and other problems which require terabytes of data to processed in parallel. One key aspect of Hadoop is that it is made up of commodity servers. Hence server, disk crashes or network issues are assumed to be norm rather than an exception.
The Hadoop paradigm is made of the Map-Reduce  & the HDFS parts. The Map-Reduce has 2 major components to it. The Map part takes as input key- value pairs and emits a transformed key-value pair. For e.g Map could count the number of occurrences of words or created an inverted index of a word and its location in a document. The Reduce part takes as input the emitted key-value pairs of the Map output and performs another operation on the inputs from the Map part for.e.g summing up the counts of words. A great tutorial on Map-Reduce can be found at http://developer.yahoo.com/hadoop/tutorial/module4.html
The HDFS (Hadoop Distributed File System) is the special storage that is used in tandem with the Map-Reduce algorithm. The HDFS distributes data among Datanodes. A Namenode maintains the meta data of where individual pieces of data are stored.
To get started with Apache Hadoop download a stable release of Hadoop from
a) Install Hadoop on your system.
Apache Hadoop requires Java to be installed. Download and install Java on your machine from
b) After you have installed java set $JAVA_HOME in
/usr/share/hadoop/templates/conf/hadoop-env.sh
export JAVA_HOME=${JAVA_HOME}
c) Create a user hduser in group hadoop
For this click  Applications->Other->User & Groups
Choose Add User – hduser  &  Add Group – hadoop
Choose properties and add hduser to the hadoop group.
Standalone Operation
Under root do
/usr/sbin/sshd
then
$ssh localhost


If you cannot do a ssh to localhost with passphrase then do the following
$ ssh-keygen -t dsa -P ” -f ~/.ssh/id_dsa
$ cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys
Now  re-run
$ssh localhost – This time it should go fine
Create a directory input and copy *.xml files from conf/
$mkdir input
$cp /usr/share/hadoop/templates/conf/*.xml input
Then execute the following. This searches for the string “dfs*” in all the XML files under the input directory
$/usr/bin/hadoop jar /usr/share/hadoop/hadoop-examples-1.0.3.jar grep input output ‘dfs[a-z.]+’
You should see
[root@localhost hadoop]# /usr/bin/hadoop jar /usr/share/hadoop/hadoop-examples-1.0.3.jar grep input output ‘dfs[a-z.]+’
12/06/10 13:00:51 INFO util.NativeCodeLoader: Loaded the native-hadoop library
..
…
12/06/10 13:01:45 INFO mapred.JobClient:     Reduce output records=38
12/06/10 13:01:45 INFO mapred.JobClient:     Virtual memory (bytes) snapshot=0
12/06/10 13:01:45 INFO mapred.JobClient:     Map output records=38
Where it indicates that there are 38 such record strings with dfs* in it.
If you get an error
java.lang.OutOfMemoryError: Java heap space
then increase the heap size from 128 to 1024 as below
…
mapred.child.java.opts
-server -Xmx1024m -Djava.net.preferIPv4Stack=true
….
Pseudo distributed mode
In the pseudo distributed mode separate Java processes are started for the Job Tracker which schedules tasks, the Task tracker which executes tasks and the Namenode which contains the data
Note: Though I created user “hduser” in the group hadoop and also added the user to the wheel group with admin privileges all the commands were issued under root as I was getting a permission denied in the hduser account.
a) Execute the following commands under root
$mkdir -p /home/hduser/hadoop/tmp
$chown hduser:hadoop /home/hduser/hadoop/tmp
$chmod 750 /home/hduser/hadoop/tmp
b) Do the following
Note: Files  core-site.xml, mapred-site,xml & hdfs-site.xml exist under
/usr/share/hadoop/templates/conf & /etc/hadoop
It appears that Apache hadoop gives precedence to /etc/hadoop. So add the following between
In file /etc/hadoop/core-site.xml
hadoop.tmp.dir
/home/hduser/hadoop/tmp
A base for other temporary directories.
fs.default.name
hdfs://localhost:54310
The name of the default file system.  Either the
literal string “local” or a host:port for NDFS.
true
In /etc/hadoop/mapred-site.xml
mapred.job.tracker
localhost:54311
true
In /etc/hadoop/hdfs-site.xml add
dfs.replication
1
c) Since the pseudo distributed mode will use the HDFS file system we need to format this.So run the following command
$/usr/bin/hadoop namenode -format
[root@localhost hduser]#  /usr/bin/hadoop namenode -format
12/06/10 15:48:16 INFO namenode.NameNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG:   host = localhost.localdomain/127.0.0.1
STARTUP_MSG:   args = [-format]
STARTUP_MSG:   version = 1.0.3
STARTUP_MSG:   build = https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.0 -r 1335192; compiled by ‘hortonfo’ on Tue May  8 20:16:59 UTC 2012
************************************************************/
…
12/06/10 15:48:17 INFO common.Storage: Image file of size 110 saved in 0 seconds.
12/06/10 15:48:18 INFO common.Storage: Storage directory /home/hduser/hadoop/tmp/dfs/name has been successfully formatted.
12/06/10 15:48:18 INFO namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at localhost.localdomain/127.0.0.1
d) Now start all the Hadoop processes
$/usr/sbin/start-all.sh
starting namenode, logging to /var/log/hadoop/root/hadoop-root-namenode-localhost.localdomain.out
localhost: starting datanode, logging to /var/log/hadoop/root/hadoop-root-datanode-localhost.localdomain.out
localhost: starting secondarynamenode, logging to /var/log/hadoop/root/hadoop-root-secondarynamenode-localhost.localdomain.out
starting jobtracker, logging to /var/log/hadoop/root/hadoop-root-jobtracker-localhost.localdomain.out ocalhost: starting tasktracker, logging to /var/log/hadoop/root/hadoop-root-tasktracker-localhost.localdomain.out
Verify that all processes have started by executing /usr/java/jdk1.7.0_04/bin/jps
[root@localhost hduser]# /usr/java/jdk1.7.0_04/bin/jps
10971 DataNode
10866 NameNode
11077 SecondaryNameNode
11147 JobTracker
11264 TaskTracker
11376 Jps
You will see JobTracker,taskTracker,NameNode,DataNode and SecondaryNameNode
You can also do netstat -plten | grep java
tcp        0      0 0.0.0.0:50090               0.0.0.0:*                   LISTEN      0          166832     11077/java
tcp        0      0 0.0.0.0:50060               0.0.0.0:*                   LISTEN      0          167407     11264/java
tcp        0      0 0.0.0.0:50030               0.0.0.0:*                   LISTEN      0          166747     11147/java
tcp        0      0 0.0.0.0:50070               0.0.0.0:*                   LISTEN      0          165669     10866/java
tcp        0      0 0.0.0.0:50010               0.0.0.0:*                   LISTEN      0          166951     10971/java
tcp        0      0 0.0.0.0:50075               0.0.0.0:*                   LISTEN      0          166955     10971/java
tcp        0      0 127.0.0.1:55839             0.0.0.0:*                   LISTEN      0          166816     11264/java
tcp        0      0 0.0.0.0:50020               0.0.0.0:*                   LISTEN      0          165843     10971/java
tcp        0      0 127.0.0.1:54310             0.0.0.0:*                   LISTEN      0          165535     10866/java
tcp        0      0 127.0.0.1:54311             0.0.0.0:*                   LISTEN      0          166733     11147/java
e) Now copy files from your local directory /home/hduser/input  to the HDFS file system
Now you can check the web interface for JobTracker & NameNode
This is as per mapred-site.xml & hdfs-site.xml in /conf directory. They are at
ñJobTracker – http://localhost:50030/
f) Copy files from your local directory to HDFS
$/usr/bin/hadoop dfs -copyFromLocal /home/hduser/input /user/hduser/input
Ensure that the files have been copies by listing the contents of HDFS
g) Check that files have been copied
$/usr/bin/hadoop dfs -ls /user/hduser/input
Found 9 items
-rw-r–r–   1 root supergroup       7457 2012-06-10 10:31 /user/hduser/input/capacity-scheduler.xml
-rw-r–r–   1 root supergroup       2447 2012-06-10 10:31 /user/hduser/input/core-site.xml
-rw-r–r–   1 root supergroup       2300 2012-06-10 10:31 /user/hduser/input/core-site_old.xml
-rw-r–r–   1 root supergroup       5044 2012-06-10 10:31 /user/hduser/input/hadoop-policy.xml
-rw-r–r–   1 root supergroup       7595 2012-06-10 10:31 /user/hduser/input/hdfs-site.xml
h) Now execute the grep functionality
[root@localhost hduser]# /usr/bin/hadoop jar /usr/share/hadoop/hadoop-examples-1.0.3.jar grep /user/hduser/input /user/hduser/output ‘dfs[a-z.]+’
12/06/10 10:34:22 INFO util.NativeCodeLoader: Loaded the native-hadoop library
….
12/06/10 10:34:23 INFO mapred.JobClient: Running job: job_201206101010_0003
12/06/10 10:34:24 INFO mapred.JobClient:  map 0% reduce 0%
12/06/10 10:34:48 INFO mapred.JobClient:  map 11% reduce 0%
…
…
12/06/10 10:35:21 INFO mapred.JobClient:  map 88% reduce 22%
12/06/10 10:35:24 INFO mapred.JobClient:  map 100% reduce 22%
12/06/10 10:35:27 INFO mapred.JobClient:  map 100% reduce 29%
12/06/10 10:35:36 INFO mapred.JobClient:  map 100% reduce 100%
12/06/10 10:35:42 INFO mapred.JobClient: Job complete: job_201206101010_0003
….
12/06/10 10:36:16 INFO mapred.JobClient:     Reduce input groups=3
12/06/10 10:36:16 INFO mapred.JobClient:     Combine output records=0
12/06/10 10:36:16 INFO mapred.JobClient:     Physical memory (bytes) snapshot=180502528
12/06/10 10:36:16 INFO mapred.JobClient:     Reduce output records=36
12/06/10 10:36:16 INFO mapred.JobClient:     Virtual memory (bytes) snapshot=695119872
12/06/10 10:36:16 INFO mapred.JobClient:     Map output records=36
i) Check the result
[root@localhost hduser]# /usr/bin/hadoop dfs -cat /user/hduser/output/*
6          dfs.data.dir
2          dfs.
2          dfs.block.access.token.enable
2          dfs.cluster.administrators
2          dfs.datanode.address
2          dfs.datanode.data.dir.perm
2          dfs.datanode.http.address
2          dfs.datanode.kerberos.principal
2          dfs.datanode.keytab.file
2          dfs.exclude
…..
j)/usr/sbin/stop-all.sh
Have fun with hadoop…