CONFIGURE AND RUN HADOOP AND HDFS

 To install Hadoop can also be run on a single-node in a pseudo-distributed mode where each Hadoop daemon runs in a separate Java process.

Procedure:

Step 1: Download Hadoop 0.18.3 from the following website viz.    http://hadoop.apache.org/common/releases.html

Step 2: Extract the hadoop tar.gz file as follows:-

$tar -xvzf hadoop-0.20.2.tar.gz

Step 3: Set the PATH and CLASSPATH variables appropriately in your .profile file and load it.

Example:

export JAVA_HOME=/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.9

Step 4: The following files will have to be modified to complete the Hadoop setup:

 

Setup NAMENODE service:

Open the file and enter the following in between the <configuration></configuration> tag:

vi /home/hadoop/hadoop/conf/core-site.xml

<configuration>

 <property>

  <name>hadoop.tmp.dir</name>

  <value>ipaddress:port</value>

  <description>A base for other temporary directories.</description>

 </property>

 

 <property>

  <name>fs.default.name</name>

  <value>/home/hadoop/hdfsdrive</value> 

 </property>

</configuration>

 

 

Setup JOBTRACKER service:

By default, the folder contains /home//hadoop/hadoop/conf/mapred-site.xml file which has to be renamed/copied with the name mapred-site.xml:

vi /home/hadoop/hadoop/conf/mapred-site.xml

The mapred-site.xml file is used to specify which framework is being used for MapReduce.
We need to enter the following content in between the <configuration></configuration> tag:

<configuration>

 <property>

  <name>mapred.job.tracker</name>

  <value>ipaddress:host</value>

 </property>

</configuration>


Setup JOBTRACKER service:

vi /home/hadoop/hadoop/conf/masters

ipaddress

Setup SLAVES service:

It is used to specify the directories which will be used as the namenode and the datanode on that host.

vi /home/hadoop/hadoop/conf/slaves

ipaddress (namenode)

Step 5: To create a necessary directory named path in hdfs

            mkdir /home/hadoop/hdfsdrive

Step 6: To format the Namenode

            Hadoop namenode –format

Step 7: To create password-less setup:

            ssh-keygen

            ssh-copy-id –I /home/hadoop/.ssh/id-rsa.pub hadoop@ipaddress

Step 8: Start hadoop service

            start-all.sh

Step 9: Stop hadoop service:

            stop-all.sh

Comments

Post a Comment

Popular posts from this blog

How to create Animated 3d chart with R.

  The rgl package is the best option to build 3d charts in R. Please see this post for an introduction to 3d scatterplots using it. It also provides the plot3d() and play3d() functions that allow to animate the 3d chart, and eventually to export the result at a .gif format. Here is an application to the famous iris dataset, with a nice animated 3d scatterplot chart. Arguments x, y, z vectors of points to be plotted. Any reasonable way of defining the coordinates is acceptable. See the function  xyz.coords  for details. xlab, ylab, zlab labels for the coordinates. type For the default method, a single character indicating the type of item to plot. Supported types are: 'p' for points, 's' for spheres, 'l' for lines, 'h' for line segments from z = 0, and 'n' for nothing. For the mesh3d method, one of 'shade', 'wire', or 'dots'. Partial matching is used. col the color to be used for plotted...
Read more

Linux/Unix Commands frequently used

  Linux/Unix Commands ls  --- lists your files ls -l  --- lists your files in 'long format', which contains lots of useful information, e.g. the exact size of the file, who owns the file and who has the right to look at it, and when it was last modified. ls -a  --- lists all files, including the ones whose filenames begin in a dot, which you do not always want to see. There are many more options, for example to list files by size, by date, recursively etc. more  filename  --- shows the first part of a file, just as much as will fit on one screen. Just hit the space bar to see more or  q  to quit. You can use  / pattern  to search for a pattern. gzip  filename  --- compresses files, so that they take up much less space. Usually text files compress to about half their original size, but it depends very much on the size of the file and the nature of the co...
Read more

R Programming Introduction

R Download Link- https://cran.r-project.org/bin/windows/base/old/3.4.0/ R Studio Download link- https://www.rstudio.com/products/rstudio/download/  R is a programming language and software environment for statistical analysis, graphics representation and reporting. R was created by Ross Ihaka and Robert Gentleman at the University of Auckland, New Zealand, and is currently developed by the R Development Core Team. The core of R is an interpreted computer language which allows branching and looping as well as modular programming using functions. R allows integration with the procedures written in the C, C++, .Net, Python or FORTRAN languages for efficiency. R is freely available under the GNU General Public License, and pre-compiled binary versions are provided for various operating systems like Linux, Windows and Mac. R is free software distributed under a GNU-style copy left, and an official part of the GNU project called  GNU S . Evolution of R R was initially writ...
Read more