Saturday, April 12, 2014

Configure and run Hadoop 2 mapreduce job history server

#1 - Create the yarn pid dir
<HADOOP_YARN_HOME>/work/pids

#2 - Enable log aggregation
 <property>
        <name>yarn.log-aggregation-enable</name>
        <value>true</value>
 </property>

#3 - Add defult yarn pid dir to the yarnenv.sh

# default pid directory 
if [ "$YARN_PID_DIR" = "" ]; then
  YARN_LOG_DIR="$HADOOP_YARN_HOME/work/pids"
fi

#4 - After you start your dfs (<HADOOP_HOME>/sbin/start-dfs.sh) and yarn (<HADOOP_HOME>/sbin/start-yarn.sh) start your yarn mapreduce history server
[hduser@hostname sbin]$ ./mr-jobhistory-daemon.sh start historyserver
starting historyserver, logging to <HADOOP_HOME>/logs/mapred-hduser-historyserver-step-cca-sjc-lnx-lab9.out
[hduser@hostname sbin]$ 

For a nice hadoop 2 multi cluster quick installation click here