前陣子參加了趨勢的雲端設計比賽，雖然沒寫出什麼東西，不過卻學了這套軟體，說穿了就是把自己寫好的程式，將他分散化到不同的server上運行，不過小弟是窮人，沒有那麼多的server，所以就只能先裝在自己的筆電上面，以下就是紀錄我的安裝過程：
Environment: Ubuntu 9.04 Hadoop (現在最新的版本是0.20)
需要安裝:
Java1.6
sudo apt-get install sun-java6-jdk
JDK會被放在/usr/lib/jvm/java-6-sun
這個時候可以去check一下/etc/jvm 應該會像
# /etc/jvm # # This file defines the default system JVM search order. Each # JVM should list their JAVA_HOME compatible directory in this file. # The default system JVM is the first one available from top to # bottom. /usr/lib/jvm/java-6-sun /usr/lib/jvm/java-gcj /usr/lib/jvm/ia32-java-1.5.0-sun /usr/lib/jvm/java-1.5.0-sun /usrssh rsync sudo apt-get install ssh rsync
因為Hadoop之間要用SSH溝通
Adding Hadoop User
sudo addgroup hadoop sudo adduser --ingroup hadoop hadoop
設定SSH
su - hadoop ssh-keygen -t rsa -P "" cat $HOME/.ssh/id_rsa.pub >> $HOME/.ssh/authorized_keys這樣是建立使用SSH連線不用密碼 接著可以用ssh localhost看看是否正常
Install Hadoop
cd /opt sudo tar zxvf hadoop-0.20.0.tar.gz sudo mv hadoop-0.20.0 hadoop sudo chown -R hadoop:hadoop hadoop
也可以使用人家包好的deb http://www.cloudera.com/hadoop-deb
Configuration
/opt/hadoop/conf/hadoop-env
修改路徑
export JAVA_HOME=/usr/lib/jvm/java-6-sun
/opt/hadoop/conf/hadoop-site.xml
修改內容
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!-- Put site-specific property overrides in this file. -->
<configuration>
<property>
<name>hadoop.tmp.dir</name>
<value>/your/path/to/hadoop/tmp/dir/hadoop-${user.name}</value>
<description>A base for other temporary directories.</description>
</property>
<property>
<name>fs.default.name</name>
<value>hdfs://localhost:54310</value>
<description>The name of the default file system. A URI whose
scheme and authority determine the FileSystem implementation. The
uri's scheme determines the config property (fs.SCHEME.impl) naming
the FileSystem implementation class. The uri's authority is used to
determine the host, port, etc. for a FileSystem.</description>
</property>
<property>
<name>mapred.job.tracker</name>
<value>localhost:54311</value>
<description>The host and port that the MapReduce job tracker runs
at. If "local", then jobs are run in-process as a single map
and reduce task.
</description>
</property>
<property>
<name>dfs.replication</name>
<value>1</value>
<description>Default block replication.
The actual number of replications can be specified when the file is created.
The default is used if replication is not specified in create time.
</description>
</property>
</configuration>Formatting the name node
建立namenode
/opt/hadoop/bin/hadoop namenode -format
啟動Hadoop
/opt/hadoop/bin/start-all.sh
Hadoop Web Interfaces
Hadoop 提供了Web 介面 (conf/hadoop-default.xml):
http://localhost:50030/ - web UI for MapReduce job tracker(s)
http://localhost:50060/ - web UI for task tracker(s)
http://localhost:50070/ - web UI for HDFS name node(s)
