{"id":952,"date":"2021-03-11T11:47:43","date_gmt":"2021-03-11T03:47:43","guid":{"rendered":"https:\/\/www.linuxdevops.cn\/?p=952"},"modified":"2021-03-16T11:45:34","modified_gmt":"2021-03-16T03:45:34","slug":"hadoop-set-up-a-single-node-cluster","status":"publish","type":"post","link":"https:\/\/www.linuxdevops.cn\/2021\/03\/hadoop-set-up-a-single-node-cluster\/","title":{"rendered":"Hadoop\uff1a\u8bbe\u7f6e\u5355\u8282\u70b9\u7fa4\u96c6\u3002"},"content":{"rendered":"
\u672c\u6587\u6863\u4ecb\u7ecd\u4e86\u5982\u4f55\u8bbe\u7f6e\u548c\u914d\u7f6e\u5355\u8282\u70b9Hadoop\u5b89\u88c5\uff0c\u4ee5\u4fbf\u60a8\u53ef\u4ee5\u4f7f\u7528Hadoop MapReduce\u548cHadoop\u5206\u5e03\u5f0f\u6587\u4ef6\u7cfb\u7edf\uff08HDFS\uff09\u5feb\u901f\u6267\u884c\u7b80\u5355\u7684\u64cd\u4f5c\u3002<\/p>\n
Linux\u6240\u9700\u7684\u8f6f\u4ef6\u5305\u62ec\uff1a<\/p>\n
\u5982\u679c\u60a8\u7684\u7fa4\u96c6\u6ca1\u6709\u5fc5\u9700\u7684\u8f6f\u4ef6\uff0c\u5219\u9700\u8981\u5b89\u88c5\u5b83\u3002<\/p>\n
\u4f8b\u5982\u5728Ubuntu Linux\u4e0a\uff1a<\/p>\n
$ sudo apt-get install ssh\n $ sudo apt-get install pdsh<\/code><\/pre>\n\u4e0b\u8f7d<\/h2>\n
\u8981\u83b7\u5f97Hadoop\u53d1\u884c\u7248\uff0c\u8bf7\u4ece\u5176\u4e2d\u4e00\u4e2aApache Download Mirrors<\/a>\u4e0b\u8f7d\u6700\u65b0\u7684\u7a33\u5b9a\u7248\u672c\u3002<\/p>\nwget https:\/\/mirrors.tuna.tsinghua.edu.cn\/apache\/hadoop\/common\/hadoop-3.3.0\/hadoop-3.3.0.tar.gz<\/code><\/pre>\n\u51c6\u5907\u542f\u52a8Hadoop\u96c6\u7fa4<\/h2>\n
\u89e3\u538b\u7f29\u4e0b\u8f7d\u7684Hadoop\u53d1\u884c\u7248\u3002\u5728\u53d1\u884c\u7248\u4e2d\uff0c\u7f16\u8f91\u6587\u4ef6etc\/hadoop\/hadoop-env.sh<\/code>\u6765\u5b9a\u4e49\u4e00\u4e9b\u53c2\u6570\uff0c\u5982\u4e0b\u6240\u793a\uff1a<\/p>\ntar -zxvf hadoop-3.3.0.tar.gz<\/code><\/pre>\n #\u8bbe\u7f6e\u4e3aJava\u5b89\u88c5\u7684\u6839\u76ee\u5f55\n export JAVA_HOME=\/usr\/lib\/jdk1.8.0_212<\/code><\/pre>\n\u5c1d\u8bd5\u4ee5\u4e0b\u547d\u4ee4\uff1a<\/p>\n
$ bin\/hadoop<\/code><\/pre>\n\u8fd9\u5c06\u663e\u793ahadoop\u811a\u672c\u7684\u7528\u6cd5\u6587\u6863\u3002<\/p>\n
\u73b0\u5728\uff0c\u60a8\u53ef\u4ee5\u4ee5\u4e09\u79cd\u53d7\u652f\u6301\u7684\u6a21\u5f0f\u4e4b\u4e00\u542f\u52a8Hadoop\u96c6\u7fa4\uff1a<\/p>\n
\n- \u672c\u5730\uff08\u72ec\u7acb\uff09\u6a21\u5f0f<\/a><\/li>\n
- \u4f2a\u5206\u5e03\u5f0f\u6a21\u5f0f<\/a><\/li>\n
- \u5168\u5206\u5e03\u5f0f\u6a21\u5f0f<\/a><\/li>\n<\/ul>\n
\u72ec\u7acb\u8fd0\u884c<\/h2>\n
\u9ed8\u8ba4\u60c5\u51b5\u4e0b\uff0cHadoop\u88ab\u914d\u7f6e\u4e3a\u4ee5\u975e\u5206\u5e03\u5f0f\u6a21\u5f0f\u4f5c\u4e3a\u5355\u4e2aJava\u8fdb\u7a0b\u8fd0\u884c\u3002\u8fd9\u5bf9\u4e8e\u8c03\u8bd5\u5f88\u6709\u7528\u3002<\/p>\n
\u4e0b\u9762\u7684\u793a\u4f8b\u590d\u5236\u89e3\u538b\u7f29\u7684conf\u76ee\u5f55\u4ee5\u7528\u4f5c\u8f93\u5165\uff0c\u7136\u540e\u67e5\u627e\u5e76\u663e\u793a\u7ed9\u5b9a\u6b63\u5219\u8868\u8fbe\u5f0f\u7684\u6bcf\u4e2a\u5339\u914d\u9879\u3002\u8f93\u51fa\u88ab\u5199\u5165\u7ed9\u5b9a\u7684\u8f93\u51fa\u76ee\u5f55\u3002<\/p>\n
$ mkdir input\n $ cp etc\/hadoop\/*.xml input\n $ bin\/hadoop jar share\/hadoop\/mapreduce\/hadoop-mapreduce-examples-3.2.2.jar grep input output 'dfs[a-z.]+'\n $ cat output\/*<\/code><\/pre>\n\u4f2a\u5206\u5e03\u5f0f\u64cd\u4f5c<\/h2>\n
Hadoop\u4e5f\u53ef\u4ee5\u4ee5\u4f2a\u5206\u5e03\u5f0f\u6a21\u5f0f\u5728\u5355\u8282\u70b9\u4e0a\u8fd0\u884c\uff0c\u5176\u4e2d\u6bcf\u4e2aHadoop\u5b88\u62a4\u7a0b\u5e8f\u90fd\u5728\u5355\u72ec\u7684Java\u8fdb\u7a0b\u4e2d\u8fd0\u884c\u3002<\/p>\n
\u914d\u7f6e<\/h3>\n
\u4f7f\u7528\u4ee5\u4e0b\u5185\u5bb9\uff1a<\/p>\n
etc\/hadoop\/core-site.xml:<\/p>\n
<configuration>\n <property>\n <name>fs.defaultFS<\/name>\n <value>hdfs:\/\/localhost:9000<\/value>\n <\/property>\n<\/configuration><\/code><\/pre>\netc\/hadoop\/hdfs-site.xml:<\/p>\n
<configuration>\n <property>\n <name>dfs.replication<\/name>\n <value>1<\/value>\n <\/property>\n<\/configuration><\/code><\/pre>\n\u8bbe\u7f6e\u65e0\u5bc6\u7801SSH<\/h3>\n
\u73b0\u5728\u68c0\u67e5\u60a8\u662f\u5426\u53ef\u4ee5\u5728\u4e0d\u4f7f\u7528\u5bc6\u7801\u7684\u60c5\u51b5\u4e0bSSH\u5230\u672c\u5730\u4e3b\u673a\uff1a<\/p>\n
$ ssh localhost<\/code><\/pre>\n\u5982\u679c\u6ca1\u6709\u5bc6\u7801\u5c31\u65e0\u6cd5SSH\u5230\u672c\u5730\u4e3b\u673a\uff0c\u8bf7\u6267\u884c\u4ee5\u4e0b\u547d\u4ee4\uff1a<\/p>\n
$ ssh-keygen -t rsa -P '' -f ~\/.ssh\/id_rsa\n $ cat ~\/.ssh\/id_rsa.pub >> ~\/.ssh\/authorized_keys\n $ chmod 0600 ~\/.ssh\/authorized_keys<\/code><\/pre>\n\u6267\u884c<\/h3>\n
\u4ee5\u4e0b\u8bf4\u660e\u662f\u5728\u672c\u5730\u8fd0\u884cMapReduce\u4f5c\u4e1a\u3002\u5982\u679c\u8981\u5728YARN\u4e0a\u6267\u884c\u4f5c\u4e1a\uff0c\u8bf7\u53c2\u9605YARN\u5728\u5355\u8282\u70b9\u4e0a<\/a>\u3002<\/p>\n\n- \n
\u683c\u5f0f\u5316\u6587\u4ef6\u7cfb\u7edf\uff1a<\/p>\n
$ bin\/hdfs namenode -format<\/code><\/pre>\n<\/li>\n- \n
\u542f\u52a8NameNode\u5b88\u62a4\u7a0b\u5e8f\u548cDataNode\u5b88\u62a4\u7a0b\u5e8f\uff1a<\/p>\n
$ sbin\/start-dfs.sh<\/code><\/pre>\nhadoop\u5b88\u62a4\u7a0b\u5e8f\u65e5\u5fd7\u8f93\u51fa\u5c06\u5199\u5165$ HADOOP_LOG_DIR<\/code>\u76ee\u5f55\uff08\u9ed8\u8ba4\u4e3a$ HADOOP_HOME\/logs<\/code>\uff09\u3002<\/p>\n<\/li>\n- \n
\u6d4f\u89c8Web\u754c\u9762\u4ee5\u67e5\u627eNameNode\uff1b\u9ed8\u8ba4\u60c5\u51b5\u4e0b\uff0c\u5b83\u5728\u4ee5\u4e0b\u4f4d\u7f6e\u53ef\u7528\uff1a<\/p>\n
\n- NameNode- http:\/\/localhost:9870\/<\/a><\/li>\n<\/ul>\n<\/li>\n
- \n
\u8bbe\u7f6e\u6267\u884cMapReduce\u4f5c\u4e1a\u6240\u9700\u7684HDFS\u76ee\u5f55\uff1a<\/p>\n
$ bin\/hdfs dfs -mkdir \/user\n $ bin\/hdfs dfs -mkdir \/user\/<\/code><\/pre>\n<\/li>\n- \n
\u5c06\u8f93\u5165\u6587\u4ef6\u590d\u5236\u5230\u5206\u5e03\u5f0f\u6587\u4ef6\u7cfb\u7edf\u4e2d\uff1a<\/p>\n
$ bin\/hdfs dfs -mkdir input\n $ bin\/hdfs dfs -put etc\/hadoop\/*.xml input<\/code><\/pre>\n<\/li>\n- \n
\u8fd0\u884c\u63d0\u4f9b\u7684\u4e00\u4e9b\u793a\u4f8b\uff1a<\/p>\n
$ bin\/hadoop jar share\/hadoop\/mapreduce\/hadoop-mapreduce-examples-3.2.2.jar grep input output 'dfs[a-z.]+'<\/code><\/pre>\n<\/li>\n- \n
\u68c0\u67e5\u8f93\u51fa\u6587\u4ef6\uff1a\u5c06\u8f93\u51fa\u6587\u4ef6\u4ece\u5206\u5e03\u5f0f\u6587\u4ef6\u7cfb\u7edf\u590d\u5236\u5230\u672c\u5730\u6587\u4ef6\u7cfb\u7edf\u5e76\u68c0\u67e5\u5b83\u4eec\uff1a<\/p>\n
$ bin\/hdfs dfs -get output output\n $ cat output\/*<\/code><\/pre>\n\u6216\u8005<\/p>\n
\u67e5\u770b\u5206\u5e03\u5f0f\u6587\u4ef6\u7cfb\u7edf\u4e0a\u7684\u8f93\u51fa\u6587\u4ef6\uff1a<\/p>\n
$ bin\/hdfs dfs -cat output\/*<\/code><\/pre>\n<\/li>\n- \n
\u5b8c\u6210\u540e\uff0c\u4f7f\u7528\u4ee5\u4e0b\u547d\u4ee4\u505c\u6b62\u5b88\u62a4\u8fdb\u7a0b\uff1a<\/p>\n
$ sbin\/stop-dfs.sh<\/code><\/pre>\n<\/li>\n<\/ol>\n\u5728\u5355\u4e2a\u8282\u70b9\u4e0a\u7684YARN<\/h3>\n
\u60a8\u53ef\u4ee5\u901a\u8fc7\u8bbe\u7f6e\u4e00\u4e9b\u53c2\u6570\u5e76\u53e6\u5916\u8fd0\u884cResourceManager\u5b88\u62a4\u7a0b\u5e8f\u548cNodeManager\u5b88\u62a4\u7a0b\u5e8f\uff0c\u4ee5\u4f2a\u5206\u5e03\u5f0f\u6a21\u5f0f\u5728YARN\u4e0a\u8fd0\u884cMapReduce\u4f5c\u4e1a\u3002<\/p>\n
\u4ee5\u4e0b\u6307\u4ee4\u5047\u5b9a\u4e0a\u8ff0\u6307\u4ee4<\/a>\u76841.\u301c4.\u6b65\u9aa4\u5df2\u6267\u884c\u3002<\/p>\n\n- \n
\u914d\u7f6e\u53c2\u6570\u5982\u4e0b\uff1a<\/p>\n
etc\/hadoop\/mapred-site.xml<\/code>:<\/p>\n\n \n mapreduce.framework.name<\/name>\n yarn<\/value>\n <\/property>\n \n mapreduce.application.classpath<\/name>\n $HADOOP_MAPRED_HOME\/share\/hadoop\/mapreduce\/*:$HADOOP_MAPRED_HOME\/share\/hadoop\/mapreduce\/lib\/*<\/value>\n <\/property>\n<\/configuration><\/code><\/pre>\netc\/hadoop\/yarn-site.xml<\/code>:<\/p>\n\n \n yarn.nodemanager.aux-services<\/name>\n mapreduce_shuffle<\/value>\n <\/property>\n \n yarn.nodemanager.env-whitelist<\/name>\n JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,CLASSPATH_PREPEND_DISTCACHE,HADOOP_YARN_HOME,HADOOP_MAPRED_HOME<\/value>\n <\/property>\n<\/configuration><\/code><\/pre>\n\u542f\u52a8ResourceManager\u5b88\u62a4\u7a0b\u5e8f\u548cNodeManager\u5b88\u62a4\u7a0b\u5e8f\uff1a<\/p>\n
$ sbin\/start-yarn.sh<\/code><\/pre>\n<\/li>\n- \n
\u6d4f\u89c8Web\u754c\u9762\u4ee5\u627e\u5230ResourceManager\uff1b\u9ed8\u8ba4\u60c5\u51b5\u4e0b\uff0c\u5b83\u5728\u4ee5\u4e0b\u4f4d\u7f6e\u53ef\u7528\uff1a<\/p>\n
\n- ResourceManager- http:\/\/localhost:8088\/<\/a><\/li>\n<\/ul>\n<\/li>\n
- \n
\u8fd0\u884cMapReduce\u4f5c\u4e1a\u3002<\/p>\n<\/li>\n
- \n
\u5b8c\u6210\u540e\uff0c\u4f7f\u7528\u4ee5\u4e0b\u547d\u4ee4\u505c\u6b62\u5b88\u62a4\u8fdb\u7a0b\uff1a<\/p>\n
$ sbin\/stop-yarn.sh<\/code><\/pre>\n<\/li>\n<\/ol>\n\u5168\u5206\u5e03\u5f0f\u8fd0\u884c<\/h2>\n
\u6709\u5173\u8bbe\u7f6e\u5b8c\u5168\u5206\u5e03\u5f0f\u7684\u975e\u91cd\u8981\u7fa4\u96c6\u7684\u4fe1\u606f\uff0c\u8bf7\u53c2\u89c1\u201c\u7fa4\u96c6\u8bbe\u7f6e\u201d<\/a>\u3002<\/p>\n","protected":false},"excerpt":{"rendered":"Hadoop\uff1a\u8bbe\u7f6e\u5355\u8282\u70b9\u7fa4\u96c6\u3002 \u76ee\u7684 \u5148\u51b3\u6761\u4ef6 \u652f\u6301\u5e73\u53f0 \u5fc5\u5907\u8f6f\u4ef6 \u5b89\u88c5\u8f6f\u4ef6 \u4e0b\u8f7d \u51c6\u5907\u542f\u52a8Hadoop\u96c6\u7fa4 <\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[17,18,74],"tags":[],"yoast_head":"\n
Hadoop\uff1a\u8bbe\u7f6e\u5355\u8282\u70b9\u7fa4\u96c6\u3002 - Linux\u81ea\u52a8\u5316\u8fd0\u7ef4<\/title>\n\n\n\n\n\n\n\n\n\n\n\n\n