大數(shù)據(jù)之---偽分布式部署之終極篇-創(chuàng)新互聯(lián)
------------------------------軟件版本--------------------------------------
目前創(chuàng)新互聯(lián)已為超過(guò)千家的企業(yè)提供了網(wǎng)站建設(shè)、域名、網(wǎng)絡(luò)空間、成都網(wǎng)站托管、企業(yè)網(wǎng)站設(shè)計(jì)、平武網(wǎng)站維護(hù)等服務(wù)，公司將堅(jiān)持客戶(hù)導(dǎo)向、應(yīng)用為本的策略，正道將秉承"和諧、參與、激情"的文化，與客戶(hù)和合作伙伴齊心協(xié)力一起成長(zhǎng)，共同發(fā)展。
RHEL6.8 hadoop2.8.1 apache-maven-3.3.9 findbugs-1.3.9 protobuf-2.5.0.tar.gz jdk-8u45
------------------------------軟件版本---------------------------------------
1.Hadoop
宏觀: Hadoop為主的生態(tài)圈 hadoop flume........
狹義: apache hadoop hadoop.apache.org
2.Hadoop(存儲(chǔ)+計(jì)算+資源和作業(yè)調(diào)度)
hadoop1.x
HDFS 存儲(chǔ)
MapReduce 計(jì)算+資源和作業(yè)調(diào)度
hadoop2.x 企業(yè)正在用
HDFS 存儲(chǔ)
MapReduce 計(jì)算
YARN 資源和作業(yè)調(diào)度平臺(tái) 計(jì)算組件都會(huì)on yarn
hadoop3.x ???
EC技術(shù)：Erasure Encoding 簡(jiǎn)稱(chēng)EC，是Hadoop3給HDFS拓展的一種新特性，用來(lái)解決存儲(chǔ)空間文件。
YARN：提供YARN的時(shí)間軸服務(wù)V.2，以便用戶(hù)和開(kāi)發(fā)人員可以對(duì)其進(jìn)行測(cè)試，并提供反饋意見(jiàn)。
優(yōu)化Hadoop Shell腳本
重構(gòu)Hadoop Client Jar包
支持隨機(jī)Container
MapReduce任務(wù)級(jí)本地優(yōu)化
支持多個(gè)NameNode
部分默認(rèn)服務(wù)端口被改變
支持文件系統(tǒng)連接器
DataNode內(nèi)部添加了負(fù)載均衡
重構(gòu)后臺(tái)程序和任務(wù)對(duì)管理
2.Maven部署
blog
2.1解壓
[root@hadoop1 softwore]# pwd |
2.2配置mavne目錄
2.3查看配置文件和 解壓我們準(zhǔn)備好的倉(cāng)庫(kù)文件
3.Hadoop編譯
3.1解壓
3.2查看pom.xml
3.3查看BUILDING.txt
Requirements:編譯軟件環(huán)境要求
* Unix System
* JDK 1.7+
* Maven 3.0 or later
* Findbugs 1.3.9 (if running findbugs)
* ProtocolBuffer 2.5.0
* CMake 2.6 or newer (if compiling native code), must be 3.0 or newer on Mac
* Zlib devel (if compiling native code)
* openssl devel (if compiling native hadoop-pipes and to get the best HDFS encryption performance)
* Linux FUSE (Filesystem in Userspace) version 2.6 or above (if compiling fuse_dfs)
* Internet connection for first build (to fetch all Maven and Hadoop dependencies)
3.4 JDK部署
| [root@hadoop1 softwore]# tar -zxvf jdk-8u45-linux-x64.gz -C /usr/java[root@hadoop1 softwore]# ls -ld /usr/java/* drwxr-xr-x 8 root root 4096 12月 15 2016 /usr/java/djdk1.7.0_79 drwxr-xr-x 8 uucp 143 4096 4月 11 2015 /usr/java/jdk1.8.0_45 drwxr-xr-x 8 uucp 143 4096 10月 7 2015 /usr/java/jdk1.8.0_65 [root@hadoop1 softwore]# vim /etc/profile export JAVA_HOME=/usr/java/jdk1.8.0_45 [root@hadoop1 softwore]# source /etc/profile |
3.5 MAVEN
[root@hadoop000 hadoop-2.8.1-src]# mvn --version
Apache Maven 3.3.9 (bb52d8502b132ec0a5a3f4c09453c07478323dc5; 2015-11-11T00:41:47+08:00)
Maven home: /opt/software/apache-maven-3.3.9
Java version: 1.8.0_45, vendor: Oracle Corporation
Java home: /usr/java/jdk1.8.0_45/jre
Default locale: en_US, platform encoding: UTF-8
OS name: "linux", version: "2.6.32-431.el6.x86_64", arch: "amd64", family: "unix"
[root@hadoop000 hadoop-2.8.1-src]#
3.6 FINDBUGS
[root@hadoop000 findbugs-1.3.9]# findbugs -version
1.3.9
[root@hadoop000 findbugs-1.3.9]#
3.7 PROTOCBUF
[root@hadoop000 local]# protoc --version
libprotoc 2.5.0
[root@hadoop000 local]#
3.8 OTHER
3.9 編譯
mvn clean package -Pdist,native -DskipTests -Dtar
3.10 解讀
4.hadoop部署
單機(jī)部署 進(jìn)程沒(méi)有
偽分布式部署 進(jìn)程存在+1節(jié)點(diǎn) 開(kāi)發(fā)
集群部署 進(jìn)程存在+n節(jié)點(diǎn) 開(kāi)發(fā)/生產(chǎn)
下載的包: src 源代碼包里面不包含jar 小
不帶src或者帶bin 編譯好的組件 大
4.1解壓
tar -xzvf hadoop-2.8.1.tar.gz
chown -R root:root hadoop-2.8.1
4.2解讀解壓文件
[root@hadoop000 hadoop-2.8.1]# ll
total 148
drwxrwxr-x. 2 root root 4096 Jun 2 2017 bin
drwxrwxr-x. 3 root root 4096 Jun 2 2017 etc
drwxrwxr-x. 2 root root 4096 Jun 2 2017 include
drwxrwxr-x. 3 root root 4096 Jun 2 2017 lib
drwxrwxr-x. 2 root root 4096 Jun 2 2017 libexec
-rw-rw-r--. 1 root root 99253 Jun 2 2017 LICENSE.txt
-rw-rw-r--. 1 root root 15915 Jun 2 2017 NOTICE.txt
-rw-r--r--. 1 root root 1366 Jun 2 2017 README.txt
drwxrwxr-x. 2 root root 4096 Jun 2 2017 sbin
drwxrwxr-x. 4 root root 4096 Jun 2 2017 share
[root@hadoop000 hadoop-2.8.1]#
bin 執(zhí)行命令的shell
etc 配置文件
lib 庫(kù)
sbin 啟動(dòng)和關(guān)閉hadoop
share jar
[root@hadoop000 hadoop-2.8.1]# rm -f bin/*.cmd
[root@hadoop000 hadoop-2.8.1]# rm -f sbin/*.cmd
[root@hadoop000 hadoop-2.8.1]#
[root@hadoop000 hadoop-2.8.1]# ll bin
total 348
-rwxrwxr-x. 1 root root 139387 Jun 2 2017 container-executor
-rwxrwxr-x. 1 root root 6514 Jun 2 2017 hadoop
-rwxrwxr-x. 1 root root 12330 Jun 2 2017 hdfs
-rwxrwxr-x. 1 root root 6237 Jun 2 2017 mapred
-rwxrwxr-x. 1 root root 1776 Jun 2 2017 rcc
-rwxrwxr-x. 1 root root 156812 Jun 2 2017 test-container-executor
-rwxrwxr-x. 1 root root 14416 Jun 2 2017 yarn
[root@hadoop000 hadoop-2.8.1]#
4.3配置環(huán)境變量
[root@hadoop000 ~]# vi /etc/profile
export HADOOP_HOME=/opt/software/hadoop-2.8.1
export PATH=$HADOOP_HOME/bin:$PROTOC_HOME/bin:$FINDBUGS_HOME/bin:$MVN_HOME/bin:$JAVA_HOME/bin:$PATH
[root@hadoop000 ~]# source /etc/profile
[root@hadoop000 ~]# which hadoop
/opt/software/hadoop-2.8.1/bin/hadoop
4.4配置core-site文件
etc/hadoop/core-site.xml:
<configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://localhost:9000</value>
</property>
</configuration>
etc/hadoop/hdfs-site.xml:
<configuration>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
</configuration>
4.5ssh
[root@hadoop000 ~]# rm -rf .ssh
[root@hadoop000 ~]# ssh-keygen
[root@hadoop000 ~]# cd .ssh
[root@hadoop000 .ssh]# ll
total 8
-rw-------. 1 root root 1671 May 13 21:47 id_rsa
-rw-r--r--. 1 root root 396 May 13 21:47 id_rsa.pub
[root@hadoop000 .ssh]# cat id_rsa.pub >> authorized_keys
[root@hadoop000 .ssh]#
[root@hadoop000 ~]# ssh localhost date
The authenticity of host 'localhost (::1)' can't be established.
RSA key fingerprint is ec:85:86:32:22:94:d1:a9:f2:0b:c5:12:3f:ba:e2:61.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'localhost' (RSA) to the list of known hosts.
Sun May 13 21:49:14 CST 2018
[root@hadoop000 ~]#
[root@hadoop000 ~]#
[root@hadoop000 ~]# ssh localhost date
Sun May 13 21:49:17 CST 2018
[root@hadoop000 ~]#
4.6 Format the filesystem:
$ bin/hdfs namenode -format
4.7 java home配置
[root@hadoop000 hadoop]# vi hadoop-env.sh
export JAVA_HOME=/usr/java/jdk1.8.0_45
4.8 Start NameNode daemon and DataNode daemon:
[root@hadoop000 hadoop-2.8.1]# sbin/start-dfs.sh
Starting namenodes on [localhost]
localhost: starting namenode, logging to /opt/software/hadoop-2.8.1/logs/hadoop-root-namenode-hadoop000.out
localhost: starting datanode, logging to /opt/software/hadoop-2.8.1/logs/hadoop-root-datanode-hadoop000.out
Starting secondary namenodes [0.0.0.0]
The authenticity of host '0.0.0.0 (0.0.0.0)' can't be established.
RSA key fingerprint is ec:85:86:32:22:94:d1:a9:f2:0b:c5:12:3f:ba:e2:61.
Are you sure you want to continue connecting (yes/no)? yes
0.0.0.0: Warning: Permanently added '0.0.0.0' (RSA) to the list of known hosts.
0.0.0.0: starting secondarynamenode, logging to /opt/software/hadoop-2.8.1/logs/hadoop-root-secondarynamenode-hadoop000.out
[root@hadoop000 hadoop-2.8.1]#
[root@hadoop000 hadoop-2.8.1]#
[root@hadoop000 hadoop-2.8.1]#
[root@hadoop000 hadoop-2.8.1]#
[root@hadoop000 hadoop-2.8.1]# jps
16243 Jps
15943 DataNode
5127 Launcher
16139 SecondaryNameNode
15853 NameNode
[root@hadoop000 ~]# hdfs dfs -put jepson.log /
[root@hadoop000 ~]#
[root@hadoop000 ~]#
[root@hadoop000 ~]# hdfs dfs -ls /
Found 1 items
-rw-r--r-- 3 root supergroup 6 2018-05-13 21:57 /jepson.log
[root@hadoop000 ~]#
[root@hadoop000 ~]# hdfs dfs -cat /jepson.log
A
5
6
[root@hadoop000 ~]#
[root@hadoop000 ~]#
[root@hadoop000 ~]# cat jepson.log
A
5
6
[root@hadoop000 ~]#
0513作業(yè):
1.hadoop編譯
2.hdfs偽分布式部署
3.以上兩篇blog
另外有需要云服務(wù)器可以了解下創(chuàng)新互聯(lián)scvps.cn，海內(nèi)外云服務(wù)器15元起步，三天無(wú)理由+7*72小時(shí)售后在線(xiàn)，公司持有idc許可證，提供“云服務(wù)器、裸金屬服務(wù)器、高防服務(wù)器、香港服務(wù)器、美國(guó)服務(wù)器、虛擬主機(jī)、免備案服務(wù)器”等云主機(jī)租用服務(wù)以及企業(yè)上云的綜合解決方案，具有“安全穩(wěn)定、簡(jiǎn)單易用、服務(wù)可用性高、性?xún)r(jià)比高”等特點(diǎn)與優(yōu)勢(shì)，專(zhuān)為企業(yè)上云打造定制，能夠滿(mǎn)足用戶(hù)豐富、多元化的應(yīng)用場(chǎng)景需求。
當(dāng)前名稱(chēng)：大數(shù)據(jù)之---偽分布式部署之終極篇-創(chuàng)新互聯(lián)
轉(zhuǎn)載源于：http://muchs.cn/article48/dddcep.html
成都網(wǎng)站建設(shè)公司_創(chuàng)新互聯(lián)，為您提供手機(jī)網(wǎng)站建設(shè)、建站公司、ChatGPT、網(wǎng)站營(yíng)銷(xiāo)、定制開(kāi)發(fā)、定制網(wǎng)站
聲明：本網(wǎng)站發(fā)布的內(nèi)容（圖片、視頻和文字）以用戶(hù)投稿、用戶(hù)轉(zhuǎn)載內(nèi)容為主，如果涉及侵權(quán)請(qǐng)盡快告知，我們將會(huì)在第一時(shí)間刪除。文章觀點(diǎn)不代表本網(wǎng)站立場(chǎng)，如需處理請(qǐng)聯(lián)系客服。電話(huà)：028-86922220；郵箱：631063699@qq.com。內(nèi)容未經(jīng)允許不得轉(zhuǎn)載，或轉(zhuǎn)載時(shí)需注明來(lái)源： 創(chuàng)新互聯(lián)
猜你還喜歡下面的內(nèi)容
- 怎么在Asp.net中使用Eval和Bind對(duì)數(shù)據(jù)進(jìn)行綁定-創(chuàng)新互聯(lián)
- Python中cmp()函數(shù)有什么用-創(chuàng)新互聯(lián)
- Apache中如何實(shí)現(xiàn)ab壓力測(cè)試、工作模式與目錄屬性?xún)?yōu)化-創(chuàng)新互聯(lián)
- Python實(shí)現(xiàn)矩陣相乘的方法有哪些-創(chuàng)新互聯(lián)
- 【c++】期末測(cè)試1-2-創(chuàng)新互聯(lián)
- [附源碼]計(jì)算機(jī)畢業(yè)設(shè)計(jì)大學(xué)生心理測(cè)評(píng)系統(tǒng)-創(chuàng)新互聯(lián)
- 如何使用javascript中的享元模式-創(chuàng)新互聯(lián)
- 租賃類(lèi)APP開(kāi)發(fā)報(bào)價(jià)分析 2022-06-06
- 嘉定APP開(kāi)發(fā)為什么原生開(kāi)發(fā)效果會(huì)更好 2020-12-14
- 手機(jī)app開(kāi)發(fā)考核指標(biāo)該不該硬性？ 2022-07-16
- APP開(kāi)發(fā)女裝批發(fā)，打開(kāi)傳統(tǒng)市場(chǎng)新局面 2022-06-22
- 低壓電器APP開(kāi)發(fā)功能特點(diǎn)有哪些 2021-04-26
- 網(wǎng)絡(luò)友好度——APP開(kāi)發(fā)應(yīng)重視的問(wèn)題 2022-07-15
- 北京APP開(kāi)發(fā)公司的優(yōu)勢(shì)在哪兒？ 2023-03-05
- 日漸多樣化的App開(kāi)發(fā)手段和方式 2022-07-26
- 醫(yī)美咨詢(xún)app開(kāi)發(fā)對(duì)于平臺(tái)來(lái)說(shuō)有哪些作用 2022-06-13
- APP開(kāi)發(fā)價(jià)格表是多少錢(qián)？ 2023-02-06
- 企業(yè)移動(dòng)APP開(kāi)發(fā)提升質(zhì)量和效率必須重視這四點(diǎn) 2020-11-30
- APP開(kāi)發(fā)需要哪些技術(shù)的支撐呢？ 2022-11-28