一、概述
讓客戶滿意是我們工作的目標(biāo),不斷超越客戶的期望值來自于我們對(duì)這個(gè)行業(yè)的熱愛。我們立志把好的技術(shù)通過有效、簡(jiǎn)單的方式提供給客戶,將通過不懈努力成為客戶在信息化領(lǐng)域值得信任、有價(jià)值的長(zhǎng)期合作伙伴,公司提供的服務(wù)項(xiàng)目有:申請(qǐng)域名、網(wǎng)絡(luò)空間、營銷軟件、網(wǎng)站建設(shè)、黃石網(wǎng)站維護(hù)、網(wǎng)站推廣。1.實(shí)驗(yàn)環(huán)境基于以前搭建的haoop HA;
2.spark HA所需要的zookeeper環(huán)境前文已經(jīng)配置過,此處不再重復(fù)。
3.所需軟件包為:scala-2.12.3.tgz、spark-2.2.0-bin-hadoop2.7.tar
4.主機(jī)規(guī)劃
bd1 bd2 bd3 | Worker |
bd4 bd5 | Master、Worker |
二、配置Scala
1.解壓并拷貝
[root@bd1 ~]# tar -zxf scala-2.12.3.tgz [root@bd1 ~]# cp -r scala-2.12.3 /usr/local/
2.配置環(huán)境變量
[root@bd1 ~]# vim /etc/profile export SCALA_HOME=/usr/local/scala export PATH=:$SCALA_HOME/bin:$PATH [root@bd1 ~]# source /etc/profile
3.驗(yàn)證
[root@bd1 ~]# scala -version Scala code runner version 2.12.3 -- Copyright 2002-2017, LAMP/EPFL and Lightbend, Inc.
三、配置Spark
1.解壓并拷貝
[root@bd1 ~]# tar -zxf spark-2.2.0-bin-hadoop2.7.tgz [root@bd1 ~]# cp spark-2.2.0-bin-hadoop2.7 /usr/local/spark
2.配置環(huán)境變量
[root@bd1 ~]# vim /etc/profile export SCALA_HOME=/usr/local/scala export PATH=:$SCALA_HOME/bin:$PATH [root@bd1 ~]# source /etc/profile
3.修改spark-env.sh #文件不存在需要拷貝模板
[root@bd1 conf]# vim spark-env.sh export JAVA_HOME=/usr/local/jdk export HADOOP_HOME=/usr/local/hadoop export HADOOP_CONF_DIR=/usr/local/hadoop/etc/hadoop export SCALA_HOME=/usr/local/scala export SPARK_DAEMON_JAVA_OPTS="-Dspark.deploy.recoveryMode=ZOOKEEPER -Dspark.deploy.zookeeper.url=bd4:2181,bd5:2181 -Dspark.deploy.zookeeper.dir=/spark" export SPARK_WORKER_MEMORY=1g export SPARK_WORKER_CORES=2 export SPARK_WORKER_INSTANCES=1
4.修改spark-defaults.conf #文件不存在需要拷貝模板
[root@bd1 conf]# vim spark-defaults.conf spark.master spark://master:7077 spark.eventLog.enabled true spark.eventLog.dir hdfs://master:/user/spark/history spark.serializer org.apache.spark.serializer.KryoSerializer
5.在HDFS文件系統(tǒng)中新建日志文件目錄
hdfs dfs -mkdir -p /user/spark/history hdfs dfs -chmod 777 /user/spark/history
6.修改slaves
[root@bd1 conf]# vim slaves bd1 bd2 bd3 bd4 bd5
四、同步到其他主機(jī)
1.使用scp同步Scala到bd2-bd5
scp -r /usr/local/scala root@bd2:/usr/local/ scp -r /usr/local/scala root@bd3:/usr/local/ scp -r /usr/local/scala root@bd4:/usr/local/ scp -r /usr/local/scala root@bd5:/usr/local/
2.同步Spark到bd2-bd5
scp -r /usr/local/spark root@bd2:/usr/local/ scp -r /usr/local/spark root@bd3:/usr/local/ scp -r /usr/local/spark root@bd4:/usr/local/ scp -r /usr/local/spark root@bd5:/usr/local/
五、啟動(dòng)集群并測(cè)試HA
1.啟動(dòng)順序?yàn)椋簔ookeeper-->hadoop-->spark
2.啟動(dòng)spark
bd4:
[root@bd4 sbin]# cd /usr/local/spark/sbin/ [root@bd4 sbin]# ./start-all.sh starting org.apache.spark.deploy.master.Master, logging to /usr/local/spark/logs/spark-root-org.apache.spark.deploy.master.Master-1-bd4.out bd4: starting org.apache.spark.deploy.worker.Worker, logging to /usr/local/spark/logs/spark-root-org.apache.spark.deploy.worker.Worker-1-bd4.out bd2: starting org.apache.spark.deploy.worker.Worker, logging to /usr/local/spark/logs/spark-root-org.apache.spark.deploy.worker.Worker-1-bd2.out bd3: starting org.apache.spark.deploy.worker.Worker, logging to /usr/local/spark/logs/spark-root-org.apache.spark.deploy.worker.Worker-1-bd3.out bd5: starting org.apache.spark.deploy.worker.Worker, logging to /usr/local/spark/logs/spark-root-org.apache.spark.deploy.worker.Worker-1-bd5.out bd1: starting org.apache.spark.deploy.worker.Worker, logging to /usr/local/spark/logs/spark-root-org.apache.spark.deploy.worker.Worker-1-bd1.out [root@bd4 sbin]# jps 3153 DataNode 7235 Jps 3046 JournalNode 7017 Master 3290 NodeManager 7116 Worker 2958 QuorumPeerMain
bd5:
[root@bd5 sbin]# ./start-master.sh starting org.apache.spark.deploy.master.Master, logging to /usr/local/spark/logs/spark-root-org.apache.spark.deploy.master.Master-1-bd5.out [root@bd5 sbin]# jps 3584 NodeManager 5602 RunJar 3251 QuorumPeerMain 8564 Master 3447 DataNode 8649 Jps 8474 Worker 3340 JournalNode
3.停掉bd4的Master進(jìn)程
[root@bd4 sbin]# kill -9 7017 [root@bd4 sbin]# jps 3153 DataNode 7282 Jps 3046 JournalNode 3290 NodeManager 7116 Worker 2958 QuorumPeerMain
五、總結(jié)
一開始時(shí)想把Master放到bd1和bd2上,但是啟動(dòng)Spark后發(fā)現(xiàn)兩個(gè)節(jié)點(diǎn)上都是Standby。然后修改配置文件轉(zhuǎn)移到bd4和bd5上,才順利運(yùn)行。換言之Spark HA的Master必須位于Zookeeper集群上才能正常運(yùn)行,即該節(jié)點(diǎn)上要有JournalNode這個(gè)進(jìn)程。
另外有需要云服務(wù)器可以了解下創(chuàng)新互聯(lián)scvps.cn,海內(nèi)外云服務(wù)器15元起步,三天無理由+7*72小時(shí)售后在線,公司持有idc許可證,提供“云服務(wù)器、裸金屬服務(wù)器、高防服務(wù)器、香港服務(wù)器、美國服務(wù)器、虛擬主機(jī)、免備案服務(wù)器”等云主機(jī)租用服務(wù)以及企業(yè)上云的綜合解決方案,具有“安全穩(wěn)定、簡(jiǎn)單易用、服務(wù)可用性高、性價(jià)比高”等特點(diǎn)與優(yōu)勢(shì),專為企業(yè)上云打造定制,能夠滿足用戶豐富、多元化的應(yīng)用場(chǎng)景需求。
當(dāng)前名稱:spark2.2.0高可用搭建-創(chuàng)新互聯(lián)
網(wǎng)站鏈接:http://muchs.cn/article18/csgidp.html
成都網(wǎng)站建設(shè)公司_創(chuàng)新互聯(lián),為您提供企業(yè)建站、小程序開發(fā)、網(wǎng)站導(dǎo)航、移動(dòng)網(wǎng)站建設(shè)、搜索引擎優(yōu)化、微信小程序
聲明:本網(wǎng)站發(fā)布的內(nèi)容(圖片、視頻和文字)以用戶投稿、用戶轉(zhuǎn)載內(nèi)容為主,如果涉及侵權(quán)請(qǐng)盡快告知,我們將會(huì)在第一時(shí)間刪除。文章觀點(diǎn)不代表本網(wǎng)站立場(chǎng),如需處理請(qǐng)聯(lián)系客服。電話:028-86922220;郵箱:631063699@qq.com。內(nèi)容未經(jīng)允許不得轉(zhuǎn)載,或轉(zhuǎn)載時(shí)需注明來源: 創(chuàng)新互聯(lián)
猜你還喜歡下面的內(nèi)容