Spark平臺上提交作業(yè)到集群生成的日志文件是什么

今天就跟大家聊聊有關Spark平臺上提交作業(yè)到集群生成的日志文件是什么，可能很多人都不太了解，為了讓大家更加了解，小編給大家總結(jié)了以下內(nèi)容，希望大家根據(jù)這篇文章可以有所收獲。

創(chuàng)新互聯(lián)服務項目包括白云網(wǎng)站建設、白云網(wǎng)站制作、白云網(wǎng)頁制作以及白云網(wǎng)絡營銷策劃等。多年來，我們專注于互聯(lián)網(wǎng)行業(yè)，利用自身積累的技術(shù)優(yōu)勢、行業(yè)經(jīng)驗、深度合作伙伴關系等，向廣大中小型企業(yè)、政府機構(gòu)等提供互聯(lián)網(wǎng)行業(yè)的解決方案，白云網(wǎng)站推廣取得了明顯的社會效益與經(jīng)濟效益。目前，我們服務的客戶以成都為中心已經(jīng)輻射到白云省份的部分城市，未來相信會繼續(xù)擴大服務區(qū)域并繼續(xù)獲得客戶的支持與信任！

Created by Wang, Jerry, last modified on Aug 16, 2015

./spark-class org.apache.spark.deploy.worker.Worker spark://NKGV50849583FV1:7077
NKGV50849583FV1:~/devExpert/spark-1.4.1/bin # ./spark-class org.apache.spark.deploy.worker.Worker spark://NKGV50849583FV1:7077
added by Jerry: loading load-spark-env.sh !!!1
added by Jerry:…
/root/devExpert/spark-1.4.1/conf
added by Jerry, number of Jars: 1
added by Jerry, launch_classpath: /root/devExpert/spark-1.4.1/assembly/target/scala-2.10/spark-assembly-1.4.1-hadoop2.4.0.jar
added by Jerry,RUNNER:/usr/jdk1.7.0_79/bin/java
added by Jerry, printf argument list: org.apache.spark.deploy.worker.Worker spark://NKGV50849583FV1:7077
added by Jerry, I am in if-else branch: /usr/jdk1.7.0_79/bin/java -cp /root/devExpert/spark-1.4.1/conf/:/root/devExpert/spark-1.4.1/assembly/target/scala-2.10/spark-assembly-1.4.1-hadoop2.4.0.jar:/root/devExpert/spark-1.4.1/lib_managed/jars/datanucleus-rdbms-3.2.9.jar:/root/devExpert/spark-1.4.1/lib_managed/jars/datanucleus-core-3.2.10.jar:/root/devExpert/spark-1.4.1/lib_managed/jars/datanucleus-api-jdo-3.2.6.jar -Xms512m -Xmx512m -XX:MaxPermSize=256m org.apache.spark.deploy.worker.Worker spark://NKGV50849583FV1:7077
Using Spark’s default log4j profile: org/apache/spark/log4j-defaults.properties
15/08/16 12:55:28 INFO Worker: Registered signal handlers for [TERM, HUP, INT]

15/08/16 12:55:28 WARN Utils: Your hostname, NKGV50849583FV1 resolves to a loopback address: 127.0.0.1; using 10.128.184.131 instead (on interface eth0)
15/08/16 12:55:28 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address
15/08/16 12:55:29 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform… using builtin-java classes where applicable
15/08/16 12:55:29 INFO SecurityManager: Changing view acls to: root
15/08/16 12:55:29 INFO SecurityManager: Changing modify acls to: root
15/08/16 12:55:29 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(root); users with modify permissions: Set(root)
15/08/16 12:55:30 INFO Slf4jLogger: Slf4jLogger started
15/08/16 12:55:30 INFO Remoting: Starting remoting
15/08/16 12:55:30 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkWorker@10.128.184.131:42568]
15/08/16 12:55:30 INFO Utils: Successfully started service ‘sparkWorker’ on port 42568.

15/08/16 12:55:30 INFO Worker: Starting Spark worker 10.128.184.131:42568 with 8 cores, 30.4 GB RAM

15/08/16 12:55:30 INFO Worker: Running Spark version 1.4.1
15/08/16 12:55:30 INFO Worker: Spark home: /root/devExpert/spark-1.4.1
15/08/16 12:55:30 INFO Utils: Successfully started service ‘WorkerUI’ on port 8081.
15/08/16 12:55:30 INFO WorkerWebUI: Started WorkerWebUI at http://10.128.184.131:8081

15/08/16 12:55:30 INFO Worker: Connecting to master akka.tcp://sparkMaster@NKGV50849583FV1:7077/user/Master…
15/08/16 12:55:30 INFO Worker: Successfully registered with master spark://NKGV50849583FV1:7077

If I quit the worker session,

I can also observe this in master’s log:

28 15/08/16 12:55:30 INFO Master: Registering worker 10.128.184.131:42568 with 8 cores, 30.4 GB RAM
29 15/08/16 13:00:19 WARN Master: Removing worker-20150816125530-10.128.184.131-42568 because we got no heartbeat in 60 seconds
30 15/08/16 13:00:19 INFO Master: Removing worker worker-20150816125530-10.128.184.131-42568 on 10.128.184.131:42568

看完上述內(nèi)容，你們對Spark平臺上提交作業(yè)到集群生成的日志文件是什么有進一步的了解嗎？如果還想了解更多知識或者相關內(nèi)容，請關注創(chuàng)新互聯(lián)行業(yè)資訊頻道，感謝大家的支持。

當前標題：Spark平臺上提交作業(yè)到集群生成的日志文件是什么
鏈接地址：http://muchs.cn/article16/pgocdg.html

成都網(wǎng)站建設公司_創(chuàng)新互聯(lián)，為您提供品牌網(wǎng)站設計、定制開發(fā)、品牌網(wǎng)站建設、用戶體驗、關鍵詞優(yōu)化、服務器托管

廣告

聲明：本網(wǎng)站發(fā)布的內(nèi)容（圖片、視頻和文字）以用戶投稿、用戶轉(zhuǎn)載內(nèi)容為主，如果涉及侵權(quán)請盡快告知，我們將會在第一時間刪除。文章觀點不代表本網(wǎng)站立場，如需處理請聯(lián)系客服。電話：028-86922220；郵箱：631063699@qq.com。內(nèi)容未經(jīng)允許不得轉(zhuǎn)載，或轉(zhuǎn)載時需注明來源： 創(chuàng)新互聯(lián)