這篇文章將為大家詳細(xì)講解有關(guān)怎么進(jìn)行SparkSQL部署與簡(jiǎn)單使用,文章內(nèi)容質(zhì)量較高,因此小編分享給大家做個(gè)參考,希望大家閱讀完這篇文章后對(duì)相關(guān)知識(shí)有一定的了解。
創(chuàng)新互聯(lián)公司2013年成立,是專業(yè)互聯(lián)網(wǎng)技術(shù)服務(wù)公司,擁有項(xiàng)目成都網(wǎng)站設(shè)計(jì)、網(wǎng)站建設(shè)網(wǎng)站策劃,項(xiàng)目實(shí)施與項(xiàng)目整合能力。我們以讓每一個(gè)夢(mèng)想脫穎而出為使命,1280元阿榮做網(wǎng)站,已為上家服務(wù),為阿榮各地企業(yè)和個(gè)人服務(wù),聯(lián)系電話:18982081108? JDK:1.8.0_45 64位
? hadoop-2.6.0-cdh6.7.0
? Scala:2.11.8
? spark-2.3.1-bin-2.6.0-cdh6.7.0(需要自己編譯)
? hive-1.1.0-cdh6.7.0
? MySQL5.6
#元數(shù)據(jù)存在MySQL,啟動(dòng)MySQL
[root@hadoop001 ~]# su mysqladmin [mysqladmin@hadoop001 root]$ cd ~ [mysqladmin@hadoop001 ~]$ service mysql start Starting MySQL [ OK ]
#啟動(dòng)HDFS
[hadoop@hadoop001 sbin]$ ./start-dfs.sh
#配置SparkSQL 的hive-site.xml
[hadoop@hadoop001 ~]$ cp $HIVE_HOME/conf/hive-site.xml $SPARK_HOME/conf/
#spark-sehll方式啟動(dòng):
[hadoop@hadoop001 bin]$ ./spark-shell --master local[2] \ --jars ~/software/mysql-connector-java-5.1.34-bin.jar scala> spark.sql("use hive_data2").show(false) scala> spark.sql("select * from emp").show(false) +-----+------+---------+----+----------+-------+------+------+ |empno|ename |job |mgr |hiredate |salary |comm |deptno| +-----+------+---------+----+----------+-------+------+------+ |7369 |SMITH |CLERK |7902|1980-12-17|800.0 |null |20 | |7499 |ALLEN |SALESMAN |7698|1981-2-20 |1600.0 |300.0 |30 | |7521 |WARD |SALESMAN |7698|1981-2-22 |1250.0 |500.0 |30 | |7566 |JONES |MANAGER |7839|1981-4-2 |2975.0 |null |20 | |7654 |MARTIN|SALESMAN |7698|1981-9-28 |1250.0 |1400.0|30 | |7698 |BLAKE |MANAGER |7839|1981-5-1 |2850.0 |null |30 | |7782 |CLARK |MANAGER |7839|1981-6-9 |2450.0 |null |10 | |7788 |SCOTT |ANALYST |7566|1987-4-19 |3000.0 |null |20 | |7839 |KING |PRESIDENT|null|1981-11-17|5000.0 |null |10 | |7844 |TURNER|SALESMAN |7698|1981-9-8 |1500.0 |0.0 |30 | |7876 |ADAMS |CLERK |7788|1987-5-23 |1100.0 |null |20 | |7900 |JAMES |CLERK |7698|1981-12-3 |950.0 |null |30 | |7902 |FORD |ANALYST |7566|1981-12-3 |3000.0 |null |20 | |7934 |MILLER|CLERK |7782|1982-1-23 |1300.0 |null |10 | |8888 |HIVE |PROGRAM |7839|1988-1-23 |10300.0|null |null | +-----+------+---------+----+----------+-------+------+------+
#spark-sql方式啟動(dòng):
[hadoop@hadoop001 bin]$ ./spark-sql --master local[2] \ --driver-class-path ~/software/mysql-connector-java-5.1.34-bin.jar #進(jìn)入數(shù)據(jù)庫(kù) spark-sql> use hive_data2; 18/08/30 20:36:52 INFO HiveMetaStore: 0: get_database: hive_data2 18/08/30 20:36:52 INFO audit: ugi=hadoop ip=unknown-ip-addr cmd=get_database: hive_data2 Time taken: 0.114 seconds #查詢數(shù)據(jù) spark-sql> select * from emp; 18/08/30 20:37:05 INFO DAGScheduler: Job 0 finished: processCmd at CliDriver.java:376, took 1.292944 s 7369 SMITH CLERK 7902 1980-12-17 800.0 NULL 20 7499 ALLEN SALESMAN 7698 1981-2-20 1600.0 300.0 30 7521 WARD SALESMAN 7698 1981-2-22 1250.0 500.0 30 7566 JONES MANAGER 7839 1981-4-2 2975.0 NULL 20 7654 MARTIN SALESMAN 7698 1981-9-28 1250.0 1400.0 30 7698 BLAKE MANAGER 7839 1981-5-1 2850.0 NULL 30 7782 CLARK MANAGER 7839 1981-6-9 2450.0 NULL 10 7788 SCOTT ANALYST 7566 1987-4-19 3000.0 NULL 20 7839 KING PRESIDENT NULL 1981-11-17 5000.0 NULL 10 7844 TURNER SALESMAN 7698 1981-9-8 1500.0 0.0 30 7876 ADAMS CLERK 7788 1987-5-23 1100.0 NULL 20 7900 JAMES CLERK 7698 1981-12-3 950.0 NULL 30 7902 FORD ANALYST 7566 1981-12-3 3000.0 NULL 20 7934 MILLER CLERK 7782 1982-1-23 1300.0 NULL 10 8888 HIVE PROGRAM 7839 1988-1-23 10300.0 NULL NULL
關(guān)于怎么進(jìn)行SparkSQL部署與簡(jiǎn)單使用就分享到這里了,希望以上內(nèi)容可以對(duì)大家有一定的幫助,可以學(xué)到更多知識(shí)。如果覺得文章不錯(cuò),可以把它分享出去讓更多的人看到。
名稱欄目:怎么進(jìn)行SparkSQL部署與簡(jiǎn)單使用-創(chuàng)新互聯(lián)
當(dāng)前地址:http://muchs.cn/article28/heocp.html
成都網(wǎng)站建設(shè)公司_創(chuàng)新互聯(lián),為您提供標(biāo)簽優(yōu)化、做網(wǎng)站、搜索引擎優(yōu)化、營(yíng)銷型網(wǎng)站建設(shè)、用戶體驗(yàn)、網(wǎng)站策劃
聲明:本網(wǎng)站發(fā)布的內(nèi)容(圖片、視頻和文字)以用戶投稿、用戶轉(zhuǎn)載內(nèi)容為主,如果涉及侵權(quán)請(qǐng)盡快告知,我們將會(huì)在第一時(shí)間刪除。文章觀點(diǎn)不代表本網(wǎng)站立場(chǎng),如需處理請(qǐng)聯(lián)系客服。電話:028-86922220;郵箱:631063699@qq.com。內(nèi)容未經(jīng)允許不得轉(zhuǎn)載,或轉(zhuǎn)載時(shí)需注明來(lái)源: 創(chuàng)新互聯(lián)
猜你還喜歡下面的內(nèi)容