hadoop的Hive實(shí)戰(zhàn)操作sogou的日志

動(dòng)手實(shí)戰(zhàn)操作搜狗日志文件

創(chuàng)新互聯(lián)專注于企業(yè)成都營(yíng)銷網(wǎng)站建設(shè)、網(wǎng)站重做改版、正陽(yáng)網(wǎng)站定制設(shè)計(jì)、自適應(yīng)品牌網(wǎng)站建設(shè)、H5響應(yīng)式網(wǎng)站、電子商務(wù)商城網(wǎng)站建設(shè)、集團(tuán)公司官網(wǎng)建設(shè)、成都外貿(mào)網(wǎng)站制作、高端網(wǎng)站制作、響應(yīng)式網(wǎng)頁(yè)設(shè)計(jì)等建站業(yè)務(wù)，價(jià)格優(yōu)惠性價(jià)比高，為正陽(yáng)等各大城市提供網(wǎng)站開發(fā)制作服務(wù)。

本節(jié)中所用到的內(nèi)容是來(lái)自搜狗實(shí)驗(yàn)室，網(wǎng)址為：

http://www.sogou.com/labs/dl/q.html

用戶可以根據(jù)自己的Spark機(jī)器實(shí)際的內(nèi)存配置等情況選擇下載不同的數(shù)據(jù)版本，為了讓所有的學(xué)習(xí)者都可以成功操作日志，我們使用的是迷你版本的tar.gz格式的文件，其大小為384K，下載地址：http://www.sogou.com/labs/resource/q.php

日志格式如下：

單挑記錄：

00:00:002982199073774412[360安全衛(wèi)士]8 3download.it.com.cn/softweb/software/firewall/antivirus/20067/17938.html

該文件的格式如下所示：

訪問時(shí)間\t用戶ID\t查詢?cè)~\t該URL在返回結(jié)果中的排名\t用戶點(diǎn)擊的順序號(hào)\t用戶點(diǎn)擊的URL

在hive中創(chuàng)建表格：

create table sogou(sTime string,userId string,Keyword string,Paiming int,Dianji int,URL string) row format delimited

fields terminated by '\t' lines terminated by '\n' stored as textfile

結(jié)果如下：

hive> create table sogou(sTime string,userId string,Keyword string,Paiming int,Dianji int,URL string) row format delimited

> fields terminated by '\t' lines terminated by '\n' stored as textfile;

Time taken: 4.422 seconds

導(dǎo)入數(shù)據(jù)：

load data local inpath '/home/dyq/Documents/SogouQ.sample' overwrite into table sogou;

結(jié)果顯示：

hive> load data local inpath '/home/dyq/Documents/SogouQ.sample' overwrite into table sogou;

Loading data to table default.sogou

Time taken: 3.567 seconds

查看表的建立情況：

hive> desc sogou;

stime string

userid string

keyword string

paiming int

dianji int

url string

Time taken: 2.515 seconds, Fetched: 6 row(s)

查看數(shù)據(jù)導(dǎo)入情況：

查看記錄條數(shù)：

select count(*) from sogou;

可以看到查詢過程很漫長(zhǎng)，效率不高的說(shuō)

hive> select count(*) from sogou;

WARNING: Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. tez, spark) or using Hive 1.X releases.

Query ID = dyq_20160828114648_be63db30-cce4-47f0-aa1d-2e0f626c274c

Total jobs = 1

Launching Job 1 out of 1

Number of reduce tasks determined at compile time: 1

In order to change the average load for a reducer (in bytes):

set hive.exec.reducers.bytes.per.reducer=<number>

In order to limit the maximum number of reducers:

set hive.exec.reducers.max=<number>

In order to set a constant number of reducers:

set mapreduce.job.reduces=<number>

Starting Job = job_1472347049416_0004, Tracking URL = http://ubuntu:8088/proxy/application_1472347049416_0004/

Kill Command = /opt/hadoop-2.6.2/bin/hadoop job -kill job_1472347049416_0004

Hadoop job information for Stage-1: number of mappers: 1; number of reducers: 1

2016-08-28 11:47:28,067 Stage-1 map = 0%, reduce = 0%

2016-08-28 11:48:09,628 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 8.1 sec

2016-08-28 11:48:35,383 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 10.68 sec

MapReduce Total cumulative CPU time: 10 seconds 680 msec

Ended Job = job_1472347049416_0004

MapReduce Jobs Launched:

Stage-Stage-1: Map: 1 Reduce: 1 Cumulative CPU: 10.68 sec HDFS Read: 906303 HDFS Write: 105 SUCCESS

Total MapReduce CPU Time Spent: 10 seconds 680 msec

10000

Time taken: 109.492 seconds, Fetched: 1 row(s)

導(dǎo)出數(shù)據(jù)到外部文件

insert overwrite local directory '/home/dyq/Documents/SogouQdept' select a.Keyword,a.URL from sogou a;

只導(dǎo)出2個(gè)字段，看看情況

hive> insert overwrite local directory '/home/dyq/Documents/SogouQdept' select a.Keyword,a.URL from sogou a;

WARNING: Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. tez, spark) or using Hive 1.X releases.

Query ID = dyq_20160828115140_2bc274b4-7845-46c2-9ac7-e45fcc5b7b23

Total jobs = 1

Launching Job 1 out of 1

Number of reduce tasks is set to 0 since there's no reduce operator

Starting Job = job_1472347049416_0005, Tracking URL = http://ubuntu:8088/proxy/application_1472347049416_0005/

Kill Command = /opt/hadoop-2.6.2/bin/hadoop job -kill job_1472347049416_0005

Hadoop job information for Stage-1: number of mappers: 1; number of reducers: 0

2016-08-28 11:51:54,594 Stage-1 map = 0%, reduce = 0%

2016-08-28 11:52:06,398 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 3.14 sec

MapReduce Total cumulative CPU time: 3 seconds 140 msec

Ended Job = job_1472347049416_0005

Moving data to local directory /home/dyq/Documents/SogouQdept

MapReduce Jobs Launched:

Stage-Stage-1: Map: 1 Cumulative CPU: 3.14 sec HDFS Read: 901894 HDFS Write: 178503 SUCCESS

Total MapReduce CPU Time Spent: 3 seconds 140 msec

Time taken: 28.859 seconds

查看sogouQdept文件的內(nèi)容：

新聞標(biāo)題：hadoop的Hive實(shí)戰(zhàn)操作sogou的日志
文章地址：http://muchs.cn/article0/gjsjio.html

成都網(wǎng)站建設(shè)公司_創(chuàng)新互聯(lián)，為您提供Google、App開發(fā)、定制開發(fā)、服務(wù)器托管、品牌網(wǎng)站建設(shè)、品牌網(wǎng)站設(shè)計(jì)

聲明：本網(wǎng)站發(fā)布的內(nèi)容（圖片、視頻和文字）以用戶投稿、用戶轉(zhuǎn)載內(nèi)容為主，如果涉及侵權(quán)請(qǐng)盡快告知，我們將會(huì)在第一時(shí)間刪除。文章觀點(diǎn)不代表本網(wǎng)站立場(chǎng)，如需處理請(qǐng)聯(lián)系客服。電話：028-86922220；郵箱：631063699@qq.com。內(nèi)容未經(jīng)允許不得轉(zhuǎn)載，或轉(zhuǎn)載時(shí)需注明來(lái)源：創(chuàng)新互聯(lián)

猜你還喜歡下面的內(nèi)容