FederatedHDFS+beeline+hiveserver2遇到的坑

遇到的坑：

創(chuàng)新互聯(lián)建站長期為上1000家客戶提供的網(wǎng)站建設(shè)服務(wù)，團(tuán)隊(duì)從業(yè)經(jīng)驗(yàn)10年，關(guān)注不同地域、不同群體，并針對不同對象提供差異化的產(chǎn)品和服務(wù)；打造開放共贏平臺(tái)，與合作伙伴共同營造健康的互聯(lián)網(wǎng)生態(tài)環(huán)境。為利通企業(yè)提供專業(yè)的成都網(wǎng)站制作、成都網(wǎng)站設(shè)計(jì)、外貿(mào)營銷網(wǎng)站建設(shè)，利通網(wǎng)站改版等技術(shù)服務(wù)。擁有十余年豐富建站經(jīng)驗(yàn)和眾多成功案例,為您定制開發(fā)。

1、 Hive的任務(wù)會(huì)從臨時(shí)目錄移動(dòng)數(shù)據(jù)到數(shù)據(jù)倉庫目錄，默認(rèn)hive使用/tmp作為臨時(shí)目錄，用戶通常使用/user/hive/warehouse/作為數(shù)據(jù)倉庫目錄。在Federated HDFS情況下，/tmp 和 /user視為兩個(gè)不同的ViewFS mount table，所以hive任務(wù)在這兩個(gè)目錄之間移動(dòng)數(shù)據(jù)。Federated HDFS不支持這樣做，所以任務(wù)會(huì)失敗。

報(bào)錯(cuò)信息：

ERROR : Failed with exception Unable to move sourceviewfs://cluster9/tmp/.hive-staging_hive_2015-07-29_12-34-11_306_6082682065011532871-5/-ext-10002to destinationviewfs://cluster9/user/hive/warehouse/tandem.db/cust_loss_alarm_unit

org.apache.hadoop.hive.ql.metadata.HiveException: Unable to movesourceviewfs://cluster9/tmp/warehouse/.hive-staging_hive_2015-07-29_12-34-11_306_6082682065011532871-5/-ext-10002to destinationviewfs://cluster9/user/hive/warehouse/tandem.db/cust_loss_alarm_unit

        atorg.apache.hadoop.hive.ql.metadata.Hive.moveFile(Hive.java:2521)

        atorg.apache.hadoop.hive.ql.exec.MoveTask.moveFile(MoveTask.java:105)

        atorg.apache.hadoop.hive.ql.exec.MoveTask.execute(MoveTask.java:222)

        at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:160)

        atorg.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:88)

        atorg.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1640)

        atorg.apache.hadoop.hive.ql.Driver.execute(Driver.java:1399)

        atorg.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1183)

        atorg.apache.hadoop.hive.ql.Driver.run(Driver.java:1049)

        atorg.apache.hadoop.hive.ql.Driver.run(Driver.java:1044)

        at org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:144)

        atorg.apache.hive.service.cli.operation.SQLOperation.access$100(SQLOperation.java:69)

        atorg.apache.hive.service.cli.operation.SQLOperation$1$1.run(SQLOperation.java:196)

        atjava.security.AccessController.doPrivileged(Native Method)

        atjavax.security.auth.Subject.doAs(Subject.java:415)

        atorg.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)

        at org.apache.hive.service.cli.operation.SQLOperation$1.run(SQLOperation.java:208)

        atjava.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)

        atjava.util.concurrent.FutureTask.run(FutureTask.java:262)

        atjava.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

        atjava.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

        atjava.lang.Thread.run(Thread.java:745)

Caused by: java.io.IOException: Renames across Mount points notsupported

        atorg.apache.hadoop.fs.viewfs.ViewFileSystem.rename(ViewFileSystem.java:444)

        atorg.apache.hadoop.hive.ql.metadata.Hive.moveFile(Hive.java:2509)

        ... 21 more

相關(guān)代碼：

org.apache.hadoop.fs.viewfs.ViewFileSystem

 /**

    // Alternate 1: renames within same file system -valid but we disallow

    // Alternate 2: (as described in next para - valid butwe have disallowed it

    //

    // Note we compare the URIs. the URIs include the linktargets.

    // hence we allow renames across mount links as longas the mount links

    // point to the same target.

    if (!re***c.targetFileSystem.getUri().equals(

             resDst.targetFileSystem.getUri())) {

      throw new IOException("Renames acrossMount points not supported");

    }

    */

   

    //

    // Alternate 3 : renames ONLY within the the samemount links.

    //

    if (re***c.targetFileSystem!=resDst.targetFileSystem) {

      throw new IOException("Renames acrossMount points not supported");

    }

 

Workaround:

a、在hdfs中 創(chuàng)建 /user/hive/warehouse/staging 目錄，賦予777權(quán)限

然后添加配置：

<property>

    <name>hive.exec.stagingdir</name>

 <value>/user/hive/warehouse/staging/.hive-staging</value>

</property>

b、 只創(chuàng)建一個(gè)加載點(diǎn)如 /cluser   然后在此加載點(diǎn)下創(chuàng)建/tmp  /user等目錄，最后修改hive相關(guān)目錄的默認(rèn)值。

2、 當(dāng)查詢返回結(jié)果集很大的時(shí)候，beeline客戶端會(huì)卡住或out-of-memory

報(bào)錯(cuò)信息：

org.apache.thrift.TException: Error in calling method FetchResults

        atorg.apache.hive.jdbc.HiveConnection$SynchronizedHandler.invoke(HiveConnection.java:1271)

        atcom.sun.proxy.$Proxy0.FetchResults(Unknown Source)

        atorg.apache.hive.jdbc.HiveQueryResultSet.next(HiveQueryResultSet.java:363)

        at org.apache.hive.beeline.BufferedRows.<init>(BufferedRows.java:42)

        atorg.apache.hive.beeline.BeeLine.print(BeeLine.java:1756)

        atorg.apache.hive.beeline.Commands.execute(Commands.java:806)

        atorg.apache.hive.beeline.Commands.sql(Commands.java:665)

        atorg.apache.hive.beeline.BeeLine.dispatch(BeeLine.java:974)

        atorg.apache.hive.beeline.BeeLine.execute(BeeLine.java:810)

        atorg.apache.hive.beeline.BeeLine.begin(BeeLine.java:767)

        at org.apache.hive.beeline.BeeLine.mainWithInputRedirection(BeeLine.java:480)

        atorg.apache.hive.beeline.BeeLine.main(BeeLine.java:463)

        atsun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

        atsun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)

        atsun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

        atjava.lang.reflect.Method.invoke(Method.java:606)

        atorg.apache.hadoop.util.RunJar.run(RunJar.java:221)

        at org.apache.hadoop.util.RunJar.main(RunJar.java:136)

Caused by: java.lang.OutOfMemoryError: Java heap space

        atjava.lang.Double.valueOf(Double.java:521)

       

Workaround：

查看源碼發(fā)現(xiàn)：beeline獲取結(jié)果集有兩種模式一種增量模式，一種buffer模式

org.apache.hive.beeline.BeeLine

 

int print(ResultSet rs) throws SQLException {

    String format = getOpts().getOutputFormat();

    OutputFormat f = (OutputFormat)formats.get(format);

 

    if (f == null) {

     error(loc("unknown-format", new Object[] {

          format,formats.keySet()}));

      f = new TableOutputFormat(this);

    }

 

    Rows rows;

 

    if (getOpts().getIncremental()) {

      rows = new IncrementalRows(this,rs); //增量模式

    } else {

      rows = new BufferedRows(this, rs);buffer模式

    }

    return f.print(rows);

  }

org.apache.hive.beeline.BeeLineOpts

private boolean incremental = false; //默認(rèn)為buffer模式

但是通過beeline –help沒有發(fā)現(xiàn)相關(guān)設(shè)置

beeline --help

Usage: java org.apache.hive.cli.beeline.BeeLine

   -u <databaseurl>              the JDBC URL to connect to

   -n <username>                  the username to connect as

   -p<password>                  the password to connect as

   -d <driverclass>              the driver class to use

   -i <initfile>                 script file for initialization

   -e<query>                     query that should be executed

   -f <execfile>                 script file that should be executed

   -w (or) --password-file <password file> the password file to read password from

   --hiveconfproperty=value       Use value for given property

   --hivevarname=value           hive variable name and value

                                  This is Hive specific settings in which variables

                                  can be set at session level and referenced in Hive

                                  commands or queries.

  --color=[true/false]           control whether color is used for display

  --showHeader=[true/false]       show column namesin query results

  --headerInterval=ROWS;         the interval between which heades are displayed

  --fastConnect=[true/false]      skip buildingtable/column list for tab-completion

  --autoCommit=[true/false]       enable/disableautomatic transaction commit

  --verbose=[true/false]         show verbose error messages and debug info

   --showWarnings=[true/false]    display connection warnings

   --showNestedErrs=[true/false]   displaynested errors

  --numberFormat=[pattern]        formatnumbers using DecimalFormat pattern

  --force=[true/false]           continue running script even after errors

  --maxWidth=MAXWIDTH            the maximum width of the terminal

   --maxColumnWidth=MAXCOLWIDTH    themaximum width to use when displaying columns

  --silent=[true/false]          be more silent

  --autosave=[true/false]        automatically save preferences

  --outputformat=[table/vertical/csv2/tsv2/dsv/csv/tsv]  format mode forresult display

                                  Note that csv, and tsv are deprecated - use csv2, tsv2 instead

  --truncateTable=[true/false]    truncatetable column when it exceeds length

   --delimiterForDSV=DELIMITER    specify the delimiter for delimiter-separated values output format (default: |)

  --isolation=LEVEL              set the transaction isolation level

   --nullemptystring=[true/false]  set to true toget historic behavior of printing null as empty string

  --help                         display this message

Beeline version 1.1.0-cdh6.4.3 by Apache Hive

但是沒關(guān)系通過

beeline -u jdbc:hive2://10.17.28.173:10000–n xxxx -pxxxx --incremental=true 還是能進(jìn)入增量模式

網(wǎng)站名稱：FederatedHDFS+beeline+hiveserver2遇到的坑
文章分享：http://muchs.cn/article8/jcpoip.html

成都網(wǎng)站建設(shè)公司_創(chuàng)新互聯(lián)，為您提供移動(dòng)網(wǎng)站建設(shè)、網(wǎng)站排名、微信公眾號(hào)、網(wǎng)站策劃、企業(yè)網(wǎng)站制作、網(wǎng)站收錄

廣告

聲明：本網(wǎng)站發(fā)布的內(nèi)容（圖片、視頻和文字）以用戶投稿、用戶轉(zhuǎn)載內(nèi)容為主，如果涉及侵權(quán)請盡快告知，我們將會(huì)在第一時(shí)間刪除。文章觀點(diǎn)不代表本網(wǎng)站立場，如需處理請聯(lián)系客服。電話：028-86922220；郵箱：631063699@qq.com。內(nèi)容未經(jīng)允許不得轉(zhuǎn)載，或轉(zhuǎn)載時(shí)需注明來源： 創(chuàng)新互聯(lián)

成都app開發(fā)公司