hadoopbalancerhbasebalancer

Hadoop 均衡器

成都創(chuàng)新互聯(lián)自2013年起，是專業(yè)互聯(lián)網(wǎng)技術(shù)服務(wù)公司，擁有項(xiàng)目網(wǎng)站建設(shè)、成都網(wǎng)站設(shè)計(jì)網(wǎng)站策劃，項(xiàng)目實(shí)施與項(xiàng)目整合能力。我們以讓每一個(gè)夢(mèng)想脫穎而出為使命，1280元浪卡子做網(wǎng)站,已為上家服務(wù),為浪卡子各地企業(yè)和個(gè)人服務(wù),聯(lián)系電話:13518219792

Hadoop在運(yùn)行過程中，其datanode的塊會(huì)越來越不平衡，不平衡的集群會(huì)導(dǎo)致部分datanode相對(duì)更繁忙。

Hadoop的均衡器是一個(gè)守護(hù)進(jìn)程。它會(huì)重新分配塊，將塊從忙碌的datanode移到相對(duì)空閑的datanode。同時(shí)堅(jiān)持復(fù)本策略，將復(fù)本分散到不同機(jī)架，以降低數(shù)據(jù)損壞率。

集群均衡標(biāo)準(zhǔn)：每個(gè)datanode的使用率和集群的使用率非常接近，差距不超過給定的閥值。

datanode使用率：該節(jié)點(diǎn)上已使用的空間與空間總量之間的比率；

集群的使用率：集群中已使用的空間與集群的總空間總量之間的比率。

為了降低集群負(fù)荷、避免干擾其他用戶，均衡器后臺(tái)運(yùn)行，并限制不同節(jié)點(diǎn)之間復(fù)制數(shù)據(jù)的帶寬，默認(rèn)值為1MB/s。

Hadoop的均衡器默認(rèn)是關(guān)閉的，啟動(dòng)需要執(zhí)行%start-balancer.sh，集群均衡后會(huì)自動(dòng)停止。

HBase 負(fù)載均衡

HBase的均衡器默認(rèn)五分鐘執(zhí)行一次，由 hbase.balancer.period屬性設(shè)置。均衡器運(yùn)行后會(huì)嘗試均勻分配region到所有region服務(wù)器。

先確定一個(gè)region分配計(jì)劃，然后開始移動(dòng)region。

通過均衡器的開關(guān)類控制開啟和關(guān)閉狀態(tài)。

HBase的均衡器默認(rèn)是開啟的，并周期性運(yùn)行。

下面是官網(wǎng)的解釋說明：

Hadoop Balancer

Over time, the distribution of blocks across datanodes can become unbalanced. An unbalanced cluster can affect locality for MapReduce, and it puts a greater strain on the highly utilized datanodes, so it’s best avoided.

The balancer program is a Hadoop daemon that redistributes blocks by moving them from overutilized datanodes to underutilized datanodes, while adhering to the block replica placement policy that makes data loss unlikely by placing block replicas on different racks (seeReplica Placement). It moves blocks until the cluster is deemed to be balanced, which means that the utilization of every datanode (ratio of used space on the node to total capacity of the node) differs from the utilization of the cluster (ratio of used space on the cluster to total capacity of the cluster) by no more than a given threshold percentage. You can start the balancer with:

% start-balancer.sh

The-threshold argument specifies the threshold percentage that defines what it means for the cluster to be balanced. The flag is optional; if omitted, the threshold is 10%. At any one time, only one balancer may be running on the cluster.

The balancer runs until the cluster is balanced, it cannot move any more blocks, or it loses contact with the namenode. It produces a logfile in the standard log directory, where it writes a line for every iteration of redistribution that it carries out. Here is the output from a short run on a small cluster (slightly reformatted to fit the page):

Time Stamp      Iteration#  Bytes Already Moved  ...Left To Move  ...Being Moved
Mar 18, 2009 5:23:42 PM  0                 0 KB        219.21 MB       150.29 MB
Mar 18, 2009 5:27:14 PM  1            195.24 MB         22.45 MB       150.29 MB
The cluster is balanced. Exiting...
Balancing took 6.072933333333333 minutes

The balancer is designed to run in the background without unduly taxing the cluster or interfering with other clients using the cluster. It limits the bandwidth that it uses to copy a block from one node to another. The default is a modest 1 MB/s, but this can be changed by setting thedfs.datanode.balance.bandwidthPerSec property in hdfs-site.xml, specified in bytes.

HBase Load Balancing

The master has a built-in feature, called the balancer. By default, the balancer runs every five minutes, and it is configured by thehbase.balancer.period property. Once the balancer is started, it will attempt to equal out the number of assigned regions per region server so that they are within one region of the average number per server. The call first determines a new assignment plan, which describes which regions should be moved where. Then it starts the process of moving the regions by calling theunassign() method of the administrative API iteratively.

The balancer has an upper limit on how long it is allowed to run, which is configured using thehbase.balancer.max.balancing property anddefaults to half of the balancer period value, or two and a half minutes.

You can control the balancer by means of the balancer switch: either use the shell’s balance_switch command to toggle the balancer status between enabled and disabled, or use thebalanceSwitch() API method to do the same. When you disable the balancer, it no longer runs as expected.

The balancer can be explicitly started using the shell’s balancer command, or using thebalancer() API method. The time-controlled invocation mentioned previously calls this method implicitly. It will determine if there is any work to be done and returntrue if that is the case. The return value of false means that it was not able to run the balancer, because either it was switched off, there was no work to be done (all is balanced), or something else was prohibiting the process. One example for this is the region in transition list (seeMain page): if there is a region currently in transition, the balancer will be skipped.

Instead of relying on the balancer to do its work properly, you can use the move command and API method to assign regions to other servers. This is useful when you want to control where the regions of a particular table are assigned. SeeRegion Hotspotting for an example.

當(dāng)前題目：hadoopbalancerhbasebalancer
文章來源：http://muchs.cn/article6/pdjpig.html

成都網(wǎng)站建設(shè)公司_創(chuàng)新互聯(lián)，為您提供外貿(mào)建站、網(wǎng)站改版、用戶體驗(yàn)、網(wǎng)站導(dǎo)航、定制開發(fā)、網(wǎng)站設(shè)計(jì)

聲明：本網(wǎng)站發(fā)布的內(nèi)容（圖片、視頻和文字）以用戶投稿、用戶轉(zhuǎn)載內(nèi)容為主，如果涉及侵權(quán)請(qǐng)盡快告知，我們將會(huì)在第一時(shí)間刪除。文章觀點(diǎn)不代表本網(wǎng)站立場(chǎng)，如需處理請(qǐng)聯(lián)系客服。電話：028-86922220；郵箱：631063699@qq.com。內(nèi)容未經(jīng)允許不得轉(zhuǎn)載，或轉(zhuǎn)載時(shí)需注明來源：創(chuàng)新互聯(lián)

猜你還喜歡下面的內(nèi)容