CDH5.16.1集群企業(yè)真正離線部署

?.準備?作

1.離線部署大綱

• MySQL離線部署
• CM離線部署
• Parcel?件離線源部署

2.規(guī)劃

linux版本:CentOS 7.2

創(chuàng)新互聯(lián)公司是一家專業(yè)提供李滄企業(yè)網站建設,專注與網站制作、做網站、成都h5網站建設、小程序制作等業(yè)務。10年已為李滄眾多企業(yè)、政府機構等服務。創(chuàng)新互聯(lián)專業(yè)網絡公司優(yōu)惠進行中。

節(jié)點	MySQL組件	Parcel?件離線源	CM服務進程	?數(shù)據(jù)組件
hadoop001	MySQL	Parcel	Alert Publisher Event Server	NN RM DN NM ZK
hadoop002			Alert Publisher Event Server	DN NM ZK
hadoop003			Host Monitor Service Monitor	DN NM ZK

3.下載源

• CM
  cloudera-manager-centos7-cm5.16.1x8664.tar.gz

• Parcel
  CDH-5.16.1-1.cdh6.16.1.p0.3-el7.parcel
  CDH-5.16.1-1.cdh6.16.1.p0.3-el7.parcel.sha1
  manifest.json

• JDK
  JDK8
  下載jdk-8u202-linux-x64.tar.gz

• MySQL
  MYSQL5.7
  下載mysql-5.7.26-el7-x86_64.tar.gz

• MySQL JDBC jar
  mysql-connector-java-5.1.47.jar
  下載完成后要重命名去掉版本號

  mv mysql-connector-java-5.1.47.jar mysql-connector-java.jar  

?.集群節(jié)點初始化

1.阿里云購買3臺虛擬機

(最低配置 2core 8G)，選擇按量付費

CentOS7.2

2.當前筆記本(win)hosts配置文件

路徑: C:\Windows\System32\drivers\etc\hosts

39.97.188.249   hadoop001   hadoop001
39.97.225.112   hadoop002   hadoop002
39.97.224.68    hadoop003   hadoop003

注意：IP是你虛擬機公網IP

3.設置所有節(jié)點的hosts文件

echo '172.17.144.104 hadoop001' >> /etc/hosts
echo '172.17.144.103 hadoop002' >> /etc/hosts
echo '172.17.144.105 hadoop003' >> /etc/hosts
#檢查
cat /etc/hosts

127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6

172.17.144.104 hadoop001
172.17.144.103 hadoop002
172.17.144.105 hadoop003

注意:IP為內網IP

4.關閉所有節(jié)點防火墻及清空規(guī)則

云主機

我們使用的云主機，無論阿里云還是騰訊云的防火墻都是關閉的，所以我們不需要關閉服務器的防火墻。但是，我們需要檢查下是否自動開啟了web訪問端口，如果沒有，則自己添加

(1)打開安全組配置

進入之后點擊配置規(guī)則

(2)添加安全組規(guī)則

注意:

1.點擊藍色感嘆號會有規(guī)則說明

2.授權對象如果在公司內需要設置網段，就按照上圖，將ip網段規(guī)定好。不限制的話就直接0.0.0.0/0

內網服務器

最好在內部服務器部署時就將防火墻關閉，如果不行就暫時關閉，等部署成功再開啟

systemctl stop firewalld

systemctl disable firewalld

iptables -F   

5.關閉所有節(jié)點selinux

阿里云服務器已經將selinux關閉了，所以不用配置

自己的服務器很可能會開啟selinux，這樣就需要關閉了

將SELINUX=disabled設置進去，之后重啟才會生效

vim /etc/selinux/config 
# This file controls the state of SELinux on the system.
# SELINUX= can take one of these three values:
#     enforcing - SELinux security policy is enforced.
#     permissive - SELinux prints warnings instead of enforcing.
#     disabled - No SELinux policy is loaded.
SELINUX=disabled
# SELINUXTYPE= can take one of three two values:
#     targeted - Targeted processes are protected,
#     minimum - Modification of targeted policy. Only selected processes are protected. 
#     mls - Multi Level Security protection.
SELINUXTYPE=targeted

6.設置所有節(jié)點時區(qū)一致及時鐘同步

阿里云已經將節(jié)點時區(qū)和時間做了同步

我們實操下公司環(huán)境的時區(qū)時間同步

6.1時區(qū)

[root@hadoop001 ~]# timedatectl
      Local time: Tue 2019-05-28 15:37:53 CST
  Universal time: Tue 2019-05-28 07:37:53 UTC
        RTC time: Tue 2019-05-28 15:37:53
       Time zone: Asia/Shanghai (CST, +0800)
     NTP enabled: yes
NTP synchronized: yes
 RTC in local TZ: yes
      DST active: n/a
#查看命令幫助，學習?關重要，?需百度，太low
[root@hadoop001 ~]# timedatectl --help
timedatectl [OPTIONS...] COMMAND ...
Query or change system time and date settings.
-h --help Show this help message
--version Show package version
--no-pager Do not pipe output into a pager
--no-ask-password Do not prompt for password
-H --host=[USER@]HOST Operate on remote host
-M --machine=CONTAINER Operate on local container
--adjust-system-clock Adjust system clock when changing local RTC mode
Commands:
status Show current time settings
set-time TIME Set system time
set-timezone ZONE Set system time zone
list-timezones Show known time zones
set-local-rtc BOOL Control whether RTC is in local time
set-ntp BOOL Control whether NTP is enabled
#查看哪些時區(qū)
[root@hadoop001 ~]# timedatectl list-timezones
Africa/Abidjan
Africa/Accra
Africa/Addis_Ababa
Africa/Algiers
Africa/Asmara
Africa/Bamako
#所有節(jié)點設置亞洲上海時區(qū)
[root@hadoop001 ~]# timedatectl set-timezone Asia/Shanghai
[root@hadoop002 ~]# timedatectl set-timezone Asia/Shanghai
[root@hadoop003 ~]# timedatectl set-timezone Asia/Shanghai

6.2.時間

#所有節(jié)點安裝ntp
[root@hadoop001 ~]# yum install -y ntp
#選取hadoop001為ntp的主節(jié)點
[root@hadoop001 ~]# vi /etc/ntp.conf
#time
server 0.asia.pool.ntp.org
server 1.asia.pool.ntp.org
server 2.asia.pool.ntp.org
server 3.asia.pool.ntp.org
#當外部時間不可用時，可使用本地硬件時間
server 127.127.1.0 iburst local clock
#允許哪些網段的機器來同步時間   修改成自己的內網網段
restrict 172.17.144.0 mask 255.255.255.0 nomodify notrap
#開啟ntpd及查看狀態(tài)
[root@hadoop001 ~]# systemctl start ntpd
[root@hadoop001 ~]# systemctl status ntpd
● ntpd.service - Network Time Service
Loaded: loaded (/usr/lib/systemd/system/ntpd.service; enabled; vendor preset: d
isabled)
Active: active (running) since Sat 2019-05-11 10:15:00 CST; 11min ago
Main PID: 18518 (ntpd)
CGroup: /system.slice/ntpd.service
!"18518 /usr/sbin/ntpd -u ntp:ntp -g
May 11 10:15:00 hadoop001 systemd[1]: Starting Network Time Service...
May 11 10:15:00 hadoop001 ntpd[18518]: proto: precision = 0.088 usec
May 11 10:15:00 hadoop001 ntpd[18518]: 0.0.0.0 c01d 0d kern kernel time sync enabl
ed
May 11 10:15:00 hadoop001 systemd[1]: Started Network Time Service.
#驗證
[root@hadoop001 ~]# ntpq -p
remote refid st t when poll reach delay offset jitter
==============================================================================
LOCAL(0) .LOCL. 10 l 726 64 0 0.000 0.000 0.000
#其他從節(jié)點停?禁?ntpd服務
[root@hadoop002 ~]# systemctl stop ntpd
[root@hadoop002 ~]# systemctl disable ntpd
Removed symlink /etc/systemd/system/multi-user.target.wants/ntpd.service.
[root@hadoop002 ~]# /usr/sbin/ntpdate hadoop001
11 May 10:29:22 ntpdate[9370]: adjust time server 172.19.7.96 offset 0.000867 sec
#每天凌晨同步hadoop001節(jié)點時間[root@hadoop002 ~]# crontab -e
00 00 * * * /usr/sbin/ntpdate hadoop001
[root@hadoop003 ~]# systemctl stop ntpd
[root@hadoop004 ~]# systemctl disable ntpd
Removed symlink /etc/systemd/system/multi-user.target.wants/ntpd.service.
[root@hadoop005 ~]# /usr/sbin/ntpdate hadoop001
11 May 10:29:22 ntpdate[9370]: adjust time server 172.19.7.96 offset 0.000867 sec
#每天凌晨同步hadoop001節(jié)點時間
[root@hadoop003 ~]# crontab -e
00 00 * * * /usr/sbin/ntpdate hadoop001

7.JDK部署

mkdir /usr/java
tar -xzvf jdk-8u45-linux-x64.tar.gz -C /usr/java/
#切記必須修正所屬?戶及?戶組
chown -R root:root /usr/java/jdk1.8.0_45
[root@hadoop001 cdh6.16.1]# vim /etc/profile
export JAVA_HOME=/usr/java/jdk1.8.0_45
export PATH=${JAVA_HOME}/bin:${PATH}
source /etc/profile
which java

如果節(jié)點過多，那么就只做一臺鏡像模板，將基礎工作完成之后，分發(fā)克隆。。（最好請運維小哥哥做~）

8.hadoop001節(jié)點離線部署MySQL5.7

（按照生產標準）

8.1 解壓及創(chuàng)建文件夾

#解壓
[root@hadoop001 cdh6.16.1]# tar xzvf mysql-5.7.11-linux-glibc2.5-x86_64.tar.gz -C /usr/local/
#切換目錄
[root@hadoop001 cdh6.16.1]# cd /usr/local/
#修改mysql名稱
[root@hadoop001 local]# mv mysql-5.7.11-linux-glibc2.5-x86_64/ mysql
#創(chuàng)建文件夾
[root@hadoop001 local]# mkdir mysql/arch mysql/data mysql/tmp

8.2 創(chuàng)建my.cnf

rm /etc/my.cnf

vim /etc/my.cnf

[client]
port            = 3306
socket          = /usr/local/mysql/data/mysql.sock
default-character-set=utf8mb4

[mysqld]
port            = 3306
socket          = /usr/local/mysql/data/mysql.sock

skip-slave-start

skip-external-locking
key_buffer_size = 256M
sort_buffer_size = 2M
read_buffer_size = 2M
read_rnd_buffer_size = 4M
query_cache_size= 32M
max_allowed_packet = 16M
myisam_sort_buffer_size=128M
tmp_table_size=32M

table_open_cache = 512
thread_cache_size = 8
wait_timeout = 86400
interactive_timeout = 86400
max_connections = 600

# Try number of CPU's*2 for thread_concurrency
#thread_concurrency = 32 

#isolation level and default engine 
default-storage-engine = INNODB
transaction-isolation = READ-COMMITTED

server-id  = 1739
basedir     = /usr/local/mysql
datadir     = /usr/local/mysql/data
pid-file     = /usr/local/mysql/data/hostname.pid

#open performance schema
log-warnings
sysdate-is-now

binlog_format = ROW
log_bin_trust_function_creators=1
log-error  = /usr/local/mysql/data/hostname.err
log-bin = /usr/local/mysql/arch/mysql-bin
expire_logs_days = 7

innodb_write_io_threads=16

relay-log  = /usr/local/mysql/relay_log/relay-log
relay-log-index = /usr/local/mysql/relay_log/relay-log.index
relay_log_info_file= /usr/local/mysql/relay_log/relay-log.info

log_slave_updates=1
gtid_mode=OFF
enforce_gtid_consistency=OFF

# slave
slave-parallel-type=LOGICAL_CLOCK
slave-parallel-workers=4
master_info_repository=TABLE
relay_log_info_repository=TABLE
relay_log_recovery=ON

#other logs
#general_log =1
#general_log_file  = /usr/local/mysql/data/general_log.err
#slow_query_log=1
#slow_query_log_file=/usr/local/mysql/data/slow_log.err

#for replication slave
sync_binlog = 500

#for innodb options 
innodb_data_home_dir = /usr/local/mysql/data/
innodb_data_file_path = ibdata1:1G;ibdata2:1G:autoextend

innodb_log_group_home_dir = /usr/local/mysql/arch
innodb_log_files_in_group = 4
innodb_log_file_size = 1G
innodb_log_buffer_size = 200M

#根據(jù)生產需要，調整pool size 
innodb_buffer_pool_size = 2G
#innodb_additional_mem_pool_size = 50M #deprecated in 5.6
tmpdir = /usr/local/mysql/tmp

innodb_lock_wait_timeout = 1000
#innodb_thread_concurrency = 0
innodb_flush_log_at_trx_commit = 2

innodb_locks_unsafe_for_binlog=1

#innodb io features: add for mysql5.5.8
performance_schema
innodb_read_io_threads=4
innodb-write-io-threads=4
innodb-io-capacity=200
#purge threads change default(0) to 1 for purge
innodb_purge_threads=1
innodb_use_native_aio=on

#case-sensitive file names and separate tablespace
innodb_file_per_table = 1
lower_case_table_names=1

[mysqldump]
quick
max_allowed_packet = 128M

[mysql]
no-auto-rehash
default-character-set=utf8mb4

[mysqlhotcopy]
interactive-timeout

[myisamchk]
key_buffer_size = 256M
sort_buffer_size = 256M
read_buffer = 2M
write_buffer = 2M

8.3 創(chuàng)建用戶組及用戶

[root@hadoop001 local]# groupadd -g 101 dba
[root@hadoop001 local]# useradd -u 514 -g dba -G root -d /usr/local/mysql mysqladmin

[root@hadoop001 local]# id mysqladmin
uid=514(mysqladmin) gid=101(dba) groups=101(dba),0(root)

## 一般不需要設置mysqladmin的密碼，直接從root或者LDAP用戶sudo切換

8.4 copy 環(huán)境變量配置文件

copy 環(huán)境變量配置文件(隱藏文件)至mysqladmin用戶的home目錄中,為了以下步驟配置個人環(huán)境變量

cp /etc/skel/.* /usr/local/mysql  

8.5 配置環(huán)境變量

[root@hadoop001 local]# vi mysql/.bash_profile

# .bash_profile

# Get the aliases and functions
if [ -f ~/.bashrc ]; then
        . ~/.bashrc
fi
# User specific environment and startup programs
export MYSQL_BASE=/usr/local/mysql
export PATH=${MYSQL_BASE}/bin:$PATH

unset USERNAME

#stty erase ^H
set umask to 022
umask 022
PS1=`uname -n`":"'$USER'":"'$PWD'":>"; export PS1

8.6 賦權限和用戶組 切換用戶mysqladmin 安裝

[root@hadoop001 local]# chown  mysqladmin:dba /etc/my.cnf 
[root@hadoop001 local]# chmod  640 /etc/my.cnf  
[root@hadoop001 local]# chown -R mysqladmin:dba /usr/local/mysql
[root@hadoop001 local]# chmod -R 755 /usr/local/mysql 

8.7 配置服務及開機自啟動

[root@hadoop001 local]#  cd /usr/local/mysql
#將服務文件拷貝到init.d下，并重命名為mysql
[root@hadoop001 mysql]# cp support-files/mysql.server /etc/rc.d/init.d/mysql 
#賦予可執(zhí)行權限
[root@hadoop001 mysql]# chmod +x /etc/rc.d/init.d/mysql
#刪除服務
[root@hadoop001 mysql]# chkconfig --del mysql
#添加服務
[root@hadoop001 mysql]# chkconfig --add mysql
[root@hadoop001 mysql]# chkconfig --level 345 mysql on

8.8 安裝libaio及安裝mysql的初始db

[root@hadoop001 mysql]# yum -y install libaio

[root@hadoop001 mysql]# su - mysqladmin
Last login: Tue May 28 17:04:49 CST 2019 on pts/0
hadoop001:mysqladmin:/usr/local/mysql:>bin/mysqld \
> --defaults-file=/etc/my.cnf \
> --user=mysqladmin \
> --basedir=/usr/local/mysql/ \
> --datadir=/usr/local/mysql/data/ \
> --initialize

在初始化時如果加上 –initial-insecure，則會創(chuàng)建空密碼的 root@localhost 賬號，否則會創(chuàng)建帶密碼的 root@localhost 賬號，密碼直接寫在 log-error 日志文件中（在5.6版本中是放在 ~/.mysql_secret 文件里，更加隱蔽，不熟悉的話可能會無所適從）

8.9 查看臨時密碼

#查看密碼
hadoop001:mysqladmin:/usr/local/mysql/data:>cat hostname.err |grep password
2019-05-28T09:28:40.447701Z 1 [Note] A temporary password is generated for root@localhost: J=<z#diyC4fh

8.10 啟動

hadoop001:mysqladmin:/usr/local/mysql:>/usr/local/mysql/bin/mysqld_safe --defaults-file=/etc/my.cnf &
[1] 21740
hadoop001:mysqladmin:/usr/local/mysql:>2019-05-28T09:38:16.127060Z mysqld_safe Logging to '/usr/local/mysql/data/hostname.err'.
2019-05-28T09:38:16.196799Z mysqld_safe Starting mysqld daemon with databases from /usr/local/mysql/data
#按兩次回車

##退出mysqladmin用戶
##查看mysql進程號
[root@hadoop001 mysql]#ps -ef|grep mysql
mysqlad+ 21740     1  0 17:38 pts/0    00:00:00 /bin/sh /usr/local/mysql/bin/mysqld_safe --defaults-file=/etc/my.cnf
mysqlad+ 22557 21740  0 17:38 pts/0    00:00:00 /usr/local/mysql/bin/mysqld --defaults-file=/etc/my.cnf --basedir=/usr/local/mysql --datadir=/usr/local/mysql/data --plugin-dir=/usr/local/mysql/lib/plugin --log-error=/usr/local/mysql/data/hostname.err --pid-file=/usr/local/mysql/data/hostname.pid --socket=/usr/local/mysql/data/mysql.sock --port=3306
root     22609  9194  0 17:39 pts/0    00:00:00 grep --color=auto mysql
##通過mysql進程號查看mysql端口號
[root@hadoop001 mysql]# netstat -nlp|grep 22557
#切換成mysqladmin
[root@hadoop001 mysql]# su - mysqladmin
Last login: Tue May 28 17:24:45 CST 2019 on pts/0
hadoop001:mysqladmin:/usr/local/mysql:>
##查看mysql是否運行
hadoop001:mysqladmin:/usr/local/mysql:>service mysql status
MySQL running (22557)[  OK  ]

8.11 登錄及修改用戶密碼

#初始密碼
hadoop001:mysqladmin:/usr/local/mysql:>mysql -uroot -p'J=<z#diyC4fh'   
mysql: [Warning] Using a password on the command line interface can be insecure.
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 2
Server version: 5.7.11-log

Copyright (c) 2000, 2016, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
#重置密碼
mysql> alter user root@localhost identified by 'ruozedata123';
mysql>GRANT ALL PRIVILEGES ON *.* TO 'root'@'%' IDENTIFIED BY 'ruozedata123' ;
#刷權限
mysql> flush privileges;

8.12 重啟

9.創(chuàng)建CDH的元數(shù)據(jù)庫和用戶、 amon服務的數(shù)據(jù)庫及用戶

mysql> CREATE DATABASE `cmf`  DEFAULT CHARACTER SET utf8;

mysql> GRANT ALL PRIVILEGES ON cmf.* TO 'cmf'@'%' IDENTIFIED BY 'ruozedata123' ;

mysql> create database amon default character set utf8;

mysql> GRANT ALL PRIVILEGES ON amon.* TO 'amon'@'%' IDENTIFIED BY 'ruozedata123' ;

--刷權限
mysql> flush privileges;

10.部署 mysql JDBC jar

[root@hadoop001 cdh6.16.1]# mkdir -p /usr/share/java
[root@hadoop001 cdh6.16.1]# ls -lh
total 3.5G
-rw-r--r-- 1 root root 2.0G May 15 10:01 CDH-5.16.1-1.cdh6.16.1.p0.3-el7.parcel
-rw-r--r-- 1 root root   41 May 14 20:17 CDH-5.16.1-1.cdh6.16.1.p0.3-el7.parcel.sha1
-rw-r--r-- 1 root root 803M May 15 09:38 cloudera-manager-centos7-cm5.16.1_x86_64.tar.gz
-rw-r--r-- 1 root root 166M May 14 20:21 jdk-8u45-linux-x64.gz
-rw-r--r-- 1 root root  65K May 14 20:17 manifest.json
-rw-r--r-- 1 root root 523M May 15 09:28 mysql-5.7.11-linux-glibc2.5-x86_64.tar.gz
-rw-r--r-- 1 root root 984K May 15 09:10 mysql-connector-java-5.1.47.jar
#mysql的jar包一定要去掉版本號~，有坑
[root@hadoop001 cdh6.16.1]# cp mysql-connector-java-5.1.47.jar /usr/share/java/mysql-connector-java.jar

三.CDH離線部署

1.部署CM Server 和Agent

1.1 所有節(jié)點創(chuàng)建?錄及解壓

mkdir /opt/cloudera-manager
tar -zxvf cloudera-manager-centos7-cm5.16.1_x86_64.tar.gz -C /opt/cloudera-manager
/
sed -i "s/server_host=localhost/server_host=hadoop001/g" /opt/cloudera-manager/cm-5.16.1/etc/cloudera-scm-agent/config.ini

1.2 所有節(jié)點修改config.ini

所有節(jié)點修改agent的配置，指向server的節(jié)點hadoop001

sed -i "s/server_host=localhost/server_host=hadoop001/g" /opt/cloudera-manager/cm-5.16.1/etc/cloudera-scm-agent/config.ini

1.3 主節(jié)點修改server的配置

vi /opt/cloudera-manager/cm-5.16.1/etc/cloudera-scm-server/db.properties

com.cloudera.cmf.db.type=mysql
com.cloudera.cmf.db.host=hadoop001
com.cloudera.cmf.db.name=cmf
com.cloudera.cmf.db.user=cmf
com.cloudera.cmf.db.password=ruozedata123
com.cloudera.cmf.db.setupType=EXTERNAL

1.4 所有節(jié)點創(chuàng)建cloudera-scm用戶

#創(chuàng)建cloudera-scm
useradd --system --home=/opt/cloudera-manager/cm-5.16.1/run/cloudera-scm-server/ --no-create-home --shell=/bin/false --comment "Cloudera SCM User" cloudera-scm
#修改cloudera-manager的權限
chown -R cloudera-scm:cloudera-scm /opt/cloudera-manager

1.5 所有節(jié)點修改cloudera-manager用戶名用戶組

chown -R cloudera-scm:cloudera-scm /opt/cloudera-manager

2.hadoop001節(jié)點部署離線parcel源

2.1 部署離線parcel源

mkdir -p /opt/cloudera/parcel-repo

[root@hadoop001 opt]# cd ~/cdh6.16.1/
[root@hadoop001 cdh6.16.1]# ls -lh
total 3.5G
-rw-r--r-- 1 root root 2.0G May 15 10:01 CDH-5.16.1-1.cdh6.16.1.p0.3-el7.parcel
-rw-r--r-- 1 root root   41 May 14 20:17 CDH-5.16.1-1.cdh6.16.1.p0.3-el7.parcel.sha1
-rw-r--r-- 1 root root 803M May 15 09:38 cloudera-manager-centos7-cm5.16.1_x86_64.tar.gz
-rw-r--r-- 1 root root 166M May 14 20:21 jdk-8u45-linux-x64.gz
-rw-r--r-- 1 root root  65K May 14 20:17 manifest.json
-rw-r--r-- 1 root root 523M May 15 09:28 mysql-5.7.11-linux-glibc2.5-x86_64.tar.gz
-rw-r--r-- 1 root root 984K May 15 09:10 mysql-connector-java-5.1.47.jar

[root@hadoop001 cdh6.16.1]# mv CDH-5.16.1-1.cdh6.16.1.p0.3-el7.parcel.sha1 /opt/cloudera/parcel-repo/CDH-5.16.1-1.cdh6.16.1.p0.3-el7.parcel.sha

#切記mv時，重命名去掉1，不然在部署過程CM認為如上?件下載未完整，會持續(xù)下載

[root@hadoop001 cdh6.16.1]# mv CDH-5.16.1-1.cdh6.16.1.p0.3-el7.parcel /opt/cloudera/parcel-repo/

[root@hadoop001 cdh6.16.1]# mv manifest.json  /opt/cloudera/parcel-repo/

如果你是通過網絡下載的parcel包，我們就需要對CDH-5.16.1-1.cdh6.16.1.p0.3-el7.parcel 進行校驗，防止文件損壞!!!

[root@hadoop001 parcel-repo]# cat CDH-5.16.1-1.cdh6.16.1.p0.3-el7.parcel.sha
703728dfa7690861ecd3a9bcd412b04ac8de7148
#計算下載文件的值，進行對比
[root@hadoop001 parcel-repo]# sha1sum CDH-5.16.1-1.cdh6.16.1.p0.3-el7.parcel
703728dfa7690861ecd3a9bcd412b04ac8de7148  CDH-5.16.1-1.cdh6.16.1.p0.3-el7.parcel
#相同，可以正常使用

2.2 目錄修改用戶及用戶組

chown -R cloudera-scm:cloudera-scm /opt/cloudera/parcel-repo/

3.所有節(jié)點創(chuàng)建大數(shù)據(jù)軟件安裝目錄、用戶及用戶組權限

mkdir -p /opt/cloudera/parcels
chown -R cloudera-scm:cloudera-scm /opt/cloudera/

4.hadoop001節(jié)點啟動Server

4.1 啟動server
/opt/cloudera-manager/cm-5.16.1/etc/init.d/cloudera-scm-server start
4.2 阿?云web界?，設置該hadoop001節(jié)點防?墻放開7180端?
4.3 等待1min，打開 http://hadoop001:7180 賬號密碼:admin/admin
4.4 假如打不開，去看server的log，根據(jù)錯誤仔細排查錯誤 
    log路徑在/opt/cloudera-manager/cm-5.16.1/log/cloudera-scm-server

5.所有節(jié)點啟動Agent

/opt/cloudera-manager/cm-5.16.1/etc/init.d/cloudera-scm-agent start

6.接下來，全部Web界面操作

http://hadoop001:7180/

賬號密碼:admin/admin

7.歡迎使?Cloudera Manager--最終?戶許可條款與條件。勾選

8.歡迎使?Cloudera Manager--您想要部署哪個版本？選擇Cloudera Express免費版本

9.感謝您選擇Cloudera Manager和CDH

10.為CDH集群安裝指導主機。選擇[當前管理的主機]，全部勾選

11.選擇存儲庫

12.集群安裝--正在安裝選定Parcel

假如本地parcel離線源配置正確，則"下載"階段瞬間完成，其余階段視節(jié)點數(shù)與內部?絡情況決定。

13.檢查主機正確性

13.1.建議將/proc/sys/vm/swappiness設置為最?值10。
swappiness值控制操作系統(tǒng)嘗試交換內存的積極；
swappiness=0：表示最?限度使?物理內存，之后才是swap空間；
swappiness=100：表示積極使?swap分區(qū)，并且把內存上的數(shù)據(jù)及時搬遷到swap空間；
如果是混合服務器，不建議完全禁?swap，可以嘗試降低swappiness。
臨時調整：
sysctl vm.swappiness=10
永久調整：
cat << EOF >> /etc/sysctl.conf
# Adjust swappiness value
vm.swappiness=10
EOF
13.2.已啟?透明???壓縮，可能會導致重?性能問題，建議禁?此設置。
臨時調整：
echo never > /sys/kernel/mm/transparent_hugepage/defrag
echo never > /sys/kernel/mm/transparent_hugepage/enabled
永久調整：
cat << EOF >> /etc/rc.d/rc.local
# Disable transparent_hugepage
echo never > /sys/kernel/mm/transparent_hugepage/defrag
echo never > /sys/kernel/mm/transparent_hugepage/enabled
EOF
# centos7.x系統(tǒng)，需要為"/etc/rc.d/rc.local"?件賦予執(zhí)?權限
chmod +x /etc/rc.d/rc.local

14.自定義服務，選擇部署Zookeeper、 HDFS、 Yarn服務

15.自定義角色分配

16.數(shù)據(jù)庫設置

連接測試失敗的可能原因:
(1)mysql JDBC jar包沒有放到/usr/share/java或jar包沒有去掉版本號
(2)建數(shù)據(jù)庫cmf 和amon的時候，沒有將權限設置成%
(3)設置完權限之后，沒有flush privileges;

17.審改設置，默認即可

18.?次運?

19.恭喜您!

20.主頁

四.報錯

1.在數(shù)據(jù)庫設置測試時發(fā)生報錯

報錯信息

ERROR 226616765@scm-web-17:com.cloudera.server.web.common.JsonResponse:
JsonResponse created with throwable: com.cloudera.server.web.cmf.MessageException:
A package was not selected.

原因:
測試連接時，等待時間過長，我就點了返回鍵重新加載，然后出現(xiàn)packet找不到的異常。
解決:
返回到選擇大數(shù)據(jù)組件的頁面后，重新進行操作，就可以測試成功了。

本文名稱：CDH5.16.1集群企業(yè)真正離線部署
文章分享：http://muchs.cn/article42/jcgihc.html

成都網站建設公司_創(chuàng)新互聯(lián)，為您提供響應式網站、App設計、用戶體驗、網站設計、軟件開發(fā)、小程序開發(fā)

廣告

聲明：本網站發(fā)布的內容（圖片、視頻和文字）以用戶投稿、用戶轉載內容為主，如果涉及侵權請盡快告知，我們將會在第一時間刪除。文章觀點不代表本網站立場，如需處理請聯(lián)系客服。電話：028-86922220；郵箱：631063699@qq.com。內容未經允許不得轉載，或轉載時需注明來源： 創(chuàng)新互聯(lián)