kubernetes部署Ingress訪問(wèn)代理與負(fù)載均衡器

Kubernetes中的pod都有獨(dú)立的內(nèi)部IP(外部不可訪問(wèn))，通過(guò)Service可以對(duì)多個(gè)pod進(jìn)行負(fù)載均衡和故障轉(zhuǎn)移，Service可以具有ClusterIP、NodeIP或LoadBanlancer模式。目前，ClusterIP只能內(nèi)部訪問(wèn)，需通過(guò)kubectl proxy代理出來(lái)，NodeIP是跟Node綁定的、遷移性差，LoadBanlancer的每個(gè)服務(wù)都有獨(dú)立的IP地址，管理、使用不便。有沒(méi)有一個(gè)固定的獨(dú)立IP、自動(dòng)節(jié)點(diǎn)漂移的解決方案呢?以前這樣的功能基本上都用Nginx來(lái)實(shí)現(xiàn)，現(xiàn)在Kubernetes有一個(gè)做好了的服務(wù)，也是基于Nginx的，就是Ingress。

如何訪問(wèn)K8S中的服務(wù)：

1、Ingress介紹

Kubernetes 暴露服務(wù)的方式目前只有三種：LoadBlancer Service、NodePort Service、Ingress;前兩種估計(jì)都應(yīng)該很熟悉，下面詳細(xì)的了解下這個(gè) Ingress

Ingress由兩部分組成：Ingress Controller 和 Ingress 服務(wù)。

Ingress Contronler 通過(guò)與 Kubernetes API 交互，動(dòng)態(tài)的去感知集群中 Ingress 規(guī)則變化，然后讀取它，按照自定義的規(guī)則，規(guī)則就是寫(xiě)明了哪個(gè)域名對(duì)應(yīng)哪個(gè)service，生成一段 Nginx 配置，再寫(xiě)到 Nginx-ingress-control的 Pod 里，這個(gè)Ingress Contronler 的pod里面運(yùn)行著一個(gè)nginx服務(wù)，控制器會(huì)把生成的nginx配置寫(xiě)入/etc/nginx.conf文件中，然后 reload 一下使用配置生效。以此來(lái)達(dá)到域名分配置及動(dòng)態(tài)更新的問(wèn)題。

看個(gè)簡(jiǎn)單的圖方便理解：

image.png

ingress控制器有兩種：nginx和haproxy 這里是以nginx為講解。

2、部署一個(gè)Nginx Ingress

ingress的部署文件在github Ingress 倉(cāng)庫(kù)找到. 針對(duì)官方配置我們單獨(dú)添加了 nodeselector 指定，綁定LB地址 以方便DNS 做解析。

$ls

default-backend.yamljenkins-ingress.ymlnginx-ingress-controller-rbac.ymlnginx-ingress-controller.yaml

---

default-backend.yaml：這是官方要求必須要給的默認(rèn)后端，提供404頁(yè)面的。它還提供了一個(gè)http檢測(cè)功能，檢測(cè)nginx-ingress-controll健康狀態(tài)的，通過(guò)每隔一定時(shí)間訪問(wèn)nginx-ingress-controll的/healthz頁(yè)面，如是沒(méi)有響應(yīng)就

返回404之類的錯(cuò)誤碼。

nginx-ingress-controller-rbac.yml:這ingress的RBAC授權(quán)文件

nginx-ingress-controller.yaml：這是控制器的部署文件。

jenkins-ingress.yml：這是Ingress服務(wù)文件，這個(gè)可以是任意web程序，里面配置域名與service的對(duì)應(yīng)關(guān)系，Ingress稱之為規(guī)則。

catnginx-ingress-controller-rbac.yml

#apiVersion:v1

#kind:Namespace

#metadata:#這里是創(chuàng)建一個(gè)namespace，因?yàn)榇薾amespace早有了就不用再創(chuàng)建了

#name:kube-system

---

apiVersion:v1

kind:ServiceAccount

metadata:

name:nginx-ingress-serviceaccount#創(chuàng)建一個(gè)serveerAcount

namespace:kube-system

---

apiVersion:rbac.authorization.k8s.io/v1beta1

kind:ClusterRole

metadata:

name:nginx-ingress-clusterrole#這個(gè)ServiceAcount所綁定的集群角色

rules:

-apiGroups:

-""

resources:#此集群角色的權(quán)限，它能操作的API資源

-configmaps

-endpoints

-nodes

-pods

-secrets

verbs:

-list

-watch

-apiGroups:

-""

resources:

-nodes

verbs:

-get

-apiGroups:

-""

resources:

-services

verbs:

-get

-list

-watch

-apiGroups:

-"extensions"

resources:

-ingresses

verbs:

-get

-list

-watch

-apiGroups:

-""

resources:

-events

verbs:

-create

-patch

-apiGroups:

-"extensions"

resources:

-ingresses/status

verbs:

-update

---

apiVersion:rbac.authorization.k8s.io/v1beta1

kind:Role

metadata:

name:nginx-ingress-role#這是一個(gè)角色，而非集群角色

namespace:kube-system

rules:#角色的權(quán)限

-apiGroups:

-""

resources:

-configmaps

-pods

-secrets

-namespaces

verbs:

-get

-apiGroups:

-""

resources:

-configmaps

resourceNames:

#Defaultsto"-"

#Here:"-"

#Thishastobeadaptedifyouchangeeitherparameter

#whenlaunchingthenginx-ingress-controller.

-"ingress-controller-leader-nginx"

verbs:

-get

-update

-apiGroups:

-""

resources:

-configmaps

verbs:

-create

-apiGroups:

-""

resources:

-endpoints

verbs:

-get

-create

-update

---

apiVersion:rbac.authorization.k8s.io/v1beta1

kind:RoleBinding#角色綁定

metadata:

name:nginx-ingress-role-nisa-binding

namespace:kube-system

roleRef:

apiGroup:rbac.authorization.k8s.io

kind:Role

name:nginx-ingress-role

subjects:

-kind:ServiceAccount

name:nginx-ingress-serviceaccount#綁定在這個(gè)用戶

namespace:kube-system

---

apiVersion:rbac.authorization.k8s.io/v1beta1

kind:ClusterRoleBinding#集群綁定

metadata:

name:nginx-ingress-clusterrole-nisa-binding

roleRef:

apiGroup:rbac.authorization.k8s.io

kind:ClusterRole

name:nginx-ingress-clusterrole

subjects:

-kind:ServiceAccount

name:nginx-ingress-serviceaccount#集群綁定到這個(gè)serviceacount

namespace:kube-system#集群角色是可以跨namespace，但是這里只指明給這個(gè)namespce來(lái)使用

$kubectlcreate-fnginx-ingress-controller-rbac.yml

serviceaccount"nginx-ingress-serviceaccount"created

clusterrole"nginx-ingress-clusterrole"created

role"nginx-ingress-role"created

rolebinding"nginx-ingress-role-nisa-binding"created

clusterrolebinding"nginx-ingress-clusterrole-nisa-binding"created

RBAC創(chuàng)建完后，就創(chuàng)建default backend服務(wù)：

$catdefault-backend.yaml

apiVersion:extensions/v1beta1

kind:Deployment

metadata:

name:default-http-backend

labels:

k8s-app:default-http-backend

namespace:kube-system

spec:

replicas:1

template:

metadata:

labels:

k8s-app:default-http-backend

spec:

terminationGracePeriodSeconds:60

containers:

-name:default-http-backend

#Anyimageispermissableaslongas:

#1.Itservesa404pageat/

#2.Itserves200ona/healthzendpoint

image:gcr.io/google_containers/defaultbackend:1.0

livenessProbe:

httpGet:

path:/healthz#這個(gè)URI是nginx-ingress-controller中nginx里配置好的localtion

port:8080

scheme:HTTP

initialDelaySeconds:30#30s檢測(cè)一次/healthz

timeoutSeconds:5

ports:

-containerPort:8080

resources:

limits:

cpu:10m

memory:20Mi

requests:

cpu:10m

memory:20Mi

nodeSelector:#指定調(diào)度到些Node,以便后面DNS解析

kubernetes.io/hostname:10.3.1.17

---

apiVersion:v1

kind:Service#為defaultbackend創(chuàng)建一個(gè)service

metadata:

name:default-http-backend

namespace:kube-system

labels:

k8s-app:default-http-backend

spec:

ports:

-port:80

targetPort:8080

selector:

k8s-app:default-http-backend

創(chuàng)建：

1

2

3

$kubectlcreate-fdefault-backend.yaml

deployment"default-http-backend"created

service"default-http-backend"created

root@ubuntu15:/data/ingress#kubectlgetrs,pod,svc-nkube-system

NAMEDESIREDCURRENTREADYAGE

rs/default-http-backend-857b544d941111m

NAMEREADYSTATUSRESTARTSAGE

po/default-http-backend-857b544d94-bwgjd1/1Running01m

NAMETYPECLUSTER-IPEXTERNAL-IPPORT(S)AGE

svc/default-http-backendClusterIP10.254.208.14480/TCP1m

創(chuàng)建好default backend后就要?jiǎng)?chuàng)建nginx-ingress-controller了：

$catnginx-ingress-controller.yaml apiVersion:extensions/v1beta1 kind:Deployment metadata: name:nginx-ingress-controller labels: k8s-app:nginx-ingress-controller namespace:kube-system spec: replicas:1 template: metadata: labels: k8s-app:nginx-ingress-controller spec: #hostNetworkmakesitpossibletouseipv6andtopreservethesourceIPcorrectlyregardlessofdockerconfiguration #however,itisnotaharddependencyofthenginx-ingress-controlleritselfanditmaycauseissuesifport10254alreadyistakenonthehost #thatsaid,sincehostPortisbrokenonCNI(https://github.com/kubernetes/kubernetes/issues/31307)wehavetousehostNetworkwhereCNIisused #likewithkubeadm #hostNetwork:true#注釋表示不使用宿主機(jī)的80口， terminationGracePeriodSeconds:60 hostNetwork:true#表示容器使用和宿主機(jī)一樣的網(wǎng)絡(luò) serviceAccountName:nginx-ingress-serviceaccount#引用前面創(chuàng)建的serviceacount containers: -image:gcr.io/google_containers/nginx-ingress-controller:0.9.0-beta.1#容器使用的鏡像 name:nginx-ingress-controller#容器名 readinessProbe:#啟動(dòng)這個(gè)服務(wù)時(shí)要驗(yàn)證/healthz端口10254會(huì)在運(yùn)行的node上監(jiān)聽(tīng)。 httpGet: path:/healthz port:10254 scheme:HTTP livenessProbe: httpGet: path:/healthz port:10254 scheme:HTTP initialDelaySeconds:10#每隔10做健康檢查 timeoutSeconds:1 ports: -containerPort:80 hostPort:80#80映射到80 -containerPort:443 hostPort:443 env: -name:POD_NAME valueFrom: fieldRef: fieldPath:metadata.name -name:POD_NAMESPACE valueFrom: fieldRef: fieldPath:metadata.namespace args: -/nginx-ingress-controller ---default-backend-service=$(POD_NAMESPACE)/default-http-backend #---default-ssl-certificate=$(POD_NAMESPACE)/ingress-secret#這是啟用Https時(shí)用的 nodeSelector:#指明運(yùn)行在哪，此IP要和defaultbackend是同一個(gè)IP kubernetes.io/hostname:10.3.1.17#上面映射到了hostport80，確保此IP80，443沒(méi)有占用.

這個(gè)控制器就是一個(gè)deployment ，里面運(yùn)行一個(gè)容器gcr.io/google_containers/nginx-ingress-controller:0.9.0-beta.1 ，有點(diǎn)像nginx容器，現(xiàn)在創(chuàng)建：

1

2

$kubectlcreate-fnginx-ingress-controller.yaml

deployment"nginx-ingress-controller"created

root@ubuntu15:/data/ingress#kubectlgetrs,pod,svc-nkube-system

NAMEDESIREDCURRENTREADYAGE

rs/default-http-backend-857b544d9411112m

rs/nginx-ingress-controller-8576d4545d11027s

NAMEREADYSTATUSRESTARTSAGE

po/default-http-backend-857b544d94-bwgjd1/1Running012m

po/nginx-ingress-controller-8576d4545d-9tjnv0/1ContainerCreating027s

NAMETYPECLUSTER-IPEXTERNAL-IPPORT(S)AGE

svc/default-http-backendClusterIP10.254.208.14480/TCP12m

現(xiàn)在ingress controller 控制器已部署好了，那么如何使用了，那就要寫(xiě)一個(gè)ingress規(guī)則了，此處就以已存在的jenkins服務(wù)為例，配置如何使用域名訪問(wèn)這個(gè)service:

$kubectlgetsvc,ep

NAMETYPECLUSTER-IPEXTERNAL-IPPORT(S)AGE

svc/jenkinsserviceNodePort10.254.70.478080:30002/TCP3h

NAMEENDPOINTSAGE

ep/jenkinsservice172.30.10.15:8080,172.30.11.7:80803h

現(xiàn)在寫(xiě)個(gè)jenkins service的Ingress 規(guī)則：

$catjenkins-ingress.yml

apiVersion:extensions/v1beta1

kind:Ingress

metadata:

name:jenkins-ingress

namespace:default#服務(wù)在哪個(gè)空間內(nèi)就寫(xiě)哪個(gè)空間

annotations:

kubernetes.io/ingress.class:"nginx"

spec:

rules:

-host:ingress.jenkins.com#此service的訪問(wèn)域名

http:

paths:

-backend:

serviceName:jenkinsservice

servicePort:8080

創(chuàng)建它：

$kubectlcreate-fjenkins-ingress.yml ingress"jenkins-ingress"created $kubectlgetingress NAMEHOSTSADDRESSPORTSAGE jenkins-ingressingress.jenkins.com8010s

到這里就已經(jīng)部署完成了，配置好域名后，就可以用此域名來(lái)訪問(wèn)了：

image.png

部署完成了，現(xiàn)在看下nginx-ingress-controller 里nginx配置文件發(fā)生了哪些變化：

upstreamdefault-jenkinsservice-8080{ least_conn; server172.30.10.15:8080max_fails=0fail_timeout=0; server172.30.11.7:8080max_fails=0fail_timeout=0; } upstreamupstream-default-backend{ least_conn; server172.30.11.6:8080max_fails=0fail_timeout=0; } server{ server_nameingress.jenkins.com; listen[::]:80; location/{ ... proxy_passhttp://default-jenkinsservice-8080; ... } }

這些配置都是ingress-controller 自已寫(xiě)入的，動(dòng)態(tài)更新就是它能通過(guò)K8S API感知到service的endpoint 發(fā)生了變化，然后修改nginx配置并執(zhí)行reload.

至此，部署完成。

Ingress還有很多部署方式，比如配置https訪問(wèn)的, 以后再寫(xiě)。