今天是一个值得庆祝的日子, 终于把研究了半年未果的k8s ingress反向代理通过traefik这个工具给试验出来了, 这里感谢我的甜宝宝对我的支持, 因为有你, 我才有动力继续前行.

traefik 是一款开源的反向代理与负载均衡工具。它最大的优点是能够与常见的微服务系统直接整合，可以实现自动化动态配置。目前支持 Docker, Swarm, Mesos/Marathon, Mesos, Kubernetes, Consul, Etcd, Zookeeper, BoltDB, Rest API 等等后端模型。

由于微服务架构以及 Docker 技术和 kubernetes 编排工具最近几年才开始逐渐流行，所以一开始的反向代理服务器比如 nginx、apache 并未提供其支持，所以才会出现 Ingress Controller 来做 kubernetes 和前端负载均衡器如 nginx 之间做衔接；即 Ingress Controller 的存在就是为了能跟 kubernetes 交互，又能写 nginx 配置，还能 reload 它，这是一种折中方案；而 traefik 天生就是提供了对 kubernetes 的支持，也就是说 traefik 本身就能跟 kubernetes API 交互，感知后端变化，因此可以得知:  traefik 完全已经取代了 Ingress Controller 作为与k8s交互的代理，在此给大家这里整体架构如下:

为什么选择 traefik？

Golang 编写，单文件部署，与系统无关，同时也提供小尺寸 Docker 镜像。
支持 Docker/Etcd 后端，天然连接我们的微服务集群。
内置 Web UI，管理相对方便。
自动配置 ACME(Let’s Encrypt) 证书功能。
性能尚可，我们也没有到压榨 LB 性能的阶段，易用性更重要。

除了这些以外，traefik 还有以下特点：
Restful API 支持。
支持后端健康状态检查，根据状态自动配置。
支持动态加载配置文件和 graceful 重启。
支持 WebSocket 和 HTTP/2。

目前网上很多资料虽然能够成功在k8s下搭建ingress+traefik实现cluster内网的代理功能, 但如何将这个反向代理IP暴露在我们外网, 也就是不依赖于AWS等公有云, 在我们on-prem(本地k8s网络环境)通过定义hostnetwork去实现将我们的node ip直接作为我们在cluster下创建的反向代理ip,  将是我们本次需要给大家介绍的内容.

我们接着上一篇Kubernates1.9+Docker17离线安装部署, 给大家展开如何实现k8s通过使用Ingress+Traefik实现集群反向代理功能.

1.获取Traefik支持K8s仓库脚本

# cd /kube/

# git clone https://github.com/containous/traefik.git

# cd traefik/examples/k8s/

# ll

total 36
-rw-r--r-- 1 root root  140 Aug 26 04:23 cheese-default-ingress.yaml
-rw-r--r-- 1 root root 1805 Aug 26 04:23 cheese-deployments.yaml
-rw-r--r-- 1 root root  519 Aug 26 04:23 cheese-ingress.yaml
-rw-r--r-- 1 root root  509 Aug 26 04:23 cheese-services.yaml
-rw-r--r-- 1 root root  504 Aug 26 04:23 cheeses-ingress.yaml
-rw-r--r-- 1 root root 1144 Aug 26 06:40 traefik-deployment.yaml
-rw-r--r-- 1 root root 1206 Aug 26 04:23 traefik-ds.yaml
-rw-r--r-- 1 root root  694 Aug 26 04:23 traefik-rbac.yaml
-rw-r--r-- 1 root root  466 Aug 26 04:40 ui.yaml

这个目录下就是示例 Traefik 启动所需要的 yaml 文件，Traefik 提供了适配各个类型服务编排的部署方式，kubernetes 启动方式支持 Deployment 和 DaemonSet，二选一都可以, 这里笔者选择Deployment

2. Traefik脚本配置

还记得我之前提到的hostnetwork吗? 

这里我们需要在部署之前将我们的traefik-deployment.yaml下添加一行 hostNetwork: true

这样子就能保证我们的Traefik反向代理的会将自己Endpoints IP配置成我们的NodeIP, 供我们与K8S同一网段的设备访问.

我们可以这么配置:

# vi traefik-deployment.yaml

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: traefik-ingress-controller
  namespace: kube-system
---
kind: Deployment
apiVersion: extensions/v1beta1
metadata:
  name: traefik-ingress-controller
  namespace: kube-system
  labels:
    k8s-app: traefik-ingress-lb
spec:
  replicas: 1
  selector:
    matchLabels:
      k8s-app: traefik-ingress-lb
  template:
    metadata:
      labels:
        k8s-app: traefik-ingress-lb
        name: traefik-ingress-lb
    spec:
      hostNetwork: true
      serviceAccountName: traefik-ingress-controller
      terminationGracePeriodSeconds: 60
      containers:
      - image: traefik
        name: traefik-ingress-lb
        ports:
        - name: http
          containerPort: 80
        - name: admin
          containerPort: 8080
        args:
        - --api
        - --kubernetes
        - --logLevel=INFO
---
kind: Service
apiVersion: v1
metadata:
  name: traefik-ingress-service
  namespace: kube-system
spec:
  selector:
    k8s-app: traefik-ingress-lb
  ports:
    - protocol: TCP
      port: 80
      name: web
    - protocol: TCP
      port: 8080
      name: admin
  type: NodePort

# vi traefik-rbac.yaml

---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: traefik-ingress-controller
rules:
  - apiGroups:
      - ""
    resources:
      - services
      - endpoints
      - secrets
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - extensions
    resources:
      - ingresses
    verbs:
      - get
      - list
      - watch
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: traefik-ingress-controller
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: traefik-ingress-controller
subjects:
- kind: ServiceAccount
  name: traefik-ingress-controller
  namespace: kube-system

这样子我们就完成了K8S脚本的初始修改, 接下来我们就可以直接开始我们的Traefik安装

3. Traefik安装

# kubectl apply -f traefik-rbac.yaml

clusterrole "traefik-ingress-controller" created
clusterrolebinding "traefik-ingress-controller" created

# kubectl apply -f traefik-deployment.yaml

serviceaccount "traefik-ingress-controller" created
deployment "traefik-ingress-controller" created
service "traefik-ingress-service" created

# kubectl apply -f ui.yaml

service "traefik-web-ui" created
ingress "traefik-web-ui" created

# kubectl get pods --all-namespaces -o wide

NAME                                          READY     STATUS    RESTARTS   AGE       IP             NODE
…
traefik-ingress-controller-795ffb7d78-ppw94   1/1       Running   0          5m        10.110.16.15   kube-node1

可以看到我们的traefik-ingress pods成功的打开了我们的hostnetwork, 这里显示的是我们的node ip而不是pod内网 ip, 也就是我们可以直接通过访问node ip 10.110.16.15来连接traefik反向代理, 利用traefik来分配我们的K8S的容器连接.

# kubectl get service --all-namespaces

NAMESPACE      NAME                        TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                       AGE
default        frontend                    ClusterIP   10.110.87.1      <none>        80/TCP                        4h
default        kubernetes                  ClusterIP   10.96.0.1        <none>        443/TCP                       1d
default        my-nginx                    ClusterIP   10.102.200.83    <none>        80/TCP                        4h
kube-system    kube-dns                    ClusterIP   10.96.0.10       <none>        53/UDP,53/TCP                 1d
kube-system    kubernetes-dashboard        NodePort    10.108.187.244   <none>        443:32666/TCP                 1d
kube-system    tiller-deploy               ClusterIP   10.111.251.3     <none>        44134/TCP                     1d
kube-system    traefik-ingress-service     NodePort    10.100.70.138    <none>        80:31087/TCP,8080:31017/TCP   13m
kube-system    traefik-web-ui              ClusterIP   10.96.54.167     <none>        80/TCP                        13m
newegg-nginx   newegg-nginx-newegg-nginx   NodePort    10.97.200.124    <none>        80:32239/TCP                  8h

我们可以通过两种方式去访问我们的traefik后台管理员界面.

1).通过Node Port方式

我们可以在与K8S node同网段的机器去访问其Node暴露的端口31017

2).直接通过ui.yaml配置traefik+ingress反向代理, 这样我们就可以通过host域名直接访问后台

# vi ui.yaml

---
apiVersion: v1
kind: Service
metadata:
  name: traefik-web-ui
  namespace: kube-system
spec:
  selector:
    k8s-app: traefik-ingress-lb
  ports:
  - name: web
    port: 80
    targetPort: 8080
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: traefik-web-ui
  namespace: kube-system
spec:
  rules:
  - host: traefik-ui.k8s
    http:
      paths:
      - path: /
        backend:
          serviceName: traefik-web-ui
          servicePort: web

在本地MacOS系统添加traefik-ui.k8s解析到10.110.16.15

# echo "10.110.16.15 traefik-ui.k8s" >> /etc/hosts

我们可以同样利用traefik反向代理, 通过http域名的方式访问admin后台.

4.部署两个服务nginx1-7和nginx1-8，配置Traefik去负载这两个服务：

配置一个deployment类型的1.7版本nginx实例, 并利用service开启其内部80端口监听.

# vi nginx1-7.yaml

apiVersion: v1
kind: Service
metadata:
  name: frontend
spec:
  ports:
    - port: 80
      targetPort: 80
  selector:
    app: nginx1-7
---
apiVersion: apps/v1beta1
kind: Deployment
metadata:
  name: nginx1-7-deployment
spec:
  replicas: 2
  template:
    metadata:
      labels:
        app: nginx1-7
    spec:
      containers:
      - name: nginx
        image: nginx:1.7.9
        ports:
        - containerPort: 80

配置一个deployment类型的1.8版本nginx实例, 并利用service开启其内部80端口监听.

# vi nginx1-8.yaml

apiVersion: v1
kind: Service
metadata:
  name: my-nginx
spec:
  ports:
    - port: 80
      targetPort: 80
  selector:
    app: nginx1-8
---
apiVersion: apps/v1beta1
kind: Deployment
metadata:
  name: nginx1-8-deployment
spec:
  replicas: 2
  template:
    metadata:
      labels:
        app: nginx1-8
    spec:
      containers:
      - name: nginx
        image: nginx:1.8
        ports:
        - containerPort: 80

通过利用Ingress绑定1.7与1.8nginx实例下的serviceName, 给其定义不同的域名, 实现traefik反向代理.

# vi traefik.yaml

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: traefik-ingress
  namespace: default
spec:
  rules:
  - host: traefik.nginx.io
    http:
      paths:
      - path: /
        backend:
          serviceName: my-nginx
          servicePort: 80
  - host: traefik.frontend.io
    http:
      paths:
      - path: /
        backend:
          serviceName: frontend
          servicePort: 80

我们这里通过配置MacOS本地Host DNS, 将所有的DNS记录指向我们暴露在外网的traefik IP, 从而让traefik作为代理接管所有访问K8S内部实例的连接, 实际的原理和我们之前用过的apache, nginx基本一致.

# echo "10.110.16.15 traefik.nginx.io traefik.frontend.io" >> /etc/hosts

这样子我们就可以直接在MacOS浏览器下访问traefik.nginx.io与traefik.frontend.io从而实现traefik反向代理.

5.配置HTTPS访问traefik管理员后台

1).创建并配置https证书并保存到k8s secret下.

# mkdir -p /opt/k8s/ssl

# cd /opt/k8s/ssl

# openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout tls.key -out tls.crt -subj "/CN=traefik-ui.k8s"

# kubectl create secret generic traefik-cert --from-file=tls.crt --from-file=tls.key -n kube-system

2).配置traefik反向代理并保存到k8s configmap下.

# mkdir -p /opt/k8s/conf/

# cd /opt/k8s/ssl/

# vi traefik-ui.toml

defaultEntryPoints = ["http","https"]
[entryPoints]
  [entryPoints.http]
  address = ":80"
    [entryPoints.http.redirect]
    regex = "^http://traefik-ui.k8s/(.*)"
    replacement = "https://traefik-ui.k8s/$1"
  [entryPoints.https]
  address = ":443"
    [entryPoints.https.tls]
      [[entryPoints.https.tls.certificates]]
      certFile = "/opt/k8s/ssl/tls.crt"
      keyFile = "/opt/k8s/ssl/tls.key"

# kubectl create configmap traefik-ui-conf --from-file=traefik-ui.toml -n kube-system

这样子我们就成功配置traefik反向代理, 仅使http://traefik-ui.k8s会重定向到https://traefik-ui.k8s, 实现https加密访问.

3).创建traefik deployment实例并支持https反向代理.

# vi traefik-deployment-ssl.yaml 

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: traefik-ingress-controller
  namespace: kube-system
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: traefik-ingress-lb
  namespace: kube-system
  labels:
    k8s-app: traefik-ingress-lb
spec:
  replicas: 1
  selector:
    matchLabels:
      k8s-app: traefik-ingress-lb
  template:
    metadata:
      labels:
        k8s-app: traefik-ingress-lb
        name: traefik-ingress-lb
    spec:
      terminationGracePeriodSeconds: 60
      hostNetwork: true
      restartPolicy: Always
      serviceAccountName: traefik-ingress-controller
      volumes:
      - name: ssl
        secret:
          secretName: traefik-cert
      - name: config
        configMap:
          name: traefik-ui-conf
      containers:
      - image: traefik
        name: traefik-ingress-lb
        volumeMounts:
        - mountPath: "/opt/k8s/ssl"
          name: "ssl"
        - mountPath: "/opt/k8s/conf"
          name: "config"
        ports:
        - name: http
          containerPort: 80
          hostPort: 80
        - name: https
          containerPort: 443
        - name: admin
          containerPort: 8080
          hostPort: 8080
        args:
        - --configFile=/opt/k8s/conf/traefik-ui.toml
        - --web
        - --web.address=:8080
        - --kubernetes
---
kind: Service
apiVersion: v1
metadata:
  name: traefik-ingress-service
  namespace: kube-system
spec:
  selector:
    k8s-app: traefik-ingress-lb
  ports:
    - protocol: TCP
      port: 80
      name: web
    - protocol: TCP
      port: 443
      name: https
    - protocol: TCP
      port: 8080
      name: admin
  type: NodePort

这里我们在原来的traefik deployment实例基础上将之前配置的secret与configmap导入该deployment下, 并mount ssl与conf目录用来让traefik获取我们在k8s node目录下保存的相应SSL证书与traefik反向代理配置.

最后我们通过service类型打开我们的deployment 443端口.

4).创建traefik管理员界面的ingress, 从而实现https域名访问.

# vi ui-ssl.yaml

---
apiVersion: v1
kind: Service
metadata:
  name: traefik-web-ui
  namespace: kube-system
spec:
  selector:
    k8s-app: traefik-ingress-lb
  ports:
  - name: web
    port: 80
    targetPort: 8080
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: traefik-web-ui
  namespace: kube-system
  annotations:
    kubernetes.io/ingress.class: traefik
spec:
  tls:
    - secretName: traefik-cert
  rules:
  - host: traefik-ui.k8s
    http:
      paths:
      - path: /
        backend:
          serviceName: traefik-web-ui
          servicePort: web

最后我们访问https://traefik-ui.k8s 验证结果.

这样子我们就成功的开启了traefik后台admin界面的https加密连接...

更多traefik+ingress相关内容: https://docs.traefik.io/user-guide/kubernetes/