如何在 k8s 上 部署 prometheus + grafana ...

依照慣例，來刷一下存在感 XD

今天來介紹如何在 k8s 上 部署 prometheus + grafana ...

1. 先拉取 prometheus 、 grafana 、 node-exporter 的 docker image
2. 匯出 prometheus 、 grafana 、 node-exporter 的 docker image
3. 利用 scp 把資料丟到遠端伺服器上
4. 因為這個 node-exporter image tag 版本我 tag 為 latest ，所以在匯入 kind local registry 之前必須要改 tag ，把 latest 改成 v1.3.1
因為

kind 的 local registry 不吃 latest ！！！
kind 的 local registry 不吃 latest ！！！
kind 的 local registry 不吃 latest ！！！

指令如下：

auxo@auxo-virtual-machine:~/tmp/docker-imgs$ docker tag prom/node-exporter:latest prom/node-exporter:v1.3.1

4.1. 把 prometheus 、 grafana 、 node-exporter 的 docker image load 到 k8s (kind) 的 local registry
auxo@auxo-virtual-machine:~/tmp/docker-imgs$ kind load docker-image grafana/grafana:8.5.5 --name test-cluster
Image: "grafana/grafana:8.5.5" with ID "sha256:a04af134d9fccb02a8fd584c38b1fc3c38393b851f05ce5740758c468e3a38c1" not yet present on node "test-cluster-worker2", loading...
Image: "grafana/grafana:8.5.5" with ID "sha256:a04af134d9fccb02a8fd584c38b1fc3c38393b851f05ce5740758c468e3a38c1" not yet present on node "test-cluster-worker", loading...
Image: "grafana/grafana:8.5.5" with ID "sha256:a04af134d9fccb02a8fd584c38b1fc3c38393b851f05ce5740758c468e3a38c1" not yet present on node "test-cluster-control-plane", loading...
Image: "grafana/grafana:8.5.5" with ID "sha256:a04af134d9fccb02a8fd584c38b1fc3c38393b851f05ce5740758c468e3a38c1" not yet present on node "test-cluster-worker3", loading...
auxo@auxo-virtual-machine:~/tmp/docker-imgs$ kind load docker-image prom/node-exporter:v1.3.1 --name test-cluster
Image: "prom/node-exporter:v1.3.1" with ID "sha256:1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26" not yet present on node "test-cluster-worker2", loading...
Image: "prom/node-exporter:v1.3.1" with ID "sha256:1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26" not yet present on node "test-cluster-worker", loading...
Image: "prom/node-exporter:v1.3.1" with ID "sha256:1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26" not yet present on node "test-cluster-control-plane", loading...
Image: "prom/node-exporter:v1.3.1" with ID "sha256:1dbe0e931976487e20e5cfb272087e08a9779c88fd5e9617ed7042dd9751ec26" not yet present on node "test-cluster-worker3", loading...
...

5. 在 k8s 中利用建立 DaemonSet 的方式安裝 node-exporter
首先來看一下 DaemonSet 的 yaml 寫法：

apiVersion: apps/v1
kind: DaemonSet
metadata:
  labels:
    app.kubernetes.io/component: exporter
    app.kubernetes.io/name: node-exporter
  name: node-exporter
  namespace: kube-ops
spec:
  selector:
    matchLabels:
      app.kubernetes.io/component: exporter
      app.kubernetes.io/name: node-exporter
  template:
    metadata:
      labels:
        app.kubernetes.io/component: exporter
        app.kubernetes.io/name: node-exporter
    spec:
      containers:
      - args:
        - --path.sysfs=/host/sys
        - --path.rootfs=/host/root
        - --no-collector.wifi
        - --no-collector.hwmon
        - --collector.filesystem.ignored-mount-points=^/(dev|proc|sys|var/lib/docker/.+|var/lib/kubelet/pods/.+)($|/)
        - --collector.netclass.ignored-devices=^(veth.*)$
        name: node-exporter
        image: prom/node-exporter:v1.3.1
        ports:
          - containerPort: 9100
            protocol: TCP
        resources:
          limits:
            cpu: 250m
            memory: 180Mi
          requests:
            cpu: 102m
            memory: 180Mi
        volumeMounts:
        - mountPath: /host/sys
          mountPropagation: HostToContainer
          name: sys
          readOnly: true
        - mountPath: /host/root
          mountPropagation: HostToContainer
          name: root
          readOnly: true
      volumes:
      - hostPath:
          path: /sys
        name: sys
      - hostPath:
          path: /
        name: root

5.1 建立一個給 node-exporter 使用的 service
kind: Service
apiVersion: v1
metadata:
  name: node-exporter-svc
  namespace: kube-ops
  annotations:
      prometheus.io/scrape: 'true'
      prometheus.io/port:   '9100'
spec:
  selector:
      app.kubernetes.io/component: exporter
      app.kubernetes.io/name: node-exporter
  ports:
  - name: node-exporter
    protocol: TCP
    port: 9100
    targetPort: 9100
	
注意這個

spec:
  selector:
      app.kubernetes.io/component: exporter
      app.kubernetes.io/name: node-exporter

這個跟 node-exporter 的 DaemonSet 裡面的 

  labels:
    app.kubernetes.io/component: exporter
    app.kubernetes.io/name: node-exporter
	
是一致的。

所以它會為每個被 select 到的 pod 套用 service

如果你 describe 這個 svc 來看，就會看到有三個 endpoints

Endpoints: 10.244.1.13:9100,10.244.2.17:9100,10.244.3.12:9100
( DaemonSet 會為每個 node 建立一個 node-exporter 的 pod 。)

5.2 安裝 kube-state-metrics
5.2.1. 因為會用到 registry.k8s.io/kube-state-metrics/kube-state-metrics:v2.5.0 ，先 docker pull 回來。

docker pull registry.k8s.io/kube-state-metrics/kube-state-metrics:v2.5.0

5.2.2. load 到 kind 中。

auxo@auxo-virtual-machine:~/tmp/docker-imgs$ kind load docker-image registry.k8s.io/kube-state-metrics/kube-state-metrics:v2.5.0 --name test-cluster
Image: "registry.k8s.io/kube-state-metrics/kube-state-metrics:v2.5.0" with ID "sha256:b781b8478c274a77dde24bf15b78795d1d564a08a1bdea43b6f38e1f366e15dd" not yet present on node "test-cluster-worker2", loading...
Image: "registry.k8s.io/kube-state-metrics/kube-state-metrics:v2.5.0" with ID "sha256:b781b8478c274a77dde24bf15b78795d1d564a08a1bdea43b6f38e1f366e15dd" not yet present on node "test-cluster-worker", loading...
Image: "registry.k8s.io/kube-state-metrics/kube-state-metrics:v2.5.0" with ID "sha256:b781b8478c274a77dde24bf15b78795d1d564a08a1bdea43b6f38e1f366e15dd" not yet present on node "test-cluster-control-plane", loading...
Image: "registry.k8s.io/kube-state-metrics/kube-state-metrics:v2.5.0" with ID "sha256:b781b8478c274a77dde24bf15b78795d1d564a08a1bdea43b6f38e1f366e15dd" not yet present on node "test-cluster-worker3", loading...

5.2.3. 先前往 https://github.com/kubernetes/kube-state-metrics
5.2.4. 先 clone 這個專案回來。

git clone https://github.com/kubernetes/kube-state-metrics.git

5.2.5. 找到 kube-state-metrics/examples/standard/ ， apply 這目錄中的所有 yamls

auxo@auxo-virtual-machine:~/tmp/kind/yaml/misc/kube-state-metrics-example$ kubectl apply -f ./standard/
clusterrolebinding.rbac.authorization.k8s.io/kube-state-metrics created
clusterrole.rbac.authorization.k8s.io/kube-state-metrics created
deployment.apps/kube-state-metrics created
serviceaccount/kube-state-metrics created
service/kube-state-metrics created

這個動作做完以後，會把 kube-state-metrics 裝在 ns/kube-system 下面。

6. 在 k8s 中安裝 prometheus

6.1. 首先建立一個 ConfigMap 設定檔。這個設定檔會變成 volume mount 在 pod 上，告訴 prometheus 去哪裡撈 metrics 的資料。也就是說 prometheus 要做的工作都設定在這裡。所以如果要在 prometheus 上新增工作或是刪除工作都要在這裡做設定。

6.2. 透過 RBAC 建立 prometheus 的使用者跟相關的角色權限配置。
6.2.1. 建立 ServiceAccount

apiVersion: v1
kind: ServiceAccount
metadata:
  name: prometheus
  namespace: kube-ops

6.2.2. 建立一個 ClusterRole ，這個 Role 裡面會配置 ServiceAccount 的權限。配置錯誤的話有些 metrics 就可能會讀不到。

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata: 
  name: prometheus
rules:
- apiGroups:
  - ""
  resources: 
  - nodes/metrics
  verbs:
  - get
- nonResourceURLs:
  - /metrics
  verbs:
  - get

- apiGroups:
  - ""
  resources:
  - secrets
  - resourcequotas
  - replicationcontrollers
  - limitranges
  - persistentvolumeclaims
  - persistentvolumes
  - namespaces
  verbs:
  - list
  - watch

- apiGroups:
  - ""
  resources: 
  - nodes
  - pods
  - services
  - endpoints
  - nodes/proxy
  - configmaps
  verbs:
  - get
  - list
  - watch
  
- apiGroups:
  - apps
  resources:
  - statefulsets
  - daemonsets
  - deployments
  - replicasets
  verbs:
  - list
  - watch
  
- apiGroups:
  - batch
  resources:
  - cronjobs
  - jobs
  verbs:
  - list
  - watch

- apiGroups:
  - autoscaling
  resources:
  - horizontalpodautoscalers
  verbs:
  - list
  - watch
  
- apiGroups:
  - authentication.k8s.io
  resources:
  - tokenreviews
  - subjectaccessreviews
  verbs:
  - create
  
- apiGroups:
  - policy
  resources:
  - poddisruptionbudgets
  verbs:
  - list
  - watch
  
- apiGroups:
  - certificates.k8s.io
  resources:
  - certificatesigningrequests
  verbs:
  - list
  - watch
  
- apiGroups:
  - storage.k8s.io
  resources:
  - storageclasses
  - volumeattachments
  verbs:
  - list
  - watch
  
- apiGroups:
  - admissionregistration.k8s.io
  resources:
  - mutatingwebhookconfigurations
  - validatingwebhookconfigurations
  verbs:
  - list
  - watch
  
- apiGroups:
  - networking.k8s.io
  resources:
  - networkpolicies
  - ingresses
  verbs:
  - list
  - watch
  
- apiGroups:
  - coordination.k8s.io
  resources:
  - leases
  verbs:
  - list
  - watch

如果重複對同一個 apiGroups 底下的資源定義了不同的 Verb ，會自動運算做 Verbs 的"聯集"。
如果我要更新這個 ClusterRole ( 或是 Role ) 的權限配置，可以在運行的 k8s 上直接 kubectl apply 就可以更新權限。

6.2.3. 建立 ClusterRoleBinding ，把這個 ServiceAccount 跟 ClusterRole 綁在一起。

6.3. 為 prometheus 配置 deployment 。

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: prometheus
  name: prometheus
  namespace: kube-ops
spec:
  replicas: 1
  selector:
    matchLabels:
      app: prometheus
  template:
    metadata:
      labels:
        app: prometheus
    spec:
      serviceAccountName: prometheus
      containers:
        - image: prom/prometheus:v2.36.0
          name: prometheus
          command:
            - "/bin/prometheus"
          # 注意這裡的 prometheus.yml 會透過 ConfigMap mount 到 /etc/prometheus 下面。
          args:
            - "--config.file=/etc/prometheus/prometheus.yml"
            - "--storage.tsdb.path=/prometheus"
            - "--storage.tsdb.retention.time=7d"
            - "--web.enable-lifecycle"
          ports:
            - containerPort: 9090
              protocol: TCP
              name: http
          volumeMounts:
            - mountPath: "/prometheus"
              # subPath: prometheus
              name: prometheus-2-36-pvc
            - mountPath: "/etc/prometheus"
              name: config-volume
          resources:
            requests:
              cpu: 100m
              memory: 512Mi
            limits:
              cpu: 100m
              memory: 512Mi
      securityContext:
        runAsUser: 0
      # 做好準備給容器 mount 使用
      volumes:
        - name: prometheus-2-36-pvc
          persistentVolumeClaim:
            claimName: prometheus-pvc
        - configMap:
            name: prometheus-config
          name: config-volume


7. 在 k8s 上安裝 grafana
7.1. 去 grafana 網站上下載 yaml 配置檔範例

https://grafana.com/docs/grafana/latest/installation/kubernetes/

7.2 修改 image 的版本和 namespace。
看你剛剛塞進 kind 的 image 是哪個版本，就改成那個版本 (這裡以 8.5.5 為範例)。
看想裝在哪個 namespace 記得也要改 namespace ( 這裡以 kube-ops 為範例 )。

---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: grafana-pvc
  # 我改了這裡。
  namespace: kube-ops
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: grafana
  name: grafana
  # 我改了這裡。
  namespace: kube-ops
spec:
  selector:
    matchLabels:
      app: grafana
  template:
    metadata:
      labels:
        app: grafana
    spec:
      securityContext:
        fsGroup: 472
        supplementalGroups:
          - 0
      containers:
        - name: grafana
		  # 我改了這裡。
          image: grafana/grafana:8.5.5
          imagePullPolicy: IfNotPresent
          ports:
            - containerPort: 3000
              # 因為這裡有定義 port 3000 為 http-grafana 所以下面的 svc 才能直接用
			  name: http-grafana
              protocol: TCP
          readinessProbe:
            failureThreshold: 3
            httpGet:
              path: /robots.txt
              port: 3000
              scheme: HTTP
            initialDelaySeconds: 10
            periodSeconds: 30
            successThreshold: 1
            timeoutSeconds: 2
          livenessProbe:
            failureThreshold: 3
            initialDelaySeconds: 30
            periodSeconds: 10
            successThreshold: 1
            tcpSocket:
              port: 3000
            timeoutSeconds: 1
          resources:
            limits:
              memory: 750Mi
              cpu: 250m
            requests:
              cpu: 250m
              memory: 750Mi
          volumeMounts:
            - mountPath: /var/lib/grafana
              name: grafana-pv
      volumes:
        - name: grafana-pv
          persistentVolumeClaim:
            claimName: grafana-pvc
---

apiVersion: v1
kind: Service
metadata:
  name: grafana
  # 我改了這裡。
  namespace: kube-ops
spec:
  ports:
    - port: 3000
      protocol: TCP
	  # 見上面 port 的相關定義。
      targetPort: http-grafana
  selector:
    app: grafana
  sessionAffinity: None
  type: LoadBalancer

8. 設定使用 grafana
8.1. 打開 grafana 的 GUI 。
如果是 kind 那種 lightweight k8s 要做 port-forward 才能存取。

auxo@auxo-virtual-machine:~/tmp/kind/yaml$ kubectl port-forward svc/grafana --address=0.0.0.0 33000:3000 -n kube-ops 1>/dev/null 2>/dev/null&

8.2. 登入 grafana ，帳密預設是 admin/admin ，第一次登入以後會要求改密碼。 

8.3. 打開 grafana 安裝 data source ，指向 prometheus service 的 endpoint
查詢 endpoint 的方法：

auxo@auxo-virtual-machine:~/tmp/kind/yaml$ kubectl describe svc/prometheus | grep -i endpoint
Endpoints:                10.244.3.6:9090

所以 endpoint 就是 http://10.244.3.6:9090
注意這裡使用的是 cluster ip 。

9. import dashboard
在 grafana 網站上有一堆人家定義好的 dashboard 可以套，可以參考一下。
https://grafana.com/grafana/dashboards/

9.1. 找到喜歡的 dashboard 以後，下載它 ( 以 json 的格式) ，然後在 grafana 的 GUI 上 import 就可以了。

9.1.1. 以 kubernetes cluster 這個 dashboard 為例，先找到這個網頁
https://grafana.com/grafana/dashboards/7249

9.1.2. 注意頁面右邊有個 Download JSON
https://grafana.com/api/dashboards/7249/revisions/1/download

9.1.3. 在 grafana 的 GUI 上 import 這個 json 就可以了。