[Prometheus] kube-prometheus-stack Alert 설정 (Prometheus- Opertaor)

• https://yjwang.tistory.com/entry/Prometheus-kube-prometheus-stack-Rule-%EC%B6%94%EA%B0%80-Prometheus-Opertaor
  위 게시글과 동일하게 kube-prometheus-stack 혹은 prometheus-operator 설치할 때 사용한 vaules.yaml를 사용할 예정입니다.

1. Slack Alarm 추가

---

사전에 Slack App에서 Incomming WebHook을 설정해놓으셔야합니다.

refer to

---

• https://grafana.com/blog/2020/02/25/step-by-step-guide-to-setting-up-prometheus-alertmanager-with-slack-pagerduty-and-gmail/

values 수정

---

global 내용 수정

alertmanager:
(생략)
  config:
    global:
      resolve_timeout: 5m
      slack_api_url: https://hooks.slack.com/services/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
    route:
      group_by: ['job']
      group_wait: 30s
      group_interval: 5m
      repeat_interval: 2m
      receiver: 'null'
      receiver: 'slack'
      routes:
      - match:
          alertname: Watchdog
        receiver: 'null'
    receivers:
    - name: 'null'
    - name: 'slack'
      slack_configs:
              - channel: '#kanban'
                username: 'prometheus'
                send_resolved: false
                icon_url: https://avatars3.githubusercontent.com/u/3380462
                title: |-
                 [{{ .Status | toUpper }}{{ if eq .Status "firing" }}:{{ .Alerts.Firing | len }}{{ end }}] {{ .CommonLabels.alertname }} for {{ .CommonLabels.job }}
                text: >-
                   {{ range .Alerts -}}
                   *Alert:* {{ .Annotations.title }}{{ if .Labels.severity }} - `{{ .Labels.severity }}`{{ end }}

                   *Description:* {{ .Annotations.description }}

                   *Details:*
                     {{ range .Labels.SortedPairs }} • *{{ .Name }}:* `{{ .Value }}`
                     {{ end }}
                   {{ end }}
    templates:
    - '/etc/alertmanager/config/*.tmpl'

적용

---

values.yaml 변경 사항 적용

root@master01:~/kube-prometheus-stack# helm upgrade -n prometheus -f values.yaml p1 prometheus-community/kube-prometheus-stack 
Release "p1" has been upgraded. Happy Helming!
NAME: p1
LAST DEPLOYED: Thu Feb  4 14:28:59 2021
NAMESPACE: prometheus
STATUS: deployed
REVISION: 13
NOTES:
kube-prometheus-stack has been installed. Check its status by running:
  kubectl --namespace prometheus get pods -l "release=p1"

Visit https://github.com/prometheus-operator/kube-prometheus for instructions on how to create & configure Alertmanager and Prometheus instances using the Operator.

확인

---

AlertManager > Status

아래와 같이 변경된 내용 확인

메시지 확인

---

Slack Message 확인