[Kolla-ansible] prometheus custom rule 추가하기

아래와 같이 kolla config 경로에 rules 파일을 생성해줍니다.

# ls -l /etc/kolla/config/prometheus/
total 20
-rw-r--r-- 1 root root 13995 Aug  6 03:41 first_rules.rules

 

파일 내용은 아래와 같습니다.

rule 샘플은 링크를 참고합니다.

https://github.com/samber/awesome-prometheus-alerts

# cat /etc/kolla/config/prometheus/first_rules.rules 
groups:
- name: nodeExpoter
  rules:
  - alert: HostOutOfMemory
    expr: node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes * 100 < 10
    for: 2m
    labels:
      severity: warning
    annotations:
      summary: Host out of memory (instance {{ $labels.instance }})
      description: "Node memory is filling up (< 10% left)\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"

  - alert: HostMemoryUnderMemoryPressure
    expr: rate(node_vmstat_pgmajfault[1m]) > 1000
    for: 2m
    labels:
      severity: warning
    annotations:
      summary: Host memory under memory pressure (instance {{ $labels.instance }})
      description: "The node is under heavy memory pressure. High rate of major page faults\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"
# cat /etc/kolla/config/prometheus/first_rules.rules 
groups:
- name: nodeExpoter
  rules:
  - alert: HostOutOfMemory
    expr: node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes * 100 < 10
    for: 2m
    labels:
      severity: warning
    annotations:
      summary: Host out of memory (instance {{ $labels.instance }})
      description: "Node memory is filling up (< 10% left)\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"

  - alert: HostMemoryUnderMemoryPressure
    expr: rate(node_vmstat_pgmajfault[1m]) > 1000
    for: 2m
    labels:
      severity: warning
    annotations:
      summary: Host memory under memory pressure (instance {{ $labels.instance }})
      description: "The node is under heavy memory pressure. High rate of major page faults\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"

 

이후 reconfigure 이나 deploy를 진행하면 자동으로 prometheus.yml 과 rule이 추가됩니다.

# kolla-ansible -i all-in-one deploy -t prometheus

# ls -l /etc/kolla/prometheus-server/
total 24
-rw-rw---- 1 root root  1284 Jul 30 05:04 config.json
-rw-rw---- 1 root root 13995 Aug  6 03:41 first_rules.rules
-rw-rw---- 1 root root   952 Aug  6 03:41 prometheus.yml

# grep -ir 'rule' /etc/kolla/prometheus-server/prometheus.yml
rule_files:
- /etc/prometheus/first_rules.rules