Prometheus on K8S (using helm)

refer to

---

• https://github.com/prometheus-community/helm-charts

prerequisite

---

!!Check your cluster version & helm version

Kubernetes 1.16+
Helm 3+

-

[root@master01 helm]# kubectl version
Client Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.2", GitCommit:"f5743093fd1c663cb0cbc89748f730662345d44d", GitTreeState:"clean", BuildDate:"2020-09-16T13:41:02Z", GoVersion:"go1.15", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.2", GitCommit:"f5743093fd1c663cb0cbc89748f730662345d44d", GitTreeState:"clean", BuildDate:"2020-09-16T13:32:58Z", GoVersion:"go1.15", Compiler:"gc", Platform:"linux/amd64"}

[root@master01 helm]# helm version
version.BuildInfo{Version:"v3.4.2", GitCommit:"23dd3af5e19a02d4f4baa5b2f242645a1a3af629", GitTreeState:"clean", GoVersion:"go1.14.13"}

Add repository

[root@master01 helm]# helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
"prometheus-community" has been added to your repositories

[root@master01 helm]# helm repo update
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "prometheus-community" chart repository
...Successfully got an update from the "stable" chart repository
Update Complete. ⎈Happy Helming!⎈

Check chart lists on prometheus-community repo

[root@master01 helm]# helm search repo prometheus-community
NAME                                                  CHART VERSION    APP VERSION    DESCRIPTION                                       
prometheus-community/alertmanager                     0.3.0            v0.21.0        The Alertmanager handles alerts sent by client ...
prometheus-community/kube-prometheus-stack            12.11.3          0.44.0         kube-prometheus-stack collects Kubernetes manif...
prometheus-community/prometheus                       13.2.0           2.24.0         Prometheus is a monitoring system and time seri...
prometheus-community/prometheus-adapter               2.10.1           v0.8.2         A Helm chart for k8s prometheus adapter           
prometheus-community/prometheus-blackbox-exporter     4.10.1           0.18.0         Prometheus Blackbox Exporter                      
prometheus-community/prometheus-cloudwatch-expo...    0.12.1           0.8.0          A Helm chart for prometheus cloudwatch-exporter   
prometheus-community/prometheus-consul-exporter       0.4.0            0.4.0          A Helm chart for the Prometheus Consul Exporter   
prometheus-community/prometheus-couchdb-exporter      0.2.0            1.0            A Helm chart to export the metrics from couchdb...
prometheus-community/prometheus-druid-exporter        0.9.0            v0.8.0         Druid exporter to monitor druid metrics with Pr...
prometheus-community/prometheus-elasticsearch-e...    4.0.1            1.1.0          Elasticsearch stats exporter for Prometheus       
prometheus-community/prometheus-kafka-exporter        0.2.0            v1.2.0         A Helm chart to export the metrics from Kafka i...
prometheus-community/prometheus-mongodb-exporter      2.8.1            v0.10.0        A Prometheus exporter for MongoDB metrics         
prometheus-community/prometheus-mysql-exporter        1.0.1            v0.12.1        A Helm chart for prometheus mysql exporter with...
prometheus-community/prometheus-nats-exporter         2.5.1            0.6.2          A Helm chart for prometheus-nats-exporter         
prometheus-community/prometheus-node-exporter         1.12.0           1.0.1          A Helm chart for prometheus node-exporter         
prometheus-community/prometheus-operator              9.3.2            0.38.1         DEPRECATED - This chart will be renamed. See ht...
prometheus-community/prometheus-postgres-exporter     1.6.0            0.8.0          A Helm chart for prometheus postgres-exporter     
prometheus-community/prometheus-pushgateway           1.5.1            1.3.0          A Helm chart for prometheus pushgateway           
prometheus-community/prometheus-rabbitmq-exporter     0.5.6            v0.29.0        Rabbitmq metrics exporter for prometheus          
prometheus-community/prometheus-redis-exporter        4.0.0            1.11.1         Prometheus exporter for Redis metrics             
prometheus-community/prometheus-snmp-exporter         0.1.1            0.19.0         Prometheus SNMP Exporter                          
prometheus-community/prometheus-stackdriver-exp...    1.6.1            0.6.0          Stackdriver exporter for Prometheus               
prometheus-community/prometheus-statsd-exporter       0.2.0            0.18.0         A Helm chart for prometheus stats-exporter        
prometheus-community/prometheus-to-sd                 0.3.1            0.5.2          Scrape metrics stored in prometheus format and ...

Install Prometheus stack

---

Check values in charts

[root@master01 helm]# helm show values prometheus-community/kube-prometheus-stack |more
...

Install helm chart

[root@master01 helm]# kubectl create namespace prometheus
namespace/prometheus created

[root@master01 helm]# helm install prometheus-stack prometheus-community/kube-prometheus-stack -n prometheus
NAME: prometheus-stack
LAST DEPLOYED: Wed Jan 13 07:10:54 2021
NAMESPACE: prometheus
STATUS: deployed
REVISION: 1
NOTES:
kube-prometheus-stack has been installed. Check its status by running:
  kubectl --namespace prometheus get pods -l "release=prometheus-stack"

Visit https://github.com/prometheus-operator/kube-prometheus for instructions on how to create & configure Alertmanager and Prometheus instances using the Operator.

Check resources

[root@master01 ~]# kubectl get all -n prometheus
NAME                                                         READY   STATUS    RESTARTS   AGE
pod/alertmanager-prometheus-stack-kube-prom-alertmanager-0   2/2     Running   0          55s
pod/prometheus-prometheus-stack-kube-prom-prometheus-0       2/2     Running   1          54s
pod/prometheus-stack-grafana-656bff5fc5-kbmqj                2/2     Running   0          72s
pod/prometheus-stack-kube-prom-operator-67cd9846bd-24t65     1/1     Running   0          72s
pod/prometheus-stack-kube-state-metrics-5d7dbfcccd-rn9vx     1/1     Running   0          72s
pod/prometheus-stack-prometheus-node-exporter-5gk8l          1/1     Running   0          72s
pod/prometheus-stack-prometheus-node-exporter-dmnsq          1/1     Running   0          72s
pod/prometheus-stack-prometheus-node-exporter-fjh8r          1/1     Running   0          72s
pod/prometheus-stack-prometheus-node-exporter-pb7m8          1/1     Running   0          72s
pod/prometheus-stack-prometheus-node-exporter-w6vq6          1/1     Running   0          72s

NAME                                                TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                      AGE
service/alertmanager-operated                       ClusterIP   None             <none>        9093/TCP,9094/TCP,9094/UDP   55s
service/prometheus-operated                         ClusterIP   None             <none>        9090/TCP                     54s
service/prometheus-stack-grafana                    ClusterIP   10.100.126.91    <none>        80/TCP                       72s
service/prometheus-stack-kube-prom-alertmanager     ClusterIP   10.100.54.1      <none>        9093/TCP                     72s
service/prometheus-stack-kube-prom-operator         ClusterIP   10.100.32.202    <none>        443/TCP                      72s
service/prometheus-stack-kube-prom-prometheus       ClusterIP   10.100.78.206    <none>        9090/TCP                     72s
service/prometheus-stack-kube-state-metrics         ClusterIP   10.100.217.2     <none>        8080/TCP                     72s
service/prometheus-stack-prometheus-node-exporter   ClusterIP   10.100.137.162   <none>        9100/TCP                     73s

NAME                                                       DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
daemonset.apps/prometheus-stack-prometheus-node-exporter   5         5         5       5            5           <none>          72s

NAME                                                  READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/prometheus-stack-grafana              1/1     1            1           72s
deployment.apps/prometheus-stack-kube-prom-operator   1/1     1            1           72s
deployment.apps/prometheus-stack-kube-state-metrics   1/1     1            1           72s

NAME                                                             DESIRED   CURRENT   READY   AGE
replicaset.apps/prometheus-stack-grafana-656bff5fc5              1         1         1       72s
replicaset.apps/prometheus-stack-kube-prom-operator-67cd9846bd   1         1         1       72s
replicaset.apps/prometheus-stack-kube-state-metrics-5d7dbfcccd   1         1         1       72s

NAME                                                                    READY   AGE
statefulset.apps/alertmanager-prometheus-stack-kube-prom-alertmanager   1/1     55s
statefulset.apps/prometheus-prometheus-stack-kube-prom-prometheus       1/1     54s

[root@master01 ~]# kubectl get secret -n prometheus prometheus-stack-grafana -o jsonpath="{.data.admin-password}" | base64 --decode ; echo
prom-operator

이후 port-forward 및 NodePort로 Service Type을 변경해서 Grafana dashboard를 구성한다.