Ubuntu 20.04
Sponsored Link

Kubernetes : Deploy Prometheus2020/08/25

 
Deploy Prometheus to monitor metrics in Kubernetes Cluster.
This example is based on the environment like follows.
-----------+---------------------------+--------------------------+------------
           |                           |                          |
       eth0|10.0.0.30              eth0|10.0.0.51             eth0|10.0.0.52
+----------+-----------+   +-----------+----------+   +-----------+----------+
|   [ dlp.srv.world ]  |   | [ node01.srv.world ] |   | [ node02.srv.world ] |
|      Master Node     |   |      Worker Node     |   |      Worker Node     |
+----------------------+   +----------------------+   +----------------------+

[1]
A Persistent storage is needed for Prometheus.
On this example, install NFS Server on Master Node and configure [/home/nfsshare] directory as NFS share as external persistent storage, and also configure dynamic volume provisioning with NFS plugin like the example of [1], [2], [3].
[2] Install Prometheus chart with Helm.
# output config and change some settings

root@dlp:~#
helm inspect values stable/prometheus > prometheus.yaml

root@dlp:~#
vi prometheus.yaml
alertmanager:
.....
.....
    line 213: uncomment and specify [storageClass] to use
    storageClass: "nfs-client"
.....
.....

server:
.....
.....
    line 803: uncomment and specify [storageClass] to use
    storageClass: "nfs-client"
.....
.....
pushgateway:
.....
.....
    line 1134: uncomment and specify [storageClass] to use
    storageClass: "nfs-client"

# create a namespace for Prometheus

root@dlp:~#
kubectl create namespace monitoring

namespace/monitoring created
root@dlp:~#
helm install prometheus --namespace monitoring -f prometheus.yaml stable/prometheus

NAME: prometheus
LAST DEPLOYED: Tue Aug 24 19:14:04 2020
NAMESPACE: monitoring
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
The Prometheus server can be accessed via port 80 on the following DNS name from within your cluster:
prometheus-server.monitoring.svc.cluster.local


Get the Prometheus server URL by running these commands in the same shell:
  export POD_NAME=$(kubectl get pods --namespace monitoring -l "app=prometheus,component=server" -o jsonpath="{.items[0].metadata.name}")
  kubectl --namespace monitoring port-forward $POD_NAME 9090


The Prometheus alertmanager can be accessed via port 80 on the following DNS name from within your cluster:
prometheus-alertmanager.monitoring.svc.cluster.local


Get the Alertmanager URL by running these commands in the same shell:
  export POD_NAME=$(kubectl get pods --namespace monitoring -l "app=prometheus,component=alertmanager" -o jsonpath="{.items[0].metadata.name}")
  kubectl --namespace monitoring port-forward $POD_NAME 9093
#################################################################################
######   WARNING: Pod Security Policy has been moved to a global property.  #####
######            use .Values.podSecurityPolicy.enabled with pod-based      #####
######            annotations                                               #####
######            (e.g. .Values.nodeExporter.podSecurityPolicy.annotations) #####
#################################################################################


The Prometheus PushGateway can be accessed via port 9091 on the following DNS name from within your cluster:
prometheus-pushgateway.monitoring.svc.cluster.local


Get the PushGateway URL by running these commands in the same shell:
  export POD_NAME=$(kubectl get pods --namespace monitoring -l "app=prometheus,component=pushgateway" -o jsonpath="{.items[0].metadata.name}")
  kubectl --namespace monitoring port-forward $POD_NAME 9091

For more information on running Prometheus, visit:
https://prometheus.io/

root@dlp:~#
kubectl get pods -n monitoring -o wide

NAME                                            READY   STATUS    RESTARTS   AGE     IP            NODE               NOMINATED NODE   READINESS GATES
prometheus-alertmanager-77455866bd-p8t9d        2/2     Running   0          2m12s   10.244.1.23   node01.srv.world   <none>           <none>
prometheus-kube-state-metrics-c65b87574-9rmz6   1/1     Running   0          2m12s   10.244.1.22   node01.srv.world   <none>           <none>
prometheus-node-exporter-rth4v                  1/1     Running   0          2m12s   10.0.0.51     node01.srv.world   <none>           <none>
prometheus-node-exporter-z7tbn                  1/1     Running   0          2m12s   10.0.0.52     node02.srv.world   <none>           <none>
prometheus-pushgateway-c454fc4-sn6bh            1/1     Running   0          2m12s   10.244.2.15   node02.srv.world   <none>           <none>
prometheus-server-dc6d7575c-5fcjh               2/2     Running   0          2m12s   10.244.2.16   node02.srv.world   <none>           <none>

# if access from outside of cluster, set port-forwarding

root@dlp:~#
kubectl port-forward -n monitoring service/prometheus-server --address 0.0.0.0 9090:80

[3] If you deploy Granafa, too, It's possible like follows.
# output config and change some settings

root@dlp:~#
helm inspect values stable/grafana > grafana.yaml

root@dlp:~#
vi grafana.yaml
line 215: enable [persistence]
line 216: uncomment and change to your [storageClass]
persistence:
  type: pvc
  enabled: true
  storageClassName: nfs-client

root@dlp:~#
helm install grafana --namespace monitoring -f grafana.yaml stable/grafana

NAME: grafana
LAST DEPLOYED: Tue Aug 24 19:33:46 2020
NAMESPACE: monitoring
STATUS: deployed
REVISION: 1
NOTES:
1. Get your 'admin' user password by running:

   kubectl get secret --namespace monitoring grafana -o jsonpath="{.data.admin-password}" | base64 --decode ; echo

2. The Grafana server can be accessed via port 80 on the following DNS name from within your cluster:

   grafana.monitoring.svc.cluster.local

   Get the Grafana URL to visit by running these commands in the same shell:

     export POD_NAME=$(kubectl get pods --namespace monitoring -l "app.kubernetes.io/name=grafana,app.kubernetes.io/instance=grafana" -o jsonpath="{.items[0].metadata.name}")
     kubectl --namespace monitoring port-forward $POD_NAME 3000

3. Login with the password from step 1 and the username: admin

root@dlp:~#
kubectl get pods -n monitoring

NAME                                            READY   STATUS    RESTARTS   AGE
grafana-7b6499754f-74786                        1/1     Running   0          41s
.....
.....

# if access from outside of cluster, set port-forwarding

root@dlp:~#
kubectl port-forward -n monitoring service/grafana --address 0.0.0.0 3000:80

[4]
If you access to Prometheus UI from a Host in cluster, access to the URL below with an Web browser.
⇒ http://prometheus-server.monitoring.svc.cluster.local
If you set port-fowarding, access to the URL below on a client computer in your local network.
⇒ http://(Master Node Hostname or IP address):(setting port)/
That's OK if following Prometheus UI is displayed.
[5]
If you access to Granafa from a Host in cluster, access to the URL below with an Web browser.
⇒ http://grafana.monitoring.svc.cluster.local
If you set port-fowarding, access to the URL below on a client computer in your local network.
⇒ http://(Master Node Hostname or IP address):(setting port)/
That's OK if following Granafa UI is displayed.
For [admin] password, it's possible to confirm with the command below.
⇒ kubectl get secret --namespace monitoring grafana -o jsonpath="{.data.admin-password}" | base64 --decode ; echo
Matched Content