How to update prometheus alerting rules in kubernetes cluster (kube-prometheus-stack) [closed]

I installed prometheus and grafana in my kubernetes cluster with kube-prometheus-stack that is a part of prometheus-community:

helm install prometheus prometheus-community/kube-prometheus-stack -n prometheus-community

Now I want to add some extra alerting rules using helm upgrade command. I can add this rules manually via prometheus or grafana UI but this method doesn’t suit for me. So, the question is: how can I update prometheus alerting rules in kubernetes cluster using some extra rules from local yaml file?

I tried to upgrade promtehteus release with helm upgrade command. For this purpose I created a local configuration file prometheus.yaml. I copied prometheus configuration from control panel (status/config section in the navigation panel) and pasted it in prometheus.yaml. Also I added in rule_files section path to my local alert_config.yaml.

...
rule_files:
- ./alert_config.yaml
...

This alert_config.yaml contains following manifest:

groups:
  - name: Host
    rules:
      - alert: HostOutOfMemory
        expr: (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes * 100 < 50) * on(instance) group_left (nodename) node_uname_info{ nodename=~".+" }
        for: 2m
        labels:
          severity: warning
        annotations:
          summary: Host out of memory (instance { { $labels.instance } })
          description: "Node memory is filling up (< 50% left)\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"

Then I executed helm upgrade:

helm upgrade -f prometheus.yml prometheus prometheus-community/kube-prometheus-stack -n prometheus-community

And got the output:

Release "prometheus" has been upgraded. Happy Helming!
NAME: prometheus
LAST DEPLOYED: Mon Nov 13 14:28:55 2023
NAMESPACE: prometheus-community
STATUS: deployed
REVISION: 2
NOTES:
kube-prometheus-stack has been installed. Check its status by running:
  kubectl --namespace prometheus-community get pods -l "release=prometheus"

But after that new prometheus alert rule didn’t appear.
I also tried to change some global values (scrape_interval and scrape_timeout) in order to check that helm upgrade execution really does something. But executing helm upgrade with this changed values also had no effect.

Enviroment

kubernetes server version: v1.23.9

kubernetes client version: v1.26.0

helm version: v3.13.1

  • Try restarting the Prometheus pod. I had a similar issue when I added a relabelling config, which took effect only when I restarted the Prometheus pod manually after helm upgrade.

    – 

  • It looks like your update doesn’t result in pod recreation. Maybe helm doesn’t think that pod is affected by changes? But you can force it to.

    – 

Leave a Comment