I installed prometheus and grafana in my kubernetes cluster with kube-prometheus-stack that is a part of prometheus-community:
helm install prometheus prometheus-community/kube-prometheus-stack -n prometheus-community
Now I want to add some extra alerting rules using helm upgrade command. I can add this rules manually via prometheus or grafana UI but this method doesn’t suit for me. So, the question is: how can I update prometheus alerting rules in kubernetes cluster using some extra rules from local yaml file?
I tried to upgrade promtehteus release with helm upgrade command. For this purpose I created a local configuration file prometheus.yaml. I copied prometheus configuration from control panel (status/config section in the navigation panel) and pasted it in prometheus.yaml. Also I added in rule_files section path to my local alert_config.yaml.
...
rule_files:
- ./alert_config.yaml
...
This alert_config.yaml contains following manifest:
groups:
- name: Host
rules:
- alert: HostOutOfMemory
expr: (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes * 100 < 50) * on(instance) group_left (nodename) node_uname_info{ nodename=~".+" }
for: 2m
labels:
severity: warning
annotations:
summary: Host out of memory (instance { { $labels.instance } })
description: "Node memory is filling up (< 50% left)\n VALUE = {{ $value }}\n LABELS = {{ $labels }}"
Then I executed helm upgrade:
helm upgrade -f prometheus.yml prometheus prometheus-community/kube-prometheus-stack -n prometheus-community
And got the output:
Release "prometheus" has been upgraded. Happy Helming!
NAME: prometheus
LAST DEPLOYED: Mon Nov 13 14:28:55 2023
NAMESPACE: prometheus-community
STATUS: deployed
REVISION: 2
NOTES:
kube-prometheus-stack has been installed. Check its status by running:
kubectl --namespace prometheus-community get pods -l "release=prometheus"
But after that new prometheus alert rule didn’t appear.
I also tried to change some global values (scrape_interval and scrape_timeout) in order to check that helm upgrade execution really does something. But executing helm upgrade with this changed values also had no effect.
Enviroment
kubernetes server version: v1.23.9
kubernetes client version: v1.26.0
helm version: v3.13.1
Try restarting the Prometheus pod. I had a similar issue when I added a relabelling config, which took effect only when I restarted the Prometheus pod manually after helm upgrade.
It looks like your update doesn’t result in pod recreation. Maybe helm doesn’t think that pod is affected by changes? But you can force it to.