How to Stop Noisy Prometheus Alerts with Recording Rules and promtool Tests

Alert fatigue usually starts with a rule that is technically correct but operationally weak. A long PromQL expression gets pasted directly into an alert, the threshold is crossed for a minute or two, and suddenly the team is handling notifications that do not represent useful incidents.

Prometheus gives you better building blocks than that. Recording rules let you precompute reusable signals, query functions explain how functions such as rate() and absent_over_time() really behave, and rule unit tests let you verify alert timing offline. If you surface Prometheus-native rules inside Grafana, Grafana shows them as data source-managed and, for Prometheus, the view is read-only, so the rule files remain the source of truth. See Prometheus alerting in Grafana.

This guide is for operators who already scrape application metrics and want calmer, more predictable alerting. The examples below are lab patterns only. Review them before using them in production.

Why noisy alerts happen

Most noisy alerts have one or more of these problems:

• the alert evaluates a long expression directly instead of a named signal
• the rule has no meaningful pending time
• the team never tested the alert against synthetic series before deployment
• missing telemetry is mixed into availability alerting
• the expected fire time is guessed instead of calculated

A cleaner approach is to precompute the signal once, alert on that named metric, and test the rule timing before reloading Prometheus.

What we are going to build

This lab creates a small Prometheus-native rule set with two goals:

1. precompute a 5-minute API error ratio with a recording rule
2. alert only when that ratio stays above 2% for 10 minutes

That keeps dashboards and alerts aligned around the same metric instead of duplicating a harder-to-read query in multiple places.

Prerequisites

Before you start, make sure you have:

• a Prometheus server already scraping the target metrics
• access to prometheus.yml and your rule files
• the promtool binary available locally
• a known evaluation interval

Prometheus documents that rules are loaded through rule_files, can be syntax-checked with promtool, and are only applied on reload if the files are valid. See Defining recording rules and promtool.

Create a recording rule first

If an alert depends on a long ratio expression, every future maintainer has to read that full query during an incident. Recording rules exist specifically to precompute frequently reused expressions and store them as new time series. Prometheus also notes that rules inside the same group run sequentially and share the same evaluation time, which matters later when the alert reads a freshly recorded metric from the same group. See Defining recording rules.

Create rules/api.rules.yml:

groups:
  - name: api-slo
    interval: 1m
    rules:
      - record: job:http_error_ratio5m
        expr: |
          sum by (job) (
            rate(http_requests_total{status=~"5.."}[5m])
          )
          /
          sum by (job) (
            rate(http_requests_total[5m])
          )

Expected lab result:

• dashboards can query job:http_error_ratio5m
• alert rules stay shorter and easier to review
• the ratio is calculated once per evaluation instead of re-expanded everywhere

Load the file from prometheus.yml:

global:
  evaluation_interval: 1m

rule_files:
  - rules/*.rules.yml

Add an alert that requires sustained failure

Now add an alert that uses the recorded metric:

groups:
  - name: api-slo
    interval: 1m
    rules:
      - record: job:http_error_ratio5m
        expr: |
          sum by (job) (
            rate(http_requests_total{status=~"5.."}[5m])
          )
          /
          sum by (job) (
            rate(http_requests_total[5m])
          )

      - alert: APIHighErrorRate
        expr: job:http_error_ratio5m{job="api"} > 0.02
        for: 10m
        labels:
          severity: page
          team: platform
        annotations:
          summary: "API error ratio above 2% for {{ $labels.job }}"
          description: "The 5-minute HTTP error ratio for {{ $labels.job }} has stayed above 2% for 10 minutes."

This version is intentionally simpler than a raw inline ratio alert. It also avoids a brittle annotation that prints the exact floating-point value, which makes unit tests easier to maintain.

Expected lab behavior:

• the rule becomes pending when the recorded ratio first exceeds 2%
• it becomes firing only after the condition stays true for 10 minutes
• the annotation text is stable enough for rule testing

Validate syntax before touching the server

Before reloading Prometheus, run the documented rule check:

promtool check rules rules/api.rules.yml

Prometheus documents promtool check rules for validating rule files and lists the available flags in the promtool command reference.

Expected result:

• parsed rule output on success
• a non-zero exit status if YAML or rule content is invalid

This step catches indentation mistakes and malformed expressions before they reach a running server.

Unit test the timing with promtool

Syntax validation is useful, but it does not tell you whether the alert fires at the time you think it does. For that, use promtool test rules. Prometheus documents the unit-test file format, including exp_labels and exp_annotations, and it states that expected alerts must include the expanded labels and annotations of the alert. See Unit testing for rules.

Create tests/api-alerts.test.yml:

rule_files:
  - ../rules/api.rules.yml

evaluation_interval: 1m
fuzzy_compare: true

tests:
  - name: api error ratio timing
    interval: 1m
    input_series:
      - series: 'http_requests_total{job="api",status="500"}'
        values: '0+3x15'
      - series: 'http_requests_total{job="api",status="200"}'
        values: '0+97x15'

    promql_expr_test:
      - expr: job:http_error_ratio5m{job="api"}
        eval_time: 1m
        exp_samples:
          - labels: 'job:http_error_ratio5m{job="api"}'
            value: 0.03

    alert_rule_test:
      - eval_time: 10m
        alertname: APIHighErrorRate
        exp_alerts: []

      - eval_time: 11m
        alertname: APIHighErrorRate
        exp_alerts:
          - exp_labels:
              job: api
              severity: page
              team: platform
            exp_annotations:
              summary: 'API error ratio above 2% for api'
              description: 'The 5-minute HTTP error ratio for api has stayed above 2% for 10 minutes.'

Run the test:

promtool test rules tests/api-alerts.test.yml

Why does this fixture expect firing at 11m rather than 16m? Two Prometheus behaviors matter here:

• rules in the same group are evaluated sequentially at the same evaluation time, so the alert can see the freshly written job:http_error_ratio5m immediately in that cycle
• rate() uses the samples available in the range and extrapolates to the ends of the time range, rather than waiting for a completely “full” 5-minute warm-up window

With 1-minute sample spacing, the recording rule already has two counter samples at 1m, so the ratio is available and above 2% there. That means the for: 10m countdown starts around 1m, making 10m still too early and 11m the first reliable firing point for this synthetic lab.

Expected lab result:

• the recorded metric exists by 1m
• the alert is not firing at 10m
• the alert is firing at 11m

That is exactly the kind of timing mistake worth catching before operators depend on the rule.

Detect missing metrics separately

Telemetry loss is a different failure mode from a high error rate, so give it its own alert. Prometheus documents absent() and absent_over_time() in the query functions reference. absent_over_time() returns a series only when the entire range is empty.

A practical example is:

- alert: APIMetricsMissing
  expr: absent_over_time(http_requests_total{job="api"}[10m])
  labels:
    severity: warning
    team: platform
  annotations:
    summary: "API metrics missing for api"
    description: "No http_requests_total samples were seen for the last 10 minutes."

As written, this alert fires after the last 10 minutes are empty. If you later add for: 5m, you are intentionally turning that into roughly 15 minutes of missing data before the alert can fire. Make that total delay explicit in the annotation or a nearby rule comment.

Keep this alert separate because it answers a different question:

• APIHighErrorRate asks whether the service is unhealthy
• APIMetricsMissing asks whether your telemetry disappeared

What Grafana will show

If your Prometheus data source has Manage alerts via Alerting UI enabled, Grafana fetches Prometheus-defined rules and displays them as data source-managed rules. Grafana’s Prometheus alerting docs are explicit that, for Prometheus, this view is read-only, so edits still belong in Prometheus rule files rather than the Grafana UI. See Prometheus alerting.

That makes this workflow practical:

1. edit rule files in version control
2. validate with promtool
3. reload Prometheus through your standard process
4. use Grafana to view the resulting rules and alert state

Cleanup

This lab does not create cloud resources, so there are no usage charges to clean up.

If you used it only for practice:

1. remove rules/api.rules.yml
2. remove tests/api-alerts.test.yml
3. reload or redeploy Prometheus through your normal operational path
4. confirm the recorded metric and test alerts no longer appear

Final thoughts

The cleanest way to reduce Prometheus alert noise is not to tweak thresholds blindly. It is to make the rule lifecycle more disciplined: record important signals once, alert on those named metrics, separate missing-telemetry detection from service-failure logic, and prove alert timing offline with promtool before anything reaches Grafana or Alertmanager.

If you want structured hands-on practice with dashboards, PromQL, exporters, and alerting workflows, see Monitoring with Grafana & Prometheus.

Sources

• Defining recording rules
• Query functions
• Unit testing for rules
• promtool
• Prometheus alerting