Instrument a Python API with Prometheus and Verify the Counter End to End

Lab environment

This walkthrough is based on a verified disposable lab run on Ubuntu 26.04 x86_64 with 6 vCPU, approximately 12 GB RAM, Docker Engine 29.9.1, and Docker Compose v5.6.0. The runtime is a single-process Python HTTP server plus Prometheus, both started with Docker Compose.

The API is intentionally small and uses Python’s standard-library http.server. Python’s documentation is explicit that http.server is not recommended for production and only implements basic security checks, so treat this lab as a learning exercise for instrumentation and scraping, not as a production web-server pattern. (docs.python.org)

Prerequisites

You need:

• Docker Engine and Docker Compose
• curl
• unused localhost ports 18000 and 19090
• the files in the GitHub source link at the end of this article, opened in a new local working directory

Work from the linked source folder and use the included README.md, app.py, Dockerfile, compose.yaml, prometheus.yml, requirements.txt, and .env.example files. Before you start, copy .env.example to .env so Compose receives the pinned image values used in the verified run.

cp .env.example .env

Because both published ports bind to 127.0.0.1, the lab is only reachable from the local machine unless you use SSH port forwarding. That is a useful safety default for a disposable monitoring exercise.

What this lab proves

This lab answers a narrow but important question: can a Python API increment a Prometheus counter on one route, expose that metric on /metrics, and can Prometheus then scrape and return the expected value?

That is smaller than “the application is healthy.” In Prometheus, the built-in up series tells you whether a scrape succeeded for a target, while a Counter in the Python client tells you that the instrumented process observed a counted event. Those two signals are related, but they prove different things. (prometheus.io)

If this style of scripting and service automation is useful to you, the Python for DevOps Automation training goes further into Python-based infrastructure work.

Review the instrumentation

The application exposes three routes:

• /health returns a JSON health response
• /hello returns a greeting and increments a counter
• /metrics exposes Prometheus-format metrics

The key part of app.py is:

"""Small lab-only HTTP API. Python's built-in server is not a production server."""
from http.server import BaseHTTPRequestHandler, HTTPServer
from prometheus_client import Counter, CONTENT_TYPE_LATEST, generate_latest

REQUESTS = Counter('lab_api_requests_total', 'Requests to the lab greeting endpoint')

class Handler(BaseHTTPRequestHandler):
    def do_GET(self):
        if self.path == '/metrics':
            body, mime = generate_latest(), CONTENT_TYPE_LATEST
        elif self.path == '/health':
            body, mime = b'{"status":"ok"}\n', 'application/json'
        elif self.path == '/hello':
            REQUESTS.inc()
            body, mime = b'{"message":"Hello from the Network Nuts lab"}\n', 'application/json'
        else:
            self.send_error(404)
            return
        self.send_response(200)
        self.send_header('Content-Type', mime)
        self.send_header('Content-Length', str(len(body)))
        self.end_headers()
        self.wfile.write(body)

if __name__ == '__main__':
    HTTPServer(('0.0.0.0', 8000), Handler).serve_forever()

This is a clean teaching pattern because each route demonstrates a different layer of behavior. /health proves the API answered one ordinary request. /hello proves the application path you care about ran and incremented the in-process metric. /metrics proves the process can expose its metrics to Prometheus. The Prometheus Python client documents that Counters only go up, reset when the process restarts, and gain the _total suffix when exposed. (prometheus.github.io)

Start the stack

From the linked source folder, run the exact setup used for the lab:

cp .env.example .env
docker compose up -d --build

This starts two services:

• api on 127.0.0.1:18000
• prometheus on 127.0.0.1:19090

The Compose definition also keeps a few sensible safeguards in place for a disposable lab: the API container is read-only, both containers drop Linux capabilities, and both use no-new-privileges:true. Leave those controls enabled; nothing in this walkthrough requires weakening them.

Prometheus is configured to scrape the API every five seconds:

global:
  scrape_interval: 5s
scrape_configs:
  - job_name: python-lab
    static_configs:
      - targets: ['api:8000']

Prometheus’s own getting-started material describes this scrape model directly: Prometheus collects metrics by scraping HTTP endpoints at the interval you configure. That timing matters when you verify fresh data. (prometheus.io)

Verify the API first

Before you look at metrics, confirm the API is reachable:

curl --retry 10 --retry-connrefused --retry-delay 1 -fsS http://127.0.0.1:18000/health

In the verified run, the response was:

{"status":"ok"}

That result proves the Python process is listening and can answer /health. It does not yet prove that Prometheus has scraped /metrics.

Send exactly 20 counted requests

Now generate the application activity that should increment the counter:

for i in $(seq 1 20); do curl -fsS http://127.0.0.1:18000/hello > /dev/null || break; done

In the verified run, all 20 requests succeeded.

Be strict here. The recorded outcome for this article comes from a fresh process and exactly 20 successful /hello requests. Do not send one extra warm-up call to /hello first. Because this metric is a Counter, re-running the loop without restarting the service will increase the value beyond 20 rather than replace it. (prometheus.github.io)

Wait for Prometheus to scrape fresh data

Prometheus scrapes on a five-second interval in this lab, so wait long enough for a fresh scrape after the requests:

sleep 6

This pause is not cosmetic. Prometheus query results reflect the most recently scraped sample available at query time; they are not a live callback into your Python process. PromQL instant queries are evaluated at a point in time using the latest sample Prometheus has for each matching series. (prometheus.io)

Confirm scrape health with the up metric

Now query Prometheus for the built-in up series:

curl -fsSG http://127.0.0.1:19090/api/v1/query \
  --data-urlencode 'query=up{job="python-lab"}'

In the verified run, Prometheus returned one sample for instance="api:8000" with value 1.

Prometheus documents up{job="<job-name>", instance="<instance-id>"} as 1 when the target is reachable and the scrape succeeded, or 0 when the scrape failed. So here, up=1 means Prometheus successfully reached the API’s metrics endpoint for the python-lab job. It does not prove that /hello behaved correctly; it only proves scrape success. (prometheus.io)

For the browser-side check, open the Prometheus targets page at http://127.0.0.1:19090/targets and confirm the python-lab target is up.

Actual Prometheus browser screenshot. Captured 17 October 2026, 11:57 IST. Open full-size image

That target view is useful during troubleshooting because it shows the scrape target state directly. If the target is down, fix scrape reachability before you reason about counter values.

Query the application counter

Once scrape health is confirmed, query the application metric itself:

curl -fsSG http://127.0.0.1:19090/api/v1/query \
  --data-urlencode 'query=lab_api_requests_total'

In the verified run, Prometheus returned 20.

The observed end-to-end results were:

• /health returned {"status":"ok"}
• 20 /hello requests succeeded
• up{job="python-lab"} returned 1
• lab_api_requests_total returned 20

Rendered output from actual lab commands. Captured 17 October 2026, 11:57 IST. Open full-size image

That is the exact evidence chain this lab was built to demonstrate: the Python process handled 20 requests to /hello, incremented the in-memory Counter 20 times, exposed the metric on /metrics, and Prometheus later scraped and returned that value.

Interpret the number correctly

A raw Counter value is cumulative. In this lab, lab_api_requests_total = 20 means the latest scraped value for that time series was 20. It does not mean 20 requests per second, and it does not mean “20 during the last scrape interval.” The Prometheus query language documentation separates simple instant selection from functions that derive rates or windowed increases over time. (prometheus.io)

For example, these are useful follow-up queries:

rate(lab_api_requests_total[5m])

increase(lab_api_requests_total[5m])

According to the current Prometheus functions reference, rate() calculates the per-second average rate of increase over the selected range, while increase() calculates the total increase across that range and automatically adjusts for counter resets. That makes them better questions for dashboards and alert logic than a bare cumulative value. (prometheus.io)

Those are examples for interpretation only. They were not the measured verification target in this lab. The measured target was simply that a fresh process plus 20 successful /hello requests produced a scraped counter value of 20.

Limits of this lab

This walkthrough is intentionally small, and its limits matter.

First, the server is Python’s standard-library HTTPServer, which the Python docs do not recommend for production use. Second, the counter is stored in-process, so a restart resets it. Third, the runtime is a single-process API container, which keeps the mental model simple but does not prove anything about multi-worker Python servers, shared metrics state, or multiple replicas behind a load balancer. (docs.python.org)

That simplicity is a feature for learning. Once you can explain exactly why this counter became 20, you are in a much better position to reason about more complex instrumentation later.

Cleanup

When you are finished, remove the disposable lab resources:

docker compose down --volumes --remove-orphans

This stops the Compose project and removes the disposable data volumes created for it.

Example source files

The supporting code and configuration on GitHub contain the complete example files discussed in this article.

Sources

• Python http.server documentation
• Prometheus Python client: Counter
• Prometheus: Jobs and instances
• Prometheus: Getting started with Prometheus
• Prometheus: Querying basics
• Prometheus: Query functions