Wiki / Integrations

Prometheus Integration

Last updated

Prometheus Integration

Expose CertTracker monitor data as Prometheus metrics via the textfile collector pattern: a small Python script polls the CertTracker API, writes a .prom file, and node_exporter (which you already have on every box) scrapes it automatically. No HTTP server to run, no new container, no new firewall rule.

Community-validated recipe. Written from Prometheus and node_exporter docs; not exercised in our own monitoring stack. Corrections welcome via the community forum.

Plan required. API access is a Business and Enterprise plan feature. See API access for token creation.

Architecture

+-------------------+         +-----------------------+
|  CertTracker      |  HTTPS  |  Prometheus host      |
|                   <----------|                       |
|  /api/v1/monitors |         |  collect.py (cron)    |
|                   |         |   |                   |
+-------------------+         |   v                   |
                              |   certtracker.prom    |
                              |   |                   |
                              |   v                   |
                              |  node_exporter        |
                              |   --collector.textfile|
                              |   |                   |
                              |   v                   |
                              |  Prometheus scraper   |
                              +-----------------------+

Step 1 — Mint a CertTracker API token

Administration → API Tokens → Create. Name it prometheus-prod. Copy the value.

Step 2 — Drop the collection script

Save this on whatever host runs node_exporter (typically every monitored Linux box, but for this we only need ONE — pick your prometheus server or any always-on box):

sudo install -d -o nobody -g nogroup -m 0755 /var/lib/node_exporter/textfile_collector
sudo install -d -o nobody -g nogroup -m 0750 /etc/certtracker

# Token file, readable only by the collector user
sudo tee /etc/certtracker/token > /dev/null <<<'ct_live_xxxxxxxxxxxxxxxxxxxxxxxx'
sudo chmod 0640 /etc/certtracker/token
sudo chown root:nogroup /etc/certtracker/token

sudo tee /usr/local/bin/certtracker_collect > /dev/null <<'EOF'
#!/usr/bin/env python3
"""Poll CertTracker and write a Prometheus textfile-collector .prom file."""
import json
import os
import sys
import time
import urllib.request

BASE_URL  = os.environ.get("CT_BASE", "https://certtracker.12squared.com").rstrip("/")
TOKEN_FILE = os.environ.get("CT_TOKEN_FILE", "/etc/certtracker/token")
OUT_FILE  = os.environ.get(
    "CT_OUT_FILE",
    "/var/lib/node_exporter/textfile_collector/certtracker.prom",
)
TIMEOUT   = float(os.environ.get("CT_TIMEOUT", "15"))


def main() -> int:
    try:
        with open(TOKEN_FILE) as f:
            token = f.read().strip()
    except OSError as exc:
        print(f"token file unreadable: {exc}", file=sys.stderr)
        return 2

    req = urllib.request.Request(
        f"{BASE_URL}/api/v1/monitors",
        headers={"Authorization": f"Bearer {token}"},
    )
    try:
        with urllib.request.urlopen(req, timeout=TIMEOUT) as r:
            payload = json.loads(r.read())
    except Exception as exc:
        # Write a sentinel metric so node_exporter still scrapes something useful
        with open(OUT_FILE + ".tmp", "w") as f:
            f.write(f'# CertTracker poll failed: {exc}\n')
            f.write('certtracker_up 0\n')
        os.replace(OUT_FILE + ".tmp", OUT_FILE)
        return 1

    now = int(time.time())
    lines = [
        '# HELP certtracker_up 1 if the CertTracker API returned a successful response.',
        '# TYPE certtracker_up gauge',
        f'certtracker_up 1',
        '# HELP certtracker_monitor_count Number of monitors in the authenticated company.',
        '# TYPE certtracker_monitor_count gauge',
        f'certtracker_monitor_count {payload.get("count", 0)}',
        '# HELP certtracker_days_to_expiry Days until certificate expiration (negative = expired).',
        '# TYPE certtracker_days_to_expiry gauge',
        '# HELP certtracker_advisory_count Number of open advisories for this monitor.',
        '# TYPE certtracker_advisory_count gauge',
        '# HELP certtracker_response_ms Last-measured TLS handshake response time, ms.',
        '# TYPE certtracker_response_ms gauge',
        '# HELP certtracker_cert_status Service-check style: 0=OK, 1=WARN, 2=CRIT, 3=UNKNOWN.',
        '# TYPE certtracker_cert_status gauge',
    ]

    for m in payload.get("monitors", []):
        fqdn  = m.get("fqdn") or "unknown"
        port  = m.get("port") or 0
        label = f'fqdn="{fqdn}",port="{port}"'

        days = m.get("days_to_expiry")
        if isinstance(days, int):
            lines.append(f'certtracker_days_to_expiry{{{label}}} {days}')

        advs = m.get("advisory_count", 0) or 0
        lines.append(f'certtracker_advisory_count{{{label}}} {advs}')

        rt = m.get("response_time_ms")
        if isinstance(rt, (int, float)):
            lines.append(f'certtracker_response_ms{{{label}}} {rt}')

        # Compute service-check style status
        cert_status = (m.get("certificate_status") or "").lower()
        status_str  = (m.get("status") or "").lower()
        avail_str   = (m.get("availability_status") or "").lower()
        if cert_status == "expired" or (isinstance(days, int) and days < 0):
            code = 2
        elif isinstance(days, int) and days <= 7:
            code = 2
        elif status_str == "unreachable":
            code = 2
        elif isinstance(days, int) and days <= 30:
            code = 1
        elif avail_str == "failure":
            code = 1
        elif advs > 0:
            code = 1
        elif not m.get("last_polled"):
            code = 3
        else:
            code = 0
        lines.append(f'certtracker_cert_status{{{label}}} {code}')

    with open(OUT_FILE + ".tmp", "w") as f:
        f.write("\n".join(lines) + "\n")
    os.replace(OUT_FILE + ".tmp", OUT_FILE)
    return 0


if __name__ == "__main__":
    sys.exit(main())
EOF

sudo chmod +x /usr/local/bin/certtracker_collect

The script writes to a temp file then os.replace()s it atomically — node_exporter never sees a half-written file.

Step 3 — Run it on a schedule

Cron is fine, every 5 min matches CertTracker's poll cadence:

sudo tee /etc/cron.d/certtracker_collect > /dev/null <<'EOF'
*/5 * * * * nobody /usr/local/bin/certtracker_collect
EOF

Test once manually:

sudo -u nobody /usr/local/bin/certtracker_collect
cat /var/lib/node_exporter/textfile_collector/certtracker.prom | head -30

Should print the HELP/TYPE banners followed by gauge lines per monitor.

Step 4 — Verify node_exporter picks it up

Confirm node_exporter is running with the textfile collector enabled and pointed at the right directory:

ps -ef | grep node_exporter | grep collector.textfile
# Expect: --collector.textfile.directory=/var/lib/node_exporter/textfile_collector

If not, edit the unit (typically /etc/default/node_exporter or /etc/systemd/system/node_exporter.service) and add the flag. Reload + restart.

Then curl node_exporter's metrics endpoint and grep for certtracker_:

curl -s http://localhost:9100/metrics | grep certtracker_ | head -10

If you see real values, Prometheus is now scraping them on its normal cadence.

Step 5 — Alert rules

Drop this in your Prometheus alerts directory (/etc/prometheus/alerts/certtracker.yml or wherever your rule files live):

groups:
  - name: certtracker
    interval: 1m
    rules:
      - alert: CertTrackerAPIDown
        expr: certtracker_up == 0
        for: 15m
        labels:
          severity: warning
        annotations:
          summary: "CertTracker API is unreachable"

      - alert: CertExpired
        expr: certtracker_days_to_expiry < 0
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "Certificate expired: {{ $labels.fqdn }}:{{ $labels.port }}"
          description: "{{ $value }} days past expiry"

      - alert: CertExpiringSoon
        expr: certtracker_days_to_expiry >= 0 and certtracker_days_to_expiry <= 7
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "Cert expires in {{ $value | printf \"%.0f\" }} days: {{ $labels.fqdn }}:{{ $labels.port }}"

      - alert: CertExpiringInWindow
        expr: certtracker_days_to_expiry > 7 and certtracker_days_to_expiry <= 30
        for: 1h
        labels:
          severity: warning
        annotations:
          summary: "Cert expires in {{ $value | printf \"%.0f\" }} days: {{ $labels.fqdn }}:{{ $labels.port }}"

      - alert: OpenAdvisories
        expr: certtracker_advisory_count > 0
        for: 1h
        labels:
          severity: warning
        annotations:
          summary: "{{ $value }} open advisory(ies): {{ $labels.fqdn }}:{{ $labels.port }}"

      - alert: CertCheckCritical
        expr: certtracker_cert_status == 2
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "{{ $labels.fqdn }}:{{ $labels.port }} cert is CRITICAL"

Reload Prometheus (systemctl reload prometheus or curl the /-/reload endpoint if you have it enabled).

Step 6 — Dashboards

If you use Grafana, a useful starting query set:

Panel PromQL
Cert count by status count by (status) (certtracker_cert_status)
Top 20 closest to expiry topk(20, -certtracker_days_to_expiry)
Average response time avg(certtracker_response_ms)
Open advisories sum(certtracker_advisory_count)

Troubleshooting

Symptom Likely cause
.prom file shows certtracker_up 0 and an error comment Script ran but API failed. Check token + plan.
.prom file missing Script never ran. Check cron + /etc/certtracker/token perms (must be readable by user nobody).
node_exporter doesn't expose the metrics --collector.textfile.directory flag missing or pointing somewhere else.
Metrics work locally but not in Prometheus Prometheus scrape job for that node_exporter target may be down.

What's not in this recipe

  • A dedicated HTTP exporter that runs as its own process and exposes /metrics directly. Possible but adds a daemon to maintain. The textfile collector pattern reuses what you already have.
  • Per-monitor labels for tags — the API doesn't currently surface CertTracker tag membership. If/when it does, the script will gain a tag label per gauge.
  • Histogram of response_ms over time — gauges don't aggregate well. If you want latency distributions, write certtracker_response_ms_bucket lines using the histogram exposition format.

Contributions via the community forum welcome — particularly a Grafana dashboard JSON export.