Wiki / Integrations

Datadog Integration

Last updated

Datadog Integration

Surface CertTracker monitor health in Datadog as both service checks (OK/WARN/CRIT, drives alerts) and metrics (days-to-expiry, advisory count, response time — driveable as dashboards and SLOs). Drop a small Python file in the Datadog Agent's checks.d/ directory, configure once, done.

Community-validated recipe. CertTracker doesn't run a Datadog Agent in-house — this guide reflects the standard custom-check pattern from Datadog's docs and may need version-specific tweaks. Corrections welcome via the community forum.

Plan required. API access is a Business and Enterprise plan feature. See API access for token creation.

Architecture

+-------------------+         +-----------------------+
| Datadog Agent     |         |  CertTracker          |
|                   |  HTTPS  |                       |
|  certtracker.py   ----------->|  /api/v1/monitors   |
|  (custom check)   |         |                       |
|       |           |         +-----------------------+
|       v           |
| service_check()   --→ Datadog Monitors → Alerts
| gauge()           --→ Datadog Metrics  → Dashboards
+-------------------+

Step 1 — Mint a CertTracker API token

Standard flow from API access: Administration → API Tokens → Create. Name it datadog-prod. Copy the value.

Step 2 — Drop the check on your Datadog Agent host

Place this file at /etc/datadog-agent/checks.d/certtracker.py:

# certtracker.py — Datadog custom check
#
# Polls CertTracker's /api/v1/monitors endpoint and emits:
#   * one service_check per monitor (OK/WARN/CRIT)
#   * gauge metrics for days_to_expiry, advisory_count, response_ms
#
# Configure in /etc/datadog-agent/conf.d/certtracker.yaml.

from datadog_checks.base import AgentCheck
import requests


class CertTrackerCheck(AgentCheck):
    SERVICE_CHECK_NAME = "certtracker.cert"

    def check(self, instance):
        base_url = instance.get("base_url", "https://certtracker.12squared.com").rstrip("/")
        token    = instance["token"]
        timeout  = float(instance.get("timeout", 15))

        try:
            r = requests.get(
                f"{base_url}/api/v1/monitors",
                headers={"Authorization": f"Bearer {token}"},
                timeout=timeout,
            )
        except requests.RequestException as exc:
            self.service_check(
                "certtracker.api", AgentCheck.CRITICAL,
                message=f"CertTracker API unreachable: {exc}",
            )
            return

        if r.status_code != 200:
            self.service_check(
                "certtracker.api", AgentCheck.CRITICAL,
                message=f"CertTracker API HTTP {r.status_code}: {r.text[:200]}",
            )
            return

        self.service_check("certtracker.api", AgentCheck.OK)
        payload = r.json()

        for m in payload.get("monitors", []):
            fqdn  = m.get("fqdn", "unknown")
            port  = m.get("port", 0)
            label = f"{fqdn}:{port}"
            tags  = [
                f"cert_fqdn:{fqdn}",
                f"cert_port:{port}",
                f"cert_label:{label}",
            ]

            # Severity from CertTracker's own classification
            days = m.get("days_to_expiry")
            advs = m.get("advisory_count", 0)
            status_str = (m.get("status") or "").lower()
            avail = (m.get("availability_status") or "").lower()
            cert_status = (m.get("certificate_status") or "").lower()

            if cert_status == "expired" or (isinstance(days, int) and days < 0):
                status = AgentCheck.CRITICAL
                msg = f"Certificate has expired ({days}d past)"
            elif isinstance(days, int) and days <= 7:
                status = AgentCheck.CRITICAL
                msg = f"Certificate expires in {days}d"
            elif status_str == "unreachable":
                status = AgentCheck.CRITICAL
                msg = "Endpoint unreachable"
            elif isinstance(days, int) and days <= 30:
                status = AgentCheck.WARNING
                msg = f"Certificate expires in {days}d"
            elif avail == "failure":
                status = AgentCheck.WARNING
                msg = "TLS handshake failing"
            elif advs > 0:
                status = AgentCheck.WARNING
                msg = f"{advs} open advisory" + ("ies" if advs != 1 else "")
            else:
                status = AgentCheck.OK
                msg = f"Certificate valid for {days}d" if isinstance(days, int) else "OK"

            self.service_check(self.SERVICE_CHECK_NAME, status, tags=tags, message=msg)

            # Metrics
            if isinstance(days, int):
                self.gauge("certtracker.days_to_expiry", days, tags=tags)
            self.gauge("certtracker.advisory_count", advs, tags=tags)
            rt = m.get("response_time_ms")
            if isinstance(rt, (int, float)):
                self.gauge("certtracker.response_ms", rt, tags=tags)

Make it readable by the Datadog user:

sudo chown dd-agent:dd-agent /etc/datadog-agent/checks.d/certtracker.py
sudo chmod 644 /etc/datadog-agent/checks.d/certtracker.py

Step 3 — Configure the check

Create /etc/datadog-agent/conf.d/certtracker.yaml:

init_config:

instances:
  - base_url: https://certtracker.12squared.com
    token: ct_live_xxxxxxxxxxxxxxxxxxxxxxxx
    timeout: 15
    min_collection_interval: 300   # 5 minutes; CertTracker polls slower than this
    tags:
      - source:certtracker
      - env:prod

min_collection_interval: 300 matches CertTracker's typical poll cadence — running the check more often just wastes API calls without producing new data.

Tighten the file's permissions since it contains the token:

sudo chown dd-agent:dd-agent /etc/datadog-agent/conf.d/certtracker.yaml
sudo chmod 640 /etc/datadog-agent/conf.d/certtracker.yaml

Step 4 — Restart the Agent

sudo systemctl restart datadog-agent

# Verify the check loaded and ran
sudo -u dd-agent datadog-agent check certtracker | head -50

Expected output includes a Service Checks block with one entry per monitor and Metrics blocks for each gauge.

Step 5 — Configure Datadog Monitors (Alerts)

In the Datadog UI: Monitors → New Monitor → Service Check.

Field Value
Service Check certtracker.cert
Scope cert_label:* (or filter by cert_fqdn, env)
Status Alert on CRITICAL, optionally on WARNING
Message Cert posture changed: {{value}}. See https://certtracker.12squared.com/monitors
Notify your existing PagerDuty / Slack / email destinations

Repeat for certtracker.api if you want notification when the API itself is unreachable (token revoked, plan downgraded, etc.).

For metric-based monitors:

Metric Suggested alert
certtracker.days_to_expiry value:<=7 → alert
certtracker.advisory_count value:>0 → warning
certtracker.response_ms value:>1000 → warning, only if you care about CertTracker latency

Step 6 — Build dashboards (optional)

A useful starting set of widgets:

  • Heatmap of certtracker.days_to_expiry grouped by cert_label — instantly see which certs are closest to expiry.
  • Top list of certtracker.advisory_count desc, top 20 — your noisiest monitors.
  • Service check summary for certtracker.cert — counts of OK/WARN/CRIT/UNKNOWN.

Troubleshooting

Symptom Likely cause
Check status: Initial check did not return any data YAML config not found or syntax error; check datadog-agent status
certtracker.api is CRITICAL with HTTP 402 Plan isn't Business/Enterprise. Upgrade in Administration → Billing in CertTracker.
certtracker.api is CRITICAL with HTTP 401 Token revoked. Mint a new one and update the YAML.
Metrics don't appear in dashboards Service checks and gauges have different retention paths; wait 5 min after first successful check.
Hostname tagging issues The check doesn't auto-correlate cert_fqdn with Datadog's host tag. If you want that linkage, add hostname: <fqdn> in the per-monitor metric calls.

What's not in this recipe

  • Auto-discovery via Datadog AD for K8s-resident CertTracker deployments — not applicable; CertTracker is SaaS.
  • Datadog integrations marketplace listing — would require Datadog Partner Network onboarding on our side. Not done; on the long-term backlog.
  • APM trace correlation — the check doesn't emit traces. The metrics + service checks are the right shape for posture monitoring; APM is overkill here.

Contributions via the community forum welcome — particularly a monitors.json export for the suggested Datadog Monitors.