Set up monitoring

You'll finish this page with a Prometheus instance scraping the Vigo server, the five bundled Grafana dashboards loaded, and alert rules firing on the conditions that matter: stale envoys, convergence failures, license drift, and CVE counts.

When you'd use this: any deployment where someone other than you needs to know about fleet anomalies. For a single-operator home lab, vigocli doctor + the Web UI dashboard is enough; Prometheus + Grafana starts paying for itself around five envoys or when you need history.

When you'd skip this: single-laptop development. The /metrics endpoint exists either way — you can come back when you need it.

The Vigo server exposes Prometheus metrics at GET /metrics on the REST port (default 8443).

Available Metrics

Fleet Gauges

Compliance

Check-ins

The vigo_checkin_age_seconds histogram uses custom buckets: 5s, 15s, 30s, 45s, 60s, 120s, 300s, 600s, 900s, 1800s, 3600s, 7200s. The sub-60s buckets straddle the 30s staleness floor (freshness.MinStaleThreshold) so the histogram can measure how close healthy check-ins get to the stale boundary under a fast (1s-default) cadence. This replaced per-envoy last-seen gauges to avoid cardinality explosion at scale.

gRPC

Covers all unary RPCs (CheckIn, ReportResult, ReportTraits, Register). Bidirectional streams (AgentStream) are not included — they use the check-in and delta metrics instead.

Runs

Orchestration

Webhooks

| vigo_webhook_duration_seconds | Histogram | | Webhook delivery latency including retries |

SMTP

License

FleetIndex

Server

Bootstrap

Secrets

Security

| vigo_security_cve_critical | Gauge | | Fleet-wide count of critical CVEs across all scanned hosts | | vigo_security_cve_high | Gauge | | Fleet-wide count of high CVEs across all scanned hosts | | vigo_security_cve_medium | Gauge | | Fleet-wide count of medium CVEs across all scanned hosts | | vigo_security_cve_low | Gauge | | Fleet-wide count of low CVEs across all scanned hosts | | vigo_security_hardening_avg | Gauge | | Average hardening score across all scanned hosts (0-100) | | vigo_security_rootkit_warnings | Gauge | | Fleet-wide count of rootkit warnings | | vigo_security_malware_detections | Gauge | | Fleet-wide count of malware detections | | vigo_security_integrity_issues | Gauge | | Fleet-wide count of file integrity issues | | vigo_security_scanned_hosts | Gauge | | Number of hosts with security scan data |

Convergence

Risk

Config

Swarm & Spanner

Fleet-total gauges for the swarm substrate, the content subsystems, and the spanner roster. All are fleet-wide totals with no per-entity labels. They refresh every 5 minutes (independent of metrics_interval) — the underlying aggregators rebuild from per-envoy traits, and swarm counts change slowly.

The *_divergent_* gauges are the health signal — a non-zero value means envoys disagree on content that should be identical, which warrants investigation.

AI Assistant Metrics

Defined in server/ai/metrics.go, registered separately.

Scrape Configuration

Add the Vigo server to your Prometheus scrape config. The metrics endpoint is on the REST port (8443), which is HTTPS — the job needs scheme: https and a tls_config that accepts the server's self-signed certificate. /metrics requires no authentication.

scrape_configs:
  - job_name: vigo
    scheme: https
    metrics_path: /metrics
    tls_config:
      insecure_skip_verify: true   # self-signed server cert
    static_configs:
      - targets: ['vigo-server:8443']
    scrape_interval: 30s

Grafana Dashboard Suggestions

Fleet Overview Panel

• Total Envoys: vigo_nodes_total
• Active Envoys (1h): vigo_nodes_active_total
• Compliance %: vigo_convergence_pct * 100
• Check-in Rate: rate(vigo_checkins_total[5m])

Compliance Breakdown

• Stacked gauge or pie chart using vigo_compliance_status grouped by status
• Time-series of vigo_convergence_pct for trend analysis

Run Performance

• P95 run duration: histogram_quantile(0.95, rate(vigo_run_duration_seconds_bucket[5m]))
• Failure rate: rate(vigo_checkins_total{status="failure"}[5m])
• Drift rate: rate(vigo_drift_detected_total[5m])

gRPC Latency

• P99 check-in latency: histogram_quantile(0.99, rate(vigo_grpc_request_duration_seconds_bucket{method="/vigo.VigoAgent/CheckIn"}[5m]))
• Error rate by method: rate(vigo_grpc_requests_total{code!="OK"}[5m])

Orchestration

• Task throughput: rate(vigo_task_runs_total[5m])
• Task failure rate: rate(vigo_task_runs_total{status="failed"}[5m])
• Workflow success rate: rate(vigo_workflow_runs_total{status="complete"}[5m]) / rate(vigo_workflow_runs_total[5m])

Webhooks

• Delivery success rate: rate(vigo_webhook_deliveries_total{status="success"}[5m]) / rate(vigo_webhook_deliveries_total[5m])
• P95 delivery latency: histogram_quantile(0.95, rate(vigo_webhook_duration_seconds_bucket[5m]))
• Alert on rate(vigo_webhook_deliveries_total{status="failed"}[5m]) > 0

License

• Envoy utilization: vigo_license_envoys_used / vigo_license_envoys_max * 100
• Days until expiry: vigo_license_days_remaining
• Expiry date: timestamp(vigo_license_expiry_timestamp) for Grafana absolute time display
• Alert on vigo_license_stage > 0 (any violation)
• Alert on vigo_license_envoys_used / vigo_license_envoys_max >= 0.9 (approaching limit — hard gate will start rejecting at 1.0)
• Alert on vigo_license_envoys_used >= vigo_license_envoys_max (at capacity — next enrollment will be rejected)
• Alert on increase(vigo_license_enrollments_blocked_total[1h]) > 0 (any hard-gate rejection — operator needs to act)
• Alert on increase(vigo_license_enforcement_errors_total[5m]) > 0 (internal enforcement errors — investigate, gate is failing closed)
• Alert on vigo_license_days_remaining < 30 (license expiring soon)

Server Health

• Uptime: vigo_uptime_seconds (detects restarts in Grafana)
• Version: vigo_build_info label version (track rollout across instances)
• Active streams: vigo_streams_active (monitor agent connectivity)
• Alert on vigo_uptime_seconds < 300 for recent restart detection
• Alert on vigo_streams_active == 0 when vigo_nodes_active_total > 0 (agents not connecting via stream)

Security Alerts

• Alert on rate(vigo_sig_verify_failed_total[5m]) > 0 to detect signature verification failures
• Alert on vigo_config_reload_errors_total increases
• Alert on rate(vigo_secret_resolution_errors_total[5m]) > 0

Check-in Staleness

• Median check-in age: histogram_quantile(0.5, rate(vigo_checkin_age_seconds_bucket[5m]))
• Alert when histogram_quantile(0.95, ...) exceeds your expected check-in interval

Metrics Refresh Interval

Gauge metrics (node counts, compliance, database size) are refreshed on a configurable interval:

tuning:
  metrics_interval: "30s"    # default

See Tuning Parameters for all tuning options.

Grafana dashboards

Pre-built Grafana dashboards for monitoring Vigo via Prometheus.

Download

Install via Vigo configcrates (recommended)

For any host that runs Grafana under Vigo management, drop the vigo-grafana-dashboards configcrate into the role and converge:

# stacks/scaffolding/roles.vgo
- name: grafana-host
  configcrates:
    - grafana                  # installs the daemon
    - vigo-grafana-dashboards  # drops all six dashboards into provisioning

The configcrate sources the dashboard JSONs from stacks/templates/vigo-dashboards/ and writes them to /etc/grafana/provisioning/dashboards/vigo/. Grafana picks them up on its next provisioning scan (default 30 s) — no UI import, no service restart.

The companion vigo-prometheus-alerts configcrate ships alerts.yaml to /etc/prometheus/rules/vigo-alerts.yaml and notifies the Prometheus service to reload. Pair it with the standard prometheus configcrate — which already mounts rule_files: /etc/prometheus/rules/*.yml.

Manual import

For hosts where Vigo isn't managing the monitoring stack, import each dashboard by hand:

1. Download the JSON file using the link in the table above
2. Open Grafana (typically http://localhost:3000)
3. Go to Dashboards → Import
4. Upload the JSON file or paste its contents
5. Select your Prometheus datasource when prompted

Fleet Overview Dashboard

Download vigo-overview.json

Row 1 — Fleet Overview

Row 2 — Compliance Breakdown

Row 3 — Check-in & gRPC

Row 4 — Convergence & Drift

| Drift Corrections / min | Time series | rate(vigo_drift_detected_total[5m]) * 60 | | Configcrate Actions | Time series by action | rate(vigo_configcrate_actions_total[5m]) * 60 |

Row 5 — Operations

Row 6 — Infrastructure

Row 7 — Webhooks & SMTP

Row 8 — Secrets

Row 9 — AI Assistant

Row 10 — Security Posture

Row 11 — Risk Posture

| Low Risk | Stat (green) | vigo_risk_distribution{level="low"} | | Medium Risk | Stat (blue) | vigo_risk_distribution{level="medium"} | | High Risk | Stat (orange) | vigo_risk_distribution{level="high"} | | Critical Risk | Stat (red) | vigo_risk_distribution{level="critical"} |

Security Posture Dashboard

Download vigo-security.json

Dedicated security monitoring dashboard with CVE tracking, hardening scores, and threat detection.

Row 1 — CVE Overview

Row 2 — Hardening

Row 3 — Threat Detection

Row 4 — Coverage

Convergence Dashboard

Download vigo-convergence.json

Per-envoy convergence status breakdown, compliance trends, and check-in health.

Row 1 — Status Breakdown

Row 2 — Trends

Row 3 — Check-in Health

Risk Posture Dashboard

Download vigo-risk.json

Fleet-wide risk scoring with distribution breakdown and trend tracking.

Row 1 — Fleet Risk

Row 2 — Distribution

Row 3 — Trends

Publish Bursts Dashboard

Download vigo-bursts.json

Watch a config publish ripple through the fleet end-to-end. Every panel reads from a metric already exported by vigosrv (see server/metrics/registry.go); the dashboard adds vertical annotations at each vigo_config_publishes_total increment so every burst-shaped change has a visible cause.

Row 1 — Burst at a Glance

Row 2 — gRPC Behaviour

Row 3 — Policy Delivery

Row 4 — Convergence & Auth

The time-to-converge panel measures the wall-clock seconds from a config version becoming current to each envoy first applying it — one sample per envoy per forward version adoption (routine re-converges are excluded). The tail tracks your check-in cadence: a long-interval fleet converges slower because envoys only notice the change at their next check-in. Samples come from the streaming check-in path (the default daemon mode); one-shot vigo run invocations are not counted, since they are operator-initiated and not part of a publish wave, and neither are envoys whose persistent state store failed to open (offline convergence disabled — a degraded state that logs a warning).

Label cardinality. Every panel here aggregates over method / code / status / scope — fixed-cardinality labels. Per-envoy series are never introduced; at fleet scale (25K+ envoys) per-envoy labels would blow Prometheus's series budget.

Prometheus Configuration

Add the Vigo server as a scrape target in prometheus.yml. :8443 is HTTPS, so the job needs scheme: https plus a tls_config for the self-signed certificate:

scrape_configs:
  - job_name: vigo
    scheme: https
    metrics_path: /metrics
    tls_config:
      insecure_skip_verify: true   # self-signed server cert
    static_configs:
      - targets: ['vigo-server:8443']
    scrape_interval: 30s

Alert rules

A curated alert-rules file ships alongside the Grafana dashboards:

[grafana/alerts.yaml](../grafana/alerts.yaml)

Nineteen Prometheus rules across convergence (failed / degraded / diverged / offline), runs (p95 duration), security (sig-verify failures, critical CVEs), operability (config reload errors, secret resolution errors), license (enforcement errors, days remaining, stage), swarm (spanner bolt roster), and bursts (sustained gRPC rate, TLS handshake spike, sig-verify failures, in-flight CheckIn high-water, publish→converge p95 over SLO). Every rule references a metric defined in server/metrics/registry.go; thresholds are starting points — tune to your fleet.

Load into a stock Prometheus:

# prometheus.yml
rule_files:
  - "/etc/prometheus/vigo-alerts.yaml"

Or import into Grafana via Alerting → Alert rules → Import. Route alerts to PagerDuty / Slack / ServiceNow / Loki via the integrations section above.

What's next

• A metric is high and you don't know which envoy → drill in via vigocli envoys list then the envoy detail page.
• You need an alert routed to PagerDuty / Slack / ServiceNow → see the integrations section above; same config block, different endpoint.
• You want events as queryable logs alongside the metrics → Ship events to Grafana Loki.
• A metric value looks wrong → Troubleshoot common issues.

Verified on Vigo 0.51.6 · 2026-05-13.