Prometheus Metrics
Monitor Wiretastix VPN performance and usage with Prometheus metrics and Grafana dashboards.
Wiretastix exposes detailed metrics about VPN peer connections, traffic usage, and uptime through a Prometheus-compatible endpoint. This guide covers metric configuration, Prometheus setup, Grafana dashboards, and alerting rules.
Overview
Available Metrics
Wiretastix currently exposes peer-level metrics:
| Metric | Type | Labels | Description |
|---|---|---|---|
wiretastix_peer_tx | Gauge | pk, id, user | Transmitted bytes per peer |
wiretastix_peer_rx | Gauge | pk, id, user | Received bytes per peer |
wiretastix_peer_uptime | Gauge | pk, id, user | Peer uptime/last handshake timestamp |
Labels:
pk- Peer public key (WireGuard)id- Peer ID (Wiretastix database ID)user- Username/nickname (from OIDC, “unset” if not authenticated)
Update Frequency
Metrics are updated every 10 seconds by default. This provides near-real-time visibility into VPN usage.
Why These Metrics Matter
Traffic monitoring:
- Identify bandwidth-heavy users
- Detect unusual traffic patterns
- Plan capacity and infrastructure
Connection monitoring:
- Alert on disconnected peers
- Track connection quality (handshake freshness)
- Monitor user activity patterns
Usage reporting:
- Generate usage reports per user
- Analyze peak usage times
- Audit VPN access
Configuration
Enable Metrics Endpoint
Metrics have their own dedicated metrics config section – a separate HTTP server from the API server (apiserver), with its own binding. Configure in wiretastix.yaml:
metrics:
path: /metrics
# Bind to specific interface and port
binding: 0.0.0.0:9100
# Or bind to all interfaces on different port
# binding: :9100
If left unset, binding defaults to 127.0.0.1:9100 and path to /metrics.
This endpoint has no authentication of its own. It isn’t secret-leaking, but it does reveal internal network topology – peer counts, WireGuard public keys, nicknames, and traffic – so if you bind it to anything other than a loopback/internal address, put it behind a reverse proxy that enforces access control (IP allow-list, auth) rather than exposing it directly.
Security considerations:
# Option 1: Bind to localhost only (access via SSH tunnel)
metrics:
binding: 127.0.0.1:9100
# Option 2: Bind to internal interface only
metrics:
binding: 10.0.1.10:9100
# Option 3: Bind to all interfaces (use firewall/reverse-proxy ACLs to restrict)
metrics:
binding: 0.0.0.0:9100
Firewall Configuration
If exposing metrics on a network interface, protect with firewall rules:
Using UFW
# Allow from Prometheus server only
sudo ufw allow from 10.0.1.50 to any port 9100 proto tcp
# Or allow from monitoring network
sudo ufw allow from 10.0.1.0/24 to any port 9100 proto tcp
Using nftables
Add to your /etc/nftables.conf:
define prometheus_server = 10.0.1.50
define wiretastix_metrics_if = ens192 # Internal interface
chain input {
type filter hook input priority 0; policy drop;
# ... other rules ...
# Allow Prometheus metrics from specific server
iifname $wiretastix_metrics_if ip saddr $prometheus_server tcp dport 9100 accept
# Or from monitoring network
# iifname $wiretastix_metrics_if ip saddr 10.0.1.0/24 tcp dport 9100 accept
}
Apply:
sudo nft -f /etc/nftables.conf
Test Metrics Endpoint
# Local test
curl http://localhost:9100/metrics
# Remote test
curl http://vpn.example.com:9100/metrics
Expected output:
# HELP wiretastix_peer_rx RX traffic by peer
# TYPE wiretastix_peer_rx gauge
wiretastix_peer_rx{id="1",pk="ABC123...=",user="john.doe"} 1048576000
wiretastix_peer_rx{id="2",pk="DEF456...=",user="jane.smith"} 524288000
# HELP wiretastix_peer_tx TX traffic by peer
# TYPE wiretastix_peer_tx gauge
wiretastix_peer_tx{id="1",pk="ABC123...=",user="john.doe"} 2097152000
wiretastix_peer_tx{id="2",pk="DEF456...=",user="jane.smith"} 1048576000
# HELP wiretastix_peer_uptime Peer Uptime
# TYPE wiretastix_peer_uptime gauge
wiretastix_peer_uptime{id="1",pk="ABC123...=",user="john.doe"} 1697472000
wiretastix_peer_uptime{id="2",pk="DEF456...=",user="jane.smith"} 1697475600
Prometheus Setup
Install Prometheus
Debian/Ubuntu
# Add Prometheus user
sudo useradd --no-create-home --shell /bin/false prometheus
# Download Prometheus
wget https://github.com/prometheus/prometheus/releases/download/v2.45.0/prometheus-2.45.0.linux-amd64.tar.gz
tar xvf prometheus-2.45.0.linux-amd64.tar.gz
# Install binaries
sudo cp prometheus-2.45.0.linux-amd64/prometheus /usr/local/bin/
sudo cp prometheus-2.45.0.linux-amd64/promtool /usr/local/bin/
# Set ownership
sudo chown prometheus:prometheus /usr/local/bin/prometheus
sudo chown prometheus:prometheus /usr/local/bin/promtool
# Create directories
sudo mkdir /etc/prometheus
sudo mkdir /var/lib/prometheus
sudo chown prometheus:prometheus /etc/prometheus
sudo chown prometheus:prometheus /var/lib/prometheus
Docker
# Create prometheus.yml (see configuration below)
# Then run:
docker run -d \
--name prometheus \
-p 9090:9090 \
-v /path/to/prometheus.yml:/etc/prometheus/prometheus.yml \
-v prometheus-data:/prometheus \
prom/prometheus
Configure Prometheus
Create /etc/prometheus/prometheus.yml:
global:
scrape_interval: 15s
evaluation_interval: 15s
# Alerting configuration (optional, see Alerting section)
alerting:
alertmanagers:
- static_configs:
- targets:
- localhost:9093
# Load alert rules
rule_files:
- "wiretastix_alerts.yml"
# Scrape Wiretastix metrics
scrape_configs:
- job_name: 'wiretastix'
static_configs:
- targets: ['vpn.example.com:9100']
labels:
instance: 'wiretastix-vpn-prod'
environment: 'production'
# Optional: Increase timeout for slow networks
scrape_timeout: 10s
# Optional: Basic auth if you've added it
# basic_auth:
# username: 'prometheus'
# password: 'secret'
# Add more Wiretastix instances
- job_name: 'wiretastix-staging'
static_configs:
- targets: ['vpn-staging.example.com:9100']
labels:
instance: 'wiretastix-vpn-staging'
environment: 'staging'
Start Prometheus
Systemd Service
Create /etc/systemd/system/prometheus.service:
[Unit]
Description=Prometheus
Wants=network-online.target
After=network-online.target
[Service]
User=prometheus
Group=prometheus
Type=simple
ExecStart=/usr/local/bin/prometheus \
--config.file=/etc/prometheus/prometheus.yml \
--storage.tsdb.path=/var/lib/prometheus/ \
--web.console.templates=/etc/prometheus/consoles \
--web.console.libraries=/etc/prometheus/console_libraries
[Install]
WantedBy=multi-user.target
Enable and start:
sudo systemctl daemon-reload
sudo systemctl enable prometheus
sudo systemctl start prometheus
# Check status
sudo systemctl status prometheus
# View logs
sudo journalctl -u prometheus -f
Docker Compose
version: '3.8'
services:
prometheus:
image: prom/prometheus:latest
container_name: prometheus
ports:
- "9090:9090"
volumes:
- ./prometheus.yml:/etc/prometheus/prometheus.yml
- ./wiretastix_alerts.yml:/etc/prometheus/wiretastix_alerts.yml
- prometheus-data:/prometheus
command:
- '--config.file=/etc/prometheus/prometheus.yml'
- '--storage.tsdb.path=/prometheus'
- '--web.console.templates=/etc/prometheus/consoles'
- '--web.console.libraries=/etc/prometheus/console_libraries'
restart: unless-stopped
volumes:
prometheus-data:
Start:
docker-compose up -d
Verify Prometheus
- Access Prometheus UI: http://prometheus-server:9090
- Check targets: Status → Targets
- Verify Wiretastix target is UP
- Run test query:
wiretastix_peer_tx
Grafana Dashboard
Install Grafana
Debian/Ubuntu
# Add Grafana GPG key
sudo apt-get install -y software-properties-common
sudo add-apt-repository "deb https://packages.grafana.com/oss/deb stable main"
wget -q -O - https://packages.grafana.com/gpg.key | sudo apt-key add -
# Install
sudo apt-get update
sudo apt-get install grafana
# Start service
sudo systemctl enable grafana-server
sudo systemctl start grafana-server
Docker
docker run -d \
--name grafana \
-p 3000:3000 \
-v grafana-data:/var/lib/grafana \
grafana/grafana
Docker Compose (add to prometheus compose)
grafana:
image: grafana/grafana:latest
container_name: grafana
ports:
- "3000:3000"
volumes:
- grafana-data:/var/lib/grafana
environment:
- GF_SECURITY_ADMIN_PASSWORD=admin
- GF_USERS_ALLOW_SIGN_UP=false
restart: unless-stopped
Configure Data Source
- Access Grafana: http://grafana-server:3000 (default: admin/admin)
- Add data source:
- Configuration → Data Sources → Add data source
- Select “Prometheus”
- URL:
http://prometheus-server:9090(orhttp://prometheus:9090in Docker) - Save & Test
Import Wiretastix Dashboard
Option 1: Manual Creation
Create a new dashboard with these panels:
Panel 1: Total Traffic (Bytes)
# Query A: Total TX
sum(wiretastix_peer_tx)
# Query B: Total RX
sum(wiretastix_peer_rx)
- Visualization: Time series
- Legend:
{{__name__}}
Panel 2: Traffic per User
# TX per user
sum by (user) (rate(wiretastix_peer_tx[5m]))
# RX per user
sum by (user) (rate(wiretastix_peer_rx[5m]))
- Visualization: Time series
- Legend:
{{user}}
Panel 3: Active Peers
# Peers with handshake in last 3 minutes
count(wiretastix_peer_uptime > (time() - 180))
- Visualization: Stat
- Unit: Short
Panel 4: Peer Connection Table
# Last handshake per peer
wiretastix_peer_uptime
- Visualization: Table
- Transform: “Organize fields” to show user, id, last handshake
- Value mapping: Convert timestamp to human-readable
Panel 5: Top 10 Users by Traffic
# Top TX users
topk(10, sum by (user) (rate(wiretastix_peer_tx[1h])))
- Visualization: Bar chart
- Legend:
{{user}}
Option 2: JSON Dashboard
Create /tmp/wiretastix-dashboard.json:
{
"dashboard": {
"title": "Wiretastix VPN Monitoring",
"tags": ["wiretastix", "vpn"],
"timezone": "browser",
"panels": [
{
"id": 1,
"title": "Active Peers",
"type": "stat",
"targets": [
{
"expr": "count(wiretastix_peer_uptime > (time() - 180))",
"refId": "A"
}
],
"gridPos": {"h": 4, "w": 6, "x": 0, "y": 0}
},
{
"id": 2,
"title": "Total Traffic (Last Hour)",
"type": "timeseries",
"targets": [
{
"expr": "sum(rate(wiretastix_peer_tx[5m]))",
"legendFormat": "TX",
"refId": "A"
},
{
"expr": "sum(rate(wiretastix_peer_rx[5m]))",
"legendFormat": "RX",
"refId": "B"
}
],
"gridPos": {"h": 8, "w": 12, "x": 0, "y": 4}
},
{
"id": 3,
"title": "Traffic per User",
"type": "timeseries",
"targets": [
{
"expr": "sum by (user) (rate(wiretastix_peer_tx[5m]))",
"legendFormat": "{{user}} TX",
"refId": "A"
},
{
"expr": "sum by (user) (rate(wiretastix_peer_rx[5m]))",
"legendFormat": "{{user}} RX",
"refId": "B"
}
],
"gridPos": {"h": 8, "w": 12, "x": 12, "y": 4}
},
{
"id": 4,
"title": "Peer Connections",
"type": "table",
"targets": [
{
"expr": "wiretastix_peer_uptime",
"format": "table",
"instant": true,
"refId": "A"
}
],
"gridPos": {"h": 8, "w": 24, "x": 0, "y": 12}
}
],
"refresh": "30s",
"time": {
"from": "now-6h",
"to": "now"
}
}
}
Import via Grafana UI: Dashboards → Import → Upload JSON file
Useful Queries
Connection Monitoring
Active Peers
# Count peers with recent handshake (last 3 minutes)
count(wiretastix_peer_uptime > (time() - 180))
Stale Connections
# Peers without handshake in last 5 minutes
wiretastix_peer_uptime < (time() - 300)
Connection Duration
# How long each peer has been connected (seconds)
time() - wiretastix_peer_uptime
Traffic Analysis
Total Bandwidth (Bytes/Second)
# TX rate
rate(wiretastix_peer_tx[5m])
# RX rate
rate(wiretastix_peer_rx[5m])
# Total rate (TX + RX)
rate(wiretastix_peer_tx[5m]) + rate(wiretastix_peer_rx[5m])
Traffic per User
# Sum all peers per user
sum by (user) (rate(wiretastix_peer_tx[5m]))
sum by (user) (rate(wiretastix_peer_rx[5m]))
Top 10 Users by Traffic
# Top TX users
topk(10, sum by (user) (rate(wiretastix_peer_tx[1h])))
# Top RX users
topk(10, sum by (user) (rate(wiretastix_peer_rx[1h])))
# Top total traffic users
topk(10, sum by (user) (rate(wiretastix_peer_tx[1h]) + rate(wiretastix_peer_rx[1h])))
Total Data Transferred
# Total bytes TX (last 24 hours)
increase(wiretastix_peer_tx[24h])
# Total bytes RX (last 24 hours)
increase(wiretastix_peer_rx[24h])
# Per user (last 24 hours)
sum by (user) (increase(wiretastix_peer_tx[24h]))
Peak Usage
Peak Bandwidth
# Peak TX rate in last hour
max_over_time(sum(rate(wiretastix_peer_tx[5m]))[1h:5m])
# Peak total bandwidth
max_over_time(sum(rate(wiretastix_peer_tx[5m]) + rate(wiretastix_peer_rx[5m]))[1h:5m])
Users Online Over Time
# Count of active users (5m windows)
count(count by (user) (wiretastix_peer_uptime > (time() - 300)))
Anomaly Detection
Unusual Traffic Spikes
# TX rate more than 2x average
rate(wiretastix_peer_tx[5m]) > 2 * avg_over_time(rate(wiretastix_peer_tx[5m])[1h:5m])
New Connections
# Peers connected in last 10 minutes
wiretastix_peer_uptime > (time() - 600)
Alerting
Alert Rules
Create /etc/prometheus/wiretastix_alerts.yml:
groups:
- name: wiretastix_alerts
interval: 30s
rules:
# Alert when peer disconnects
- alert: WiretastixPeerDisconnected
expr: time() - wiretastix_peer_uptime > 300
for: 5m
labels:
severity: warning
annotations:
summary: "Wiretastix peer {{ $labels.user }} (ID: {{ $labels.id }}) disconnected"
description: "Peer {{ $labels.user }} has not had a handshake in over 5 minutes."
# Alert on high bandwidth usage
- alert: WiretastixHighBandwidth
expr: rate(wiretastix_peer_tx[5m]) + rate(wiretastix_peer_rx[5m]) > 100000000 # 100 MB/s
for: 10m
labels:
severity: warning
annotations:
summary: "High bandwidth usage by {{ $labels.user }}"
description: "Peer {{ $labels.user }} (ID: {{ $labels.id }}) is using over 100 MB/s."
# Alert when total traffic exceeds threshold
- alert: WiretastixHighTotalTraffic
expr: sum(rate(wiretastix_peer_tx[5m]) + rate(wiretastix_peer_rx[5m])) > 500000000 # 500 MB/s
for: 10m
labels:
severity: critical
annotations:
summary: "Wiretastix total traffic exceeds 500 MB/s"
description: "Total VPN traffic is {{ $value | humanize }}B/s."
# Alert on no active peers (VPN down?)
- alert: WiretastixNoPeers
expr: count(wiretastix_peer_uptime > (time() - 180)) == 0
for: 10m
labels:
severity: critical
annotations:
summary: "No active Wiretastix peers"
description: "No peers have connected in the last 10 minutes. VPN may be down."
# Alert on stale metrics (scrape failing)
- alert: WiretastixMetricsStale
expr: absent(up{job="wiretastix"}) or up{job="wiretastix"} == 0
for: 5m
labels:
severity: critical
annotations:
summary: "Wiretastix metrics unavailable"
description: "Prometheus cannot scrape Wiretastix metrics."
# Alert when user exceeds daily quota (example: 10 GB)
- alert: WiretastixUserQuotaExceeded
expr: sum by (user) (increase(wiretastix_peer_tx[24h]) + increase(wiretastix_peer_rx[24h])) > 10737418240 # 10 GB
labels:
severity: info
annotations:
summary: "User {{ $labels.user }} exceeded daily quota"
description: "User {{ $labels.user }} has transferred {{ $value | humanize1024 }}B in the last 24 hours."
# Alert on unusual connection patterns (peer connects/disconnects rapidly)
- alert: WiretastixFlappingPeer
expr: changes(wiretastix_peer_uptime[30m]) > 10
labels:
severity: warning
annotations:
summary: "Peer {{ $labels.user }} is flapping"
description: "Peer {{ $labels.user }} (ID: {{ $labels.id }}) has connected/disconnected {{ $value }} times in 30 minutes."
Load in Prometheus:
# Restart Prometheus to load rules
sudo systemctl restart prometheus
# Or reload configuration
curl -X POST http://localhost:9090/-/reload
Alertmanager Integration
For sending alerts via email, Slack, PagerDuty, etc., configure Alertmanager.
Install Alertmanager
# Download
wget https://github.com/prometheus/alertmanager/releases/download/v0.26.0/alertmanager-0.26.0.linux-amd64.tar.gz
tar xvf alertmanager-0.26.0.linux-amd64.tar.gz
# Install
sudo cp alertmanager-0.26.0.linux-amd64/alertmanager /usr/local/bin/
sudo mkdir /etc/alertmanager
Configure Alertmanager
Create /etc/alertmanager/alertmanager.yml:
global:
resolve_timeout: 5m
smtp_smarthost: 'smtp.example.com:587'
smtp_from: 'alerts@example.com'
smtp_auth_username: 'alerts@example.com'
smtp_auth_password: 'password'
route:
group_by: ['alertname', 'instance']
group_wait: 10s
group_interval: 10s
repeat_interval: 12h
receiver: 'email'
receivers:
- name: 'email'
email_configs:
- to: 'admin@example.com'
headers:
Subject: '[Wiretastix Alert] {{ .GroupLabels.alertname }}'
# Slack integration
- name: 'slack'
slack_configs:
- api_url: 'https://hooks.slack.com/services/YOUR/WEBHOOK/URL'
channel: '#monitoring'
text: "{{ range .Alerts }}{{ .Annotations.summary }}\n{{ .Annotations.description }}\n{{ end }}"
Start Alertmanager
# Systemd service
sudo systemctl start alertmanager
# Or manually
/usr/local/bin/alertmanager --config.file=/etc/alertmanager/alertmanager.yml
Monitoring Best Practices
1. Set Appropriate Retention
Configure Prometheus retention based on your needs:
# In prometheus.service
ExecStart=/usr/local/bin/prometheus \
--config.file=/etc/prometheus/prometheus.yml \
--storage.tsdb.path=/var/lib/prometheus/ \
--storage.tsdb.retention.time=90d \ # Keep 90 days
--storage.tsdb.retention.size=50GB # Or limit by size
2. Use Recording Rules
For frequently-used queries, create recording rules to pre-compute:
# Add to prometheus.yml or separate file
groups:
- name: wiretastix_recordings
interval: 60s
rules:
# Pre-compute total bandwidth
- record: wiretastix:total_bandwidth:rate5m
expr: sum(rate(wiretastix_peer_tx[5m]) + rate(wiretastix_peer_rx[5m]))
# Pre-compute per-user traffic
- record: wiretastix:user_traffic:rate5m
expr: sum by (user) (rate(wiretastix_peer_tx[5m]) + rate(wiretastix_peer_rx[5m]))
# Active peer count
- record: wiretastix:active_peers:count
expr: count(wiretastix_peer_uptime > (time() - 180))
Use in queries:
# Instead of: sum(rate(wiretastix_peer_tx[5m]) + rate(wiretastix_peer_rx[5m]))
# Use:
wiretastix:total_bandwidth:rate5m
3. Create Usage Reports
Export data periodically for reporting:
#!/bin/bash
# /usr/local/bin/wiretastix-usage-report.sh
DATE=$(date +%Y-%m-%d)
PROMETHEUS_URL="http://localhost:9090"
# Query total traffic per user (last 24 hours)
curl -s -G "${PROMETHEUS_URL}/api/v1/query" \
--data-urlencode 'query=sum by (user) (increase(wiretastix_peer_tx[24h]) + increase(wiretastix_peer_rx[24h]))' \
| jq -r '.data.result[] | "\(.metric.user),\(.value[1])"' \
> /var/log/wiretastix/usage-report-${DATE}.csv
# Email report
mail -s "Wiretastix Daily Usage Report" admin@example.com < /var/log/wiretastix/usage-report-${DATE}.csv
Schedule daily:
# Cron job
0 2 * * * /usr/local/bin/wiretastix-usage-report.sh
4. Monitor Metrics Endpoint Health
# Check metrics endpoint
curl -s -o /dev/null -w "%{http_code}" http://vpn.example.com:9100/metrics
# Expected: 200
Add to monitoring:
# prometheus.yml
scrape_configs:
- job_name: 'wiretastix-health'
static_configs:
- targets: ['vpn.example.com:9100']
metrics_path: '/metrics'
scrape_interval: 30s
scrape_timeout: 10s
5. Capacity Planning
Track trends to plan capacity:
# Average daily bandwidth (last 30 days)
avg_over_time(sum(rate(wiretastix_peer_tx[5m]) + rate(wiretastix_peer_rx[5m]))[30d:1h])
# Peak concurrent users (last 30 days)
max_over_time(count(wiretastix_peer_uptime > (time() - 300))[30d:1h])
# Growth rate (users)
(count(wiretastix_peer_uptime) - count(wiretastix_peer_uptime offset 30d)) / count(wiretastix_peer_uptime offset 30d) * 100
Troubleshooting
Metrics Not Updating
Symptom: Metrics show old data or don’t update.
Diagnosis:
# Check if Wiretastix is running
sudo systemctl status wiretastix
# Check if metrics endpoint responds
curl http://localhost:9100/metrics
# Check Wiretastix logs
sudo journalctl -u wiretastix -f
Solution: Restart Wiretastix:
sudo systemctl restart wiretastix
Prometheus Cannot Scrape
Symptom: Target shows “DOWN” in Prometheus.
Diagnosis:
# Test from Prometheus server
curl http://vpn.example.com:9100/metrics
# Check firewall
sudo ufw status
sudo nft list ruleset | grep 9100
Solution: Fix firewall or network connectivity.
High Cardinality Issues
Symptom: Prometheus uses too much memory with many peers.
Cause: Each peer creates multiple metric series (one per label combination).
Solution:
- Increase Prometheus memory
- Use recording rules to aggregate
- Set retention limits
Missing Labels
Symptom: user label shows “unset” for authenticated peers.
Cause: Peer not associated with OIDC user in database.
Solution:
# Check peer-user association
wiretastix-cli peer list
# Ensure user logged in via OIDC (not manual peer)
Advanced: Custom Metrics
Future versions of Wiretastix may expose additional metrics:
- Group-level statistics
- Rule evaluation counters
- API request counts
- Database query performance
- OIDC token refresh events
To request new metrics, open an issue on GitHub.
Summary
Wiretastix provides comprehensive Prometheus metrics for monitoring VPN performance:
Metrics exposed:
wiretastix_peer_tx- Transmitted byteswiretastix_peer_rx- Received byteswiretastix_peer_uptime- Connection status
Key setup steps:
- Configure metrics endpoint in
wiretastix.yaml - Protect endpoint with firewall
- Configure Prometheus to scrape
- Create Grafana dashboards
- Set up alerting rules
Best practices:
- Monitor peer connections and bandwidth
- Alert on disconnections and high usage
- Use recording rules for performance
- Generate usage reports
- Plan capacity based on trends
For more monitoring and security topics, see:
- Security Best Practices - Securing metrics endpoint
- FAQ & Troubleshooting - Common monitoring issues
- Reverse Proxy Setup - Proxy metrics endpoint