Grafana Dashboards

Grafana Dashboards

Access

# Port forwarding
ssh -L 3000:localhost:3000 user@vps
# Open: http://localhost:3000
# Login: admin
# Password: from ${GRAFANA_PASSWORD} in .env

Dashboard Panels

1. Total Requests — Request Rate

Chart showing requests per second for api and bot. Source: http.server.request_count metric.

2. Success Rate

Percentage of 2xx/3xx responses. Quickly identifies error spikes.

3. HTTP Errors Breakdown — 4xx/5xx Errors

Error distribution by status code (400, 401, 403, 404, 500, 502, 503).

4. Latency p50/p95/p99

Histogram from http.server.duration. Tracks performance degradation:

• p50 — typical latency
• p95 — issues for 5% of users
• p99 — outlier latency

5. Last 5xx Errors — Recent Errors with traceId

Table of recent server errors. Each row contains:

• traceId — clickable link to Jaeger
• statusCode — response code
• method + url — endpoint
• timestamp — error time

6. Service Uptime — Target Status

Status of /metrics endpoints for api and bot. Shows which services are currently available.

Error Search by traceId

Via Grafana

1. Open dashboard → Last 5xx Errors panel
2. Click on traceId in the error row
3. Jaeger UI opens with the full waterfall trace
4. Analyze:
   • Red spans — errors at specific steps
   • Long spans — performance bottlenecks
   • Missing spans — broken call chains

Via pino logs (CLI)

# Find an error
docker compose -f docker-compose.stage.yml logs | grep '"level":"error"' | head -5

# Copy traceId from output, e.g. "4e1f2a3b..."
docker compose -f docker-compose.stage.yml logs | grep "4e1f2a3b"

# Or open Jaeger UI and enter traceId manually

Via Jaeger UI Directly

1. Open http://<vps>:16686
2. Select service: terma-api or terma-bot
3. Set time range (last 15 min / 1 hour / today)
4. Click Find Traces
5. Filter by:
   • error=true — only traces with errors
   • http.status_code>=500 — server errors
   • db.system=postgresql — database queries

Auto-provisioning

Grafana dashboards and datasources are configured automatically on startup:

scripts/grafana/
├── dashboards.yml        # Dashboard definitions
├── datasources.yml       # Prometheus + Jaeger datasources
└── dashboards/
    └── terma-grand.json  # Dashboard JSON model

To update a dashboard — modify the JSON model and restart Grafana:

docker compose -f docker-compose.stage.yml restart grafana