Monitoring & Logs

Monitoring & Logs

Current Observability Stack

Component	Purpose	Port
OpenTelemetry SDK (@terma/observability)	Auto-instrumentation of HTTP, metrics & traces	—
Prometheus	Metrics collection (pull)	prometheus:9090
Jaeger	Trace collection & visualization (OTLP HTTP)	jaeger:4318 (traces), jaeger:16686 (UI)
Grafana	Dashboards (auto-provisioned)	grafana:3000
pino	Structured JSON logs with traceId	—

---

1. OpenTelemetry (Auto-instrumentation)

The packages/observability package initializes the OTel SDK at service startup:

• HTTP server (Fastify) — automatic spans via @opentelemetry/instrumentation-http
• HTTP client — W3C traceparent header injection on every outgoing request
• PrometheusExporter — metrics on port 9090 (api) / 9091 (bot)
• OTLPTraceExporter — traces sent to Jaeger (push)

Verify OTel Status

curl http://localhost:9090/metrics | head -20
curl http://localhost:3000/api/health | jq .otel

Expected /health response:

{
  "status": "ok",
  "version": "0.1.2",
  "otel": {
    "initialized": true,
    "tracer": "started",
    "meter": "started"
  },
  "uptime": 12345
}

---

2. Prometheus (Metrics)

Collection

Each service runs a built-in Prometheus exporter:

Service	Port	Endpoint
api	9090	/metrics
bot	9091	/metrics

Prometheus is configured via scripts/prometheus-conf/prometheus.yml.

Available Metrics

• http.server.duration — latency histogram (ms)
• http.server.request_count — request count by route/status
• process.memory.usage — process memory
• process.cpu.utilization — process CPU

Accessing Prometheus

# Port forwarding (without Grafana)
ssh -L 9090:localhost:9090 user@vps
# Open: http://localhost:9090

---

3. Jaeger (Tracing)

End-to-end Propagation

Telegram → bot → api → db

W3C Trace Context (traceparent) is automatically injected on every HTTP call:

• Bot → API: createApiClient injects propagation.inject()
• API → DB: HTTP-level tracing (Fastify)

View Traces in Jaeger UI

# Port forwarding
ssh -L 16686:localhost:16686 user@vps
# Open: http://localhost:16686

Searching for traces:

1. Select service terma-api or terma-bot
2. Set time range
3. Click Find Traces

Waterfall flamegraph:

grammy.command.start
  → api-client.fetch (/internal/users/ensure)
    → Fastify.route (POST /internal/users/ensure)
      → db.query (drizzle)

---

4. Grafana (Dashboards)

Grafana is auto-provisioned via scripts/grafana/datasources.yml + scripts/grafana/dashboards.yml.

Panels

1. Total Requests — request rate (api/bot)
2. Success Rate — percentage of successful requests
3. HTTP Errors Breakdown — 4xx/5xx errors by status code
4. Latency p50/p95/p99 — latency percentiles
5. Last 5xx Errors — recent errors with traceId
6. Service Uptime — Prometheus target status

Error Search by traceId

In the Last 5xx Errors panel, recent errors are displayed with clickable Jaeger links:

1. Click on a traceId in the panel
2. Jaeger UI opens with the full waterfall trace
3. Analyze latency and errors on each span

Port Forwarding

ssh -L 3000:localhost:3000 user@vps
# Open: http://localhost:3000
# Login: admin / password from ${GRAFANA_PASSWORD}

---

5. Structured Logs (pino)

All services produce JSON logs with the following fields:

{
  "time": "2025-06-15T10:30:00.123Z",
  "level": "info",
  "msg": "request completed",
  "traceId": "4e1f2a3b...",
  "spanId": "8c9d0e1f...",
  "name": "http.request",
  "method": "GET",
  "url": "/api/health",
  "statusCode": 200,
  "responseTime": 12.3
}

Viewing Logs

# Tail all service logs
docker compose -f docker-compose.stage.yml logs --tail=50 -f

# Search by traceId
docker compose -f docker-compose.stage.yml logs | grep "4e1f2a3b"

# Filter errors
docker compose -f docker-compose.stage.yml logs | grep '"level":"error"'

---

6. Docker Monitoring Services

Services are started via docker-compose.stage.yml:

# Start the monitoring stack
docker compose -f docker-compose.stage.yml up -d prometheus jaeger grafana

# Check status
docker compose -f docker-compose.stage.yml ps

# View logs
docker compose -f docker-compose.stage.yml logs -f prometheus

---

7. Health Check

Endpoint

GET /api/health → 200 OK

Response includes OTel status:

{
  "status": "ok",
  "version": "0.1.2",
  "otel": {
    "initialized": true,
    "tracer": "started",
    "meter": "started"
  },
  "uptime": 12345
}

Cron Check

*/3 * * * * curl -sf https://termagrand.ru/api/health \
  || curl -s "https://api.telegram.org/bot${BOT_TOKEN}/sendMessage" \
     -d "chat_id=${ADMIN_CHAT_ID}&text=🚨+Healthcheck+FAILED+termagrand.ru"

External Uptime Monitor

Recommended: UptimeRobot (free plan, 5-min interval) or BetterStack.

Expected status: 200 OK.

---

8. Security & 152-FZ Compliance

• Metrics — aggregated numeric values (no PII)
• Traces — technical identifiers (traceId, spanId) and URL templates
• Logs — traceId/spanId, no PII
• Request bodies are not recorded in span attributes
• HTTP URL paths without query parameters (pathname only)

---

Next Step

06 — Backup & Restore