feat(ops): export Prometheus metrics (system + game KPIs) via OpenTelemetry #159

Merged

marco merged 3 commits from feat/prometheus-metrics into main

2026-07-22 22:42:19 +02:00

marco commented

2026-07-22 22:27:23 +02:00

Owner

Summary

Export Prometheus metrics from both servers for the homelab Grafana/Prometheus stack, next to the structured logs (#156). Observability only — read-only, no gameplay/wire surface. Closes #157.

OpenTelemetry on both processes (Program.cs): runtime + ASP.NET instrumentation, Prometheus exporter, a GET /metrics scrape endpoint. Auth uses only the free instrumentation (login rate/latency via http_server_request_duration_*, kestrel connections). Same "one format everywhere" spirit as the logs — env is a label, never a different shape.
A GameMetrics server-infra singleton owns one Meter + all custom instruments and is injected into WorldTick, WorldPersister, GameSessionHandler, CommandRegistry. World/Gameplay never reference it.
Custom GameServer KPIs:
- Sim health — isommo_tick_duration_milliseconds (histogram), isommo_tick_overruns_total.
- Players — isommo_players_connected, isommo_player_connects_total, isommo_player_disconnects_total, isommo_player_session_seconds.
- Persistence — isommo_save_duration_milliseconds, isommo_last_save_age_seconds, isommo_saves_total, isommo_save_failures_total, isommo_save_size_bytes.
- World — isommo_creatures_count, isommo_items_count (total items the server tracks).
- Security (aggregate, for anti-cheat alerts) — isommo_malformed_messages_total, isommo_chat_rate_limited_total, isommo_intents_rejected_total{reason}, isommo_gm_commands_total{command,authorized}.

Design notes (from the critical-design-review, persisted in #157)

Single-threaded sim honored: tick-side recording uses in-memory instruments (no lock, no I/O in the tick loop); the world-count ObservableGauges read a sim-thread-updated snapshot (GameMetrics.UpdateWorldCounts, called from WorldTick), never World off the scrape thread. players_connected reads the thread-safe ConnectionManager.
No unbounded/PII labels: no per-player label anywhere (identity stays in the logs); gm_commands is tagged with the canonical command name (bounded), never the raw client string; intents_rejected{reason} is a fixed reason set.
/metrics is unauthenticated → internal-only: documented in docs/deploy.md (the reverse proxy must not route it; Prometheus scrapes over the internal net). The Caddy sample only routes /auth/* and /ws.
World facade untouched by logic: only added read-only count props (World.ItemCount, ItemRegistry.Count) mirroring the existing CreatureCount/PlacedItemCount. (Follow-up #158 will unify these into world.Items/world.Mobiles query views.)

How it was tested

Unit (GameMetricsTests, via MeterListener): every instrument — tick duration + overrun threshold, connect/disconnect + session seconds, save duration/count/failure/bytes, last_save_age (0 before any save, ~0 right after), the four security counters (incl. the bounded reason/command/authorized tags), and the world-count gauges.
Integration (GameServerIntegrationTests): GET /metrics returns Prometheus text exposition with the custom meter present; after a real WebSocket client connects, isommo_players_connected ≥ 1 and isommo_player_connects_total is emitted.
Live (fresh DB, just dev, real client via the debug harness) — scraped /metrics and sanity-checked the numbers:
- baseline: isommo_items_count 5 (the 5 seeded items), creatures_count 0, players_connected 0, tick histogram averaging <1 ms with 0 overruns (100 ms budget).
- after login + moves + a periodic save: player_connects_total 1, players_connected 1, saves_total 2, save_size_bytes 98 (one player blob), save_duration avg ~3 ms, last_save_age_seconds 12.8 (into a 15 s cycle). Auth /metrics shows kestrel/aspnetcore series.
- confirmed the exported names are clean/conventional (no _ms_milliseconds double suffixes).
just lint (0 warnings) + just test (all 7 projects green) + whole solution builds.

Checklist

just lint passes (zero warnings)
just test is green (whole solution builds — client + tools included)
Multi-platform preserved (OpenTelemetry packages are pure managed; no OS-specific dependency)
Docs updated (docs/deploy.md) + ## Definition of Done in #157
No self-merge — owner merges after review

## Summary Export **Prometheus metrics** from both servers for the homelab Grafana/Prometheus stack, next to the structured logs (#156). Observability only — read-only, no gameplay/wire surface. Closes #157. - **OpenTelemetry on both processes** (`Program.cs`): runtime + ASP.NET instrumentation, Prometheus exporter, a `GET /metrics` scrape endpoint. Auth uses only the free instrumentation (login rate/latency via `http_server_request_duration_*`, kestrel connections). Same "one format everywhere" spirit as the logs — env is a label, never a different shape. - **A `GameMetrics` server-infra singleton** owns one `Meter` + all custom instruments and is injected into `WorldTick`, `WorldPersister`, `GameSessionHandler`, `CommandRegistry`. `World`/`Gameplay` never reference it. - **Custom GameServer KPIs:** - Sim health — `isommo_tick_duration_milliseconds` (histogram), `isommo_tick_overruns_total`. - Players — `isommo_players_connected`, `isommo_player_connects_total`, `isommo_player_disconnects_total`, `isommo_player_session_seconds`. - Persistence — `isommo_save_duration_milliseconds`, `isommo_last_save_age_seconds`, `isommo_saves_total`, `isommo_save_failures_total`, `isommo_save_size_bytes`. - World — `isommo_creatures_count`, `isommo_items_count` (total items the server tracks). - Security (aggregate, for anti-cheat alerts) — `isommo_malformed_messages_total`, `isommo_chat_rate_limited_total`, `isommo_intents_rejected_total{reason}`, `isommo_gm_commands_total{command,authorized}`. ## Design notes (from the critical-design-review, persisted in #157) - **Single-threaded sim honored**: tick-side recording uses in-memory instruments (no lock, no I/O in the tick loop); the world-count `ObservableGauge`s read a **sim-thread-updated snapshot** (`GameMetrics.UpdateWorldCounts`, called from `WorldTick`), never `World` off the scrape thread. `players_connected` reads the thread-safe `ConnectionManager`. - **No unbounded/PII labels**: no per-player label anywhere (identity stays in the logs); `gm_commands` is tagged with the **canonical** command name (bounded), never the raw client string; `intents_rejected{reason}` is a fixed reason set. - **`/metrics` is unauthenticated → internal-only**: documented in `docs/deploy.md` (the reverse proxy must not route it; Prometheus scrapes over the internal net). The Caddy sample only routes `/auth/*` and `/ws`. - **World facade untouched by logic**: only added read-only count props (`World.ItemCount`, `ItemRegistry.Count`) mirroring the existing `CreatureCount`/`PlacedItemCount`. (Follow-up #158 will unify these into `world.Items`/`world.Mobiles` query views.) ## How it was tested - **Unit** (`GameMetricsTests`, via `MeterListener`): every instrument — tick duration + overrun threshold, connect/disconnect + session seconds, save duration/count/failure/bytes, `last_save_age` (0 before any save, ~0 right after), the four security counters (incl. the bounded `reason`/`command`/`authorized` tags), and the world-count gauges. - **Integration** (`GameServerIntegrationTests`): `GET /metrics` returns Prometheus text exposition with the custom meter present; after a real WebSocket client connects, `isommo_players_connected` ≥ 1 and `isommo_player_connects_total` is emitted. - **Live (fresh DB, `just dev`, real client via the debug harness)** — scraped `/metrics` and sanity-checked the numbers: - baseline: `isommo_items_count 5` (the 5 seeded items), `creatures_count 0`, `players_connected 0`, tick histogram averaging <1 ms with 0 overruns (100 ms budget). - after login + moves + a periodic save: `player_connects_total 1`, `players_connected 1`, `saves_total 2`, `save_size_bytes 98` (one player blob), `save_duration` avg ~3 ms, `last_save_age_seconds 12.8` (into a 15 s cycle). Auth `/metrics` shows kestrel/aspnetcore series. - confirmed the exported names are clean/conventional (no `_ms_milliseconds` double suffixes). - `just lint` (0 warnings) + `just test` (all 7 projects green) + whole solution builds. ## Checklist - [x] `just lint` passes (zero warnings) - [x] `just test` is green (whole solution builds — client + tools included) - [x] Multi-platform preserved (OpenTelemetry packages are pure managed; no OS-specific dependency) - [x] Docs updated (`docs/deploy.md`) + `## Definition of Done` in #157 - [x] No self-merge — owner merges after review

marco added 1 commit

2026-07-22 22:27:23 +02:00

feat(ops): export Prometheus metrics (system + game KPIs) via OpenTelemetry

ci / Lint & Test (pull_request) Failing after 1m45s

Details

c81304b61f

/metrics on Auth and GameServer for the Grafana/Prometheus stack: runtime +
ASP.NET instrumentation on both, plus a GameServer Meter for tick health,
player sessions, world-save timing, entity counts, and aggregate security
counters. Closes #157.

marco added 1 commit

2026-07-22 22:36:54 +02:00

test(ops): isolate GameMetrics meter per test to fix parallel-run race

ci / Lint & Test (pull_request) Failing after 38s

Details

3229f39217

The Meter name is process-global, so a MeterListener filtering by the shared
name caught other parallel tests' GameMetrics measurements and mutated the
capture List from their threads (List corruption on CI). Give each test a
unique meter name via an optional GameMetrics ctor param (default unchanged).

marco added 1 commit

2026-07-22 22:39:37 +02:00

style: collapse GameMetrics registration to one line (csharpier)

ci / Lint & Test (pull_request) Successful in 1m56s

Details

db4cf2a704

marco merged commit b603ea4e79 into main

2026-07-22 22:42:19 +02:00

marco deleted branch feat/prometheus-metrics

2026-07-22 22:42:19 +02:00

marco referenced this pull request from a commit

2026-07-22 22:42:20 +02:00