ops: export Prometheus metrics (system + game KPIs) for Grafana #157

New issue

Closed

opened 2026-07-22 22:02:42 +02:00 by marco · 0 comments

marco commented

2026-07-22 22:02:42 +02:00

Owner

Scope (what to do)

Add OpenTelemetry to both Program.cs: runtime + ASP.NET instrumentation, Prometheus exporter, /metrics scraping endpoint. Auth uses only the free instrumentation (login rate/latency via http.server.request.duration).
A GameMetrics server-infra singleton owning one Meter + instruments; injected into WorldTick, WorldPersister, ConnectionManager/GameSessionHandler. World/Gameplay never reference it.
Custom KPIs (GameServer):
- Simulation: tick_duration_ms (histogram), tick_overruns_total.
- Players: players_connected (gauge), player_connects_total, player_disconnects_total, player_session_seconds (histogram).
- Persistence: save_duration_ms (histogram), seconds_since_last_save (gauge), saves_total, save_failures_total, save_bytes.
- World (sim-thread snapshot): creatures_count, ground_items_count, corpses_count (gauges).
- Security (aggregate, for anti-cheat alerts): malformed_messages_total, chat_rate_limited_total, intents_rejected_total{reason}, gm_commands_total{command,authorized}.
docs/deploy.md: the /metrics endpoint, a sample Prometheus scrape job, and the constraint that the reverse proxy must NOT route /metrics (internal scrape only).

Design constraints (from the review)

Single-threaded sim: world-count ObservableGauges read a sim-thread-updated snapshot, never World from the scrape thread. ConnectionManager.Count (ConcurrentDictionary) is safe to read directly. Tick-side recording uses in-memory Meter instruments only — no lock, no I/O in the tick loop.
No per-player labels anywhere (unbounded cardinality / PII) — identity stays in logs (#156). All labels are bounded, fixed sets.
/metrics is unauthenticated → internal-only; not on the public proxy.

Invariants Check (persisted from the critical-design-review)

Scope ✓ — exactly the owner-approved trimmed set (gameplay-event counters dropped).
Server-authoritative N/A — no client intent / gameplay rule; a scrape is an ops pull.
GM authorization ✓ — gm_commands_total{command,authorized} observes the existing CanExecute; adds no authority path.
Identity model ✓ — aggregate metrics, no PlayerId/char label.
Protocol versioned N/A — no wire-shape/enum/DTO change.
String catalog N/A — no player-facing text.
Single-threaded sim ✓ — sim-thread recording via thread-safe instruments; world-count gauges read a sim-thread snapshot, not World; no new World lock.
World.cs HARD GATE ✓ — no metrics logic in World; only (at most) a trivial read-only count property delegating to a component.
Screen HARD GATE N/A — server-only.
Client engine-independence N/A — no client code.
Gameplay/Networking separation ✓ — metrics is infra; recording in Networking + tick orchestrator + persistence; World/Gameplay unaware.
Act on the instance ✓ — world-count snapshot is a per-tick derived aggregate from the authoritative registries, not a per-instance side-collection.
Extend by type, not switch ✓ — bounded label values at existing choke points, no new type-dispatched switch.
Server-paced actions N/A — none added.
Persistence (GameServer) ✓ — save metrics are timing/outcome only; no format/version change, no tick-loop I/O.
Persistence (Auth) N/A — no EF schema change.
Process separation ✓ — each process wires its own OTel + /metrics; config duplicated per process, no shared file.
Typed options ✓ — any new tunable is a bound-options field, default in one place.
Broadcasts/AoI N/A — no observable entity/broadcast event.
Multi-platform ✓ — OTel/Prometheus exporter are pure managed, cross-platform.
Assets required N/A — no art.
Asset naming HARD GATE N/A — no pack/tiledata asset.
ModernUO reference — divergence: ModernUO has no metrics exporter; we add OTel for our Prometheus/Grafana stack, keeping its single-thread model.
Docs & DoD same change ✓ — deploy.md + this DoD in the implementing PR.

Definition of Done

GET /metrics on both Auth (:5080) and GameServer (:5100) returns Prometheus text exposition (HTTP 200, # TYPE lines).
The GameServer /metrics exposes every custom KPI listed above (verifiable: connect a client, move, trigger a save, and the corresponding series change — players_connected ≥ 1, a tick_duration_ms bucket populated, saves_total increments, seconds_since_last_save resets).
Sending a malformed frame / a non-admin /-command increments malformed_messages_total / gm_commands_total{authorized="false"} respectively.
No metric carries a per-player/unbounded label (grep: no PlayerId/char in label keys).
World-count gauges are proven not to read World off the sim thread (code review: gauge callbacks read the snapshot only).
/metrics is documented in docs/deploy.md as internal-only (proxy must not route it) with a sample scrape job.
Base DoD: dotnet test green, whole solution builds, zero warnings; multi-platform preserved.

Export Prometheus metrics from both servers (Auth + GameServer) for the homelab Grafana/Prometheus stack, alongside the structured logs (#156, Loki). Observability only — read-only, no gameplay/wire surface. ## Scope (what to do) - Add OpenTelemetry to both `Program.cs`: runtime + ASP.NET instrumentation, Prometheus exporter, `/metrics` scraping endpoint. Auth uses only the free instrumentation (login rate/latency via `http.server.request.duration`). - A `GameMetrics` server-infra singleton owning one `Meter` + instruments; injected into `WorldTick`, `WorldPersister`, `ConnectionManager`/`GameSessionHandler`. `World`/`Gameplay` never reference it. - **Custom KPIs (GameServer):** - Simulation: `tick_duration_ms` (histogram), `tick_overruns_total`. - Players: `players_connected` (gauge), `player_connects_total`, `player_disconnects_total`, `player_session_seconds` (histogram). - Persistence: `save_duration_ms` (histogram), `seconds_since_last_save` (gauge), `saves_total`, `save_failures_total`, `save_bytes`. - World (sim-thread snapshot): `creatures_count`, `ground_items_count`, `corpses_count` (gauges). - Security (aggregate, for anti-cheat alerts): `malformed_messages_total`, `chat_rate_limited_total`, `intents_rejected_total{reason}`, `gm_commands_total{command,authorized}`. - `docs/deploy.md`: the `/metrics` endpoint, a sample Prometheus scrape job, and the constraint that the reverse proxy must NOT route `/metrics` (internal scrape only). ## Design constraints (from the review) - **Single-threaded sim**: world-count `ObservableGauge`s read a **sim-thread-updated snapshot**, never `World` from the scrape thread. `ConnectionManager.Count` (ConcurrentDictionary) is safe to read directly. Tick-side recording uses in-memory `Meter` instruments only — no lock, no I/O in the tick loop. - **No per-player labels** anywhere (unbounded cardinality / PII) — identity stays in logs (#156). All labels are bounded, fixed sets. - `/metrics` is unauthenticated → internal-only; not on the public proxy. ## Invariants Check (persisted from the critical-design-review) 1. Scope ✓ — exactly the owner-approved trimmed set (gameplay-event counters dropped). 2. Server-authoritative N/A — no client intent / gameplay rule; a scrape is an ops pull. 3. GM authorization ✓ — `gm_commands_total{command,authorized}` observes the existing `CanExecute`; adds no authority path. 4. Identity model ✓ — aggregate metrics, no PlayerId/char label. 5. Protocol versioned N/A — no wire-shape/enum/DTO change. 6. String catalog N/A — no player-facing text. 7. Single-threaded sim ✓ — sim-thread recording via thread-safe instruments; world-count gauges read a sim-thread snapshot, not World; no new World lock. 8. World.cs HARD GATE ✓ — no metrics logic in World; only (at most) a trivial read-only count property delegating to a component. 9. Screen HARD GATE N/A — server-only. 10. Client engine-independence N/A — no client code. 11. Gameplay/Networking separation ✓ — metrics is infra; recording in Networking + tick orchestrator + persistence; World/Gameplay unaware. 12. Act on the instance ✓ — world-count snapshot is a per-tick derived aggregate from the authoritative registries, not a per-instance side-collection. 13. Extend by type, not switch ✓ — bounded label values at existing choke points, no new type-dispatched switch. 14. Server-paced actions N/A — none added. 15. Persistence (GameServer) ✓ — save metrics are timing/outcome only; no format/version change, no tick-loop I/O. 16. Persistence (Auth) N/A — no EF schema change. 17. Process separation ✓ — each process wires its own OTel + /metrics; config duplicated per process, no shared file. 18. Typed options ✓ — any new tunable is a bound-options field, default in one place. 19. Broadcasts/AoI N/A — no observable entity/broadcast event. 20. Multi-platform ✓ — OTel/Prometheus exporter are pure managed, cross-platform. 21. Assets required N/A — no art. 22. Asset naming HARD GATE N/A — no pack/tiledata asset. 23. ModernUO reference — divergence: ModernUO has no metrics exporter; we add OTel for our Prometheus/Grafana stack, keeping its single-thread model. 24. Docs & DoD same change ✓ — deploy.md + this DoD in the implementing PR. ## Definition of Done - [ ] `GET /metrics` on both Auth (:5080) and GameServer (:5100) returns Prometheus text exposition (HTTP 200, `# TYPE` lines). - [ ] The GameServer `/metrics` exposes every custom KPI listed above (verifiable: connect a client, move, trigger a save, and the corresponding series change — `players_connected` ≥ 1, a `tick_duration_ms` bucket populated, `saves_total` increments, `seconds_since_last_save` resets). - [ ] Sending a malformed frame / a non-admin `/`-command increments `malformed_messages_total` / `gm_commands_total{authorized="false"}` respectively. - [ ] No metric carries a per-player/unbounded label (grep: no PlayerId/char in label keys). - [ ] World-count gauges are proven not to read `World` off the sim thread (code review: gauge callbacks read the snapshot only). - [ ] `/metrics` is documented in `docs/deploy.md` as internal-only (proxy must not route it) with a sample scrape job. - [ ] Base DoD: `dotnet test` green, whole solution builds, zero warnings; multi-platform preserved.

marco referenced this issue

2026-07-22 22:12:27 +02:00

refactor(world): read-only entity views world.Mobiles / world.Items (drop scalar count props) #158

marco referenced this issue from a commit

2026-07-22 22:26:50 +02:00

feat(ops): export Prometheus metrics (system + game KPIs) via OpenTelemetry

marco referenced this issue from a pull request that will close it,

2026-07-22 22:27:23 +02:00

feat(ops): export Prometheus metrics (system + game KPIs) via OpenTelemetry #159

marco closed this issue

2026-07-22 22:42:19 +02:00

marco referenced this issue from a commit