marco/IsoMmo

Watch

Fork 0

flaky: GameServer WebSocket integration tests time out under parallel load #43

New issue

Closed

opened 2026-07-18 05:44:03 +02:00 by marco · 2 comments

marco commented

2026-07-18 05:44:03 +02:00

Owner

GameServerIntegrationTests occasionally fail with OperationCanceledException from TestWebSocket.ReceiveAsync (a receive stalls past the 20s timeout) — seen on CI run 39 (Item_EquipAndUnequip_UpdateTheSlot) and once locally in a full parallel dotnet test. Passing in isolation and on a re-run confirms it is load-induced flakiness, not a logic bug (CI run 40, a superset of the same code, is green).

Likely root cause: thread-pool starvation. xUnit runs test classes in parallel; each GameServerFactory spins up a full host with a 100ms GameLoopService tick loop (+ the 15s PersistenceService). Many concurrent hosts starve the pool, so a broadcast (InventoryState) is delivered later than the receive window.

Scope (investigate, then pick)

Reduce per-host background pressure in tests (e.g. make the tick / persistence interval configurable and relax them in the factory, or dont register PersistenceService in tests that dont exercise it).
Or cap parallelism for the heavy integration collection (MaxParallelThreads / a dedicated non-parallel collection).
Make ReceiveUntilAsync resilient to a slow first message rather than throwing on the inner per-receive timeout.

Definition of Done

The GameServer integration tests pass green across 20 consecutive CI runs (or a documented local stress loop) with no timeout failures.
The chosen mitigation does not reduce what the tests assert (same coverage of connect/move/AoI/inventory flows).
If parallelism is capped, the full-suite wall-clock is documented so the trade-off is explicit.

`GameServerIntegrationTests` occasionally fail with `OperationCanceledException` from `TestWebSocket.ReceiveAsync` (a receive stalls past the 20s timeout) — seen on CI run 39 (`Item_EquipAndUnequip_UpdateTheSlot`) and once locally in a full parallel `dotnet test`. Passing in isolation and on a re-run confirms it is **load-induced flakiness, not a logic bug** (CI run 40, a superset of the same code, is green). **Likely root cause:** thread-pool starvation. xUnit runs test classes in parallel; each `GameServerFactory` spins up a full host with a 100ms `GameLoopService` tick loop (+ the 15s `PersistenceService`). Many concurrent hosts starve the pool, so a broadcast (`InventoryState`) is delivered later than the receive window. ## Scope (investigate, then pick) - Reduce per-host background pressure in tests (e.g. make the tick / persistence interval configurable and relax them in the factory, or dont register `PersistenceService` in tests that dont exercise it). - Or cap parallelism for the heavy integration collection (`MaxParallelThreads` / a dedicated non-parallel collection). - Make `ReceiveUntilAsync` resilient to a slow first message rather than throwing on the inner per-receive timeout. ## Definition of Done - [ ] The GameServer integration tests pass green across 20 consecutive CI runs (or a documented local stress loop) with no timeout failures. - [ ] The chosen mitigation does not reduce what the tests assert (same coverage of connect/move/AoI/inventory flows). - [ ] If parallelism is capped, the full-suite wall-clock is documented so the trade-off is explicit.

marco referenced this issue from a commit

2026-07-18 08:30:06 +02:00

test(gameserver): raise WebSocket receive timeout to 60s for busy CI runners (#43)

marco commented

2026-07-18 08:32:16 +02:00

Author

Owner

Stopgap shipped in #40 (commit 7f8d609): raised the WebSocket receive timeout from 20s to 60s. Confirmed the failure is Item_EquipAndUnequip_UpdateTheSlot timing out (OperationCanceledException from TestWebSocket.ReceiveAsync) under load on the shared runner — CI run 39/41 red, run 42 green after the bump. Could not reproduce in 38 local runs (macOS + Linux down to 0.5 CPU), so this is a busy-shared-box flake, not a product bug.

The 60s ceiling only widens the window; this issue stays open for the real fix — stop depending on wall-clock delivery (e.g. poll a server-side condition, or reduce concurrent test-host pressure). A reliable reproduction (a genuinely CPU-starved runner) is the first task.

**Stopgap shipped in #40** (commit `7f8d609`): raised the WebSocket receive timeout from 20s to 60s. Confirmed the failure is `Item_EquipAndUnequip_UpdateTheSlot` timing out (`OperationCanceledException` from `TestWebSocket.ReceiveAsync`) under load on the shared runner — CI run 39/41 red, run 42 green after the bump. Could not reproduce in 38 local runs (macOS + Linux down to 0.5 CPU), so this is a busy-shared-box flake, not a product bug. The 60s ceiling only widens the window; **this issue stays open for the real fix** — stop depending on wall-clock delivery (e.g. poll a server-side condition, or reduce concurrent test-host pressure). A reliable reproduction (a genuinely CPU-starved runner) is the first task.

marco commented

2026-07-18 13:31:35 +02:00

Author

Owner

Root cause found — and it was NOT thread starvation (my earlier hypothesis was wrong). Fixed in #40, commit ee7a00e.

The real bug: on connect, GameSessionHandler called interest.Enter (making the connection visible to the tick's AoI reconcile) before sending Welcome. Under load, the handler thread could be preempted between Enter and the Welcome send, letting a tick broadcast ItemSpawned/PlayerJoined to the connection ahead of its Welcome. The integration test (and a real client) treats Welcome as the first message and discards anything before it — dropping the sword's ItemSpawned. Because the AoI reconcile marks entities as known when it emits them, the dropped spawn is never re-sent, so the client hangs until the receive timeout (the 60s bump just made it wait longer).

Proven deterministically: widening the Enter→Welcome window with a 300ms delay makes Item_EquipAndUnequip_UpdateTheSlot fail every time with the exact CI OperationCanceledException; moving Enter after Welcome makes it pass even with that delay.

Fix: send the initial snapshot (Welcome + InventoryState) first, join the AoI second — so Welcome is always the client's first message. Also reverted the 20s→60s receive-timeout stopgap. Closing when #40 merges.

**Root cause found — and it was NOT thread starvation** (my earlier hypothesis was wrong). Fixed in #40, commit `ee7a00e`. The real bug: on connect, `GameSessionHandler` called `interest.Enter` (making the connection visible to the tick's AoI reconcile) **before** sending `Welcome`. Under load, the handler thread could be preempted between `Enter` and the `Welcome` send, letting a tick broadcast `ItemSpawned`/`PlayerJoined` to the connection **ahead of its `Welcome`**. The integration test (and a real client) treats `Welcome` as the first message and discards anything before it — dropping the sword's `ItemSpawned`. Because the AoI reconcile marks entities as *known* when it emits them, the dropped spawn is **never re-sent**, so the client hangs until the receive timeout (the 60s bump just made it wait longer). **Proven deterministically**: widening the Enter→Welcome window with a 300ms delay makes `Item_EquipAndUnequip_UpdateTheSlot` fail every time with the exact CI `OperationCanceledException`; moving `Enter` after `Welcome` makes it pass even with that delay. **Fix**: send the initial snapshot (`Welcome` + `InventoryState`) first, join the AoI second — so `Welcome` is always the client's first message. Also reverted the 20s→60s receive-timeout stopgap. Closing when #40 merges.

marco closed this issue

2026-07-18 16:47:50 +02:00

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

marco/IsoMmo#43

No description provided.

Rows
Columns