epic: world map at 133M cells - chunked format + editor/client/server migration #229
Labels
No labels
alpha:wave-0
alpha:wave-1
alpha:wave-2
alpha:wave-3
area:assets
area:combat
area:ecology
area:infra
area:render
area:scripting
area:ui
area:world
enhancement
epic
migration
post-alpha
roadmap
tech-debt
type:bug
type:chore
type:design
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
marco/IsoMmo#229
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Mandate
Build the infrastructure to author, store, render and serve a hand-drawn world map of 13,344 x 9,984 = 133,226,496 cells (417x312 chunks of 32x32; ~2,067 regions of 256x256). Procedural generation stays a bounded, opt-in, seed-logged tool - never the authoring path. Size is a requirement, not a value to reduce; the architecture must stay size-agnostic. Full brief:
SPEC_CLAUDE_MAP_EDITOR_133M.Sequencing decision (Andrea): format-first, no throwaway - no interim edits to the dense model.
Re-baselined current state (
main@7614a5d, verified)The spec analyzed an older snapshot; much has since merged:
mainworld.map2304x1728 = 3,981,312 cells, 32,557 placements / 85 kindsTileMap._z,GroundZAt,CanStep, Int16 -225..+255); world still flat (noworld.zmap)GroundBlendRendererviewport-bounded); not yet a chunk cacheStill-true seams to migrate:
IMapFormat.ReadMap(string,...)/WriteMap->(string,...)whole-string;TileMap/MapModeldensestring[,](full.Clone()per edit);Brush.FloodglobalHashSet+Queue; editorNewfixed 32x32, brush 1-9, min zoom 0.25;'S'spawn bug (write overwrites terrain, read restores as grass -> flooded map loses spawn on save/reopen).Phased plan (each item = a PR, each leaves build+test green)
IReadOnlyWorldMap+IWorldMapStore+ manifest/region/chunk (magic, endianness, CRC, catalog hash); legacy importer fromworld.map/world.statics/world.zmap; atomic temp->replace writer; fix'S'-> spawn becomes manifest metadata. Golden + property (round-trip) tests.TextMapFormatkept as legacy adapter. No editor/runtime behavior change.MapModel; delta undo/redo;Allaga mappa= manifest default-flip (single undo entry); autosave + journal recovery. The 133M world opens here.New+ 13,344x9,984 preset; big/shaped brushes; polygon/spline/stamp; statics palette (thumbnails/search/tags). Andrea draws the planet here.ProtocolVersionbump; region-border integration tests.Invariants Check (persisted from the critical design review)
No FAIL, no HARD GATE tripped.
/tpis separate);Allaga mappais offline authoring.ProtocolVersion.Currentin that PR (Fase 7).World.World.csHARD GATE OK - no logic added; delegates map queries to a component.ScreenHARD GATE OK - streaming/LRU/overview in rendering +ClientWorldcomponents;GameScreenonly wires.IReadOnlyWorldMap+ chunk/overview math inShared/Client.Core(unit-tested, no MonoGame); only GPU LRU/atlas isClientglue; editor model stays inMapEditor.Core.SharedbehindIMapFormat.IMapFormat, distinct from per-entityworld.sav; no DB/central-switch/cross-player refs.CollisionOnly/Invisible (e.g.redwood_trunk) from missing-asset.docs/world-map-format.mdreferences code+golden-tests, never mirrors the byte layout; per-PR DoD delta.Server-side validation (trust boundary)
Authoring is offline (no player intent). Runtime boundary: map+catalog hash handshake at connect (mismatched client -> diagnostic error, not silent desync) - the one wire addition ->
ProtocolVersionbump. Walkability/LOS/Z stay server-decided from the server's warm-loaded map; client copy is render/prediction only.Verification plan
Core/Sharedunit (format, importer, chunk/overview math,Allaga mappasemantics); console benchmark harness (memory/time budgets); golden + property round-trip tests; WebSocket integration for the hash handshake (Fase 7); existingWorldwalkability suite stays green on the chunk-backed map.Allaga mappa-> four corners + chunk/region borders + old spawn cell =water, survive save/reopen, statics/Z untouched, single-undo restores exactly; fault-injection mid-commit recovers old-or-new (never mixed); hash mismatch -> client diagnostic error; missingwaterin catalog -> readable error, no mutation.Allaga mappabefore/after). Harness gap: the debug harness drives the client, not the editor - editor screenshotting to be added in Fase 4.Open decisions
Allaga mappa-> draw) vs. evolve the existing 2304^2.Definition of Done (epic)
Each phase lands as its own PR with its own DoD delta, whole solution builds, zero warnings, tests green, Win+macOS honored.
PR 1 baseline recorded (
just bench) — current dense/whole-string architectureMeasured on Andrea's dev PC (Windows 10.0.26200, x64, 16 CPUs, .NET 10.0.0, ~64 GiB). Reproduce with
just bench.2304x1728 = 3,981,312 cells (measured):
ReadMap222 ms / 76 MiB alloc,MapModel.FromTileMap78 ms / 40 MiB, brush stroke 9x9 ~0 ms / 0 MiB,ToTileMap(full clone) 9 ms / 40 MiB,WriteMap88 ms / 64 MiB. Peak retained ~95 MiB.Projection to 13,344x9,984 = 133,226,496 cells:
ReadMap~7.4 s / ~2.5 GiB,WriteMap~3.0 s / ~2.1 GiB. Absolute floors: one dense ref grid ~1.0 GiB retained; three in flight ~3.0 GiB; UTF-16 map string ~254 MiB.Takeaways for PR 2+: a local edit is essentially free, but every dirty render/save touches the whole grid — the O(map) cost is in
ReadMap/FromTileMap/ToTileMap/WriteMap, not in the brush. At the target the current model needs GBs of retained heap and multi-second monolithic stalls -> confirms the chunked, addressable format is required. Numbers are informational per this PC, not a CI gate.PR: #230.
PR 2 up: #231 — chunked binary format core (additive, no runtime wiring yet).
Delivered:
IReadOnlyWorldMap+WorldMapStore(random-access, bounded cache) +WorldMapWriter(atomic region+manifest) +LegacyMapImporter(spawn->metadata, skips uniform default chunks); CRC32 per chunk+manifest;docs/world-map-format.md. 24 new tests. Headline: the store opens the full 13,344x9,984 = 133,226,496-cell world as an empty ocean by reading only the manifest — zero region files, no per-cell allocation.Stacked on #230; will rebase onto main after #230 merges. Next: PR 3 (editor model chunk-backed).
PR 3 up: #232 — chunk-backed editor model (
ChunkedEditModel, MapEditor.Core). Scoped to the model only.Delivered: overlay reads (edited chunk / base store / default); materialize-on-edit (memory bounded by the dirty set);
FloodMap= manifest default-flip (ocean initializer); incremental save (only dirty regions rewritten, untouched regions byte-identical, emptied regions deleted). 9 new tests including opens the 133M world + edits one cell with a single dirty chunk and incremental save leaves untouched regions byte-identical.Deliberately deferred to keep it reviewable: undo/redo, autosave/journal, and the
EditorGameUI wiring + authoring UX (zoom/overview/brushes) — the last is the slice where you actually see and draw the world (with screenshots). Stacked on #231/#230; rebase after they merge.Fase 4 (editor UI) — critical design review persisted
Verdict: go-with-changes. The enabling change is the render path: the editor renders by building a full dense
TileMap(_model.ToTileMap()→GridRenderer), which can't exist at 133M. Core of the design = render a viewport window, and decompose Fase 4 into sub-slices.Decisions
TileMapinMapEditor.Coreand pass its world offset to the reusedGridRenderer; the client's render hot-path stays untouched. (GeneralizingGridRendererto anIReadOnlyWorldMapsource is deferred to Fase 7 if wanted.)NewOcean) is the main flow; the 2304² map stays a fixture, not the source. A "import legacy world.map" path is kept as a secondary option.content/worlds/asterra/=manifest.bin+regions/. Never underbin/.Sub-slices
EditorGameonChunkedEditModel; viewport-window render (window+offset, rebuilt on view move);Newconfigurable + 13,344×9,984 ocean preset; open/save thecontent/worlds/asterra/dir + import a legacyworld.maponce; chunked (bounded) flood; extend--screenshotwith--open/--gotofor piloted screenshots. Outcome: open, pan, draw and save the 133M world in the editor.Invariants Check (key lines; full walk in-session)
No
✗. Most invariants are N/A (offline authoring tool: no server/wire/sim/persistence-of-game-state). Load-bearing gate: the editorScreen-analog HARD GATE (tools/IsoMmo.MapEditor/CLAUDE.md) — windowing math, chunked flood, tool/coordinate logic go inMapEditor.Core(unit-tested);EditorGameonly wires + draws. Client engine-independence ✓ (Core stays MonoGame-free). Persistence ✓ (editor writes authored map content, distinct from runtimeworld.sav). Multi-platform ✓. Assets ✓ (real art viaGridRenderer; missing = not drawn). Docs: updatedocs/map-editor.mdin the slice that changes open/save/New + render.Server-side validation
N/A here (offline). The server validating the authored world (spawn in-bounds+walkable, catalog hash) is the Fase-7 warm-load, not this slice.
Verification plan
MapEditor.Coreunit tests: visible-cell-window bounds (camera+viewport+zoom → cell AABB), bounded chunked flood (no 133M enumeration), New-preset construction, open/import/save wiring. Screenshots (required, visible): piloted via the extended--screenshot(--open <dir> --goto x y+ a scripted paint) — fresh ocean, a drawn patch, the imported legacy world. Harness item for 4a: scripted editor input so the screenshot shows authored terrain, not an empty ocean.Fase 7 (chunked world in the game) — critical design review persisted
Verdict: go-with-changes, decomposed into 7a/7b/7c. The whole runtime (movement/combat/creatures/items/spells + client render) is built on
TileMap's interface (IsWalkable/HasLineOfSight/GroundZAt/TerrainAt/CanStep/Statics/Width/Height/Spawn), backed today by a densestring[,]— which can't hold 133M cells. So Fase 7 = re-back the map, don't rewrite the consumers, + client streaming + a connect handshake.Decisions
IsoViewport+ a bounded chunk LRU; adaptGridRendererto a windowed source). Not full predictive streaming yet.IReadOnlyTileMapand addCompactWorldMap(byte[] terrain indices + Int16 Z + sparse statics + catalog→TerrainDef) implementing it; pointWorld+ systems at the interface.TileMapstays for import/tests. Low churn per file.Int16[W*H]for O(1) parity with the currentGroundZAt.Sub-slices
IReadOnlyTileMap+CompactWorldMap+ a loader fromWorldMapStore; server warm-loads asterra at startup (before connections, no tick I/O) + validates spawn; consumers switched to the interface. Parity unit tests (walkability/LOS/Z match the text map cell-for-cell). No wire change.GridRendererwindowed source. Piloted screenshots of the game on the chunked world.ProtocolVersionbump. Client sends its world hash at connect; server rejects mismatch with a diagnosticSystemNotice; WebSocket integration test for the reject path.Invariants Check (key lines; no ✗ unresolved)
Server-authoritative ✓ (authoritative compact map). Protocol versioned ✗→✓: the connect hash handshake IS a wire change → bump
ProtocolVersion.Currentin 7c. Single-threaded sim ✓ (warm-load + validate before accepting connections; O(1) in-memory queries; no file/decompress in the tick; workers touch only immutable data).World.csHARD GATE ✓ (map access delegated; compact backing inside the map component).ScreenHARD GATE ✓ (streaming/LRU in a component;GameScreenwires). Client engine-independence ✓ (window math inClient.Core/IsoViewport; GPU glue inClient). Persistence (GameServer) ✓ (map is authored content, distinct fromworld.sav; no tick I/O). Typed options ✓ (world path/stream budget onGameOptions). Broadcasts/AoI ✓ (unchanged; absolute positions from the compact map). Multi-platform ✓. Assets ✓. Docs+DoD ✓ (architecture.md runtime map path; world-map-format.md handshake).Server-side validation
Map+catalog hash handshake (mismatch → diagnostic notice, not silent desync) — the one wire addition →
ProtocolVersionbump. Walkability/LOS/Z stay server-decided from the compact authoritative map. Release validation: server refuses a world with no in-bounds walkable spawn.Verification plan
7a: parity harness (
CompactWorldMapvsTileMapcell-for-cell on the same content) + startup warm-load/validate + compact memory (nostring[,]); no screenshots. 7b: piloted client screenshots on the chunked world (fresh DB) + region-border correctness. 7c: WebSocket integration test (mismatched hash rejected; matching connects).Fase 7b — design review (client renders the chunked asterra world in-game)
Verdict: go-with-changes. Owner picked warm-load whole (client holds asterra in a
CompactWorldMap, ~127 MiB; asterra is flat → no Z sidecar). Reuses 7a's tested code; renderers already window (IsoViewport.VisibleCellBounds), so a future chunk-streaming loader is a drop-in behindIReadOnlyTileMap. No wire change (server already sends absolute positions; the map/catalog hash handshake is 7c).Invariants Check (each line of
CLAUDE.md## Design checklist):ProtocolVersionbump (handshake = 7c) · String catalog N/A (load failure = fatal startup log) · Single-threaded sim N/A (server-side) · World.cs gate N/A · Screen gate ✓ (map load inIsoGamecomposition root; windowing stays in the renderers/ClientWorld, notGameScreen) · Client engine-independence ✓ (window math already inClient.Core;CompactWorldMapinShared) · Gameplay/Networking sep ✓ · Act-on-instance N/A · Extend-by-type ✓ (source chosen by path/config, not a type switch) · Typed content-def N/A · Server-paced N/A · Persistence N/A (authored content, notworld.sav) · Process sep ✓ · Typed options: client hardcodes asterra + a dev--world/ISOMMO_WORLDoverride (notIConfigurationin a service) · Broadcasts/AoI N/A · Multi-platform ✓ (pure .NET; cross-platform paths) · Assets ✓ (realland/<landArt>or biome tint, same path as today) · Asset naming N/A · ModernUO: UO streams map blocks around the player — we consciously diverge (warm-load whole) for this slice, streaming is the aligned follow-up · Docs & DoD ✓ (updateworld-map-format.md; screenshots required).Server-side validation: nothing new — 7b is pure rendering; the trust boundary is unchanged from 7a (every move validated against the server's asterra).
Verification plan: screenshots REQUIRED (visible) — fresh DB →
just dev→ pilot the client, player spawns on asterra (~5183,4378), capture the rendered continent/coast; confirm a step into ocean is still server-rejected; whole-solution build 0 warnings +dotnet testgreen.Definition of Done (7b, delta)
world.map), proven by a piloted screenshot from a fresh DB.ProtocolVersionbump and no wire-DTO change in 7b (handshake deferred to 7c).IReadOnlyTileMap; a dev override still loads the small legacy map.dotnet testgreen.Marco aiutami invece di farti le unghie a tema pokemon 💅
Epic completata: formato chunked + migrazione editor/client/server su main (#230–#236). Il mondo
asterragira sul formato chunked.