Replication Diagnostics
See what replication costs and what each client receives, from the server and from the client.
Status: ✅ Implemented · Visibility: Public · Level: 🟣 Advanced · Category: Subscriptions
🎯 What it solves
A replication problem shows up on a client — a stale entity, a stuttering mover, a slow frame — far from where it is caused. Clients and tools need the server's own numbers alongside the stream, and developers need a way to catch a forgotten push.
⚙️ How it works (in brief)
STATScapability. A client that requests it receives, about once a second, a dense snapshot of the catalog's metrics: built-ins (typhon.tick.p50/p99/max,typhon.system.meanper system,typhon.archetype.entities,typhon.sessions,typhon.net.outBytesPerSec,typhon.subscriptions.track.p99,typhon.durability.wait.p99,typhon.sessions.closedby reason andtyphon.frames.skippedby cause, and per sessiontyphon.session.outBytesPerSec,skippedFrames,droppedCommands) and your own.typhon.durability.wait.p99is the per-tick wait — the Unit-of-Work flush, which in WAL mode isRequestFlush+WaitForDurablefor every record the tick produced together — and not a per-commit figure; read againsttyphon.tick.p99it says how much of a tick goes to durability rather than to compute. Every*.p99is a percentile over the last ten seconds of ticks (300 at 30 Hz);typhon.tick.maxis the longest tick of the last second, andtyphon.tick.p50the last second's median. Over one second a p99 of 30 samples is simply the maximum, which is why the two are separate.- Application metrics:
subs.Metric(name, unit, codec, source)(server scope, optionally labelled) andsubs.SessionMetric(…)(per session); the server segment is encoded once for every subscriber. - The same numbers without a client:
TyphonRuntime.ReadStats()returns tick p50/p99 against the target, the overrun count, the per-tick durability wait, per-system means, entities per archetype and the session figures — reading the telemetry ring and the engine directly, so it answers on an engine with no replication and no declared metrics.MapTyphonStats()(Typhon.Subscriptions.AspNetCore) serves it as JSON at/typhon/stats.json. It has no authentication and publishes system and archetype names: restrict it as you would an admin route. - Why sessions closed.
ReadStats().SessionsClosed(per reason: left, link lost, kicked, and the engine's own: silent 4001, lagging 1013, protocol, abusive, internal) andReadStats().FramesSkipped(per cause: in flight full, lagging, pool refused, oversize, gated, owed reset), cumulative, the same numbersSTATScarries. And one warning a second at most, only in a second the engine itself closed sessions: how many, every reason and skip cause of that second, and the longest tick interval. Silent closes with normal ticks are clients that stopped talking; a long tick interval is the server stalling; lag closes with in-flight-full skips are clients that cannot keep up. DEBUGcapability, granted only to sessions admitted withSessionLimits.AllowDebug(SessionLimits.God): engine sub-blocks such as the replication grid and a session's push geometry, for inspectors and recorders.- Push validator:
TYPHON_PUSH_VALIDATE=N— see Push replication. - Trace records for the profiler and the Workbench, each behind its own gate under
Typhon:Profiler:Subscriptions: the server and per-session figures once a second (kinds 68, 69), and one session's story as it happens — every frame published (71, the volume one), every frame skipped with its cause (72), every open, close and degrade step (73). See the telemetry flags. - Runtime counters on
ctx.Subscriptions(projection blocks, enter/leave flow, identity flow, send path) for server-side logging.
💻 Usage
subs.Metric("game.creatures.alive", "count", Codec.VarUInt, () => aliveCount);
subs.SessionMetric("game.session.ping", "ms", Codec.F16, session => PingOf(session));
⚠️ Guarantees & limits
- A skipped session loses nothing: gauges are windowed and counters cumulative.
- Metric names under
typhon.are reserved; a metric is refused atStartif its name collides with a built-in. DEBUGexposes server structure (grid, clusters, migrations); never grant it to untrusted clients.
🧪 Tests
- SubscriptionsSessionRecordsTests — a session's records, opened to closed (gated: run with
TYPHON__PROFILER__SUBSCRIPTIONS__ENABLED=true) - StatsBlockTests (one close counted alike through
STATS,ReadStatsand the warning) · DebugBlockTests
🔗 Related
- Related feature: Telemetry & Runtime Inspection
- Wire protocol & catalog