Backpressure & Budgets
A slow or hostile client costs itself, never the tick or the other sessions.
Status: โ Implemented ยท Visibility: Public ยท Level: ๐ฃ Advanced ยท Category: Subscriptions
๐ฏ What it solves
Clients are slow, lossy, far away or malicious. A server that queues frames for a slow client grows without bound; one that waits for it stalls everybody; one that accepts whatever a client sends can be driven off its tick.
โ๏ธ How it works (in brief)
- Skip, never queue. A session has two frame slots (one in flight, one ready). With both busy it is skipped: nothing is gathered
for it and nothing about it moves. Its next frame carries the union of everything it missed, replayed from an 8-tick push log; past
it, a
RESETre-delivers its view. - Rate classes. A session that keeps being skipped is served every 2nd, then every 4th tick, then closed with 1013 at
CloseStalledAfter. A profile can also be served every 2 or 4 ticks by declaration (Every). Each step, down or back up, reaches the application as aSessionEventKind.Degradedcarrying the level (SessionEvent.DegradeLevel: 1, 2, or 0 for full rate again) โ the place to shrink that player's interest or show a connection warning. - Acknowledgement-based lag. A send completing is not proof of delivery; clients report their last applied tick in
PING(4 Hz), and a session too far behind is skipped. A client that stops pinging is closed with 4001. - Per-session byte budget (
Session(id).SetBudget(bytesPerSecond)orSessionLimits.BytesPerSecond): the engine raises the session's detail level โ longer periods for far bands, a smaller enter budget โ and, at the last level, shrinks its radius; it never drops state. - Frame budget. A frame carries what fits
FrameBytes(or a lowerSessionLimits.FrameBytes); the records that do not fit are owed to the next frames, oldest first, and sent with the entity's state as it is then โ a burst of joins or updates is spread, never refused or lost. The view's fill is paced by bytes as well as by the enter count, and noGROUP_COMPLETEorVIEW_COMPLETEarrives while an enter is owed. A session owing more than four enter budgets is refilled by aRESET. - Overload. When the runtime is overloaded, every Sphere session is served a level lower and every profile half as often.
- Inbound. Each session's inbound byte budget (
IngressBytesPerSecond, required with commands) refuses over-budgetCOMMANDSwhole; refused commands pastAbuseRefusalsPerWindowforAbuseWindowsadjacent windows close the session with 1008. - Pools. Frames and per-entity state live in native pools under
FramePoolBudgetBytesandStatePoolBudgetBytes(the latter per observed archetype); an exhausted frame pool skips sessions (and they catch up), an exhausted state pool defers clusters (and they arrive once a block frees). Neither allocates past its budget.
| Option | Default | Effect |
|---|---|---|
CloseStalledAfter |
3 s | Skip run after which a session is closed with 1013 (degraded first) |
LagSkipRttAllowanceMs |
250 | RTT allowance of the acknowledgement-based lag skip |
FrameBytes |
256 KiB | Largest frame a session may be sent; what does not fit is owed to the next frames. An archetype whose enter record could exceed half of it is refused at Start |
FramePoolBudgetBytes / StatePoolBudgetBytes |
256 MiB / 256 MiB per archetype | Native pool ceilings |
IngressPoolBudgetBytes |
64 MiB | Inbound rings, all sessions |
MaxGroupsPerSession |
required with a group set (โค 1 048 576) | Keys a session holds per set, explicit requests and server keys together; a bound, not a reservation โ a session costs the keys it holds (Groups) |
ReliableEventBytes |
required with a reliable event | The reliable ring's budget; an emit past it is refused, Emit returning false (Events) |
โ ๏ธ Guarantees & limits
- A slower session loses rate, never state; records are absolute and skips are unions (SUB-03).
- An idle tick is not a skip (SUB-15). A client that cannot finish one frame within
CloseStalledAfteris closed with 1013 within that bound, degraded or not; a client that finishes frames, however slowly, is served at the rate it drains and is not closed. - A server stall is not the client's (SUB-15). Silence and the skip run are measured in time that passed, not in ticks, so a GC pause or a slow disk โ and the ticks the runtime then runs back to back to catch up โ neither closes nor degrades a session that kept talking.
- An exhausted state pool defers, never drops (SUB-32): a pushed cluster with no block is offered again every tick and replicated the
first tick a block frees. Its entities are invisible meanwhile, so the budget is still a sizing decision. Refusals are counted
(
RentRefusalsin the resource metrics) and the start and end of an exhaustion are logged. - Connection churn is not rate-limited by the engine. Measured on the SWG demo (1 000 player sessions, 50 Hz), ~2 000 connect/close cycles a second from one host add about 5 ms to the tick's p99, all of it GC from ASP.NET Core's per-connection objects; on one Windows box that rate also fills the TCP TIME_WAIT table in about 30 s and stalls the whole machine for seconds. Put connection-rate limiting in front of a public endpoint.
๐งช Tests
- SendPumpTests โ skip, degrade, close
- PushLodLevelTests โ detail levels under a budget
- SlowClientTests โ a stalled client closed within the bound, a slow one served at its rate, a 200 ms one never degraded, every degrade step told to the application
- SilenceUnderStallTests โ 1.5 s and 4 s server stalls close and degrade no one
- IngressHardeningTests โ inbound budget and abuse
๐ Related
- Related feature: Overload Management
- Commands & acknowledgements