Table of Contents

Parallel Entity Processing (QuerySystem.Parallel)

Multi-core entity chunking that skips per-chunk Transaction overhead for non-Versioned writes.

Status: ✅ Implemented · Visibility: Public · Level: 🟣 Advanced · Category: Runtime

🎯 What it solves

A game server with dozens of cores but only a handful of systems can't get throughput from DAG-level parallelism alone — most of the win has to come from splitting one system's entity set across workers. Doing that naively (materialize the full entity list, give every chunk its own Transaction) burns cycles on list-copying and per-chunk Transaction setup/commit before any game logic runs. Parallel entity processing removes both costs for the common case (writing non-Versioned components), so b.Parallel() scales a QuerySystem without changing its Execute loop.

⚙️ How it works (in brief)

The runtime picks one of four dispatch paths from two facts about the system: whether it has a ChangeFilter and whether it declares WritesVersioned(). Non-Versioned systems get a reusable PointInTimeAccessor (PTA) — attached once per tick to a fresh MVCC snapshot — and each worker gets its own EntityAccessor from it, so entity access has no per-entity dictionary lookup and no Transaction commit. Unfiltered non-Versioned systems go further: the entity set is never materialized into an array — each chunk iterates a contiguous slice of the View's underlying hash map directly. Filtered systems (and any system writing Versioned components) materialize a list instead — dirty-only when filtered, full-set otherwise — and Versioned writers get a real per-chunk Transaction, since the PTA cannot perform copy-on-write.

Path Selected when Prepare cost Chunk entity access
1 — Full, non-Versioned no ChangeFilter, no WritesVersioned() O(1) — no list built ctx.Accessor (PTA), zero-copy hash-map slice
2 — Filtered, non-Versioned ChangeFilter set, no WritesVersioned() O(dirty count) ctx.Accessor (PTA), pooled slice
3 — Full, Versioned no ChangeFilter, WritesVersioned() O(entity count) ctx.Transaction, pooled slice
4 — Filtered, Versioned ChangeFilter set, WritesVersioned() O(dirty count) ctx.Transaction, pooled slice

Chunk count. A system's first dispatch is split by entity count: min(round(WorkerCount × ChunksPerWorker), ceil(entities / ParallelQueryMinChunkSize)). From the second dispatch on (RuntimeOptions.CostBasedChunking, on by default), the split follows what the system measurably cost the time before. Its chunks spread over round(WorkerCount × ChunksPerWorker) while each carries 25–100 µs of work; below that it gets fewer chunks, above it up to twice as many. Workers claim the extra chunks as they come free, so the pool no longer waits on one slow last chunk.

💻 Usage

public class MovementSystem : QuerySystem
{
    protected override void Configure(SystemBuilder b) => b
        .Name("Movement")
        .Input(() => activeUnitsView)
        .Parallel();                       // Path 1: no ChangeFilter, no WritesVersioned

    protected override void Execute(TickContext ctx)
    {
        foreach (var id in ctx.Entities)
        {
            var entity = ctx.Accessor.OpenMut(id);     // PTA — no per-entity dictionary lookup
            var pos = entity.Read<EcsPosition>();
            pos.X += entity.Read<EcsVelocity>().X * ctx.DeltaTime;
            entity.Set<EcsPosition>(pos);
        }
    }
}

// Path 4 (filtered + Versioned write): same Configure, plus
// .ChangeFilter(typeof(EcsBuffs)).WritesVersioned()
// — Execute then uses ctx.Transaction.OpenMut(id) instead of ctx.Accessor.
Option Default Effect
b.Parallel() off Enables chunking; required for any of the four paths
b.ChangeFilter(...) none Narrows the dispatched set to dirty/Added entities (Paths 2/4)
b.WritesVersioned() off Switches chunk access from ctx.Accessor to a per-chunk ctx.Transaction (Paths 3/4)
b.ChunksPerWorker(factor) 1.0 Oversubscribes chunk count beyond worker count, range [1.0, 64.0]

⚠️ Guarantees & limits

  • ctx.Accessor reads all storage modes (Versioned via MVCC chain walk, SingleVersion/Transient direct) and writes SingleVersion/Transient, but throws on a Versioned write — declare WritesVersioned() instead.
  • ctx.Accessor cannot Spawn, Destroy, Commit, or Rollback. Use ctx.Commands to queue deferred spawns and destroys from inside a parallel chunk; the engine applies the buffer at the fence phase after all chunks complete. Commit and Rollback remain exclusive to ctx.Transaction.
  • A parallel system's ctx.ChunkCount can change from one tick to the next and reach twice round(WorkerCount × ChunksPerWorker). Never size per-chunk state by it; index per worker (ctx.WorkerId).
  • A dispatch's chunks tile the cluster list as it stood when the dispatch began: clusters a spawn appends while they run are walked from the next tick (rule CD-02).
  • Path 1 vs. a per-chunk Transaction: ~2.2x lower per-chunk overhead (PTA ~380µs/chunk vs. Transaction ~850µs/chunk) — only declare WritesVersioned() when actually needed.
  • All workers in a tick see the same frozen MVCC snapshot (one TSN per Attach()); the PTA is reused across ticks and across all non-Versioned parallel systems with zero steady-state allocation.
  • Scaling is good up to one CCD's worth of cores; cross-CCD EntityMap access flattens the curve on multi-CCD hardware (measured: ~89% efficiency at 8 workers, ~42% at 16 on a 2-CCD part).
  • The DAG, not the runtime, guarantees chunks across different parallel systems don't race.
  • Out of scope: cross-system pipelining (next system's chunk K starting before this system fully completes).

🧪 Tests

  • ParallelQueryTests — all four dispatch paths (ParallelQuery_NonVersioned_ChunkReceivesAccessor, ParallelQuery_WritesVersioned_ChunkReceivesTransaction), chunk partitioning, chunk-throw isolation
  • ChunksPerWorkerTests — ChunksPerWorker oversubscription factor vs. worker-count cap and entity-count cap
  • CostChunkingTests — RuntimeOptions.CostBasedChunking: the cost rule's grain, an expensive system split past the worker count, the entity rule kept when it is off or when a system sets MinChunkSize
  • ChunkClusterRangeTests — a dispatch's chunks tile the cluster list Prepare counted, even when a spawn grows it while they run (CD-02)

🔗 Related

Edit this page