09 — Querying

Code: src/Typhon.Engine/Querying/

Querying is the read side of Typhon's ECS data model. Application code asks "give me the entities matching these constraints"; the engine turns that into an execution plan, scans the smallest possible amount of state, and streams matching primary keys into a caller-owned container. On top of that one-shot pipeline sits a view system — long-lived, incrementally maintained sets of entity IDs that get push-notified by writers and report Added / Removed / Modified deltas back to the consumer.

This doc covers both layers. The query builder (EcsQuery<TArchetype>) actually lives in Ecs/ because it's archetype-shaped — but every plumbing piece behind it (ExpressionParser, PlanBuilder, ExecutionPlan, FieldEvaluator, PipelineExecutor, statistics) lives in Querying/. The view types (IView, ViewBase, ViewDelta, ViewDeltaRingBuffer, ViewRegistry) live in Querying/ too. EcsView<TArch> lives in Ecs/ for the same reason as EcsQuery.

_{Query engine components: EcsQuery (fluent API) → ExpressionParser → PlanBuilder → PipelineExecutor, feeding the view layer (ViewBase._entityIds, ViewDelta + DeltaKind, ViewRegistry / IndexMaintainer) and reading from ComponentTable / B+Tree indexes / statistics.}

1. Overview — the two pipelines

There are two distinct query paths and one shared planner.

Path	Trigger	Lifetime	Output
One-shot `Execute()`	`tx.Query<T>()....Execute()`	Runs once, allocates a `HashSet<EntityId>`, returns	Snapshot at `tx.TSN`
View (`.ToView()`)	`tx.Query<T>()....ToView()`	Registers with `ViewRegistry`, lives until `Dispose()`	Maintained `HashMap<long>` + per-tick `ViewDelta`

Both go through the same machinery:

ExpressionParser — Expression<Func<T, bool>> lambdas are normalized to Disjunctive Normal Form (OR of ANDs), one FieldPredicate[] per OR branch.
QueryResolverHelper.ResolveEvaluators — FieldPredicate (string field name + op + boxed value) becomes FieldEvaluator (16 B: field index, key type, op, byte branch index, 8-byte widened threshold). String64 predicates are rejected here (QueryResolverHelper.cs:93) — KeyType.String64 exists for the index/statistics layer only; no scalar comparator for it.
PlanBuilder — picks the most selective indexed field as the primary scan (via AdvancedSelectivityEstimator), narrows its key range, orders the rest of the evaluators by ascending estimated cardinality (most selective first → short-circuit), builds an ExecutionPlan.
PipelineExecutor — iterates the primary B+Tree, walks the MVCC revision chain (or reads SV/Transient component data directly), evaluates non-primary predicates inline, streams matching entity PKs into a caller-provided HashMap<long> / List<long>.

Views layer on top of (4) by registering an IView with the per-ComponentTable ViewRegistry. When the index maintainer commits a mutation that crosses a registered field's boundary, the registry routes a ViewDeltaEntry into each affected view's lock-free MPSC ViewDeltaRingBuffer. The view's Refresh(tx) drains the ring buffer and reconciles its HashMap<long> against the new state.

_{The pipeline executor's five steps: select the most-selective primary index → stream entity IDs via index scan → check secondary predicates → materialize components → add to the view's entity set (ViewBase._entityIds).}

2. Query API — `EcsQuery<TArchetype>`

Ecs/public/EcsQuery.cs

Constructed via tx.Query<TArchetype>() (polymorphic — includes descendant archetypes) or tx.QueryExact<TArchetype>() (only the exact archetype). The struct is mutated fluently; each call returns the modified struct value.

Three tiers of constraints

Tier	Method	What it filters	Cost
T1 — Archetype	`With<T>()`, `Without<T>()`, `Exclude<TArch>()`	Mask over `ArchetypeId` (256-bit or large bitmap)	One bit test per archetype
T2 — Enabled bits	`Enabled<T>()`, `Disabled<T>()`	Per-entity 16-bit enable mask (max 4 of each kind per query)	Cluster occupancy mask AND
T3 — Field predicates	`Where<T>(Func<T,bool>)`, `WhereField<T>(Expression<Func<T,bool>>)`, spatial predicates	Per-entity component values	Index scan + predicate eval

Where<T>(Func<T, bool>) takes a compiled delegate — opaque to the planner, evaluated post-scan via tx.Open(id).TryRead<T>(...). It chains as AND.

WhereField<T>(Expression<Func<T, bool>>) is the planner-visible form. The lambda gets parsed by ExpressionParser, broken into DNF, indexed-field-resolved, and folded into the execution plan. Multiple WhereField calls on the same component cross-product as AND of ORs (DNF normalization at EcsQuery.cs:299-321); a second call on a different component throws InvalidOperationException — use .Where<T>(lambda) for the second component. Required for incremental views.

Spatial predicates

query.WhereInAABB<Position>(minX, minY, minZ, maxX, maxY, maxZ);
query.WhereNearby<Position>(centerX, centerY, centerZ, radius);
query.WhereRay<Position>(originX, originY, originZ, dirX, dirY, dirZ, maxDist);

The component type T must be tagged [SpatialIndex] (see 07-spatial). Only one spatial predicate per query (the inline 7-double parameter array is overwritten on repeated calls). At execution, the spatial index produces candidate entity IDs and the rest of the query (archetype mask + visibility + remaining predicates) filters them.

`OrderBy` / `Skip` / `Take`

query.WhereField<Position>(p => p.X >= 0)
     .OrderByField<Position, int>(p => p.Level)
     .Skip(100).Take(50)
     .ExecuteOrdered();

OrderByField/OrderByFieldDescending require a prior WhereField to identify the component table. The OrderBy field must be indexed. Skip / Take require OrderBy (without it the result set is unordered).

Two ordered execution paths inside ExecuteOrdered. The choice turns on whether every archetype in the query mask carries per-archetype indexes on the where-component — HasClusterIndexes set and a non-null ClusterState.IndexSlots — not on how its entities are stored:

Every archetype does: K-way merge across each archetype's per-archetype B+Tree (ExecuteOrderedClustered). Each stream yields keys in sort order; the merge interleaves them. Early termination once Skip + Take entries are collected.
Any archetype does not: ExecuteTargeted() then sort by the OrderBy field (ExecuteOrderedViaSortFallback). O(n log n) — acceptable for a case the schema has to go out of its way to produce.

`EcsNavigationQueryBuilder<TArch, TSource, TTarget>`

Ecs/public/EcsNavigationQueryBuilder.cs

tx.Query<Worker>()
  .NavigateField<Worker, Building>(w => w.HomeBuildingId)
  .Where((src, tgt) => src.Active && tgt.Level >= 5)
  .Execute();

FK-join query. Created from EcsQuery.NavigateField<TSource, TTarget>(fkSelector). The FK selector must reference a long field on the source component. Wraps a plain NavigationQueryBuilder<TSource, TTarget> for the planner-side resolution. Execute() scans each target archetype's EntityMap, looks up the FK index on the source table, and joins. ToView() registers with both source and target ViewRegistry (deletions to the target invalidate the view, which is why target predicates are mandatory).

3. Plan building

`ExpressionParser` — DNF with a 16-branch cap

Querying/internals/ExpressionParser.cs

ParseDnf<T>(Expression<Func<T, bool>>):

Build an AST from the expression tree.
Apply DNF rewrite (distribute && over ||).
Collect leaves into FieldPredicate[][] — outer = OR branches, inner = AND predicates per branch.

Branch cap: MaxDnfBranches = 16 (ExpressionParser.cs:23). DNF normalization can blow up exponentially — (A||B) && (C||D) && (E||F) produces 2×2×2 = 8 branches; five ANDed pairs would explode past 16. Hit the cap → InvalidOperationException with a message telling you to split the query or reduce ANDed OR pairs. The cap aligns with the OR-mode view's 16-bit branch bitmap (see §5).

`PlanBuilder`

Querying/internals/PlanBuilder.cs

Singleton (PlanBuilder.Instance). Two stages:

OrderBySelectivity — calls AdvancedSelectivityEstimator.EstimateCardinality(table, fieldIdx, op, threshold) for each evaluator, sorts evaluators ascending by estimated count.
BuildPlanWithPrimarySelection — picks the most selective indexed evaluator as the primary scan stream, computes its [scanMin, scanMax] range, builds the ExecutionPlan.

BuildPlanAttributed is the production entry point — accepts the originating query's identity and source-location info for trace attribution (Phase 7 observability — see 12-observability).

`ExecutionPlan`

Querying/public/ExecutionPlan.cs

readonly struct (16-byte fixed fields + two arrays):

Field	Meaning
`PrimaryFieldIndex`	Index into `ComponentTable.IndexedFieldInfos`. `-1` means the planner couldn't pick a secondary index.
`PrimaryKeyType`	`KeyType` of the primary index. Valid only when `PrimaryFieldIndex >= 0`.
`PrimaryScanMin` / `PrimaryScanMax`	Long-encoded inclusive bounds for the primary scan.
`Descending`	Iterate primary index in reverse order.
`OrderedEvaluators`	All evaluators, sorted by ascending estimated cardinality (selectivity-first).
`EstimatedCounts`	Per-evaluator cardinality (parallel array).
`UsesSecondaryIndex`	`PrimaryFieldIndex >= 0`.

When the planner cannot narrow a range. SelectPrimaryStream picks a range where the predicate allows one; an all-!= predicate defeats it, because NotEqual is excluded from range selection. BuildPlanWithPrimarySelection then calls SelectFullScanStream, which picks any indexed field the predicate references and scans its full type range, leaving every evaluator to filter — so the typical NE-only query still comes out with a non-negative PrimaryFieldIndex. Two cases produce -1: OrderBy PK (orderByFieldIndex == -1, because no secondary index reproduces PK order) and the degenerate case where no evaluator carries a usable index, which WhereField makes unreachable by rejecting non-indexed fields.

A -1 plan is still executable, and PrimaryFieldIndex decides scan shape, not whether a scan happens. Every archetype is cluster-backed and carries per-archetype B+Trees, and EcsQuery dispatches straight to them. PipelineExecutor holds only predicate evaluation (EvaluateFilters) and foreign-key ordinal lookup (FindFKIndexOrdinal); it runs no scan and counts nothing, so no plan shape can route a query into an empty result through it:

Query shape	Entry point
Unordered, plan proposes a stream and the primary index's fan-out is high enough (many rows per distinct key)	`ExecuteTargeted` → `ScanPerArchetypeBTreeSelective` — Path A: range-scan the archetype's B+Tree, then verify the remaining predicates on the matched slots only
Unordered, otherwise	`ExecuteTargeted` → `ScanPerArchetypeBTree` — Path B: zone-map-prune each cluster and evaluate every predicate against the SoA column
Ordered, and every archetype in the mask carries per-archetype indexes on the where-component (`HasClusterIndexes`, non-null `IndexSlots`)	`ExecuteOrderedClustered` — K-way merge over the OrderBy field's per-archetype B+Trees
Ordered, and any archetype in the mask lacks them	`ExecuteOrderedViaSortFallback` — scan, then sort
`Count()`	`ExecuteTargeted().Count` — the same scan, then its size

All of them scan real data and return correct results (src/Typhon.Engine/Ecs/public/EcsQuery.cs). In particular, a query is never empty and Count is never 0 merely because of plan shape.

Path A vs Path B is a performance decision only. Both return the same entities, including the same MVCC born/died visibility gate on a Versioned archetype, and the planner chooses between them on the primary index's fan-out (rows per distinct key): Path A is taken when fan-out is high enough that per-key tree cost beats the per-cluster zone-map pass; at low fan-out Path B wins on every layout measured. A Transient index home always takes Path B, whose body never touches a tree and is therefore correct for either home. The equality of the two paths is asserted rather than assumed — QueryPathEquivalenceTests runs a key-type × operator × sign matrix through both with the choice forced, because while they agreed only by convention no end-to-end test could have told the difference.

`FieldEvaluator`

Querying/public/FieldEvaluator.cs

[StructLayout(LayoutKind.Sequential, Pack = 1)]
public struct FieldEvaluator  // exactly 16 bytes
{
    public byte FieldIndex;      // index into IndexedFieldInfos (max 63)
    public byte FieldSize;
    public KeyType KeyType;      // Bool..Double; String64 reserved (see below)
    public CompareOp CompareOp;  // ==, !=, <, <=, >, >=
    public byte ComponentTag;    // 0 = T1, 1 = T2 (multi-component views)
    public byte BranchIndex;     // DNF branch (0 for AND view, 0..15 for OR view)
    public ushort FieldOffset;   // byte offset within component
    public long Threshold;       // 8-byte widened constant; float/double via reinterpret
}

Evaluate(ref FieldEvaluator eval, byte* fieldPtr) dispatches on KeyType to the right primitive comparator and short-circuits on the first false. [MethodImpl(AggressiveInlining)] on every step.

Why no String64 evaluator: the Threshold slot is 8 bytes — not large enough to hold a 64-byte String64. Predicates with string fields are rejected at ResolveEvaluators time by QueryResolverHelper.MapFieldTypeToKeyType (throws NotSupportedException for FieldType.String). The String64BTree still exists as an index — it just can't be used as a scalar predicate target. The KeyType.String64 tag survives in IndexStatistics to flag "no selectivity stats for this index".

4. Execution — `PipelineExecutor`

Querying/internals/PipelineExecutor.cs

Singleton (PipelineExecutor.Instance). Three public methods:

void Execute(ExecutionPlan, FieldEvaluator[], ComponentTable, Transaction, HashMap<long> result);
void ExecuteOrdered(ExecutionPlan, FieldEvaluator[], ComponentTable, Transaction, List<long> result, int skip, int take);
int  Count(ExecutionPlan, FieldEvaluator[], ComponentTable, Transaction);

The caller owns the HashMap<long> / List<long> lifecycle. One-shot queries allocate a fresh container; views reuse ViewBase._entityIds. There is no internal buffering — matching entity PKs are streamed from the B+Tree iterator directly into the caller's container as they pass the filter chain.

Three storage-mode branches

ExecuteCoreSecondaryIndex dispatches on table.StorageMode:

Versioned → ExecutePKsTypedVersioned. The index value is the compRevFirstChunkId of the entity's revision chain. The pipeline reads EntityPK from CompRevStorageHeader (one cache line), checks MVCC visibility via tx.IsEntityVisible(entityPK), and only if non-primary filters exist does it walk the revision chain via RevisionChainReader.WalkChain(tx.TSN) to resolve the current component content chunk for predicate evaluation. No EntityMap re-lookup — the index entry already points at the chain.
SingleVersion → ExecutePKsTypedSV. Index values are component chunk IDs. EntityPK is read from offset 0 of the chunk's inline overhead. No revision chain; no MVCC — SV reads see the current in-place value.
Transient → ExecutePKsTypedNonVersioned<TKey, TransientStore>. Same shape as SV but using TransientStore accessors. Transient data lives in TransientComponentSegment and never persists.

The SV and Transient paths share the ExecutePKsTypedNonVersioned<TKey, TStore> generic — TStore is the only thing that differs structurally.

Iterator shape

For an AllowMultiple = true index, the B+Tree exposes EnumerateRangeMultiple(min, max) / EnumerateRangeMultipleDescending — one outer MoveNextKey per distinct key, an inner do { var values = enumerator.CurrentValues; ... } while (NextChunk()); for all chunks of multi-values at that key. For unique indexes it's a plain foreach (var kv in EnumerateRange(min, max)). Either way the executor streams entries through the per-entity check function (ExecuteOneVersioned / inline-evaluator-loop for SV/Transient).

`ExecuteFullScan` (it's not on `PipelineExecutor`)

The wrapper ExecuteFullScan(plan, evaluators, table, tx, HashMap<long> result) lives on EcsViewFieldReader (Ecs/public/EcsView.cs:864), not on PipelineExecutor. The typed subclass EcsViewFieldReader<T> simply delegates to PipelineExecutor.Instance.Execute(...). It exists as a thin abstraction so EcsView can call the executor without knowing the component type — the EcsView is parameterized by TArchetype, the field reader carries the component T. ExecuteOrderedScan and CountScan follow the same pattern.

What's not there (audit clarifications)

There is no "batch 256 PKs into a stackalloc buffer" mechanism. There is no Batch or BatchSize constant in PipelineExecutor. Streaming is per-entry, directly from the B+Tree enumerator into the caller's container.

5. View system

Querying/public/IView.cs, Querying/public/ViewBase.cs, Ecs/public/EcsView.cs

`IView` — registry-facing contract

public interface IView
{
    int ViewId { get; }                // monotonic per-process
    int[] FieldDependencies { get; }   // indices into ComponentTable.IndexedFieldInfos
    bool IsDisposed { get; }
}

The IView surface deliberately omits the internal ViewDeltaRingBuffer — that's supplied separately at registration time. The registry holds the buffer but not the view: storing the IView reference would root the object and prevent the GC from collecting an abandoned view, keeping its buffer subscribed and receiving commit deltas indefinitely (#865).

`ViewBase` — shared state

public abstract class ViewBase : IView, IDisposable, IEnumerable<long>
{
    internal readonly HashMap<long> _entityIds = new();                      // POH-allocated entity-PK set
    private readonly Dictionary<long, DeltaKind> _deltas = new(16);          // per-tick delta map
    protected readonly FieldEvaluator[] Evaluators;
    internal ViewDeltaRingBuffer DeltaBuffer { get; }
    // ViewId, FieldDependencies, IsDisposed, SourceFile/Line/Method (caller-attribute capture)
}

The XML doc comment on ViewBase (line 10) references View<T>, View<T1,T2> and OrView<T> — no such types exist. Don't propagate the names. The single concrete derived type is EcsView<TArchetype>.

_entityIds is the maintained working set. _deltas accumulates per-tick DeltaKind (Added / Removed / Modified) per PK; ClearDelta() drops the map. Idempotent compaction is done inline in CompactDelta: Added+Removed cancels, Modified+Removed promotes to Removed, Removed+Added becomes Modified.

`EcsView<TArchetype>`

Five constructors, five modes:

Mode	Trigger	Behavior
Membership	`EcsQuery` with no predicate (no `WhereField`, no `.Where`, no spatial, no T2 `Enabled`/`Disabled`)	Registers with `ArchetypeMembershipRegistry` (not `ViewRegistry`). `Refresh()` checks a structural epoch (O(1) gate); drains the spawn/destroy channel when it moved. Never re-executes the full query on quiet ticks.
Pull	`EcsQuery` without `WhereField` but with an opaque `.Where(lambda)`, a spatial predicate, or a T2 `Enabled`/`Disabled` filter	No evaluators, no `ViewRegistry` registration. `Refresh()` re-executes the full query every call.
Incremental (single AND branch)	One `WhereField` call	Single `ExecutionPlan`, ring buffer driven, registered with `ViewRegistry`.
Incremental (cached plan)	Same as above, with plan reuse for descriptor emission	Caches the plan in `ViewBase._cachedPlans` for per-tick `QueryPlan` span emission.
OR mode	Multiple DNF branches	One `ExecutionPlan` and one `FieldEvaluator[]` per branch. Per-entity `ushort` branch bitmap (`_branchBitmaps`) records which branches that entity currently satisfies.

EntityId-facing surface:

EntityIdEnumerator GetEntityEnumerator() — over the internal HashMap<long> keys cast to EntityId.
IReadOnlyList<EntityId> Added, Removed — caches rebuilt from ViewBase._deltas on each Refresh(), so consumers can foreach typed handles instead of raw longs.
bool Contains(EntityId) / Contains(long), int Count, bool HasChanges.

`DeltaKind` and `ViewDelta`

Querying/internals/DeltaKind.cs, Querying/public/ViewDelta.cs

internal enum DeltaKind : byte { Added = 0, Removed = 1, Modified = 2 }

public readonly struct ViewDelta
{
    public readonly DeltaView Added;     // O(1) Count, O(1) Contains
    public readonly DeltaView Removed;
    public readonly DeltaView Modified;
    public bool IsEmpty { get; }
}

ViewDelta is a zero-allocation read-only handle over ViewBase._deltas. DeltaView is a filtered IReadOnlyCollection over the same dictionary that only enumerates entries matching its DeltaKind filter. Lifetime is "until the next ClearDelta()" — do not cache across refresh cycles.

OR-mode bitmaps

When an OR view has N branches (1 ≤ N ≤ 16), each entity in _entityIds also has a ushort entry in _branchBitmaps: bit i set means "entity satisfies branch i". An entity is in the view iff its bitmap is non-zero. When an incoming delta clears the last branch, the entity is removed; when a new branch flips on, the bitmap is updated but the view membership doesn't change. This is why the DNF branch cap is 16 — the bitmap is a ushort.

`ViewRegistry`

Querying/internals/ViewRegistry.cs

One per ComponentTable. Maps fieldIndex → ViewRegistration[] (copy-on-write — readers are lock-free, writers take a Lock).

internal readonly struct ViewRegistration
{
    public readonly int ViewId;                       // NOT IView — holding the view would root it, blocking GC of abandoned views (#865)
    public readonly ViewDeltaRingBuffer DeltaBuffer;
    public readonly byte ComponentTag;                // 0 = T1, 1 = T2 (multi-component views)
}

On RegisterView(IView, DeltaBuffer) the registry inserts a ViewRegistration into each field-dependency bucket. On DeregisterView, it sweeps all buckets, removing every registration for that view (some views are registered under multiple component tags). Both RegisterView and DeregisterView also sweep registrations whose view was finalized without Dispose (DeltaBuffer.IsAbandoned), retiring the buffer and removing the entry.

`ViewDeltaRingBuffer`

Querying/internals/ViewDeltaRingBuffer.cs

MPSC lock-free ring buffer. Per-view. Default capacity 4096 (must be a power of two). SOA layout — separate arrays for the 24-byte ViewDeltaEntry, the 8-byte tsn, the 1-byte flags, the 1-byte componentTag, the 1-byte written marker — each starting at a 64-byte cache-line boundary, all backed by a single IMemoryAllocator.AllocatePinned block so the buffer accounts as one resource node.

Producer hot path is a CAS on a PaddedLong _tail; consumer hot path is a plain increment on a PaddedLong _head — PaddedLong is [StructLayout(Explicit, Size = 64)] to keep producer and consumer counters on separate cache lines. On overflow (tail - head >= capacity), the producer sets a sticky _overflow = 1 flag and returns false; the owning view's next Refresh() reacts by doing a full re-scan and reporting HasOverflow.

ViewDeltaEntry is exactly 24 bytes:

[StructLayout(LayoutKind.Sequential, Pack = 1)]
internal struct ViewDeltaEntry  // 24 bytes
{
    public long EntityPK;        //  8 B offset  0
    public KeyBytes8 BeforeKey;  //  8 B offset  8
    public KeyBytes8 AfterKey;   //  8 B offset 16
}

KeyBytes8 is an 8-byte opaque field key — sized to fit any indexed scalar exactly. Larger key types (String64) aren't supported as view predicates, so 8 B is sufficient.

6. Statistics — selectivity estimation

The planner can't pick the most selective index without per-field cardinality estimates. Typhon maintains three sketches per indexed field, rebuilt by a background worker.

`StatisticsWorker`

Querying/internals/StatisticsWorker.cs

Dedicated background thread. Wakes every PollIntervalMs. For each archetype with a ClusterState: if ArchetypeClusterState.MutationsSinceRebuild exceeds MutationThreshold and the estimated live entity count ≥ MinEntitiesForRebuild, it triggers a StatisticsRebuilder.RebuildClusterAll(...) pass. Page-granularity sampling kicks in above SamplingMinEntities. There is no per-ComponentTable sweep: cluster-backed archetypes store no entities in ComponentSegment, so such a pass would scan nothing.

`StatisticsOptions`

Querying/public/StatisticsOptions.cs

Property	Default	Meaning
`MutationThreshold`	1000	Index mutations before a rebuild triggers
`PollIntervalMs`	5000	Worker poll cadence (floor-clamped to 100)
`MinEntitiesForRebuild`	100	Skip tables smaller than this
`SamplingMinEntities`	10000	Above this, page-sample instead of full-scan
`Enabled`	`true`	Master switch

`HyperLogLog`

Querying/internals/HyperLogLog.cs

Cardinality estimator. Precision 12 → 4096 registers → 4 KB per sketch. Hashes input via the Murmur3 64-bit finalizer (3 xor-shifts + 2 multiplies — keys are already long-encoded). EstimateCardinality() applies the Flajolet et al. harmonic-mean estimator with LinearCounting small-range correction (no 32-bit large-range correction — the 64-bit hash space doesn't need it). Immutable after build; rebuilds produce a fresh instance and atomic-swap onto IndexStatistics.

`MostCommonValues` (MCV)

Querying/internals/MostCommonValues.cs

Top-100 values by count. Entries are sorted by value (not by count) for O(log K) binary search on equality probes. TotalEntities and RemainingEntries let the estimator interpolate counts for values not in the top-K. ~1.8 KB per field. Immutable after build.

`Histogram`

Querying/internals/Histogram.cs

Equi-width: 100 buckets spanning [MinValue, MaxValue]. ~1.6 KB per field. BucketCount[i] is the number of entries falling into bucket i. O(1) bucket lookup. Handles ranges that span the signed-long boundary (e.g., order-preserving float encoding) via unsigned subtraction.

`AdvancedSelectivityEstimator`

Querying/internals/AdvancedSelectivityEstimator.cs

Singleton (AdvancedSelectivityEstimator.Instance). The priority chain:

Op	Primary	Fallback
`==`	MCV exact lookup (O(log K))	B+Tree point seek (O(log N))
`!=`	`total − Equal(threshold)`	same
`<`, `<=`, `>`, `>=`	Histogram range estimation	uniform distribution over `[min, max]`

Floats/doubles skip the ±1 boundary adjustment for > / < (bit-level ±1 can cross NaN boundaries); the approximation error is at most 1 entity. Replaces the older BasicSelectivityEstimator and the removed HistogramSelectivityEstimator. When no stats exist, it degrades gracefully to a B+Tree point-seek (equality) or a uniform-distribution guess (range).

7. Pending-spawn visibility

A transaction can Spawn an entity and then immediately query — the query must see its own uncommitted writes. But pending spawns are held on Transaction.PendingSpawns (a list); they have no entry in the EntityMap yet, and crucially no entry in any secondary index, because indexes are populated by the commit pipeline.

EcsQuery handles this in two places:

Targeted scan (WhereField path): after the index-driven scan, ExecuteTargeted calls CollectPendingSpawnsWithFieldFilter(result) (EcsQuery.cs:1983). For each entry in tx.PendingSpawns: skip if in PendingDestroys, skip if not in archetype mask, then evaluate the compiled predicate (_pendingSpawnFieldFilter) via tx.Open(id).TryRead<T>(out value) && predicate(value). The compiled Func<T,bool> is deferred — it's only .Compile()d on first invocation (Expression.Compile costs ~100+ µs) and cached in a closure.
Broad scan (opaque Where<T>(Func<T,bool>)): pending spawns are reached by CollectMatching, which enumerates archetype EntityMaps — but tx.Open(id) resolves to the pending-spawn data, so opaque Where filters work transparently for pending entities.

PendingDestroys is checked first in both paths — an entity that was spawned and then destroyed in the same transaction is invisible to its own queries.

8. Subscriptions — pushing state to external clients

Replication to remote clients has its own chapter: 15-subscriptions. It does not read views: it is built from declared projections and answered from geometry, so the view machinery in §5 is unaffected by it.

Table of Contents

09 — Querying

1. Overview — the two pipelines

2. Query API — `EcsQuery<TArchetype>`

Three tiers of constraints

Spatial predicates

`OrderBy` / `Skip` / `Take`

`EcsNavigationQueryBuilder<TArch, TSource, TTarget>`

3. Plan building

`ExpressionParser` — DNF with a 16-branch cap

`PlanBuilder`

`ExecutionPlan`

`FieldEvaluator`

4. Execution — `PipelineExecutor`

Three storage-mode branches

Iterator shape

`ExecuteFullScan` (it's not on `PipelineExecutor`)

What's not there (audit clarifications)

5. View system

`IView` — registry-facing contract

`ViewBase` — shared state

`EcsView<TArchetype>`

`DeltaKind` and `ViewDelta`

OR-mode bitmaps

`ViewRegistry`

`ViewDeltaRingBuffer`

6. Statistics — selectivity estimation

`StatisticsWorker`

`StatisticsOptions`

`HyperLogLog`

`MostCommonValues` (MCV)

`Histogram`

`AdvancedSelectivityEstimator`

7. Pending-spawn visibility

8. Subscriptions — pushing state to external clients

See also

Table of Contents

09 — Querying

1. Overview — the two pipelines

2. Query API — EcsQuery<TArchetype>

Three tiers of constraints

Spatial predicates

OrderBy / Skip / Take

EcsNavigationQueryBuilder<TArch, TSource, TTarget>

3. Plan building

ExpressionParser — DNF with a 16-branch cap

PlanBuilder

ExecutionPlan

FieldEvaluator

4. Execution — PipelineExecutor

Three storage-mode branches

Iterator shape

ExecuteFullScan (it's not on PipelineExecutor)

What's not there (audit clarifications)

5. View system

IView — registry-facing contract

ViewBase — shared state

EcsView<TArchetype>

DeltaKind and ViewDelta

OR-mode bitmaps

ViewRegistry

ViewDeltaRingBuffer

6. Statistics — selectivity estimation

StatisticsWorker

StatisticsOptions

HyperLogLog

MostCommonValues (MCV)

Histogram

AdvancedSelectivityEstimator

7. Pending-spawn visibility

8. Subscriptions — pushing state to external clients

See also

2. Query API — `EcsQuery<TArchetype>`

`OrderBy` / `Skip` / `Take`

`EcsNavigationQueryBuilder<TArch, TSource, TTarget>`

`ExpressionParser` — DNF with a 16-branch cap

`PlanBuilder`

`ExecutionPlan`

`FieldEvaluator`

4. Execution — `PipelineExecutor`

`ExecuteFullScan` (it's not on `PipelineExecutor`)

`IView` — registry-facing contract

`ViewBase` — shared state

`EcsView<TArchetype>`

`DeltaKind` and `ViewDelta`

`ViewRegistry`

`ViewDeltaRingBuffer`

`StatisticsWorker`

`StatisticsOptions`

`HyperLogLog`

`MostCommonValues` (MCV)

`Histogram`

`AdvancedSelectivityEstimator`