Table of Contents

Class DagScheduler

Namespace
Typhon.Engine
Assembly
Typhon.Engine.dll

Production DAG scheduler for the Typhon Runtime. Executes a static system DAG on a pool of worker threads, with any-worker dispatch and inline continuation.

public sealed class DagScheduler : HighResolutionTimerServiceBase, IResource, IDisposable, IMetricSource
Inheritance
HighResolutionTimerServiceBase
DagScheduler
Implements
Inherited Members
Extension Methods

Remarks

Derives from Typhon.Engine.Internals.HighResolutionTimerServiceBase to leverage its three-phase wait (Sleep → Yield → Spin), self-calibrated sleep threshold, timing error tracking, and ResourceNode lifecycle.

The timer thread acts as a metronome: it resets per-tick state, bumps the generation counter (waking workers), and waits for tick completion. All dispatch decisions happen on workers (POC decision D2: any-worker dispatch, no scheduler thread).

Between ticks, each worker blocks on its own wake event — a ManualResetEventSlim constructed with spinCount: 0 — inside a loop that re-checks _tickGeneration, with a 50 ms timeout as a shutdown-liveness backstop. The three-phase wait (Sleep → Yield → Spin) belongs to the timer thread's metronome in Typhon.Engine.Internals.HighResolutionTimerServiceBase, not to the workers; _nextTickTimestamp is likewise metronome state, read by GetNextTick and never by a worker.

Constructors

DagScheduler(SystemDefinition[], int[], IReadOnlyList<Track>, int, RuntimeOptions, IResource, EventQueueBase[], ILogger)

Creates a new DAG scheduler.

public DagScheduler(SystemDefinition[] systems, int[] topologicalOrder, IReadOnlyList<Track> tracks, int deferredTrackStartIndex, RuntimeOptions options, IResource parent, EventQueueBase[] eventQueues = null, ILogger logger = null)

Parameters

systems SystemDefinition[]

System definitions from Build().

topologicalOrder int[]

Topological order from Build().

tracks IReadOnlyList<Track>

The track hierarchy in execution order — each track carries its DAGs and their resolved system indices.

deferredTrackStartIndex int

Tracks from this index onward are dispatched on demand by the runtime (e.g. Engine-Post after serial fence prep), not by the in-tick track loop.

options RuntimeOptions

Runtime configuration.

parent IResource

Parent resource node (typically Runtime).

eventQueues EventQueueBase[]

Event queues to reset at each tick start. Can be empty.

logger ILogger

Optional logger for diagnostics.

Fields

UnhandledExceptionCallback

Public hook fired when something threw that the host has to be told about. User code can subscribe to log to file, send to telemetry, request graceful shutdown, or escalate to a debugger break. Invoked on the thread that caught the exception — a worker or the tick driver — so do not block in it. Defaults to null (no-op).

Two sources, and only two. (1) The WorkerLoop's outer safety net and the tick driver's, when an exception escaped every inner try/catch — meaning an inner handler didn't catch it, which is itself a bug worth surfacing prominently. (2) Since #890, ANY failure of a system on an ENGINE track — the Fence DAG — from its ShouldRun, Prepare or Execute, on any dispatch path, because the fence is the work that makes a tick durable and a fence that did not finish must not be silent (see Typhon.Engine.DagScheduler.RecordEngineTrackFailure(System.Int32,System.String,System.Exception)).

Application-track systems keep the original contract: the per-system handlers in Typhon.Engine.DagScheduler.ProcessParallelQuery(System.Int32,System.Int32,System.Boolean), Typhon.Engine.DagScheduler.ProcessCallbackOrQuery(System.Int32,System.Int32,System.Boolean), Typhon.Engine.DagScheduler.ProcessPipeline(System.Int32,System.Int32,System.Boolean) and Typhon.Engine.DagScheduler.ExecuteInline(System.Int32,System.Int32,System.Boolean) log via LogSystemException and capture via CaptureSystemException without invoking this callback; their detail lives in Exception in the telemetry ring. 

public Action<int, string, Exception> UnhandledExceptionCallback

Field Value

Action<int, string, Exception>

Properties

AllSystemCount

Total number of systems in the DAG, including engine-internal systems (e.g. FenceExec).

public int AllSystemCount { get; }

Property Value

int

CurrentOverloadLevel

Current overload response level.

public OverloadLevel CurrentOverloadLevel { get; }

Property Value

OverloadLevel

CurrentTickNumber

Number of ticks executed so far.

public long CurrentTickNumber { get; }

Property Value

long

DispatcherWorkerId

The reserved worker slot for the dispatcher (timer) thread — always WorkerCount, one past the last real worker (#860).

public int DispatcherWorkerId { get; }

Property Value

int

Remarks

The timer thread runs system bodies in one narrow case: Typhon.Engine.DagScheduler.MarkTrackRootsReady(System.Int32[]) skips a root, and OnSystemComplete chains the successor into ExecuteInline right there. Internally the scheduler passes workerId = -1 for this thread; Typhon.Engine.DagScheduler.ToWorkerSlot(System.Int32) translates it to this slot so system code can index per-worker state on every path that runs system code.

Why a dedicated slot is safe: disjointness, not quiescence. It is tempting to argue that the inline execution happens before WakeWorkers() wakes the pool and is therefore serialized against every worker. That is NOT true across tracks: Typhon.Engine.DagScheduler.DispatchTrackMultiThreaded(System.Int32) clears _tickInProgress only after its completion barrier, so a worker that has already evaluated its while (_tickInProgress == 1 && _systemsRemaining.Value > 0) loop condition can still be inside FindReadySystem while the timer thread has advanced into the next track's Typhon.Engine.DagScheduler.MarkTrackRootsReady(System.Int32[]). The slot is safe because the dispatcher never writes a worker's slot and no worker ever writes this one — do not weaken that to "the dispatcher can share a worker slot because nothing overlaps".

Consequence: per-worker structures reachable from system code must be sized WorkerCount + 1, and WorkerId ranges over [0, WorkerCount], not [0, WorkerCount).

EventQueues

All registered event queues. Read-only view exposed for static-data introspection (the profiler builds the v7 EventQueueRecord catalog from this) — not for hot-path use; system code should go through the pre-allocated ConsumedQueues array instead.

public IReadOnlyList<EventQueueBase> EventQueues { get; }

Property Value

IReadOnlyList<EventQueueBase>

IsFenceFailed

True once an engine-track (Fence-DAG) system has thrown. Terminal — never cleared; the tick that follows one is not a tick that completed.

public bool IsFenceFailed { get; }

Property Value

bool

IsTickAborted

True once a fatal system exception has aborted a tick under AbortTickAndStop. Terminal — never cleared. No further ticks execute.

public bool IsTickAborted { get; }

Property Value

bool

LostWakeCount

Lost wakes that cost a worker its between-tick backstop (50 ms), since the scheduler was built: the backstop fired after a dispatch whose Set never reached the worker. Non-zero means the wake protocol is broken. Zero does not prove it is not: a lost wake that the next dispatch's Set rescues before the backstop fires leaves nothing to count, so at tick rates faster than the backstop most would go unseen. Per tick: LostWakes.

public long LostWakeCount { get; }

Property Value

long

MeasureWorkerIdle

Whether worker idle time is measured (WorkerIdle). A measurement switch: a few timestamps per idle spell.

public static bool MeasureWorkerIdle { get; set; }

Property Value

bool

SystemCount

Number of user-registered systems (systems whose track does not carry the EngineTag — i.e. excludes the Fence DAG). This is the count callers usually want — "the systems I registered". For the total including engine-internal systems use AllSystemCount.

public int SystemCount { get; }

Property Value

int

Systems

All registered systems across every track and DAG, indexed by system index. Used by the scheduler for dispatch, lookup, and topological reasoning — the index identity is load-bearing for chunk dispatch state and dependency wiring, so this array stays as the canonical source for everything index-keyed (including diagnostics/profiler that walk every entry by raw index).

public SystemDefinition[] Systems { get; }

Property Value

SystemDefinition[]

Remarks

When you only care about systems the user registered (counting them in a test, iterating them in a DAG-viewer that hides engine plumbing), prefer UserSystems — it filters out systems whose track carries the EngineTag (e.g. the Fence DAG) so the count is stable regardless of which engine-internal DAGs the runtime registers.

Telemetry

Telemetry ring buffer for diagnostic inspection.

public TickTelemetryRing Telemetry { get; }

Property Value

TickTelemetryRing

ThreadName

Thread name set by derived classes for diagnostics.

protected override string ThreadName { get; }

Property Value

string

Tracks

The runtime partitioning hierarchy — tracks in execution order, each carrying its DAGs. Static for the scheduler's lifetime.

public IReadOnlyList<Track> Tracks { get; }

Property Value

IReadOnlyList<Track>

UserSystems

User-registered systems only (filtered Systems with engine-tagged-track entries removed). Indices in this view do NOT match the canonical system index — use this only for iteration and counting, not for index-keyed lookups.

public SystemDefinition[] UserSystems { get; }

Property Value

SystemDefinition[]

WorkerCount

Number of worker threads.

public int WorkerCount { get; }

Property Value

int

WorkerHotSpinners

How many idle workers keep spinning inside a dispatch while the others park. A negative value keeps the legacy spin-then-yield loop. Default 0.

public static int WorkerHotSpinners { get; set; }

Property Value

int

Remarks

A spinner starts a newly ready system without a kernel wake, which is what it is for — and measured, it does not pay: on the SWG demo at 1 000 sessions, two spinners made the tick 3.9 % longer at P50 than none (5 of 6 interleaved pairs), because a spinning worker takes execution resources from its busy SMT sibling and holds the core active, while the wake it saves is paid off the critical path whenever the publisher's own chunk covers it. Kept as a knob for a box where wakes are dearer than cores.

WorkerIdle

Worker time inside dispatches since start, while MeasureWorkerIdle is on: total, idle (spinning or parked), parked, and the counts of parks, backstop expiries, idle spells and wakes publishers issued. Summed over workers, in milliseconds.

public (double InDispatchMs, double IdleMs, double ParkedMs, long Parks, long Backstops, long Spells, long Wakes) WorkerIdle { get; }

Property Value

(double InDispatchMs, double IdleMs, double ParkedMs, long Parks, long Backstops, long Spells, long Wakes)

WorkerIdleSpinBudget

How many idle iterations a worker spins before it starts yielding the core, WHILE A TICK IS IN PROGRESS. Default 4096.

public static int WorkerIdleSpinBudget { get; set; }

Property Value

int

Remarks

This is the dispatch ramp, and it is a latency knob, not a throughput one. A worker that finds no ready system inside a tick is waiting for a sibling to finish a system it depends on — work that is coming in microseconds. Yielding gives the core up, and on a box with two threads per core and every worker yielding, coming back costs tens of microseconds: measured on the SWG demo at 1 000 sessions, the last of 31 chunks of the frame stage started 1.88 ms after the first, which was 51 % of that stage's whole span and capped its parallel efficiency at 79 %.

The budget is bounded by the tick, not by the gap between ticks. Between ticks a worker parks on its own event and burns nothing — that path is untouched. This one only spins while _tickInProgress is set and there are systems left, so the worst case is a worker spinning for the remainder of a tick it has no work in, which is what a core would otherwise be idle for anyway.

Lower it on a box that is oversubscribed, or where the engine shares cores with other processes: there the yield is the right answer and the spin is theft. 0 restores the pre-#906 behaviour of yielding after the first microsecond.

WorkerParkAfterUs

How long an idle worker spins before it may park, in microseconds. Default 10: long enough for a successor that becomes ready within microseconds to be taken without a park and a wake; measured, 10 µs beat 50 µs (tick P99 8 % shorter) and matched 0.

public static int WorkerParkAfterUs { get; set; }

Property Value

int

WorkerSlotCount

Number of distinct worker slots a context can report — WorkerCount real workers plus the dispatcher slot (#860).

public int WorkerSlotCount { get; }

Property Value

int

Remarks

Two different indices share the name "worker id" — size against the right one. This value sizes arrays indexed by WorkerId, the slot a system body sees. The scheduler's own per-worker pools (_partitionViews, the tier-range view pool, ParallelTransactionAccessor.GetWorkerAccessor) are indexed by the internal workerId that chunk dispatch hands out, which is always a real pool index and never the dispatcher slot — they are correctly sized WorkerCount and must stay that way.

WorkerUtilization

Worker utilization since start, when Scheduler:Gauges:WorkerUtilization telemetry is on: summed worker time inside system and chunk bodies, summed tick wall time, and the tick count. Active ÷ (workers × wall) is the share of the pool doing useful work.

public (double ActiveMs, double TickWallMs, long Ticks, int Workers) WorkerUtilization { get; }

Property Value

(double ActiveMs, double TickWallMs, long Ticks, int Workers)

Methods

DispatchDeferredTracks()

Dispatches every runtime-deferred track (Engine-Post and beyond) in execution order. Called by TyphonRuntime.OnTickEndInternal after the serial WriteTickFence prep, so the Fence DAG sees a populated FenceContext. No-op when every deferred track is empty (e.g. parallel fence disabled).

public void DispatchDeferredTracks()

DispatchTrack(int)

Dispatches a single track by its execution-order index. The in-tick tracks are dispatched automatically each tick; the runtime calls this (via DispatchDeferredTracks()) for the deferred Engine-Post track after serial fence prep. Branches on worker count: single-threaded mode runs the track synchronously on the caller.

public void DispatchTrack(int trackIndex)

Parameters

trackIndex int

Dispose(bool)

Stops the timer thread and disposes managed resources. Follows the standard ResourceNode disposal pattern. Idempotent with Typhon.Engine.Internals.HighResolutionTimerServiceBase.StopTimerThread() — calling this after the timer has already been stopped just runs the managed-resource cleanup.

protected override void Dispose(bool disposing)

Parameters

disposing bool

DumpHangDiagnostic()

Returns a multi-line snapshot of the scheduler's current tick state, suitable for logging when a tick fails to complete. Cheap enough to call ad-hoc — O(systemCount) reads of internal counters.

public string DumpHangDiagnostic()

Returns

string

EffectiveRateDivisor(int)

How many ticks apart this system's runs are, under the gates Typhon.Engine.DagScheduler.CheckOverloadSkip(System.Int32) is applying right now (DSEL-03).

public int EffectiveRateDivisor(int sysIdx)

Parameters

sysIdx int

The system.

Returns

int

The tick spacing, at least 1.

Remarks

The LCM, because the two gates are independent and a run has to pass both. A system admitted only when tick % TickDivisor == 0 AND tick % ThrottledTickDivisor == 0 runs on the ticks divisible by both, which is their least common multiple — so tickDivisor 2 with throttledTickDivisor 3 runs every SIX ticks under load, not every two and not every three.

Why this lives here rather than being read off SystemDefinition. The throttled gate is conditional on the overload level and on the system's priority, and the runtime has to scale AmortizedDeltaTime by whatever is actually gating the system this tick. Recomputing that condition at the point of use is how the two halves drift apart: the first version of #1293 scaled by TickDivisor alone, so every throttled system silently integrated at a fraction of real time from the moment overload engaged — the very defect it was fixing, displaced into the path nobody tests.

It does NOT account for ShouldRun or a reactive change-filter skip. Those are the application's own decisions about a particular tick rather than a declared rate, and a system that skips itself knows why; the engine cannot attribute elapsed time to a condition it does not own.

ExecuteCallbacks(long, long)

Execute the callback(s) due at this tick. Called by the timer loop immediately after the three-phase wait completes.

protected override void ExecuteCallbacks(long scheduledTick, long actualTick)

Parameters

scheduledTick long

The Stopwatch timestamp that was targeted (as returned by Typhon.Engine.Internals.HighResolutionTimerServiceBase.GetNextTick()).

actualTick long

The Stopwatch timestamp when the wait actually completed. The difference abs(actualTick - scheduledTick) is the timing error for this tick.

GetNextTick()

Returns the Stopwatch timestamp of the next tick to wait for. Called by the timer loop after each callback execution to determine the next wake-up time.

protected override long GetNextTick()

Returns

long

The GetTimestamp() value at which the next tick should fire. Return MaxValue to idle indefinitely (e.g., no registrations).

Remarks

Returning a timestamp in the past causes the wait loop to exit immediately and call ExecuteCallbacks(long, long) without delay (catch-up behavior). Derived classes should use metronome-style advancement (nextTick += interval) rather than computing from current time, to prevent drift accumulation.

OnWaitComplete(long, long, long, byte)

Emit a SchedulerMetronomeWait span for the inter-tick wait that just completed. Called on the TickDriver thread by Typhon.Engine.Internals.HighResolutionTimerServiceBase.

protected override void OnWaitComplete(long scheduledTimestamp, long startTimestamp, long endTimestamp, byte phaseFlags)

Parameters

scheduledTimestamp long
startTimestamp long
endTimestamp long
phaseFlags byte

Remarks

Why this exists (issue #289 follow-up). Without this hook the timer thread's three-phase wait between ticks emits no profiler events — appearing as dead time on the trace even when the engine is intentionally throttling itself via Typhon.Engine.Internals.OverloadDetector.TickMultiplier. The span carries the multiplier and an intentClass byte (CatchUp / Throttled / Headroom) so a trace viewer can answer why the metronome was waiting.

RegisterContext<TContext>(TContext)

Bind an ambient TContext onto every registered system deriving from ChunkedCallbackSystem<TContext>. Must be called after the schedule is built (systems must already be registered) and before Start().

public void RegisterContext<TContext>(TContext context) where TContext : class

Parameters

context TContext

Type Parameters

TContext

Shutdown()

Stops the worker pool: signals shutdown, releases any worker parked between ticks, and joins them.

A tick already in flight is ABANDONED, not drained. That is not a choice so much as an admission: this method signals every worker to stop before it waits, so the systems remaining in that tick have nobody left to run them, and "waiting for the current tick to finish" could only ever mean waiting forever. A caller that needs the in-flight tick's work to land must stop dispatching and let the tick complete BEFORE calling this.

Does NOT stop the timer thread — Dispose(bool) does, via the base class, and that is the only quiescence point. The thread stays alive here but stops producing ticks: GetNextTick() returns long.MaxValue and ExecuteCallbacks(long, long) bails out, both gated at tick ENTRY. A tick already past that gate therefore runs to completion, and the increment of CurrentTickNumber is the last statement of that tick's telemetry finalizer — so the counter can advance by ONE after this method returns. Never by more: _workerShutdown is published before the join, so every later tick early-returns at the gate. Callers that need "nothing is running" must use Dispose(bool).

May legitimately be called ON the timer thread. OnTickAborted fires inside the tick-end callback and its documented response is FatalStop(), which lands here — so on that path this returns INTO the tick it just stopped, and the one-tick advance above is guaranteed rather than merely possible. Do not "fix" that by joining the timer thread here: it would be a self-join, which StopTimerThread's 2 s bound turns into a stall that returns false and changes nothing. See issue #404. 

public void Shutdown()

Start()

Starts the worker threads and the tick driver timer thread.

public void Start()