Lock-Contention Forensics (Deep Diagnostics)
Post-mortem visibility into which threads waited on which locks, for how long, and why.
Status: ๐ Planned ยท Visibility: Internal ยท Category: Profiler
๐ฏ What it solves
When a stress test shows an unexplained latency spike or an occasional transaction timeout under load, span timing alone can't tell you whether a thread was doing work or sitting blocked on a lock. Aggregate contention counters ("47 contention events averaging 340ยตs") lose the per-event detail needed to actually fix it: which lock, which two threads, what the holder was doing, and whether it's a recurring pattern. This feature is designed to answer "why did thread 7 wait 2.3ms for thread 3, and does it keep happening?" without attaching an external profiler.
โ๏ธ How it works (in brief)
The design captures contention only โ not every lock acquisition โ so the fast, uncontended path (the vast majority of lock operations) keeps its current zero-overhead behavior. A capture point fires only when a thread actually has to wait: it would record the waiter and holder thread IDs, the lock's identity, a call stack at the wait point, and the resulting wait duration, then surface each event as a span so it can appear inline in a flame graph (nested under whatever transaction triggered it) or as its own trace when contention happens outside a transaction.
๐ป Usage
Not usable yet โ no engine emission exists. The roadmap reserves wire event-kind range 70-74
(LockAcquire / LockRelease / LockWaitBegin / LockWaitEnd) for this feature; none of these event
kinds, and no contention-capture hook on AccessControl, AccessControlSmall, or ResourceAccessControl,
are implemented today.
// Illustrative only โ design complete, not implemented yet. No "Typhon:Profiler:LockContention" config
// key, no TyphonEvent factory, and no contention-capture hook exist in source today.
// var runtime = TyphonRuntime.Create(dbe, schedule => { /* ... */ });
// // ...contention on AccessControl/AccessControlSmall/ResourceAccessControl would emit
// // LockWaitBegin/LockWaitEnd spans automatically, no call-site changes required...
| Option | Default | Effect |
|---|---|---|
Typhon:Profiler:LockContention:Enabled (proposed) |
false |
Not implemented โ no such config key exists today |
โ ๏ธ Guarantees & limits
- Not implemented. No wire event kinds, capture hooks, or config surface exist in
src/Typhon.Engine/Foundation/Concurrency/internals/today; this is a complete design awaiting a build slot. - Contention-only by design โ capturing every acquire/release was estimated at 1M+ events/sec (1.8B over 30 minutes); capturing only actual waits cuts that by 100-10,000x while keeping exactly what's useful for debugging.
- Zero overhead on the fast path โ the design only does work (stack capture, event record) once a thread has already determined it must wait; threads that acquire immediately pay nothing extra.
- Holder identification is exact for exclusive waits โ
AccessControl/AccessControlSmallalready track the current exclusive holder's thread ID as part of their atomic lock state, so the design can read it with no extra bookkeeping. For shared-lock contention there is no single holder to name (multiple threads can hold a shared lock at once); the design records it as "waiting on shared readers" rather than a thread ID. - Would use the 16-bit thread-ID convention already shared by
AccessControl,AccessControlSmall, andResourceAccessControlโ no new thread-identity mechanism. - Scoped to three lock types:
AccessControl,AccessControlSmall,ResourceAccessControl. General span tracing, memory tracing, and I/O tracing are separate features, not part of this one. - Reserved wire range 70-74 is held for this feature and won't be reused by other event kinds in the meantime.
๐ Related
- Sibling features: GC Event Tracing, Off-CPU Thread Scheduling Capture, Configuration & Performance Tuning
- Sibling: Reader-Writer & Resource Lifecycle Locks โ the
AccessControl/AccessControlSmall/ResourceAccessControllock family this feature would instrument - Source (existing, related):
src/Typhon.Engine/Foundation/Concurrency/internals/AccessControl.cs,AccessControlSmall.cs,ResourceAccessControl.csโ no contention-capture code present yet