Execution EngineLeases & Dispatcher

Leases & Dispatcher

ExecutionDispatcher in Notation Durable polls for due executions, acquires time-bounded leases, renews active leases on a timer, aborts or kills deliveries on lease loss, and applies exponential backoff when deliveries fail before settlement.

Poll loop & claim priority

The dispatcher bounds concurrent work by reserving free slots from concurrency, active runs, and in-flight claims. Each tick claims due work up to the free slot count with leaseDuration and any active ID filter.

claimDue selects non-terminal executions whose nextClaimAt timestamp is at or before the runtime clock, ordered by earliest nextClaimAt and then executionId. When at least one execution is claimed and slots remain, or when an in-flight delivery finishes, the dispatcher schedules the next claim immediately; idle polls wait pollInterval.

Execution stateClaim eligibility
TerminalNever claimed again
No held leaseLatest of dueAt or redeliverAt
Held leaseLatest of pending wakes or leaseExpiresAt plus backoff delay
OptionDefault
concurrency10 active deliveries
pollInterval10 ms when idle
leaseDuration30_000 ms
renewIntervalleaseDuration / 3
abortGracePeriod1_000 ms

renewInterval must be positive and shorter than leaseDuration, and filter.executionIds cannot be empty.

Fencing tokens (dueAt vs claimedDueAt)

Each claim atomically issues an ExecutionLease carrying a monotonically increasing leaseToken and expiration timestamp. To isolate concurrent wakes, ExecutionState separates claimedDueAt for the active lease from dueAt for wakes arriving mid-delivery.

Every fenced operation – step writes, cancellation delivery, renewal, release, and acknowledgement – checks inside one atomic transaction that leaseExpiresAt is non-null and leaseToken equals lease.token. Clock expiry makes an execution claimable again, but an expired lease remains current until a successor claim or forced cancellation increments leaseToken.

When a delivery suspends, claimedDueAt clears and dueAt merges any next wake timestamp so mid-delivery wakes are preserved. Releasing a lease or claiming over an expired one keeps whichever wake timestamp is earlier.

Lease renewal & hard termination

While a delivery runs, the dispatcher renews its lease every renewInterval, retrying transient renewal errors because the lease remains valid until superseded. Losing the lease stops renewal and aborts the delivery's AbortController with LeaseLostError.

MechanismBehavior
SDK signal checkThrows InfrastructureError before advancing steps or fenced writes
Grace period abandonmentAbandons deliveries stuck past abortGracePeriod and frees the slot
Child worker abortKills the forked worker process immediately with SIGKILL
Worker IPC disconnectExits child process with code 75 if the parent runtime dies

An abandoned delivery's late resolution or rejection is logged and ignored without calling release or acknowledge.

Outcome classification & retry backoff

Normal workflow outcomes settle through acknowledge, resetting retry counts and clearing backoff timers. Infrastructure failures – such as worker crashes, lost leases, step conflicts, and aborted delivery signals – reject instead without acknowledging workflow progress.

RejectionDispatcher action
Worker exited before replyLeaves lease to expire so dying worker writes settle before reclaim
Lease lost on step writeCalls release, which returns lost without modifying state
Step conflict or runtime errorCalls release to schedule redelivery after backoff
Aborted delivery signalSkips release and acknowledge after renewal loss or grace expiry

If loading state, releasing, or acknowledging fails unexpectedly, the dispatcher lets the lease expire safely. Each unacknowledged delivery increments the failure count and schedules retry via capped exponential backoff (1_000 * 2 ** (n - 1) ms, up to 300,000 ms).