Recovery
The reconciler prevents orphaned cloud resources, duplicated creates, and broken dependency chains across process crashes, ambiguous responses, interrupted replacements, and teardowns.
Creating Records
Before invoking resource.create, createResourceOperation (packages/reconciler/src/operations/operation.create.ts) persists an intermediate "creating" state record. Subsequent runs use it to detect started attempts and reuse generated physical names.
Record Persistence
creatingRecord(params) writes lastOperation: "creating" under state:persist:<id>:creating:
- Recorded fields – Stores
id,group,type,provider,lastOperationAt, resolvedconfig,params(with generatedphysicalNamevalues),dependsOn, andretired.outputstoresresource.toState(definedKey(resource, resourceParams))for any key fields already inparams. - Name preservation –
recordedNamesreadsphysicalNamefields from"creating"records just like completed records. Retries and subsequent deploys attempt creation and lookup under the same generated name. - Planner and output isolation –
decideActiontreatsisCreatingRecord(stateNode)as"create"and setsPlanNode.mayTakeOver = true.loadOutputandreadRecordedOutputssuppress itsoutputso dependents plan against unknown values.
Deleting Unfinished Creates
When deleteResourceOperation (packages/reconciler/src/operations/operation.delete.ts) deletes a "creating" record (params.creatingParams is set), findCreatedResource inspects the provider before calling resource.delete:
| Condition | Behavior |
|---|---|
| Missing find match | Skips remote delete with reason: "not-created" and removes record |
| Undefined key field | Skips remote delete with reason: "key-unknown" and removes record |
| Defined key fields | Runs delete with resource.key, treating ResourceNotFoundError as deleted |
When resource.find exists, it runs under delete:find with creatingParams and skips on a null match; otherwise findCreatedResource checks whether any resource.key field is undefined. Skipped remote deletes emit status: "skip".
Unknown Create Outcomes
Each create attempt inside retryPending runs through @notation/durable's step.effect so a crash or indeterminate failure never blindly invokes create a second time.
Durable Effect Execution
createAttempt executes step.effect("create:remote:effect:<attempt>", ...) with these constants from packages/reconciler/src/operations/operation.create.ts:
CREATE_LOOKUP_RETRY_MS = 1_000– Durable delay (retryIn) between unknown-outcome lookups.MAX_CREATE_LOOKUP_ATTEMPTS = 10– Upper cap on lookup retries (Math.min(params.maxOperationAttempts ?? DEFAULT_MAX_OPERATION_ATTEMPTS, MAX_CREATE_LOOKUP_ATTEMPTS)).failsOnpredicate – Treats errors as terminal for the effect whenlookup === undefined || !ResourceCreationUnknownError.is(error).ResourceAlreadyExistsErrorroutes to takeover, whileResourceOperationPendingErroradvancesretryPendingwith a neweffectIdandcallbackContext.- Credential stripping –
performcallsholdCreated(resource, ...)to extract and hold credentials in memory. Onlykey, credential versions, andinvalidnames are journaled in the effect result.
Effect Lookup Functions
getLookupFunction(resource, resourceParams) selects how step.effect probes the provider after an indeterminate attempt:
| Capability | Behavior |
|---|---|
| Find operation | Calls findResourceKey and settles or returns not-found |
| Read without computed key | Calls read and treats ResourceNotFoundError as not-found |
| No lookup operation | Returns undefined and keeps the "creating" record |
For resource.find, a match returns { status: "settled", result: { key, lookedUp: true } } and null returns { status: "not-found" }. For resource.read, success or a non-throttled ResourceOperationPendingError returns settled(), while other errors – including error.throttled – rethrow so the effect retries later. When no lookup is available, the "creating" record stays in state on ResourceCreationUnknownError or EFFECT_OUTCOME_UNKNOWN.
Resource Takeovers
When an existing resource matches the target identity, takeOver adopts it into state and delegates to reconcileTakenOver (packages/reconciler/src/durable/reconcile.ts).
Takeover Paths
createResourceOperation enters takeOver(step, params, key) through three paths:
| Path | Trigger |
|---|---|
| Pre-create find | create:find locates an existing match before writing "creating" |
| Effect lookup settlement | getLookupFunction settles an indeterminate create with created.lookedUp === true |
| Already-exists error | createAttempt throws ResourceAlreadyExistsError and existingKey resolves the key |
Pre-create find covers missing records, "creating" records, and drift-recreate. On ResourceAlreadyExistsError, existingKey resolves the key via find under create:existing or returns {} when params holds the entire primary key.
Observed Parameter Mapping
takeOver reads the remote resource under create:takeover (readRemoteOperation) and calls setRecordedOutput(resource, { ...key, ...output }). Credentials receive null-version markers because existing secrets cannot be read. It then persists a "create" record under state:persist:<id>:taken-over using observedParams(resource, remote ?? {}, resourceParams):
const unread = Object.fromEntries(
set.filter(([name]) => remote[name] === undefined),
);
const { changed, replaceFields } = diffParams(resource.schema, {}, unread, {
canUpdate: Boolean(resource.update),
});
const kept = new Set(replaceFields.map((field) => field.name));
const leftOut = new Set(changed.filter((name) => !kept.has(name)));Mutable params omitted by remote are left out of the persisted record so the next comparison updates them. Unread immutable, key, or non-updatable params keep their desired values so missing read fields never trigger a spurious replacement.
Post-Takeover Reconciliation
When createResourceOperation returns { tookOver: true }, applyAction calls reconcileTakenOver(step, opts, shared, replaceWithout):
- Fresh state session – Opens a new
openStateSessionunder scopetaken-overwithstateKey:after-takeover. This prevents collisions withstate:persist:<id>:taken-over. - No repeat drift read – Runs
decideActiondirectly against the newly read record and emitsreconciler.deploy.decision. It then invokesapplyActionwithexecutionId: "${shared.executionId}/taken-over".
Takeover Guards
Two safety guards prevent a takeover from corrupting replacements or adopting the wrong remote resource.
Replacement Key Guard
During a delete-first replacement (params.replaces is set), refuseReplaced(params, key) compares resource.keyOf({ ...resourceParams, ...key }) against resource.keyOf(replaces.output) across all three takeover paths:
- Identical key refusal – If
changedKeysis empty, the existing resource may be the replaced instance still lingering in provider listings.refuseReplacedrestoresparams.replacesviarestoreRecordand throwsResourceAlreadyExistsErrorso the next deploy deletes and recreates cleanly. - Scope limit –
refuseReplacedruns only inside the immediate delete-first replacement. Later deploys resuming from"creating"records, orblocksDeleteOfdependents recreated later in the same run, haveparams.replaces === undefined.
Uniqueness Guard
When create throws ResourceAlreadyExistsError, existingKey(step, params, error) prevents ambiguous adoption:
- Computed primary keys – If
hasComputedPrimaryKey(resource)is true andresource.findis absent,existingKeythrowsResourceAlreadyExistsErrornoting the provider-assigned key. - Empty or multi-match
find– Iffindreturnsnull,existingKeythrowsResourceAlreadyExistsError. Providerfindhelpers (such asfindOnlyInPages) must fail whenever multiple remote resources match.
Readability Requirement
refuseUnreadable and takeOver enforce that resource.read is implemented before adopting an existing resource. Without read, noReadError throws and instructs the operator to delete the remote resource manually.
Failed Create Recovery
After a create error, createResourceOperation either leaves the "creating" record intact for recovery or rolls state back to its starting snapshot.
Record Restoration
restoreRecord(params, record = params.prior) writes under state:persist:<id>:restored:
| Outcome | State |
|---|---|
| Unknown outcome or failed read | Keeps the "creating" record intact for recovery |
| Replacement takeover guard failure | Restores params.replaces ?? params.prior so the next deploy replaces |
| Other create failure | Restores params.prior or clears the "creating" record to null |
restoreRecord preserves retired: record.retired ?? [] when writing a record, or writes null if record is undefined. Unknown outcomes match isOutcomeUnknown(error) (ResourceCreationUnknownError or WorkflowEffectError with code === "EFFECT_OUTCOME_UNKNOWN"), and replacement takeover guard failures occur in existingKey, refuseReplaced, or refuseUnreadable.
Deferred Validation Checks
Some provider APIs apply changes asynchronously before their read endpoint reflects them, or return incomplete metadata after succeeding. createResourceOperation and updateResourceOperation handle both via readLagsCreate and firstDeferredError (packages/reconciler/src/operations/deferred-checks.ts):
readLagsCreate2s retry – Whenresource.readLagsCreateis set,createResourceOperationpasses{ retryNotFoundMs: 2_000 }(READ_AFTER_CREATE_RETRY_MS) toreadResourceOperation. This convertsResourceNotFoundErrorintoResourceOperationPendingErrorwhile preservingcallbackContext.- Deferred error persistence –
firstDeferredErrorchecks credential validity/presence,assertComputedFields,readResourceOperation, andassertRequiredOutputsin order, capturing the firstValueValidationErrororSchemaStructureError. The caller persists"create"or"update"before throwing so state tracks the live resource; any other error throws immediately (updateResourceOperationrestores{ ...params.persistedOutput, ...params.resource.output }if the read fails).
Replacement Strategies
packages/reconciler/src/durable/reconcile.ts executes either replaceCreateFirst or replaceDeleteFirst based on replaceOrder(resource).
Create-First Replacement
replaceCreateFirst runs under scope replace:create:
- Generate new names – Calls
generateNamesStep(replace:create:physical-names). - Retire prior instance – Captures
retiredAtinreplace:create:retired-atand appendsretiredInstance(session.node, shared.dependsOn, retiredAt)tosession.node.retired ?? []. - Create beside old resource – Calls
createResourceOperationwithretiredso"creating"and"create"writes carry the retired list atomically. IndryRun, emits adeletedry-runevent withreason: "retired". - Post-takeover handling – Calls
reconcileTakenOveriftookOveris true, leaving retired instances forsweepOrphansat the end ofdeploy.
Retired Instance Cleanup
deleteRetiredInstances (packages/reconciler/src/durable/reconcile.ts) iterates node.retired ?? [] oldest first (index = 0, 1, ...) under step.scope("retired:<index>"):
- Per-instance slicing – After
deleteResourceOperationdeletes instanceindex(passingreadRetiredDeletionifdeleteTakesDependentsis set), itsremovecallback callssession.persist({ ...toPersistedState(node, node.dependsOn!), retired: instances.slice(index + 1) }, "retired:<index>"). - Error attribution – State-write errors (
writing === true, such asVersionConflict),InfrastructureError, andWorkflowCancelledErrorrethrow as-is. Provider delete refusals throwretiredDeleteError, naming dependent resources declared in code during"deploy".
Delete-First Replacement
replaceDeleteFirst runs in four phases:
deleteBlockingDependents– Underreplace:dependents, skipsopts.state.values()unless a registered class listsresource.typeinblocksDeleteOf. Otherwise it collects transitive blocking dependents viacollectDependents, checksassertOrphanDeleteAllowedandrejectChange(..., { replace: true })(respectingfindReplaceOverride), and deletes each understateKeyreplace:<encodedResourceId>withreason: "dependency-replaced"andreplacedDependency: resource.id.- Retired and active deletion – Under
replace:delete, callsdeleteRetiredInstances, readsreadDeletion(dependentsstep) ifdeleteTakesDependentsis set, and runsdeleteResourceOperationremoving the store entry. - Cycle contribution filtering – If
replaceWithout?.(new Set(deleted))returns non-empty cycle peers,withoutContributionsre-evaluatesgetParamsunderreplace:paramswithout those contributions. - Fresh session create – Opens a new
openStateSessionunderreplace:createbecause the prior store instance was deleted. CallscreateResourceOperationwithprior: undefinedandreplaces: { ...session.node, retired: [] }.
Output Backfill
When a resource decides noop despite an incomplete or stale state record, applyAction repairs state without calling update at the provider.
Missing Required Outputs
applyAction checks findMissingOutputs(resource, session.node.output) on noop resources. If it catches SchemaStructureError (for instance after a post-operation validation failure), recoverOutputs (packages/reconciler/src/durable/reconcile.ts) runs readResourceOperation under output-read.
recoverOutputs merges { ...session.node.output, ...readResult }, persists the record when !opts.dryRun, and rethrows if any required output remains absent. Resources without read fail earlier in assertOutputsRecoverable.
Metadata Rewrites
When a noop resource has all required outputs and !opts.dryRun, applyAction checks !hasSameEdges(session.node, dependsOn), recorded.provider !== resource.provider, and !sameGroup(recorded.group, resource.group). If any differ, rewriteRecord (packages/reconciler/src/durable/reconcile.ts) updates state metadata without invoking provider operations.
Destroy Traversal
destroy (packages/reconciler/src/durable/destroy.ts) acquires the deployment hold, lists state once under notation:orphans:list, and deletes every registered record in reverse dependency order.
Edge Merging
For every declared resource in opts.resources, destroy merges three edge sets into edgesById:
| Source | Edges |
|---|---|
| Code graph | getDependencyIds(resource) |
| State record | recordById.get(resource.id)?.dependsOn ?? [] |
| Retired instances | retired[].dependsOn from sweptRecords |
Merging retired edges ensures dependents left behind by an interrupted deploy are deleted before the retired instances they still reference.
Unregistered Record Retention
Before deleting anything, destroy identifies records whose type has no class in workflowRegistry(opts):
- Type-mismatched declared records – If
resource.type !== node.typeandresolveRecordClass(registry, node)isundefined,keep(node, edgesById, kept)marks the record and all transitive dependencies (including desired resources) inkept. It also emitsreconciler.orphan-deletion.skippedwithworkflow: "destroy". - Unregistered orphans –
sweepStepsrunskeepon unregistered orphans and emitsreconciler.orphan-deletion.skippedfor each. Records whosenode.typehas a registered class still delete viainstantiateRecord, even whennode.type !== declared.type.
Reverse Visit Order
destroy constructs a base order from registered orphans in sweepSteps order followed by orderDeployUnits(opts.resources).reverse() with each unit's members reversed.
It builds dependentsById while skipping intra-unit edges (unitOfId(dependent) === unitOfId(id)) so cycles tear down in exact reverse deploy order. A recursive walk visits unvisited dependentsById before deleting each target, breaking cross-unit loops by deleting the first-visited base-order record last:
- Declared resources – Deleted under
notation:destroy:<encodedId>. - Orphans – Instantiated from state via
instantiateRecordand deleted undernotation:orphans:<encodedId>.