Recovery
When a deployment stops mid-run or hits a missing non-rotatable credential, Notation preserves workflow checkpoints and stage locks in .notation/workflows.db. You can resume interrupted runs, adopt unfinished resources, clear stale locks, and apply targeted operator overrides from the CLI.
Resuming interrupted runs
Before calling any cloud API, notation deploy and notation destroy print an execution ID and resume command:
Deploying orders:prod
Execution ID 7f3e9b1a-4c2d-4a8e-9b1c-2d5e6f7a8b9c
To resume after a crash, run notation deploy --execution-id 7f3e9b1a-4c2d-4a8e-9b1c-2d5e6f7a8b9cIf the process crashes or loses connectivity, pass --execution-id to replay completed checkpoints instead of repeating finished cloud operations:
notation deploy --execution-id 7f3e9b1a-4c2d-4a8e-9b1c-2d5e6f7a8b9cnotation destroy --execution-id 7f3e9b1a-4c2d-4a8e-9b1c-2d5e6f7a8b9cOmit --stage when resuming so Notation restores the stage, workflow, and overrides from the saved execution record. Executions that ended in a terminal error cannot resume; fix the underlying cause and start a fresh run instead.
Because plaintext secrets stay strictly in process memory, a resumed run reloads external credential sources automatically and rotates rotatable credentials if an uncompleted consumer needs them.
Unfinished creates
If a crash occurs right after sending a create call, Notation has already persisted the pending resource and any generated physical name. On the next run, Notation recovers automatically:
notation planflags the unfinished create for inspection.notation deployqueries the provider by its saved name and adopts the live resource into state if found, or completescreatecleanly if absent.
Any retired instances left behind by an interrupted create-first replacement appear as delete-retired in plan and retry automatically on the next deploy. If a resource was created without saving a required output, the next deploy backfills it via read (or prompts you to recreate it with --replace when read is unsupported).
Stale holds
Before mutating resources, deploy and destroy acquire an exclusive hold on <app>:<stage>. Concurrent runs warn with the holding execution ID and wait for the lock to clear.
Normal completions and terminal errors release the hold automatically, while crashed runs preserve it so --execution-id keeps ownership. Always resume the holding execution first; only if that process is permanently lost should you clear the lock via clearDeploymentHold (see the Embedded Runtime Reference).
Forced replacements
Use --replace <id>@<hash> to force replacement of an unchanged resource, adding --override-loss-protection <id> if the resource is guarded by rejectChange. The 8-character instance hash prevents a command from targeting the wrong resource instance.
Run notation plan without @<hash> first to inspect the change and print the ready-to-run deploy command with the resolved hash:
notation plan --stage prod --replace sqs-queue:orders
notation deploy --stage prod --replace sqs-queue:orders@3a7f9c12notation plan --stage prod --replace kv:cache
notation deploy --stage prod --replace kv:cache@8e4d1b20 --override-loss-protection kv:cacheManual credentials
When a non-rotatable minted credential is needed by a consumer in a fresh or resumed process, plan and deploy fail with UnobtainableCredentialError and print recovery options.
Pass a replacement value via an environment variable using --credential <id>.<credential>=<ENV_VAR> with plan, then export the secret and apply with the resolved @<hash>:
notation plan --stage prod --credential webhook-key.secret=WEBHOOK_KEY_SECRETexport WEBHOOK_KEY_SECRET="whsec_newly_issued_value"
notation deploy --stage prod --credential webhook-key.secret@7b1e52c0=WEBHOOK_KEY_SECRETYou can also pass --replace <id>@<hash> on the credential-minting resource instead of --credential to re-mint a fresh secret and update consumers.