Explanation

How retries work

Source RetryDecisionReplay.cs, RetryResume.cs, CreateWorkQueueEntriesJunction.cs

Trax has no per-junction retry. When a manifest's run fails, the scheduler runs the manifest again, from its first junction, until it succeeds or runs out of retries and is dead-lettered. Only a manifest's run is retried this way: a train run directly, queued through the operations queueTrain mutation, or queued by a [TraxMutation] dispatch mutation with mode: QUEUE (or GraphQLOperation.Queue) has no retry, never replays and never becomes a dead letter. A train that should recover runs from a manifest (Schedule, or ScheduleOnceAsync for a single run). Building a train that recovers lists every piece it needs.

Counting failures

MaxRetries is the number of retries after the first run, so a manifest runs at most MaxRetries + 1 times before it is dead-lettered: MaxRetries(0) dead-letters on the first failure, and the default of 3 on the fourth. A failure counts only while it started inside the manifest's failure window (its FailureWindow, or the scheduler's FailureCountWindow, 24 hours by default) and after its latest resolved dead letter. So failures weeks apart never add up to a dead letter, and a requeued manifest starts fresh. A success does not reset the count inside the window, but it does end the backoff.

A cancelled run, timed out or cancelled by an operator, is not a failure: it is not retried and does not count. A run that stopped because something inside it gave up, such as an HttpClient timeout, was not cancelled: it is recorded Failed, classified Transient, and counts like any other failure. A dispatch attempt that never reached a runner (DispatchRequeued) does not count either; see JobDispatcher: Dispatch failures.

Retry delay and backoff

When a manifest's latest finished run failed and its counted failures have not exceeded MaxRetries, the next run, the retry, is delayed by an exponential backoff over the counted failures:

delay = min(DefaultRetryDelay * RetryBackoffMultiplier ^ (failureCount - 1), MaxRetryDelay)

With the defaults (5 minutes, 2.0, 1 hour):

Only a retry waits. Once a run succeeds or is cancelled, the next occurrence runs on time, however many failures are still inside the window; those failures still set the next retry's delay and still count toward the dead letter. The options are on AddScheduler, and the scheduler warns at startup when the backoff for DefaultMaxRetries retries alone reaches the failure window, since a manifest that always fails would then never be dead-lettered.

The delay is the work queue entry's ScheduledAt: the JobDispatcher skips an entry until it comes due. The comparison runs on the database's clock, so ScheduledAt is the database's time plus the delay, not the scheduler host's. A retry with no delay (DefaultRetryDelay(TimeSpan.Zero)) stores no ScheduledAt and is dispatched on the next poll, however far the host's clock runs ahead.

When a retry runs

A retry is not a timer of its own. It is the manifest's next due run, queued with that delay. A failed run does not move the point a schedule counts from (an interval counts from the last success, a cron from its last success's occurrence), so a manifest that failed while it was due is still due, and the ManifestManager queues the retry on its next cycle:

So with MaxRetries(2), DefaultRetryDelay(2 s) and the default multiplier, a due manifest that keeps failing runs three times within about ten seconds and is dead-lettered on the next cycle. The Scheduling sample runs exactly that. A requeue of the dead letter, by contrast, queues a run at once, with no backoff; see Resolve a dead letter.

Retries replay decisions

A retry exists because something after a decision failed: a tool step threw, a database timed out. A retry runs the chain from its first junction, and asking a model again costs a call per question and can be answered differently. So when a manifest's run fails, its retry, queued by the ManifestManager, and a requeue of its dead letter replay the decisions the failed run recorded: the retry takes the tracks the failed run's deciders chose instead of asking them again. Only the answers are replayed. Every ordinary junction runs again, side effects included.

The scheduler reads the run to replay from the database, never from a caller: it is the manifest's latest finished run, and only when the next run is a retry of it, that is, the run failed and its failure still counts toward the retries (the same test that applies the backoff). A failure that an acknowledged dead letter or the failure window has set aside is not retried, so the next occurrence asks afresh. It links the retry only when all of these hold:

So a retry replays at most once in a row: answers that were replayed into a failure are not replayed again, and the next retry asks afresh. A run's answers are replayed at most once, and that is enforced when an entry is dispatched: an entry naming a run that another run already replays is dispatched with its link dropped, so it runs asking afresh, and the dispatcher logs it at Information. A manifest edited between the failure and the retry asks afresh, because the answers were given about the old input. Asking afresh is never an error, and a lookup that fails is logged and asks afresh rather than holding up the retry. A dependent's failed run is dated by when its work queue entry was dispatched (work_queue.dispatched_at), which the database stamps as it stamps the parent's last success; a run with no dispatched entry falls back to its start time.

A replay that never started still counts as the one replay. A retry queued to replay that never ran (its dispatch ran out of attempts, its stored input could not be read, the stale pending reaper failed it) keeps its link, so as the manifest's latest failed run it makes the next retry ask afresh. Those answers are not offered to a later retry: a replay can be lost, but never applied twice. A requeue of such a run follows the kept link to the answers it never replayed.

A run whose replay could not be honoured, and which asked its deciders afresh instead (its metadata marked replay_abandoned), is different: its answers are its own. It counts as a run that asked its deciders itself, so the next retry may replay its answers once, and it does not use up the replay of the run it names.

The "nothing else replays the failed run" check reads the run history, so metadata cleanup keeps that history whole on every host: it never deletes a run that replays another while it keeps the run replayed. Both expired, they are deleted in the same transaction; a replay whose source is kept stays with it.

A link can still become impossible to honour after it is written: the failed run can be deleted before the retry runs, or the retry can land on a host that does not record decisions. A manifest's run then logs a warning and asks afresh, rather than failing as a manual requeue would.

Within the replay, each answer is still checked one by one: it replays only into a question asked the same way, about a state that hashes the same, and only while it is younger than ReplayAnswersFor (24 hours by default). An answer that fails any of these is asked afresh. The InMemory provider does not replay retries, because its ManifestManager dispatches without work queue entries to compare inputs against. Replay needs AddDecisionRecording() on the hosts that run the manifest's train.

Asking afresh on purpose

A manifest whose questions should always be answered on current information turns replay off:

services.AddTrax(trax =>
    trax.AddEffects(effects => effects.UsePostgres(connectionString))
        .AddMediator(assembly)
        .AddScheduler(scheduler =>
            scheduler.Schedule<IScoreLeadsTrain>(
                "score-leads",
                new ScoreLeadsInput(),
                Every.Minutes(30),
                options => options.ReplayDecisionsOnRetry(false)
            )
        )
);

The flag is stored on the manifest as replay_decisions_on_retry, so every scheduler host reads the same value. Stated in code, it is written on every seed, so ReplayDecisionsOnRetry(true) in code turns replay back on at each restart over an operator's runtime opt-out, as every stated setting does. Where operators manage the flag at runtime, leave it out of code. Left unstated, a new manifest replays and an existing one keeps its value. IOperationsService.SetManifestsReplayDecisionsOnRetryAsync sets it at runtime. Turning it off reaches a retry already queued: the write clears the link on the manifest's queued entry in the same transaction as the flag, its message counts the retries it cleared ("2 queued retry(s) no longer replay a failed run's decisions."), and the dispatcher checks the flag again when it claims an entry. An opt-out committed in the instant between that check and the run's start still lets that one run replay.

For a single occasion, the dead-letter requeues, TriggerAsync and RequeueExecutionAsync take askAfresh. A requeue asked afresh queues no link, and a trigger asked afresh clears the link of the queued retry it releases. The trigger clears it only while the entry is still queued: when the dispatcher claimed the entry first, the run replays anyway, and the trigger says so. Its ManifestTriggerResult has AlreadyDispatched set and ReplayDecisionsOf naming the run whose decisions it replays, and a warning is logged.

A requeue of an execution keeps the replay-once rule too. When a queued entry or a run, in any state, already replays the run being re-queued, RequeueExecutionAsync still queues it but asks afresh, and its message ends "It asks its deciders afresh: the decisions of execution 42 are already replayed by another run or queued entry, and are replayed once." A manifest's retry does not link a run a requeue replays, and a requeue does not link a run a retry replays. The database holds one queued entry per replayed run (the unique index ix_work_queue_unique_queued_replay), so when two hosts, two operators, or a requeue and a retry queue a replay of the same run in the same instant, the one that loses is queued again without the link and asks afresh. That holds for a retry, a dead-letter requeue, a requeue-all and RequeueExecutionAsync; nothing fails, and a requeue's message says it asked afresh. Trax.Scheduler/docs/adr/0017 records why retries replay, and why only once.

Retries resume from a checkpoint

A train whose chain declares a checkpoint (Checkpoint<TState>()) stores that state when a run reaches it. A manifest's retry and a requeue of its dead letter then resume after the failed run's latest checkpoint instead of running every step again: the new entry names the failed run in ResumeFrom, and the run skips every step before the checkpoint, including the writes it already committed. Declaring the checkpoint is the opt-in; it does not depend on ReplayDecisionsOnRetry, which still decides whether the retry replays the failed run's decisions.

The failed run is found as the replay finds it: the manifest's latest finished run, failed, a run of the manifest's train dispatched from an entry of the manifest with no subject key and exactly the input the retry is given; a dependent whose parent has succeeded again since is a new firing. The scheduler then asks whether the train's declared chain can resume after that run's latest checkpoint, following a run that was itself a resume back to the checkpoints it resumed from, so a second failure after a resume resumes from the same checkpoint. When the run stored no checkpoint, when a checkpoint no longer matches the chain or the state's shape after a deploy, or when the lookup fails, the retry reruns the chain from the top as before. A run a state machine invoked is never resumed this way.

A run has at most one queued resume (the unique index ix_work_queue_unique_queued_resume). When an operator's resumeExecution and a retry of the same run are queued in the same instant, one of them carries the resume: a retry that loses is queued without it and reruns from the top, and an operator's resume that loses is refused. requeueExecution always reruns from the top.

On InMemory

The InMemory provider has no work queue: its ManifestManager dispatches inline. It does not delay retries the way Postgres does and does not replay decisions, so retries are best tested against a real database, as the Scheduling sample's E2E suite does.

SDK Reference

Schedule | AddScheduler | AddDecisionRecording | ITraxScheduler