Checkpoint

Package Trax.CoreNamespace Trax.Core.TrainSource Monad.Checkpoint.cs, CheckpointState.cs

Stores a state the chain declares, so a later run of the same input resumes after it instead of running every step before it again.

Signature

// Train<TInput, TReturn>; MonadTask<TInput, TReturn> and Monad<TInput, TReturn> have the same, public
protected MonadTask<TInput, TReturn> Checkpoint<TState>();

Parameters

Parameter	Description
`TState`	The type in Memory to store. It must read back from JSON as the same value (see below).

Returns

MonadTask<TInput, TReturn>, the next link of the chain. Checkpoint is also available inside a track and inside a Parallel branch.

Behavior

When the run reaches the step, the TState in Memory is stored, with the track each routing step before it took. Nothing else in Memory is stored: services, data contexts and objects changed in place are not written down. A run with no data provider (a plain Train, or a host without UsePostgres, UseSqlite or UseInMemory) takes no checkpoint, and the step does nothing.

What a state must be

The state is stored as JSON and read back, so it must come back as the same value. The startup check refuses, naming the member:

a class that is not sealed, at any depth: a derived value would be read back as its base type
a member typed object, or an interface other than a framework collection
a field: the serializer writes properties only
a property with no public setter, init accessor or constructor parameter of its name
a property marked [JsonIgnore]

It also refuses a state that reaches a member marked [TraxSensitive], and a checkpoint inside a track whose routing key is sensitive, whose route is withheld from every record. A sealed record of data is always accepted, such as public sealed record CheckedFindings(Findings Findings, string Depth);.

When it is written

A checkpoint is written through a data context of its own, in its own transaction, and commits nothing else the run tracks. Writing one fails the step, classified permanent, and stores nothing, when:

the step's Trax data context still holds unsaved changes or an open transaction: a run resumed after the checkpoint would skip work that never committed.
the state's JSON is larger than the cap a requeue's stored input has: four times the mediator's MaxInputJsonBytes, 1 MiB by default.
a string in the state holds a NUL character (\u0000). Postgres cannot store one, and would read back a different value, so every provider refuses it alike. Store such data encoded (Base64, say).

A run that completes deletes its checkpoints. A failed or cancelled run's checkpoints are deleted with it by metadata cleanup, which keeps a run while a queued resume still needs it.

Resuming

A run resumes from an earlier run of the same train and the same input that failed or was cancelled, and that no resume has already completed:

How	Resumes
A manifest's retry	After the failed run's latest checkpoint, when it has one. Otherwise from the top, as before.
A dead letter's requeue	The same.
`requeueExecution`	Never: it runs the train again from the top.
`resumeExecution(id, from)`, or the dashboard's Resume and Resume from here	At the step `from` names, or after the latest checkpoint when `from` is omitted.
A state machine entering an invoking state again	Never: the state is the stage's checkpoint.

The resumed run runs Junctions() again and skips every step before its resume point in place: no junction runs and no decider is asked, but each step keeps the node id and asking it had, so the decisions after the point replay from the run it resumes. The checkpoint's state and routes go into Memory, beside the run's input and its services. A resumed run that fails again resumes from the same checkpoint next time, not from the top.

Steps after the checkpoint run again on every resume.

What can resume where

Whether a run can resume at a step is decided from the declared chain before anything runs. Every step from the resume point on must find its inputs in what the checkpoint restores, the train's input, the container, or a step after the point. A refusal names the step and the type:

Code	Why
`no-checkpoint`	The run wrote no checkpoint before the point.
`missing-input`	A step needs a type nothing restores or produces, as in "`Summarize#0` needs `Findings`".
`stale-value`	A step reads a type a skipped step overwrote, which the checkpoint does not hold.
`off-the-path`	The point is in a track of a routing step after the checkpoint, which the resumed run would not ask.
`inside-a-branch`	The point is inside a `Parallel` branch: resume at the `Parallel`, and each branch resumes from its own checkpoint.
`chain-changed`	The chain's hash differs from the one stored with the checkpoint: a deploy changed it.
`state-changed`	The state type's shape differs from the one stored, so reading it back would quietly default or change members. The shape covers each member's JSON name and type, the serializer attributes on it (`[JsonConverter]`, `[JsonNumberHandling]` and the rest), and each enum's members with their values.

A decision made before the checkpoint is not stored: a step after it that reads one makes the check refuse. Put what it needs in the state.

The resumed run checks all of this again as it starts, with the run it names: that it is a run of the same train, on the same input, that failed or was cancelled, that no resume of it completed, and that the check above still allows the resume. When one fails:

a retry or a dead letter's requeue runs from the top and logs why, as a retry always did;
an operator's resume (resumeExecution, or the dashboard's buttons) fails, classified permanent, with a ResumeRefusedException whose message is the reason: the operator asked to carry on, not to start again, so the run is not silently rerun. Requeue it to run it from the top.

The run-level refusals have codes of their own: unknown-run, another-train, not-failed, already-resumed and different-input.

Inside a Parallel

A checkpoint in a branch is stored under the branch's path. A resume into a failed Parallel runs each branch from its own latest checkpoint, and from its start when it has none. A branch whose last step is a checkpoint it reached runs nothing, and gives the join only what its checkpoint holds:

.Parallel(p => p
    .Branch("embedding", b => b.Chain<EmbeddingSimilarity>().Checkpoint<EmbeddingScores>())
    .Branch("cocitation", b => b.Chain<CoCitationOverlap>()))
.Chain<CombineSignals>()

A branch's checkpoint holds what the branch computed from Memory as the run reached the Parallel, so it is restored only when nothing before the Parallel runs again: when the run resumes at the Parallel itself, or after a main checkpoint with nothing between it and the Parallel but values handed to the chain and extractions. Resuming at a step before the Parallel runs every branch from its start.

In the declared graph

A checkpoint is a node of kind Checkpoint in the declared graph, whose input is its state, with an id such as Checkpoint<CheckedFindings>#0.

Example

using Trax.Core.Functional;
using Trax.Effect.Services.ServiceTrain;
 
public class ResearchTopicTrain : ServiceTrain<ResearchRequest, ResearchReport>
{
    protected override Task<Either<Exception, ResearchReport>> Junctions() =>
        Chain<PlanResearch>()
            .Switch<ResearchBrief, Source>(s => s
                .When(Source.Web, w => w.Chain<SearchWeb>())
                .When(Source.Papers, p => p.Chain<SearchPapers>()))
            .Scale<Findings, Depth>(s => s
                .AtLeast(Depth.Skim, k => k.Chain<SkimSources>())
                .AtLeast(Depth.CrossCheck, c => c.Chain<FetchFullTexts>()))
            .Checkpoint<CheckedFindings>()   // a crash in Summarize no longer fetches again
            .Chain<Summarize>()
            .Resolve();
}