# Checkpoint

Stores a state the chain declares, so a later run of the same input resumes after it instead of
running every step before it again.

## Signature

```csharp fragment
// Train<TInput, TReturn>; MonadTask<TInput, TReturn> and Monad<TInput, TReturn> have the same, public
protected MonadTask<TInput, TReturn> Checkpoint<TState>();
```

## Parameters

| Parameter | Description |
|---|---|
| `TState` | The type in Memory to store. It must read back from JSON as the same value (see below). |

## Returns

`MonadTask<TInput, TReturn>`, the next link of the chain. `Checkpoint` is also available inside a track and inside a `Parallel` branch.

## Behavior

When the run reaches the step, the `TState` in Memory is stored, with the track each routing step
before it took. Nothing else in Memory is stored: services, data contexts and objects changed in
place are not written down. A run with no data provider (a plain `Train`, or a host without
`UsePostgres`, `UseSqlite` or `UseInMemory`) takes no checkpoint, and the step does nothing.

### What a state must be

The state is stored as JSON and read back, so it must come back as the same value. The startup
check refuses, naming the member:

- a class that is not sealed, at any depth: a derived value would be read back as its base type
- a member typed `object`, or an interface other than a framework collection
- a field: the serializer writes properties only
- a property with no public setter, `init` accessor or constructor parameter of its name
- a property marked `[JsonIgnore]`

It also refuses a state that reaches a member marked `[TraxSensitive]`, and a checkpoint inside a
track whose routing key is sensitive, whose route is withheld from every record. A sealed record of
data is always accepted, such as `public sealed record CheckedFindings(Findings Findings, string Depth);`.

### When it is written

A checkpoint is written through a data context of its own, in its own transaction, and commits
nothing else the run tracks. Writing one fails the step, classified permanent, and stores nothing,
when:

- the step's Trax data context still holds unsaved changes or an open transaction: a run resumed
  after the checkpoint would skip work that never committed.
- the state's JSON is larger than the cap a requeue's stored input has: four times the mediator's
  `MaxInputJsonBytes`, 1 MiB by default.
- a string in the state holds a NUL character (`\u0000`). Postgres cannot store one, and would read
  back a different value, so every provider refuses it alike. Store such data encoded (Base64, say).

A run that completes deletes its checkpoints. A failed or cancelled run's checkpoints are deleted
with it by metadata cleanup, which keeps a run while a queued resume still needs it.

### Resuming

A run resumes from an earlier run of the same train and the same input that failed or was
cancelled, and that no resume has already completed:

| How | Resumes |
|---|---|
| A manifest's retry | After the failed run's latest checkpoint, when it has one. Otherwise from the top, as before. |
| A dead letter's requeue | The same. |
| `requeueExecution` | Never: it runs the train again from the top. |
| [`resumeExecution(id, from)`](/docs/sdk-reference/graphql-api/mutations/executions#resumeexecution), or the dashboard's **Resume** and **Resume from here** | At the step `from` names, or after the latest checkpoint when `from` is omitted. |
| A state machine entering an invoking state again | Never: the state is the stage's checkpoint. |

The resumed run runs `Junctions()` again and skips every step before its resume point in place: no
junction runs and no decider is asked, but each step keeps the node id and asking it had, so the
decisions after the point replay from the run it resumes. The checkpoint's state and routes go into
Memory, beside the run's input and its services. A resumed run that fails again resumes from the
same checkpoint next time, not from the top.

Steps after the checkpoint run again on every resume.

### What can resume where

Whether a run can resume at a step is decided from the declared chain before anything runs. Every
step from the resume point on must find its inputs in what the checkpoint restores, the train's
input, the container, or a step after the point. A refusal names the step and the type:

| Code | Why |
|---|---|
| `no-checkpoint` | The run wrote no checkpoint before the point. |
| `missing-input` | A step needs a type nothing restores or produces, as in "`Summarize#0` needs `Findings`". |
| `stale-value` | A step reads a type a skipped step overwrote, which the checkpoint does not hold. |
| `off-the-path` | The point is in a track of a routing step after the checkpoint, which the resumed run would not ask. |
| `inside-a-branch` | The point is inside a `Parallel` branch: resume at the `Parallel`, and each branch resumes from its own checkpoint. |
| `chain-changed` | The chain's hash differs from the one stored with the checkpoint: a deploy changed it. |
| `state-changed` | The state type's shape differs from the one stored, so reading it back would quietly default or change members. The shape covers each member's JSON name and type, the serializer attributes on it (`[JsonConverter]`, `[JsonNumberHandling]` and the rest), and each enum's members with their values. |

A decision made before the checkpoint is not stored: a step after it that reads one makes the
check refuse. Put what it needs in the state.

The resumed run checks all of this again as it starts, with the run it names: that it is a run of
the same train, on the same input, that failed or was cancelled, that no resume of it completed,
and that the check above still allows the resume. When one fails:

- a retry or a dead letter's requeue runs from the top and logs why, as a retry always did;
- an operator's resume (`resumeExecution`, or the dashboard's buttons) fails, classified permanent,
  with a `ResumeRefusedException` whose message is the reason: the operator asked to carry on, not
  to start again, so the run is not silently rerun. Requeue it to run it from the top.

The run-level refusals have codes of their own: `unknown-run`, `another-train`, `not-failed`,
`already-resumed` and `different-input`.

### Inside a Parallel

A checkpoint in a branch is stored under the branch's path. A resume into a failed `Parallel` runs
each branch from its own latest checkpoint, and from its start when it has none. A branch whose
last step is a checkpoint it reached runs nothing, and gives the join only what its checkpoint
holds:

```csharp fragment
.Parallel(p => p
    .Branch("embedding", b => b.Chain<EmbeddingSimilarity>().Checkpoint<EmbeddingScores>())
    .Branch("cocitation", b => b.Chain<CoCitationOverlap>()))
.Chain<CombineSignals>()
```

A branch's checkpoint holds what the branch computed from Memory as the run reached the
`Parallel`, so it is restored only when nothing before the `Parallel` runs again: when the run
resumes at the `Parallel` itself, or after a main checkpoint with nothing between it and the
`Parallel` but values handed to the chain and extractions. Resuming at a step before the
`Parallel` runs every branch from its start.

### In the declared graph

A checkpoint is a node of kind `Checkpoint` in the [declared graph](/docs/sdk-reference/train-methods/chain-graph), whose input is its state, with an id such as `Checkpoint<CheckedFindings>#0`.

## Example

```csharp compile
using Trax.Core.Functional;
using Trax.Effect.Services.ServiceTrain;

public class ResearchTopicTrain : ServiceTrain<ResearchRequest, ResearchReport>
{
    protected override Task<Either<Exception, ResearchReport>> Junctions() =>
        Chain<PlanResearch>()
            .Switch<ResearchBrief, Source>(s => s
                .When(Source.Web, w => w.Chain<SearchWeb>())
                .When(Source.Papers, p => p.Chain<SearchPapers>()))
            .Scale<Findings, Depth>(s => s
                .AtLeast(Depth.Skim, k => k.Chain<SkimSources>())
                .AtLeast(Depth.CrossCheck, c => c.Chain<FetchFullTexts>()))
            .Checkpoint<CheckedFindings>()   // a crash in Summarize no longer fetches again
            .Chain<Summarize>()
            .Resolve();
}
```

## See also

- [Checkpoints and branches](/docs/core/checkpoints-and-branches), what to put in a state and where to place the step
- [resumeExecution](/docs/sdk-reference/graphql-api/mutations/executions#resumeexecution)
- [Parallel](/docs/sdk-reference/train-methods/parallel)