Cancellation and timeouts in the scheduler

Source LocalWorkerService.cs, CancellationRegistry/, CancelTimedOutJobsJunction.cs, IJobSubmitter.cs

A run the scheduler dispatches can be stopped three ways: the host shuts down, someone cancels it, or it runs past its timeout. All three end in the same place, an OperationCanceledException the train sees through its token or before its next EffectJunction, and a ServiceTrain records it as Cancelled. How a train handles the token itself is in How cancellation works; this page is what the scheduler adds on top.

Shutdown and the grace period

The scheduler's polling services (the ManifestManager, the JobDispatcher, metadata cleanup) pass the host's stoppingToken to the trains they run, so a shutdown cancels them straight away.

The local worker that runs your jobs does not hand its jobs the stopping token. Each run gets its own CancellationTokenSource, and when the host signals shutdown the worker calls CancelAfter(ShutdownTimeout) on it, 30 seconds by default:

Host signals shutdown (stoppingToken fires)
        │
        ▼
┌───────────────────────────────────────────┐
│  runCts.CancelAfter(ShutdownTimeout)      │  ← grace period starts
│                                           │
│  The run carries on, and has until the    │
│  grace period ends to finish              │
│                                           │
│  Grace period over: runCts fires          │  ← the run is cancelled
└───────────────────────────────────────────┘

The grace period exists for runs in the middle of something that should not be cut off, a transaction or a call to an external system, and is long enough for most of them to finish normally. A run still going at the end is cancelled like any other.

When a run finishes, inside the grace period or after being cancelled, the worker deletes its background_job row without the stopping token. That token has already fired, and the delete is bookkeeping for finished work. Were it cancellable, every job that finished during shutdown would leave its row behind for another worker to claim after VisibilityTimeout, only to refuse it as no longer Pending.

With a BatchSize above 1, a job the worker claimed but had not started when shutdown began is not started at all. Its fetched_at is cleared so another worker can claim it at once, rather than each remaining job getting a grace period of its own.

ShutdownTimeout and BatchSize are set with ConfigureLocalWorkers:

services.AddTrax(trax => trax
    .AddEffects(effects => effects.UsePostgres(connectionString))
    .AddMediator(assembly)
    .AddScheduler(scheduler => scheduler
        .ConfigureLocalWorkers(options => options.ShutdownTimeout = TimeSpan.FromSeconds(60))
    )
);

Cancelling a run

The dashboard's cancel button, ITraxScheduler.CancelAsync(externalId) for every pending and running execution of a manifest, and CancelGroupAsync(groupId) for a manifest group, all use the same two layers:

The database flag. CancellationRequested is set on each run's metadata row. This works on any host. A run still Pending is recorded Cancelled without running when the job runner picks it up. A running one sees the flag through CancellationCheckProvider, registered by AddJunctionProgress(), before its next EffectJunction.
The registry. On the host that is running the job, the worker has registered the run's CancellationTokenSource with ICancellationRegistry under its metadata id, and the cancel calls TryCancel, which cancels the token at once. The first registration for a run holds it until that worker removes it with Unregister(metadataId, cts), so a second delivery of the same job on the same host, which does not run the train, cannot take the registration from the one that does.

Dashboard cancel, CancelAsync, CancelGroupAsync
    ├──→ SET cancel_requested = true      (always, any host)
    └──→ ICancellationRegistry.TryCancel() (the host running the job)
            ├─ found     → token cancelled → the awaited call throws
            └─ not found → nothing; the flag covers it

The registry is the only layer that stops a train built from plain Junctions: the flag is read by a junction effect, and junction effects run only around an EffectJunction. A train of plain junctions running on another host runs to the end. See Make a junction cancellable.

A cancelled run is not retried and creates no dead letter. A cancelled run of a scheduled manifest uses up the occurrence it ran for: the manifest next runs at its next scheduled time, not on the next polling cycle.

Timeouts

Every polling cycle, the ManifestManager's CancelTimedOutJobsJunction looks at each run in progress and cancels any that has run longer than its timeout, through the same two layers. A run's timeout is the TimeoutSeconds of the manifest of the run at the root of its ParentId chain, so a nested train shares the timeout of the scheduled run it belongs to, or DefaultJobTimeout (20 minutes) when that manifest sets none. DefaultJobTimeout applies only to runs a scheduler dispatched; a train run directly on the train bus is not timed out. Runs of a manifest disabled while they run are still timed out. The resolution rules are in Timeout Enforcement.

services.AddTrax(trax => trax
    .AddEffects(effects => effects.UsePostgres(connectionString))
    .AddMediator(assembly)
    .AddScheduler(scheduler => scheduler
        .DefaultJobTimeout(TimeSpan.FromMinutes(30))
        .Schedule<ISyncTrain>(
            "sync-hourly",
            new SyncInput(),
            Every.Hours(1),
            options => options.Timeout(TimeSpan.FromMinutes(10)))
    )
);

A timeout cancels the run rather than failing it. The run is recorded Cancelled and is not retried, so an hourly job that always runs past its timeout runs once an hour, not continuously. Because the cancel uses the same layers, a timed-out run on another host stops only at an EffectJunction, and one made of plain junctions is not stopped by its timeout at all. The stale-run reaper, which fails a run whose worker died, is a separate safety net; see ManifestManager.

Job submitters

IJobSubmitter has EnqueueAsync overloads that take a CancellationToken, with default implementations that call the overloads without one, so a custom submitter written before them still compiles. The built-in submitters implement them: the Postgres submitter passes the token to SaveChangesAsync, and the in-memory one passes it to the train's Run. See Job Submission.

SDK Reference

ConfigureLocalWorkers | CancelAsync / CancelGroupAsync | AddScheduler | Schedule | AddJunctionProgress