Explanation

How remote dispatch works

A train runs off the scheduler host along one of two paths. A queued run (a manifest coming due, a queue* mutation, QueueAsync) leaves through an IJobSubmitter, when the JobDispatcher takes its work queue entry. A synchronous run (a run* mutation, a [TraxQuery] query, ITrainExecutionService.RunAsync) leaves through the IRunExecutor, and its caller waits for the output. Each remote transport replaces one of the two for the trains it covers, and each has a receiver on the far side that runs the train against the same database. Which transport to choose for which train is the subject of Remote Execution; this page is about what each one does.

Routing a queued run

Every queued run goes to the default submitter, which writes a background_job row for the local workers, unless its train is routed elsewhere. A train is routed by, in order:

1. ForTrain<TTrain>() in the routing callback of a UseRemoteWorkers, UseSqsWorkers or UseLambdaWorkers call.
2. [TraxRemote] on the train class, which sends a train no call names to the first of those calls in the builder, whatever its kind.
3. The default submitter, for everything else.

A train can be routed to one submitter only, and the scheduler's build fails if two calls name it. It also fails when a [TraxRemote] train has none of the three calls to go to: running a train marked remote on the scheduler host, often the one host it was kept off for isolation, is not treated as a fallback. Routing never replaces the default submitter, so the local workers keep running every train no call routes.

Each call is its own submitter, with its own options and client. Two UseRemoteWorkers calls to two endpoints never send one endpoint's credentials, headers or signing key to the other.

The submitters

All three serialize a RemoteJobRequest: the run's metadata id and, when there is one, its input. The receiver loads the metadata from the shared database, claims the run, runs the train and records the outcome, so the result never travels back through the submitter.

The HTTP submitter reads the runner's response and fails the submit with a TrainException on a non-success status, an IsError response, or a success status whose body is not a RemoteJobResponse for the same metadata id: a proxy's page, an empty body, a misrouted BaseUrl.

A failed submit is not always a failed delivery. The dispatcher looks at the run's row. If the runner has already started the run (it ran the train and reported the failure, or is still running it when the request timed out), the run is the runner's and its outcome is recorded there, so the entry is not requeued and the train does not run twice. Only a run still Pending (the request never reached a runner, or the runner refused it before starting) counts as a failed dispatch attempt and is requeued, up to MaxDispatchAttempts.

SQS and Lambda both deliver at least once: SQS redelivers by design, and Lambda retries an asynchronous invocation with the same payload. The run's Pending metadata row is what stops a second delivery from running the train a second time, which is why the receivers check those requests for their signature but not for freshness or repeats.

Dispatching many at once

The JobDispatcher dispatches the entries of a cycle one at a time by default. That costs nothing for the default submitter, whose dispatch is a row insert, but the HTTP submitter waits for the runner's response, so fifty entries of two seconds each make a hundred-second cycle. MaxConcurrentDispatch dispatches up to that many entries of one cycle at once. Entries are claimed with FOR UPDATE SKIP LOCKED, so concurrent dispatch creates no duplicate runs. Each concurrent dispatch holds its own DI scope and database connection, so the setting is kept well below the connection pool size (100 by default with Npgsql). See Parallel Dispatch.

The run executors

UseRemoteRun and UseLambdaRun replace the IRunExecutor that ITrainExecutionService.RunAsync runs through, for every train, after the caller has been authorized and the input read on the calling host. Without either, LocalRunExecutor runs the train in process through ITrainBus.RunByNameAsync.

Both serialize a RemoteRunRequest (the train name, the input JSON and the input type) and wait for a RemoteRunResponse. On success the output is deserialized and returned. On failure they throw RemoteRunException, a TrainException, whose PublicMessage is the runner's message for a client when the failure was a train author's TrainException, and null for a transport failure or any other failure. The receiver (UseTraxRunEndpoint, or TraxLambdaFunction for a Run envelope) calls ITrainExecutionService.RunAsync itself, so the run's metadata is written to the shared database and appears on the dashboard like a local one.

Set beside a submitter, the two paths differ in what the caller gets:

Cancelling a remote run

No transport carries a cancel to the receiver. A cancel from the dashboard sets the run's cancel flag in the database, and the train sees it at its next junction boundary when the receiver registers CancellationCheckProvider, through AddJunctionProgress(). A junction already running is not interrupted. A Lambda function also cancels its own job shortly before the function's timeout, so the run can record that it was cancelled rather than be left InProgress for the reaper.

SDK Reference

UseRemoteWorkers | UseRemoteRun | UseSqsWorkers | UseLambdaWorkers | UseLambdaRun | TraxLambdaFunction | AddTraxJobRunner