Explanation

How jobs reach workers

Source JobSubmitter/, LocalWorkerService.cs

The JobDispatcher decides that a queued entry runs now. It does not run it. It creates the run's metadata row, commits, and hands the run's id and input to a job submitter (IJobSubmitter). The submitter's only job is to get that run to something that executes the JobRunner, which loads the row, runs your train and records the result.

Keeping that step behind one interface is what lets the same scheduler run its jobs on threads in its own process, on a separate worker fleet, or in AWS Lambda, without anything above the submitter changing.

Which submitter a host gets

The host hasSubmitterWhere the job runs
UsePostgres() or UseSqlite()PostgresJobSubmitter, which inserts a row into trax.background_jobOn LocalWorkerService threads in this process, started automatically
UseInMemory()The in-memory submitterInline, inside the call that submits it. Meant for tests and prototypes
OverrideSubmitter(...)YoursWherever it sends it. Replaces both defaults, and no local workers are started
UseRemoteWorkers, UseSqsWorkers or UseLambdaWorkers with routingAn HTTP, SQS or Lambda submitter, for the trains routed to itOn a runner elsewhere; every other train keeps the default. See Choosing where trains run

Only PostgresJobSubmitter is a public type. The others are selected by their builder methods.

Why the metadata row is committed first

The dispatcher commits the run's metadata row and the entry's move to Dispatched in one transaction, and calls the submitter only after it commits. The worker that picks the job up has to read that row, and the in-memory submitter runs the train inside the call itself, so the row must already be visible when the call is made.

The cost is a window where the row exists and the job may not: the submit can fail after the commit. The dispatcher handles that case, and the rule it follows is the subject of Dispatch failures.

The built-in local workers

With Postgres (or SQLite) the default submitter writes one row per job into trax.background_job, in the same database the rest of Trax uses, and LocalWorkerService runs a pool of worker tasks that take jobs from it. There is no other queue to run or connect to.

JobDispatcher ──► PostgresJobSubmitter ──► trax.background_job ──► LocalWorkerService (N workers)
                  INSERT (metadata_id,                                each: claim ► run ► delete
                  input, priority)

The pool's size, polling interval, batch size, visibility timeout and shutdown grace are LocalWorkerOptions, set with ConfigureLocalWorkers.

Worker Lifecycle

Each worker loops through three phases.

Claim, in a transaction:

SELECT * FROM trax.background_job
WHERE fetched_at IS NULL
   OR fetched_at < NOW() - make_interval(secs => :visibility_timeout)
ORDER BY priority DESC, created_at ASC
LIMIT :batch_size
FOR UPDATE SKIP LOCKED

FOR UPDATE SKIP LOCKED gives each job to exactly one worker without workers waiting on each other: a row another worker has locked is skipped, not waited for. Higher priorities go first, and jobs of the same priority in the order they were queued. The worker sets fetched_at on what it claimed and commits.

Run, in a fresh DI scope: the worker resolves the JobRunner and runs it with the run's id and input. While a job runs, the worker refreshes fetched_at on it, and on the jobs waiting behind it in its batch, every third of the visibility timeout, so a job that runs longer than the timeout keeps its claim. Cancelling a run from the dashboard or ITraxScheduler reaches the running job however long it has been running.

Delete, whether the job succeeded or failed: the worker deletes the background_job row. The delete does not take the host's stopping token, so a job that finishes inside the shutdown grace period is still removed rather than left to be claimed again. The table is only a transient queue; the run's history is in its metadata row.

When the host starts shutting down in the middle of a batch, the jobs not yet started are released (their fetched_at is cleared), so any worker can claim them at once.

Crash recovery

A worker that dies after claiming a job stops refreshing fetched_at. Once the timestamp is older than the visibility timeout, the claim query matches the job again and another worker runs it:

Worker A claims job #1 → fetched_at = 10:00:00
Worker A crashes at 10:01:00
At 10:30:00 (with the default 30-minute timeout):
Worker B's claim query finds job #1 (fetched_at < NOW() - 30m)
Worker B claims and runs job #1

The second delivery does not run the train twice. The run's metadata row decides which delivery runs it, as JobRunner: Duplicate deliveries describes.

InMemory

With UseInMemory() the submitter runs the train inside the submit call, synchronously. There are no worker threads and no tables, so tests get the scheduling API without a database. There is no work queue reader either: the InMemory ManifestManager dispatches due manifests itself, inline. Retry delays, decision replay and the stale-run reapers do not run there, which is why retries and dead letters are best tested against a real Postgres (see Administrative Trains: InMemory).

Elsewhere

The same interface takes runs off the scheduler's process. Three of the models push each run to a runner: over HTTP, through an SQS queue, or by invoking a Lambda function directly. The fourth runs a standalone worker process that takes jobs from the same background_job table the local workers use. Every model shares the one database. Choosing where trains run compares them.

A custom job submitter

A host that needs another transport implements IJobSubmitter and registers it with OverrideSubmitter. Two overloads are required; the priority and cancellation overloads have default implementations that fall back to them, so a submitter that ignores priority need not implement them:

using Trax.Scheduler.Services.JobSubmitter;
 
public class ExternalQueueJobSubmitter(IExternalJobQueue queue) : IJobSubmitter
{
    public Task<string> EnqueueAsync(long metadataId) =>
        queue.SendAsync(metadataId, input: null, priority: 0, CancellationToken.None);
 
    public Task<string> EnqueueAsync(long metadataId, object input) =>
        queue.SendAsync(metadataId, input, priority: 0, CancellationToken.None);
 
    // Optional: carry the entry's priority to the backend.
    public Task<string> EnqueueAsync(
        long metadataId,
        object input,
        int priority,
        CancellationToken cancellationToken
    ) => queue.SendAsync(metadataId, input, priority, cancellationToken);
}
using Trax.Effect.Data.Postgres.Extensions;
using Trax.Effect.Extensions;
using Trax.Mediator.Extensions;
using Trax.Scheduler.Extensions;
using Trax.Scheduler.Services.JobSubmitter;
 
services.AddTrax(trax =>
    trax.AddEffects(effects => effects.UsePostgres(connectionString))
        .AddMediator(assembly)
        .AddScheduler(scheduler =>
            scheduler.OverrideSubmitter(s =>
                s.AddScoped<IJobSubmitter, ExternalQueueJobSubmitter>()
            )
        )
);

Whatever consumes the queue runs the JobRunner the way the local workers do: it resolves IJobRunnerTrain and runs it with new RunJobRequest(metadataId, input), in a host where AddTraxJobRunner has registered it.

Moving a host off the removed Hangfire backend is covered in Migrate from Hangfire.

SDK Reference

ConfigureLocalWorkers | UseRemoteWorkers | UseSqsWorkers | AddTraxWorker | AddTraxJobRunner | OverrideSubmitter