Explanation

Administrative Trains

Source SchedulerConfigurationBuilder.Build.cs, AdminTrains.cs

The scheduler runs its own work as trains, the same way it runs yours, so every cycle has a metadata row and shows up in the dashboard when it fails. AddScheduler() registers these trains and the hosted services that run them; you never register or start them yourself, and you pass only your own assemblies to AddMediator().

TrainWhat it does
ManifestManagerCancels timed-out runs, fails stale ones, dead-letters manifests out of retries, and writes due manifests to the work queue
JobDispatcherReads the work queue, applies the capacity limits, creates each run's metadata and hands it to a job submitter
JobRunnerRuns your train on a worker and records its success on the manifest
MetadataCleanupDeletes old metadata rows; on by default with a database provider, tuned by AddMetadataCleanup()
DeadLetterCleanupTrainDeletes resolved dead letters past their retention; see Dead letter auto-purge

AdminTrains.Types lists them, interfaces and classes, including the InMemory variant of the ManifestManager. Their runs do not count toward MaxActiveJobs, they are never timed out, and the dashboard leaves them out of its statistics by default.

The hosted services

Which services run depends on the data provider. Postgres and SQLite count as a database provider; InMemory does not.

ServiceRegisteredWhat it does
SchedulerConfigBootstrapHostedServicealwaysApplies the scheduler settings saved from the dashboard or updateScheduler over the values in code, then rereads them every 5 seconds so a save on any host reaches this one
SchedulerStartupServicealwaysRuns once, before the polling services. With a database provider it first fails every InProgress run that started before this host did (RecoverStuckJobsOnStartup, on by default), then seeds the manifests the builder declared, prunes orphaned manifests and deletes manifest groups with no manifests left. InMemory only seeds
ManifestManagerPollingServicealwaysRuns the ManifestManager every ManifestManagerPollingInterval (5 seconds by default). On InMemory it runs InMemoryManifestManagerTrain, which dispatches due manifests inline instead of writing to the work queue
JobDispatcherPollingServicedatabase providerRuns the JobDispatcher every JobDispatcherPollingInterval (2 seconds by default), and sooner when woken
QueuedWorkListenerServicedatabase providerWakes the dispatcher when work is queued, so a queued run does not wait for the next poll
LocalWorkerServicedatabase provider, with the default submitterThe worker pool that runs jobs from trax.background_job; see How jobs reach workers
MetadataCleanupPollingServicedatabase providerRuns the MetadataCleanup train every CleanupInterval, and once at startup
DeadLetterCleanupPollingServicedatabase providerRuns the dead-letter purge every DeadLetterCleanupInterval (1 hour by default). Always registered, because AutoPurgeDeadLetters can be turned on at runtime

The intervals and switches are options of AddScheduler.

Startup seeding is strict: a manifest that cannot be seeded stops the host, because broken manifest configuration should not start quietly. A seed that fails on what the provider's SQL dialect calls transient (a lost or refused connection, a timeout, a deadlock or serialization failure, SQLite's busy and locked codes) is tried up to five times in all, the delay doubling from 2 seconds. Any other failure, a constraint violation included, fails at once. Pruning is the exception: it is housekeeping, so a failed prune is logged and the host starts anyway.

The stuck-job recovery has no way to know which process owns a run, so "started before this host" is the whole test. On a single host that is exactly the runs a crash left behind. Where several hosts or remote workers share the database, it also fails a run still executing elsewhere, which is why it can be turned off with RecoverStuckJobsOnStartup(false).

The work queue

Every run the scheduler starts goes through the work_queue table. The ManifestManager does not hand a due manifest to a worker; it writes a work queue entry. TriggerAsync, a dead-letter requeue, a dashboard re-run and a GraphQL queue mutation write entries too. The JobDispatcher is the only thing that reads them.

ManifestManager (schedules) ──┐
TriggerAsync, requeues ───────┼──► work_queue ──► JobDispatcher ──► job submitter ──► JobRunner
Dashboard, queue mutations ───┘

That gives capacity one place to be enforced. MaxActiveJobs, per-group limits, priority and subject serialization are all applied by the dispatcher, whatever wrote the entry. The ManifestManager can decide what is due without knowing how busy the system is, and the two run on their own intervals, so each can be tuned on its own.

An entry records the train to run (its interface's full name), the serialized input and its type, its status (Queued, then Dispatched), the manifest it came from if any, and, once dispatched, the metadata row of its run.

On Postgres a trigger on work_queue sends a notification in the transaction that queues an entry, so every dispatcher, on any host, wakes when that transaction commits and hears nothing if it rolls back. A run queued on an API host starts within moments even when the scheduler polls every 30 seconds. On SQLite the wake comes from a save in the same process. The poll stays the fallback, and the only clock for an entry whose ScheduledAt comes due, a dispatch retried after a backoff, or a manifest that is enabled again. A connection pooler in transaction mode, such as PgBouncer, does not carry LISTEN, so behind one the dispatcher polls. The listener holds one unpooled connection, shown in pg_stat_activity as trax_queued_work_listener; when it is lost the listener logs a warning and tries again after 1 second, doubling to 30.

InMemory

With InMemory there is no work queue reader. InMemoryManifestManagerTrain runs only LoadManifestsJunction, ReapFailedJobsJunction, DetermineJobsToQueueJunction and InMemoryDispatchJobsJunction, which runs each due manifest through the in-memory job submitter inline. It has no timeout cancellation, no stale Pending or InProgress recovery and no stale staged-entry sweep, and a run queued through GraphQL (mode: QUEUE) stays queued. It resolves a manifest's stored input type only among the registered trains' inputs, as the dispatcher does; a manifest whose input resolves to nothing is logged and leaves no Pending row.

Several hosts

Each service has its own way of staying safe when several hosts share the database: the ManifestManager takes an advisory lock so one host runs it per cycle, the JobDispatcher claims entries with FOR UPDATE SKIP LOCKED, and the cleanups are idempotent. Multi-Server Concurrency explains each.

Excluding your own trains from MaxActiveJobs

A high-frequency internal train of your own can be left out of the count the same way the administrative trains are:

services.AddTrax(trax =>
    trax.AddEffects(effects => effects.UsePostgres(connectionString))
        .AddMediator(assembly)
        .AddScheduler(scheduler =>
            scheduler.MaxActiveJobs(100).ExcludeFromMaxActiveJobs<ISyncTrain>()
        )
);

SDK Reference

AddScheduler | AddMetadataCleanup