ManifestManagerTrain

The ManifestManager is the first half of each polling cycle. It figures out which manifests are due for execution and writes them to the work queue. It doesn't dispatch anything; that's the JobDispatcher's job.

Chain

CancelTimedOutJobsJunction → ReapStalePendingMetadataJunction → ReapStaleInProgressMetadataJunction → LoadManifestsJunction → ResolveStaleStagedEntriesJunction → ReapFailedJobsJunction → DetermineJobsToQueueJunction → CreateWorkQueueEntriesJunction

Junctions

The timeout and stale-run junctions come first and work on runs directly, not on the loaded manifests, so the manifests are loaded after them and a run they fail is already counted in the same cycle.

CancelTimedOutJobsJunction

Finds every InProgress run past its timeout and requests cooperative cancellation. A run's timeout is resolved from the run at the root of its ParentId chain: that run's manifest TimeoutSeconds, or DefaultJobTimeout when it sets none, which applies only when a scheduler dispatched the root (a manifest, a work queue entry or a background_job row). A nested train shares its root's timeout; a train run directly on the train bus, or whose chain cannot be resolved within 16 levels, is not timed out. Runs of disabled manifests are included; the scheduler's own trains, trains in ExcludedTrainTypeNames, and trains nested in either are not. See Timeout Enforcement. Sets CancellationRequested = true in the database (picked up by CancellationCheckProvider at the next junction boundary) and attempts same-server instant cancellation via the CancellationRegistry.

ReapStalePendingMetadataJunction

Fails Pending metadata that has not been picked up within StalePendingTimeout (default: 20 minutes). Acts as a safety net for dispatch failures where the worker never started executing the job (e.g., remote worker unreachable, Lambda crashed after receiving the request).

ReapStaleInProgressMetadataJunction

Fails InProgress metadata that has not completed within StaleInProgressTimeout (default: 60 minutes). Acts as a safety net for hard crashes. Lambda hard-kills, OOM events, or process crashes where the worker dies without reaching FinishServiceTrain. This timeout should be longer than DefaultJobTimeout to allow cooperative cancellation (via CancelTimedOutJobsJunction) to propagate before force-failing.

A run whose own timeout, resolved as CancelTimedOutJobsJunction resolves it, is longer gets as much longer: it is failed at the later of StaleInProgressTimeout and that timeout plus the same grace (StaleInProgressTimeout - DefaultJobTimeout, 40 minutes with the defaults). A manifest with a three hour timeout therefore has its run failed as stale at 3 h 40 min, not at 60 minutes while it is still working.

Newly-failed metadata from both stale reapers is counted by LoadManifestsJunction in the same ManifestManager cycle, enabling dead-lettering if retries are exhausted.

Failing a run, from either reaper, also releases its subject key, so a run still pending past StalePendingTimeout or still working past its stale threshold stops holding its subject.

LoadManifestsJunction

Projects all enabled manifests into lightweight ManifestDispatchView records using a single database query with pre-computed aggregate flags (FailedCount, HasAwaitingDeadLetter, HasQueuedWork, HasActiveExecution, HasSuccessfulMetadata) the end time of the manifest's latest cancelled run, whether its latest finished run failed (LatestFinishedRunFailed), and, for dependents, when their latest successful and cancelled runs were dispatched. These flags are computed via COUNT/EXISTS subqueries pushed into the database, keeping query cost O(manifests) regardless of how large the child tables (Metadatas, DeadLetters, WorkQueues) grow.

The projection uses AsNoTracking(), the results are read-only snapshots used for scheduling decisions only. No unbounded child collections are loaded into memory.

Scaling note: LoadManifestsJunction loads all enabled manifests in a single query. For typical production deployments (up to 10K manifests), this is efficient with proper indexes.

ResolveStaleStagedEntriesJunction

Resolves work queue entries that a crash left unconfirmed. A train with DeferQueuePromotion commits its entry unconfirmed, runs OnQueue, then confirms it; a process that stopped in between leaves an entry the dispatcher will never claim. Any entry still Queued and unconfirmed after StaleStagedEntryTimeout (default: 10 minutes) is:

IWorkQueuePromotion.PromoteAsync, which an enqueue calls once its hook returns, only confirms an entry that is still Queued, so an entry this junction has already cancelled stays cancelled. The promotion methods work on every data provider, the InMemory provider included. Like the rest of the train, this runs only on the server holding the leader lock, and not at all while the ManifestManager is disabled (ManifestManagerEnabled = false). A deployment that turns the ManifestManager off everywhere leaves stranded staged entries unresolved.

ReapFailedJobsJunction

Scans loaded manifests for any whose failure count exceeds MaxRetries. MaxRetries is the number of retries after the first run, so MaxRetries(0) dead-letters on the first failure and the default of 3 on the fourth; a manifest with no counted failure is never dead-lettered. For each, it creates a DeadLetter record with status AwaitingIntervention and persists immediately.

A manifest is only reaped if it doesn't already have an unresolved dead letter. This prevents duplicate dead letters from accumulating when the same manifest fails across multiple polling cycles.

FailedCount only counts failures that started within the manifest's own failure window (manifest.failure_window_seconds, set with FailureWindow) or, when it has none, FailureCountWindow (default: 24 hours), and after the most recent dead letter resolution. When a dead letter is resolved (retried or acknowledged), the failure counter effectively resets, only new failures contribute toward the next MaxRetries threshold. This prevents retried manifests from being immediately re-dead-lettered due to historical failures that were already addressed. The window does the same for failures that were never dead-lettered: a failure a month ago neither lengthens a retry's delay nor counts toward a dead letter. A success does not reset the count inside the window, but it does end the backoff: only a run whose latest finished run failed is delayed (see CreateWorkQueueEntriesJunction).

The junction returns the list of newly created dead letters so DetermineJobsToQueueJunction can skip those manifests without re-querying the database.

DetermineJobsToQueueJunction

The decision junction. It runs two passes over the loaded manifests:

Pass 1: Time-based manifests (Cron and Interval). For each, it checks whether the manifest is due using SchedulingHelpers.ShouldRunNow(), which dispatches to either cron parsing or interval arithmetic based on the schedule type. Cron is evaluated in UTC. A cron that has never succeeded is due at its first occurrence after it was scheduled, which scheduling records on the manifest's NextScheduledRun; one written by an earlier Trax version with no success and no stored NextScheduledRun is due at once. The schedule is evaluated from the later of the manifest's last successful run and its last cancelled run, because a cancelled run consumes the occurrence it ran for: after a cancelled run the next occurrence after it is due, whatever NextScheduledRun held.

Pass 2: Dependent manifests. For each manifest with ScheduleType.Dependent, it finds the parent in the loaded set and checks whether the parent's LastSuccessfulRun is later than the start of the dependent's latest run, successful or cancelled, so a parent success that landed while the dependent was running queues it again. LoadManifestsJunction loads that start for dependents only, as the DispatchedAt of the run's work queue entry; a run with no dispatched entry falls back to its own start time, and with no run on record the dependent's own LastSuccessfulRun stands in. DispatchedAt and the parent's LastSuccessfulRun are both stamped by the database's clock, so clock skew between the dispatcher and the worker cannot requeue a dependent after every run. Before comparing timestamps, the junction verifies that the parent has at least one Completed metadata record (HasSuccessfulMetadata). If the parent has a LastSuccessfulRun timestamp but no successful metadata to back it up (e.g., metadata was truncated or pruned), the timestamp is considered stale and the dependent is not queued. See Dependent Trains.

Manifests with ScheduleType.DormantDependent are excluded from both passes. They are never auto-queued by the ManifestManager, dormant dependents must be explicitly activated at runtime by the parent train via IDormantDependentContext.

Both passes apply the same per-manifest guards before evaluating the schedule:

• Skip if the manifest's ManifestGroup has IsEnabled = false
• Skip if the manifest was just dead-lettered this cycle
• Skip if it has an AwaitingIntervention dead letter
• Skip if it has a Queued work queue entry (already waiting to be dispatched)
• Skip if it has Pending or InProgress metadata (already running)

MaxActiveJobs is deliberately not enforced here. The ManifestManager freely identifies all due manifests. The JobDispatcher handles capacity gating at dispatch time. This keeps the two concerns separate, scheduling logic doesn't need to know about system-wide capacity.

CreateWorkQueueEntriesJunction

For each manifest identified as due, creates a WorkQueue entry with:

• TrainName from the manifest's Name (the canonical interface name, e.g. MyApp.Trains.IProcessOrderTrain)
• Input / InputTypeName from the manifest's Properties / PropertyTypeName
• ManifestId linking back to the source manifest
• Priority set from the manifest's own Priority, as a manual trigger or a dead-letter requeue is (the dispatcher orders by the group's priority first)
• Status = Queued
• ScheduledAt delayed by the retry backoff when the manifest's latest finished run failed (see Retry Delay & Backoff); after a success or a cancel the entry is due at once

For dependent manifests, DependentPriorityBoost is added on top of the manifest priority.

Each entry is saved individually. If one fails (e.g., a serialization issue for a specific manifest), the others still get queued. Errors are logged per-manifest.

Group-Fair Batching

The number of entries created per cycle is limited by MaxWorkQueueEntriesPerCycle (default: 200). When more manifests are due than the limit allows, entries are distributed fairly across manifest groups: each group gets a base allocation of limit / numGroups, with overflow slots going to higher-priority groups first. This prevents a single large group from monopolizing the batch and starving smaller groups (e.g., 5000 cache manifests crowding out 15 delta manifests). Excess manifests are deferred to the next polling cycle.

Without group-fair batching, if a large group has thousands of simultaneously-eligible manifests (such as after a mass failure with retries), a flat Take(N) would consistently select manifests from that group first, leaving smaller groups perpetually deferred.

Set MaxWorkQueueEntriesPerCycle to null to disable the limit (unlimited, all due manifests are enqueued per cycle, no fairness needed).

Concurrency Model: Two-Layer Defense

The ManifestManager uses a layered approach to prevent duplicate work queue entries. Each layer addresses a different failure mode.

Outer Layer: Advisory Lock (Single-Leader Election)

The ManifestManagerPollingService acquires a PostgreSQL transaction-scoped advisory lock before invoking the train:

SELECT pg_try_advisory_xact_lock(hashtext('trax_manifest_manager'))

This is a non-blocking try-lock: if another server already holds it, the current server skips the cycle entirely and waits for the next polling tick. No server ever blocks waiting for the lock.

The lock is xact-scoped (transaction-scoped), meaning it auto-releases when the wrapping transaction commits or rolls back. The entire ManifestManagerTrain runs within this transaction, so all database changes (dead letters, WorkQueue entries) are committed atomically. If the train fails partway through, everything rolls back. No partial state.

This is the primary concurrency control. It guarantees that in a multi-server deployment, only one server evaluates manifests at a time, eliminating the TOCTOU race between LoadManifestsJunction (which reads HasQueuedWork = false) and CreateWorkQueueEntriesJunction (which inserts the entry).

Inner Layer: Logical State Guards

Even within a single-server cycle, DetermineJobsToQueueJunction applies per-manifest guards via ShouldSkipManifest before evaluating schedules:

These guards are computed from the database-projected flags in ManifestDispatchView. They are not a replacement for the advisory lock, without the lock, two servers could simultaneously see HasQueuedWork = false for the same manifest and both create entries. The guards protect against logical errors within a single evaluation cycle (e.g., a manifest that appears "due" but is already being handled).

Backstop: Unique Partial Index

As a final safety net, a unique partial index on the work_queue table prevents duplicate Queued entries for the same manifest at the database level:

CREATE UNIQUE INDEX ix_work_queue_unique_queued_manifest
    ON trax.work_queue (manifest_id)
    WHERE status = 'queued' AND manifest_id IS NOT NULL;

If the advisory lock is somehow bypassed (e.g., a bug, a code path that doesn't go through the polling service), this index causes a constraint violation on the second insert. The per-entry try/catch in CreateWorkQueueEntriesJunction catches the error and logs it. No crash, no corruption. Manual WorkQueue entries (manifest_id IS NULL) are excluded from this index.

Non-Postgres Providers

The advisory lock is only acquired when the IDataContext is backed by Entity Framework Core (DbContext). When using the InMemory provider for tests, the lock is skipped and the train runs directly, safe because InMemory implies a single-process setup.

See Multi-Server Concurrency for the full cross-service concurrency model.

What Changed

Previously, this train had an EnqueueJobsJunction as its final junction. That junction would directly create Metadata records and enqueue to the job submitter (Hangfire). MaxActiveJobs was enforced there, meaning the ManifestManager was both the scheduler and the dispatcher.

Now those responsibilities are split. The ManifestManager writes intent to the work queue. The JobDispatcher reads from it and handles the actual dispatch. This means TriggerAsync, dashboard re-runs, and scheduled manifests all converge on the same dispatch path.

SDK Reference

AddScheduler