ManifestManagerTrain
Source ManifestManager/
The ManifestManager is the first half of each polling cycle. It figures out which manifests are due for execution and writes them to the work queue. It doesn't dispatch anything; that's the JobDispatcher's job.
Chain
CancelTimedOutJobsJunction → ReapStalePendingMetadataJunction → ReapStaleInProgressMetadataJunction → LoadManifestsJunction → ResolveStaleStagedEntriesJunction → ReapFailedJobsJunction → DetermineJobsToQueueJunction → CreateWorkQueueEntriesJunctionJunctions
The timeout and stale-run junctions come first and work on runs directly, not on the loaded manifests, so the manifests are loaded after them and a run they fail is already counted in the same cycle.
CancelTimedOutJobsJunction
Finds every InProgress run past its timeout and requests cooperative cancellation. A run's timeout is resolved from the run at the root of its ParentId chain: that run's manifest TimeoutSeconds, or DefaultJobTimeout when it sets none, which applies only when a scheduler dispatched the root (a manifest, a work queue entry or a background_job row). A nested train shares its root's timeout; a train run directly on the train bus, or whose chain cannot be resolved within 16 levels, is not timed out. Runs of disabled manifests are included; the scheduler's own trains, trains in ExcludedTrainTypeNames, and trains nested in either are not. See Cancellation and timeouts. Sets CancellationRequested = true in the database (picked up by CancellationCheckProvider at the next junction boundary) and attempts same-server instant cancellation via the CancellationRegistry.
ReapStalePendingMetadataJunction
Fails Pending metadata that has not been picked up within StalePendingTimeout (default: 20 minutes). Acts as a safety net for dispatch failures where the worker never started executing the job (e.g., remote worker unreachable, Lambda crashed after receiving the request).
ReapStaleInProgressMetadataJunction
Fails InProgress metadata that has not completed within StaleInProgressTimeout (default: 60 minutes). Acts as a safety net for hard crashes. Lambda hard-kills, OOM events, or process crashes where the worker dies without reaching FinishServiceTrain. This timeout should be longer than DefaultJobTimeout to allow cooperative cancellation (via CancelTimedOutJobsJunction) to propagate before force-failing.
A run whose own timeout, resolved as CancelTimedOutJobsJunction resolves it, is longer gets as much longer: it is failed at the later of StaleInProgressTimeout and that timeout plus the same grace (StaleInProgressTimeout - DefaultJobTimeout, 40 minutes with the defaults). A manifest with a three hour timeout therefore has its run failed as stale at 3 h 40 min, not at 60 minutes while it is still working.
Newly-failed metadata from both stale reapers is counted by LoadManifestsJunction in the same ManifestManager cycle, enabling dead-lettering if retries are exhausted.
Failing a run, from either reaper, also releases its subject key, so a run still pending past StalePendingTimeout or still working past its stale threshold stops holding its subject.
LoadManifestsJunction
Projects all enabled manifests into lightweight ManifestDispatchView records using a single database query with pre-computed aggregate flags (FailedCount, HasAwaitingDeadLetter, HasQueuedWork, HasActiveExecution, HasSuccessfulMetadata) the end time of the manifest's latest cancelled run, whether its latest finished run failed (LatestFinishedRunFailed), and, for dependents, when their latest successful and cancelled runs were dispatched. These flags are computed via COUNT/EXISTS subqueries pushed into the database, keeping query cost O(manifests) regardless of how large the child tables (Metadatas, DeadLetters, WorkQueues) grow.
The projection uses AsNoTracking(), the results are read-only snapshots used for scheduling decisions only. No unbounded child collections are loaded into memory.
Scaling note: LoadManifestsJunction loads all enabled manifests in a single query. For typical production deployments (up to 10K manifests), this is efficient with proper indexes.
ResolveStaleStagedEntriesJunction
Resolves work queue entries that a crash left unconfirmed. A train with DeferQueuePromotion commits its entry unconfirmed, runs OnQueue, then confirms it; a process that stopped in between leaves an entry the dispatcher will never claim. Any entry still Queued and unconfirmed after StaleStagedEntryTimeout (default: 10 minutes) is:
| Setting | Outcome |
|---|---|
| Default | Cancelled, via IWorkQueuePromotion.CancelStaleAsync. Nothing recorded tells a hook that succeeded from one that never ran or one that rejected the mutation, so the entry is kept visible rather than run. The same call deletes the entries it cancelled on an earlier pass once they are 30 days old (retention) |
PromoteStaleStagedEntries() | Promoted, via IWorkQueuePromotion.PromoteStaleAsync, and dispatched like any other entry. The run re-executes the whole chain, so this assumes idempotent hooks and a chain that re-checks what the hook checked |
IWorkQueuePromotion.PromoteAsync, which an enqueue calls once its hook returns, only confirms an entry that is still Queued, so an entry this junction has already cancelled stays cancelled. The promotion methods work on every data provider, the InMemory provider included. Like the rest of the train, this runs only on the server holding the leader lock, and not at all while the ManifestManager is disabled (ManifestManagerEnabled = false). A deployment that turns the ManifestManager off everywhere leaves stranded staged entries unresolved.
ReapFailedJobsJunction
Scans loaded manifests for any whose failure count exceeds MaxRetries. MaxRetries is the number of retries after the first run, so MaxRetries(0) dead-letters on the first failure and the default of 3 on the fourth; a manifest with no counted failure is never dead-lettered. For each, it creates a DeadLetter record with status AwaitingIntervention and persists immediately.
A manifest is only reaped if it doesn't already have an unresolved dead letter. This prevents duplicate dead letters from accumulating when the same manifest fails across multiple polling cycles.
FailedCount only counts failures that started within the manifest's own failure window (manifest.failure_window_seconds, set with FailureWindow) or, when it has none, FailureCountWindow (default: 24 hours), and after the most recent dead letter resolution. When a dead letter is resolved (retried or acknowledged), the failure counter effectively resets, only new failures contribute toward the next MaxRetries threshold. This prevents retried manifests from being immediately re-dead-lettered due to historical failures that were already addressed. The window does the same for failures that were never dead-lettered: a failure a month ago neither lengthens a retry's delay nor counts toward a dead letter. A success does not reset the count inside the window, but it does end the backoff: only a run whose latest finished run failed is delayed (see CreateWorkQueueEntriesJunction).
The junction returns the list of newly created dead letters so DetermineJobsToQueueJunction can skip those manifests without re-querying the database.
DetermineJobsToQueueJunction
The decision junction. It runs two passes over the loaded manifests:
Pass 1: Time-based manifests (Cron and Interval). For each, it checks whether the manifest is due using SchedulingHelpers.ShouldRunNow(), which dispatches to either cron parsing or interval arithmetic based on the schedule type. Cron is evaluated in UTC. A cron that has never succeeded is due at its first occurrence after it was scheduled, which scheduling records on the manifest's NextScheduledRun; one written by an earlier Trax version with no success and no stored NextScheduledRun is due at once. The schedule is evaluated from the later of the manifest's last successful run and its last cancelled run, because a cancelled run consumes the occurrence it ran for: after a cancelled run the next occurrence after it is due, whatever NextScheduledRun held.
Pass 2: Dependent manifests. For each manifest with ScheduleType.Dependent, it finds the parent in the loaded set and checks whether the parent's LastSuccessfulRun is later than the start of the dependent's latest run, successful or cancelled, so a parent success that landed while the dependent was running queues it again. LoadManifestsJunction loads that start for dependents only, as the DispatchedAt of the run's work queue entry; a run with no dispatched entry falls back to its own start time, and with no run on record the dependent's own LastSuccessfulRun stands in. DispatchedAt and the parent's LastSuccessfulRun are both stamped by the database's clock, so clock skew between the dispatcher and the worker cannot requeue a dependent after every run. Before comparing timestamps, the junction verifies that the parent has at least one Completed metadata record (HasSuccessfulMetadata). If the parent has a LastSuccessfulRun timestamp but no successful metadata to back it up (e.g., metadata was truncated or pruned), the timestamp is considered stale and the dependent is not queued. See Dependent Trains.
Manifests with ScheduleType.DormantDependent are excluded from both passes. They are never auto-queued by the ManifestManager, dormant dependents must be explicitly activated at runtime by the parent train via IDormantDependentContext.
Both passes apply the same per-manifest guards before evaluating the schedule:
- Skip if the manifest's ManifestGroup has
IsEnabled = false - Skip if the manifest was just dead-lettered this cycle
- Skip if it has an
AwaitingInterventiondead letter - Skip if it has a
Queuedwork queue entry (already waiting to be dispatched) - Skip if it has
PendingorInProgressmetadata (already running)
MaxActiveJobs is deliberately not enforced here. The ManifestManager freely identifies all due manifests. The JobDispatcher handles capacity gating at dispatch time. This keeps the two concerns separate, scheduling logic doesn't need to know about system-wide capacity.
CreateWorkQueueEntriesJunction
For each manifest identified as due, creates a WorkQueue entry with:
TrainNamefrom the manifest'sName(the canonical interface name, e.g.MyApp.Trains.IProcessOrderTrain)Input/InputTypeNamefrom the manifest'sProperties/PropertyTypeNameManifestIdlinking back to the source manifestPriorityset from the manifest's ownPriority, as a manual trigger or a dead-letter requeue is (the dispatcher orders by the group's priority first)Status = QueuedScheduledAtdelayed by the retry backoff when the manifest's latest finished run failed (see Retry delay and backoff); after a success or a cancel the entry is due at onceReplayDecisionsOfnaming the failed run, for a retry whose failed run's decisions can be replayed (see Retries replay decisions). Every due retry is looked up in one pass, on a short-lived context of its own outside the leader transaction; a lookup that fails is logged and the retry asks afresh.
For dependent manifests, DependentPriorityBoost is added on top of the manifest priority.
Each entry is saved individually. If one fails (e.g., a serialization issue for a specific manifest), the others still get queued. Errors are logged per-manifest.
Group-Fair Batching
The number of entries created per cycle is limited by MaxWorkQueueEntriesPerCycle (default: 200). When more manifests are due than the limit allows, entries are distributed fairly across manifest groups: each group gets a base allocation of limit / numGroups, with overflow slots going to higher-priority groups first. This prevents a single large group from monopolizing the batch and starving smaller groups (e.g., 5000 cache manifests crowding out 15 delta manifests). Excess manifests are deferred to the next polling cycle.
Without group-fair batching, if a large group has thousands of simultaneously-eligible manifests (such as after a mass failure with retries), a flat Take(N) would consistently select manifests from that group first, leaving smaller groups perpetually deferred.
Set MaxWorkQueueEntriesPerCycle to null to disable the limit (unlimited, all due manifests are enqueued per cycle, no fairness needed).
Concurrency
The whole train runs inside one transaction that first takes a PostgreSQL advisory lock, so in a
multi-server deployment one server runs the ManifestManager per cycle and the others skip it. The
per-manifest guards above protect a single cycle, and a unique partial index on work_queue
refuses a second queued entry for a manifest if anything gets past both. InMemory takes no lock.
Multi-Server Concurrency explains each
layer.