IOperationsService
Package Trax.SchedulerNamespace Trax.Scheduler.Services.OperationsSource IOperationsService.cs, OperationsService.cs
The operations the dashboard and the GraphQL operations namespace share. Each action an operator can take has one method here, and both surfaces call it, so the dashboard and the API validate, authorize and report failures the same way.
Signature
namespace Trax.Scheduler.Services.Operations;
public interface IOperationsService
{
Task<OperationResult> QueueTrainAsync(QueueTrainInput input, CancellationToken ct);
Task<OperationResult> RunTrainAsync(RunTrainInput input, CancellationToken ct);
Task<OperationResult> RequeueExecutionAsync(long metadataId, CancellationToken ct);
Task<OperationResult> RequeueExecutionAsync(long metadataId, bool askAfresh, CancellationToken ct);
Task<OperationResult> ResumeExecutionAsync(long metadataId, string? from, CancellationToken ct);
Task<OperationResult> CancelExecutionsAsync(IReadOnlyCollection<long> ids, CancellationToken ct);
Task<OperationResult> CancelWorkQueueEntriesAsync(IReadOnlyCollection<long> ids, CancellationToken ct);
Task<OperationResult> SetManifestsEnabledAsync(IReadOnlyCollection<long> ids, bool enabled, CancellationToken ct);
Task<OperationResult> SetManifestsReplayDecisionsOnRetryAsync(IReadOnlyCollection<long> ids, bool replay, CancellationToken ct);
Task<OperationResult> SetManifestGroupsEnabledAsync(IReadOnlyCollection<long> ids, bool enabled, CancellationToken ct);
Task<OperationResult> SetAllManifestGroupsEnabledAsync(bool enabled, CancellationToken ct);
Task<TriggerManifestResult> TriggerManifestAsync(string externalId, TimeSpan? delay, bool askAfresh, CancellationToken ct);
Task<BatchTriggerResult> TriggerManifestsAsync(IReadOnlyCollection<long> manifestIds, bool askAfresh, CancellationToken ct);
Task<BatchTriggerResult> TriggerManifestGroupsAsync(IReadOnlyCollection<long> groupIds, CancellationToken ct);
Task<OperationResult> CancelManifestGroupsAsync(IReadOnlyCollection<long> groupIds, CancellationToken ct);
Task<ManifestExecutionStats> GetManifestExecutionStatsAsync(long manifestId, CancellationToken ct);
Task<IReadOnlyList<ManifestGroupExecutionStats>> GetManifestGroupExecutionStatsAsync(IReadOnlyCollection<long> groupIds, CancellationToken ct);
Task<IReadOnlyList<ManifestUpcomingRuns>> GetUpcomingRunsAsync(IReadOnlyCollection<long> manifestIds, int count, CancellationToken ct);
Task<LogPage> GetLogsAsync(LogQuery query, CancellationToken ct);
Task<int> CountLogsAsync(LogQuery query, CancellationToken ct);
Task<LogCount> CountLogsCappedAsync(LogQuery query, CancellationToken ct);
Task<RecordedDecisionPage> GetRecordedDecisionsAsync(long metadataId, long? afterId, int take, CancellationToken ct);
Task<WorkQueueEntryDetail?> GetWorkQueueEntryDetailAsync(long id, CancellationToken ct);
Task<MachineInstancePage> GetMachineInstancesAsync(MachineInstanceQuery query, CancellationToken ct);
Task<MachineInstanceTotal> CountMachineInstancesAsync(MachineInstanceQuery query, CancellationToken ct);
Task<MachineInstanceRecord?> GetMachineInstanceAsync(MachineInstanceKey key, CancellationToken ct);
Task<IReadOnlyList<MachineInstanceStateCount>> GetMachineInstanceStateCountsAsync(string? machine, CancellationToken ct);
Task<OperationResult> CancelWorkQueueEntryAsync(long id, CancellationToken ct);
Task<OperationResult> UpdateManifestAsync(long id, ManifestUpdate update, CancellationToken ct);
Task<OperationResult> UpdateManifestGroupAsync(long id, UpdateManifestGroupInput input, CancellationToken ct);
Task<OperationResult> UpdateSchedulerConfigAsync(UpdateSchedulerConfigInput input, CancellationToken ct);
Task<MachineInstanceRuns?> GetMachineInstanceRunsAsync(MachineInstanceKey key, CancellationToken ct);
Task<MachineInstanceTransitionPage?> GetMachineInstanceTransitionsAsync(MachineInstanceKey key, long? beforeRevision, int take, CancellationToken ct);
Task<MachineInstanceCancelResult> CancelMachineInstanceAsync(MachineInstanceKey key, CancellationToken ct);
IAsyncEnumerable<MachineInstanceRecord> WatchMachineInstancesAsync(string? machine, CancellationToken ct);
Task<IReadOnlyList<MachineDescription>> GetMachinesAsync(CancellationToken ct);
Task<MachineBoardDescription?> GetMachineBoardAsync(string machine, CancellationToken ct);
Task<DashboardMetrics> GetDashboardMetricsAsync(MetricsRange range, bool hideAdminTrains, CancellationToken ct);
Task<MetricSeries> GetMetricSeriesAsync(MetricSeriesScope scope, long? id, string? name, MetricSeriesRange range, CancellationToken ct);
Task<Attention> GetAttentionAsync(CancellationToken ct);
Task<ManifestGroupDependencyGraph?> GetManifestGroupDependencyGraphAsync(long groupId, CancellationToken ct);
Task<ManifestGroupDependencyGraph> GetGlobalManifestGroupGraphAsync(CancellationToken ct);
SchedulerConfigSnapshot GetSchedulerConfig();
}
public record OperationResult(bool Success, long? Id = null, int? Count = null, string? Message = null);Members
| Group | Members |
|---|---|
| Queueing and running | QueueTrainAsync, RunTrainAsync, RequeueExecutionAsync, ResumeExecutionAsync |
| Batch actions | CancelExecutionsAsync, CancelWorkQueueEntriesAsync, CancelWorkQueueEntryAsync, SetManifestsEnabledAsync, SetManifestsReplayDecisionsOnRetryAsync, SetManifestGroupsEnabledAsync, SetAllManifestGroupsEnabledAsync, CancelManifestGroupsAsync, TriggerManifestAsync, TriggerManifestsAsync, TriggerManifestGroupsAsync, UpdateManifestAsync |
| Settings | UpdateManifestGroupAsync, GetSchedulerConfig, UpdateSchedulerConfigAsync |
| Read models | GetManifestExecutionStatsAsync, GetManifestGroupExecutionStatsAsync, GetUpcomingRunsAsync, GetLogsAsync, CountLogsAsync, CountLogsCappedAsync, GetWorkQueueEntryDetailAsync, GetRecordedDecisionsAsync, GetDashboardMetricsAsync, GetMetricSeriesAsync, GetAttentionAsync, GetManifestGroupDependencyGraphAsync, GetGlobalManifestGroupGraphAsync |
| State-machine instances | GetMachineInstancesAsync, CountMachineInstancesAsync, GetMachineInstanceAsync, GetMachineInstanceStateCountsAsync, WatchMachineInstancesAsync, GetMachineInstanceRunsAsync, GetMachineInstanceTransitionsAsync, CancelMachineInstanceAsync |
| State-machine definitions | GetMachinesAsync, GetMachineBoardAsync |
Queueing and running
| Method | Input | Id on success |
|---|---|---|
QueueTrainAsync | QueueTrainInput(TrainName, InputJson, Priority, ScheduledAt) | the work queue entry |
RunTrainAsync | RunTrainInput(TrainName, InputJson) | the run's metadata row |
RequeueExecutionAsync | the run's metadata id | the work queue entry |
ResumeExecutionAsync | the run's metadata id and a step | the work queue entry |
QueueTrainAsync
Enqueues through ITrainExecutionService.QueueAsync, so the train's [TraxAuthorize] requirements, its OnQueue hook and its subject key apply. The entry waits for dispatch like any other. The queued run asks its deciders afresh: of the methods here, only RequeueExecutionAsync queues a run that replays an earlier one's decisions.
RunTrainAsync
Applies the per-record checks a queue applies (below), writes a Pending run and submits it at once to the job submitter the train is routed to, the same routing the job dispatcher uses (ForTrain<T>(), then [TraxRemote], then the default submitter). Nothing goes through the work queue. Once the run's row is written the submit no longer takes ct, so a caller that goes away does not abort the submit or cancel the run; the submitter's own timeouts bound it. When the submit throws and the run is still Pending, no runner started it: the run is recorded Failed and the exception is thrown to the caller. When a runner already started it (a remote runner that answered with the train's error, a call that timed out while the run went on, or an in-process submitter that ran a failing train), the run owns its outcome: the call succeeds with the run's id and the message Run {id} of {train} submitted; its outcome is pending on the run., and the run's row records how it ended.
Reading the input
Both look the train up by its interface FullName and hand the input to the mediator: QueueTrainAsync through ITrainExecutionService.QueueAsync, RunTrainAsync through ITrainExecutionService.PrepareAsync, which authorizes the caller and reads the input without writing anything. Either way InputJson is read by TrainInputReader: property names matched whatever their case (customerId, CustomerId and CUSTOMERID all fill the same property), a property given twice in any casing refused as invalid input rather than resolved to its last value, JSON reference metadata ($id, $ref, $values) not honoured, so the input is exactly the tree the caller wrote, the mediator's input size cap, and a blank input read as {}, which the input type must be buildable from.
The form a queued input is stored in, and the form a run's submitter writes for its worker, is indented and writes every member, so it is larger than the caller's JSON. Both are held to TrainInputReader.StoredInputGrowthFactor (4) times MaxInputJsonBytes, measured before anything is written or submitted.
A run applies the per-record checks a queue applies. When the train overrides OnQueue, the hook runs on the run's input before the run's row is saved, as it runs for an enqueue: on an instance in a scope of its own, with TrainInput reading the input, with the metadata.ExternalId the run executes under, with writes on IEnqueueContextAccessor.Current saved together with the run's row, and within MaxQueueHookDuration. A hook that throws refuses the run and nothing is written.
A run is otherwise a deliberate bypass of the work queue. It skips dispatch priority, group MaxActiveJobs, and the subject lock. A train that overrides QueueSubjectKey serializes its work per subject through the queue, so it is run now only inside a trusted scope, such as the dashboard's; any other caller gets a failed result telling it to queue the train instead. Use QueueTrainAsync when a run must wait its turn.
When nothing can dispatch a queued run
On a host whose store is in memory (EF Core's InMemory provider, with no database provider registered) the job dispatcher never runs, and no other process can reach the store, so a queued entry would never start. Every method here that would queue a run (QueueTrainAsync, RequeueExecutionAsync, TriggerManifestAsync, TriggerManifestsAsync, TriggerManifestGroupsAsync) refuses with OperationsService.NoDispatcherMessage and writes nothing. QueueTrainAsync and RequeueExecutionAsync first authorize the caller for the train as the mediator would, so a caller the train refuses gets the authorization failure, not this message. ITraxScheduler's dead-letter requeues, which the dashboard and the API call directly, are refused the same way. RunTrainAsync is unaffected. A relational store is never refused, so an API-only host on PostgreSQL whose scheduler runs elsewhere keeps queueing. ITraxScheduler.TriggerAsync, called from code, is not refused. See scheduler ADR 0019.
RequeueExecutionAsync
Task<OperationResult> RequeueExecutionAsync(long metadataId, CancellationToken ct);
Task<OperationResult> RequeueExecutionAsync(long metadataId, bool askAfresh, CancellationToken ct);| Parameter | Type | Description |
|---|---|---|
metadataId | long | The run (metadata row) to re-queue |
askAfresh | bool | When true, the new entry carries no replay link and the new run asks every question again; nothing else differs. The overload without it is askAfresh: false. |
Re-queues a run: queues a fresh run of the same train with the input the run recorded. The GraphQL
requeueExecution mutation and the
dashboard's Re-queue button both call it, so the two refuse the same runs with the same
messages and enqueue the same way. The dashboard calls it inside its "dashboard" trusted scope;
the API does not.
It reads the run by metadataId and refuses, with a failed result and nothing queued, when the
run does not exist (Execution {id} not found.), when its train is no longer registered
(Train {name} is no longer registered, so execution {id} cannot be re-queued.), or when its
saved input is not the input it ran with:
nothing was saved (inputs are saved only when SaveTrainParameters() is on), the parameter effect
saved a _truncated, _unserializable or _disposed placeholder in its place, or
[TraxSensitive] members were masked. Each of those would read back as default values.
The saved input is written with reference metadata ($id, $values, $ref) that the mediator
does not honour in an input it is handed, so it is first resolved with
TrainInputReader.ResolveSavedInput
into the plain tree the run was given: lists come back as lists, and an object that appeared twice
comes back in full both times. A saved input whose metadata has no plain form is refused
(Execution {id}'s saved input cannot be read back as the input it ran with: ...), and one whose
plain form is over the stored-input cap (StoredInputGrowthFactor times MaxInputJsonBytes, since
the saved form is larger than what the caller sent, and PostgreSQL's jsonb renders it back with a
space after every : and ,) gets the generic size message. The plain form is then handed over
written compactly, so jsonb's spacing does not count against the cap. It is still the run's input
as the train saw it, every member written, defaults included, so it can be larger than the JSON the
caller first sent: an input accepted close to MaxInputJsonBytes that left members out can be over
it on requeue, and is refused with the generic size message. Queue it again with the original JSON,
or raise MaxInputJsonBytes. The enqueue then goes through the
same path as QueueTrainAsync, so the train's authorization, its OnQueue hook, its subject key
and the input cap apply, and a refusal or failure is reported as it is there, with one difference:
the caller supplied no JSON, so an input that no longer reads as the train's input type (the type
changed shape after the run) is refused as
The saved input of run {id} no longer reads as {InputType.FullName}: ..., not as
Invalid InputJson. Like an enqueue's parse error, that is given only once the caller is
authorized. On success Id is the new work queue entry.
When the run has decisions to replay (it recorded a decision, or was itself queued to replay
another run), the new entry names it in ReplayDecisionsOf, so the new run
replays those decisions and takes
the tracks the original took. A requeue of a requeue therefore replays too, following the chain
back to the answers the first run recorded. A run with nothing to replay is re-queued as an
ordinary enqueue. It asks
HasDecisionsToReplay.
It is the only caller-facing method that sets the link, always to the run being re-queued; the
scheduler sets it on a manifest's retry and dead-letter requeue from its own checks (see
Retries replay decisions).
Each replayed answer is still checked against the state it is asked about now and its age, so a
question whose state changed, or whose answer is older than ReplayAnswersFor, is asked afresh.
A run's answers are replayed once. When a queued entry (a manifest's retry, an earlier requeue) or
a run in any state already replays the run being re-queued, the requeue still queues it but asks
afresh, and the success message ends
It asks its deciders afresh: the decisions of execution {id} are already replayed by another run or queued entry, and are replayed once.
The same holds when another enqueue queues a replay of the run in the same instant: the database
allows one queued entry per replayed run, so the requeue that loses is queued without the link,
asks afresh, and says so in the same message. With askAfresh: true no link is set and the message
adds nothing.
When the run has decisions to replay and the registered ITrainExecutionService does
not implement the
QueueAsync overload that takes QueueTrainOptions,
the mediator's DecisionReplayNotSupportedException is logged and thrown as a host
misconfiguration rather than reported as The enqueue was refused.
A requeue always runs the chain again from the top, whatever checkpoints the run stored. To carry on from a checkpoint instead, resume it.
ResumeExecutionAsync
Task<OperationResult> ResumeExecutionAsync(long metadataId, string? from, CancellationToken ct);| Parameter | Type | Description |
|---|---|---|
metadataId | long | The failed or cancelled run (metadata row) to resume |
from | string? | The node id of the step to resume at, as the run graph names it, or null for after the run's latest checkpoint |
Queues a run that resumes a run from a checkpoint: it skips every step before the resume point and starts from the state the checkpoint stored. It is a requeue in every check but where the run starts. It refuses, with a failed result and nothing queued, when:
- no run has the id (
Execution {id} not found.); - the run is not
FailedorCancelled(Execution {id} is {state}; only a failed or cancelled run can be resumed.); - a state machine's invoking state queued it, which only the machine retries, by entering the state again (
Execution {id} was started by a step of the state machine '{machine}', ... so it cannot be resumed. ...); - its saved input cannot be re-queued, for the reasons
RequeueExecutionAsyncgives; - a resume of it is already queued (
A resume of execution {id} is already queued ...); - its train is no longer registered, or its chain cannot be read on this host;
- the resume check refuses: the run stored no checkpoint before
from, a step after the point needs a value nothing restores, or the stored checkpoint no longer matches the running chain or the state's shape. The check's own reason is the message, unchanged.
Otherwise it enqueues through the mediator exactly as RequeueExecutionAsync does, with the run's
saved input, so the train's [TraxAuthorize], its OnQueue hook and its subject key apply, and an
authorization failure propagates. The entry names the run in ResumeFrom and the step in
ResumeAt, and, when the run has decisions to replay, in ReplayDecisionsOf as a requeue would, so
a question asked after the resume point takes the track the run took. On success Id is the new
work queue entry. The new run checks the resume again when it starts, and runs from the top, with a
warning in its log, if it can no longer be trusted.
A run has at most one queued resume. When a manifest's retry, which resumes after the failed run's latest checkpoint on its own (see Retries resume from a checkpoint), or another operator queues one between this call's checks and its insert, the database refuses the second, and this call returns the same refusal.
Failures
Both methods return a failed OperationResult with a Message for an answer the caller can act on, and throw for a fault on the server's side.
| Outcome | QueueTrainAsync | RunTrainAsync |
|---|---|---|
Blank TrainName, unknown train | failed result | failed result |
Invalid or null InputJson, a property given twice, or JSON reference metadata | failed result (Invalid InputJson: ...) | failed result, same message; no run is written |
InputJson over MaxInputJsonBytes, or a stored form over its cap | failed result, a generic message | failed result, same message; no run is written |
The train's OnQueue or QueueSubjectKey refused | failed result: The enqueue was refused: {message} for a plain TrainException, QueuedWorkCancelledException or QueueHookTimeoutException, otherwise the fixed The enqueue was refused. with the exception logged at Warning | the OnQueue hook refused: failed result under the same rule, as The run was refused: {message} or The run was refused.; no run is written |
A train that overrides QueueSubjectKey, outside a trusted scope | not applicable | failed result telling the caller to queue it instead; no run is written |
| The caller may not run the train | throws UnauthorizedAccessException | throws UnauthorizedAccessException, before the input is read |
[TraxAuthorize] train, no enforcer, not trusted | logged and thrown (TrainAuthorizationNotConfiguredException) | throws TrainAuthorizationNotConfiguredException |
Database or network failure, including one inside the OnQueue hook | logged and thrown | logged and thrown |
| The job submitter failed before a runner started the run | not applicable | the run is marked Failed with the submitter's exception, then it is logged and thrown |
| The job submitter threw after a runner started the run | not applicable | success, with the run's id; the message says its outcome is pending on the run, and the run's row records it |
ct cancelled | throws OperationCanceledException | throws OperationCanceledException before the run's row is written; after that ct is not passed to the submit, so it does not cancel the run |
A thrown failure reaches Trax.Api's error filter, which masks any type it does not know as Unexpected Execution Error, so a connection string's host and port never reach a GraphQL client.
Batch actions
The actions a list page applies to its selected rows. Each takes up to OperationsService.MaxBatchSize (1000) ids, ignores duplicates, writes only the rows that change, and returns OperationResult(true, Count: N) with the number changed, zero included. An empty list, or more ids than the limit, is a failed result and changes nothing.
| Method | Change signal |
|---|---|
CancelExecutionsAsync(ids, ct) | Execution, when at least one run was flagged |
CancelWorkQueueEntriesAsync(ids, ct) | WorkQueue |
SetManifestsEnabledAsync(ids, enabled, ct) | Manifest |
SetManifestsReplayDecisionsOnRetryAsync(ids, replay, ct) | Manifest, when any changed; WorkQueue, when a link was cleared |
SetManifestGroupsEnabledAsync(ids, enabled, ct) | ManifestGroup |
SetAllManifestGroupsEnabledAsync(enabled, ct) | ManifestGroup |
CancelManifestGroupsAsync(groupIds, ct) | Execution, when at least one run was flagged |
CancelExecutionsAsync
Each run still Pending or InProgress gets CancellationRequested, and one running on this host is also cancelled at once through ICancellationRegistry. Terminal and unknown runs are skipped, and so is a run a step of a user's state-machine draft started, which an operator may not cancel: the message counts those and ends with OperationsService.UserOwnedRunCancelRefusal, and a single such id is refused with it. A Pending run is recorded Cancelled and never run when the job runner picks it up, whatever junction providers the host registers. An InProgress run observes the flag at its next junction boundary, when the host uses the junction progress provider.
CancelWorkQueueEntriesAsync
Entries still Queued become Cancelled, in one statement, so an entry the dispatcher claims meanwhile keeps its status. An entry a user's draft queued is skipped and counted, as CancelExecutionsAsync skips its run, and CancelWorkQueueEntryAsync refuses one
SetManifestsEnabledAsync
Manifests whose IsEnabled differs
SetManifestsReplayDecisionsOnRetryAsync
Manifests whose ReplayDecisionsOnRetry differs. Turning it off also clears the replay link of each manifest's queued entry, in the same transaction as the flag, so a retry waiting out its backoff asks afresh; the message then adds {n} queued retry(s) no longer replay a failed run's decisions. See Retries replay decisions.
SetManifestGroupsEnabledAsync
Groups whose IsEnabled differs, with UpdatedAt bumped
SetAllManifestGroupsEnabledAsync
Every group whose IsEnabled differs; a separate method so that "all" is never what an empty list means
CancelManifestGroupsAsync
Every Pending or InProgress run of every manifest in the given groups, by the rule CancelExecutionsAsync applies, flagged in one update. Count is the number of runs flagged; the message also says how many of the ids named a group.
ITraxScheduler.CancelAsync and CancelGroupAsync cancel a manifest's or a group's runs by the same rule as CancelExecutionsAsync, and signal Execution the same way.
TriggerManifestAsync
Task<TriggerManifestResult> TriggerManifestAsync(string externalId, TimeSpan? delay, bool askAfresh, CancellationToken ct);
public record TriggerManifestResult(bool Success, string Message, ManifestTriggerResult? Trigger)
{
public bool StillReplaying { get; init; }
}Triggers the manifest with this external id as ITraxScheduler.TriggerAsync does, and says what the trigger did. The dashboard's Run Now and the API's triggerManifest and triggerManifestDelayed all call it. delay of null (or zero or less) means now, and a delay that would put the run past the latest time a DateTime holds (such as TimeSpan.MaxValue) is a failed result that queues nothing; an existing entry due sooner keeps its own time. Message says which happened: a new run queued, the manifest's queued run brought forward, one already due released as the trigger, or one the dispatcher claimed as the trigger reached it. Trigger carries the detail (WorkQueueId, Created, MovedForward, ScheduledAt, AlreadyDispatched, ReplayDecisionsOf). With askAfresh: true a released retry asks its deciders afresh; when the dispatcher claimed it first, the trigger still succeeds and StillReplaying is true. An unknown external id, or a host where nothing dispatches the queue, is a failed result with Trigger null and nothing changed.
UpdateManifestAsync
Task<OperationResult> UpdateManifestAsync(long id, ManifestUpdate update, CancellationToken ct);
public record ManifestUpdate(
bool? IsEnabled = null, int? MaxRetries = null, int? Priority = null, int? TimeoutSeconds = null,
bool ClearTimeout = false, ScheduleType? ScheduleType = null, string? CronExpression = null,
int? IntervalSeconds = null);Patches one manifest; a null field is left as it is. Every check runs before a field is written, so a refused patch changes nothing: a negative MaxRetries, a Priority outside the work queue's range (0 to 31), a TimeoutSeconds or IntervalSeconds of 0 or less, a switch to Cron without an expression or to Interval without an interval, a switch to Once, Dependent or DormantDependent (they need a time or a parent the patch cannot give), and a cron expression the scheduler cannot use: not 5 or 6 fields, a field out of range (99 * * * *), or one that never fires (0 0 30 2 *). The schedule is checked only when the patch changes it, so a manifest already holding a bad schedule can still be disabled. Success signals ChangeDomain.Manifest and returns the manifest's id. The API's updateManifest calls it.
Batch triggers
Task<BatchTriggerResult> TriggerManifestsAsync(IReadOnlyCollection<long> manifestIds, bool askAfresh, CancellationToken ct);
Task<BatchTriggerResult> TriggerManifestGroupsAsync(IReadOnlyCollection<long> groupIds, CancellationToken ct);
public record BatchTriggerResult(
bool Success, int Matched, int Queued, int AlreadyQueued, int TooLateToAskAfresh, int Skipped,
string Message, IReadOnlyList<BatchItemNote> Notes);
public record BatchItemNote(long Id, string Message);TriggerManifestsAsync triggers each manifest, by its database id, exactly as
ITraxScheduler.TriggerAsync triggers one by
external id: a new work queue entry marked as asked for by name, so it runs even while the manifest
is disabled, or, when the manifest already has a queued entry, that entry released as the triggered
run and brought forward to now. TriggerManifestGroupsAsync triggers the members
ITraxScheduler.TriggerGroupAsync would: the enabled manifests of each group that run on their own
schedule, leaving out Dependent and DormantDependent ones. Both take the same limits as the other
batches (an empty list or more than 1000 ids is a failed result with nothing triggered), trigger
the manifests one at a time in the order given, each with its own save, and signal WorkQueue once.
| Field | Meaning |
|---|---|
Success | False only for a batch refused as given, or on a host where nothing dispatches the queue; every count is then zero |
Matched | The ids that named a manifest, or a group, that exists |
Queued | Manifests a new entry was queued for |
AlreadyQueued | Manifests that already had a queued entry, which became the triggered run; also one the dispatcher had already claimed, which is running |
TooLateToAskAfresh | Manifests triggered with askAfresh: true whose queued retry the dispatcher claimed first, so the run replays the failed run's decisions anyway; each has a note naming that run. Always zero for a group trigger |
Skipped | Ids that named no manifest or group; each has a note |
Message | One line for an operator, such as 2 queued, 1 already queued (that entry now runs as the trigger) across 3 of 3 manifest(s). |
Notes | One line per id that did not get what the trigger asked for |
With askAfresh: true a released retry that would replay a failed run's decisions asks its
deciders afresh instead (see
Retries replay decisions). A
database failure part way is thrown, not reported; the manifests triggered before it stay
triggered, and sending the same batch again is safe, since a manifest holds at most one queued
entry.
On a relational provider each batch is one UPDATE with its state test in it, so a row another writer changes meanwhile keeps its new state. The InMemory provider has no set-based update, so there the rows are loaded, changed and saved, with the same result and count (scheduler ADR 0007).
Settings
| Method | Returns |
|---|---|
UpdateManifestGroupAsync(id, input, ct) | OperationResult |
GetSchedulerConfig() | SchedulerConfigSnapshot |
UpdateSchedulerConfigAsync(input, ct) | OperationResult |
UpdateManifestGroupAsync
Patches one manifest group's dispatch settings (MaxActiveJobs, Priority, the enable flag). Each
field of UpdateManifestGroupInput is optional, and only the fields set are written; UpdatedAt is
bumped when one changed. Success is OperationResult(true, Id: groupId, Count: N), with N the
fields written. A group that does not exist, a Priority outside 0 to 31 or a MaxActiveJobs below
1 is a failed result, and no field is written. The API's updateManifestGroup calls it; its fields
are listed under the GraphQL mutations.
GetSchedulerConfig
The scheduler settings this process runs with, read from its SchedulerConfiguration singleton
(and LocalWorkerOptions and MetadataCleanupConfiguration when registered). On a scheduler host
the stored settings row is loaded into it at startup and within seconds of any save.
UpdateSchedulerConfigAsync
Saves the named scheduler settings to the settings row (see
Stored settings). Each field of
UpdateSchedulerConfigInput is optional, and only the fields set are written. Success is
OperationResult(true, Count: N), with N the fields changed. A value outside the range the
scheduler can run with is a failed result naming each offending field, and nothing is applied: a
polling or cleanup interval outside 1 second to 30 days; a job timeout, stale-pending timeout or
metadata retention under 1 second; a negative retry count, retry delay or dead-letter retention;
any duration over ten years; a MaxActiveJobs below 1; a LocalWorkerCount outside 1 to 256; a
failure count window outside 1 second to ten years; or a backoff multiplier below 1 or not finite.
The dashboard's Server Settings page and the API's updateScheduler call it.
Read models
The numbers behind a manifest's detail cards, the manifest groups list, the logs pages, a run's recorded decisions, a work queue entry's page and the State Machines pages.
| Method | Returns |
|---|---|
GetManifestExecutionStatsAsync(manifestId, ct) | ManifestExecutionStats |
GetManifestGroupExecutionStatsAsync(groupIds, ct) | IReadOnlyList<ManifestGroupExecutionStats> |
GetUpcomingRunsAsync(manifestIds, count, ct) | IReadOnlyList<ManifestUpcomingRuns> |
GetLogsAsync(LogQuery, ct) | LogPage |
CountLogsAsync(LogQuery, ct) | int |
CountLogsCappedAsync(LogQuery, ct) | LogCount |
GetWorkQueueEntryDetailAsync(id, ct) | WorkQueueEntryDetail? |
GetRecordedDecisionsAsync(metadataId, afterId, take, ct) | RecordedDecisionPage |
GetDashboardMetricsAsync(range, hideAdminTrains, ct) | DashboardMetrics |
GetMetricSeriesAsync(scope, id, name, range, ct) | MetricSeries |
GetAttentionAsync(ct) | Attention |
GetManifestGroupDependencyGraphAsync(groupId, ct) | ManifestGroupDependencyGraph? |
GetGlobalManifestGroupGraphAsync(ct) | ManifestGroupDependencyGraph |
GetManifestExecutionStatsAsync
ManifestExecutionStats: run counts by state, the latest start of any run and the latest end of a completed run. Zeros and nulls for a manifest with no runs.
GetManifestGroupExecutionStatsAsync
One ManifestGroupExecutionStats per distinct id, in the order given: manifest count, run counts and the latest run. LastEndedRunFailed says whether the group's latest ended run (completed, failed or cancelled, by start time) failed. Zeros for a group with no manifests or runs; an empty list for no ids; more than 1000 ids throws ArgumentOutOfRangeException.
GetUpcomingRunsAsync
When each manifest runs next, one ManifestUpcomingRuns per distinct id that names a manifest, in the order given:
| Field | Meaning |
|---|---|
ManifestId | The manifest. |
ScheduleText | The schedule in plain words: Daily at 02:00 UTC, Every 5 minutes, Weekdays at 09:00 UTC, After nightly-import succeeds, On demand. A cron with no plain reading reads Cron 0 9-17 * * 1-5 (UTC). A variance adds , up to 2 minutes late. |
NextRuns | The next count times the manifest manager will find the manifest due, earliest first, in UTC. The first is now when it is due already. |
RunsAfterManifestId, RunsAfterManifestExternalId | For a dependent that can run, the manifest whose success starts it; null otherwise. |
The times come from the manager's own due-ness check, so the two cannot disagree: exclusion windows are skipped the way the manager skips them, which under the default FireOnceNow misfire policy means one catch-up run as the window ends (see Exclusion Windows). Each run is assumed to succeed when it is due; a retry's backoff, the jitter a variance adds after the next run, and the manager's polling interval are not shown. NextRuns is empty for a manual, on-demand, dependent or dormant dependent manifest, and for one that is disabled, sits in a disabled group, or has a dead letter awaiting intervention. count is clamped to 0 through OperationsService.MaxUpcomingRuns (10). An empty list for no ids; more than 1000 ids throws ArgumentOutOfRangeException.
GetLogsAsync
A LogPage in the query's Order, newest first by default, filtered as below. AfterId pages by keyset in that order (older entries newest first, newer ones oldest first) and ignores Skip; prefer it, since an offset's cost grows with its size. Take is clamped to 1 through OperationsService.MaxPageSize (500). NextCursor is the last entry's id.
CountLogsAsync
The exact number of entries matching the filter, its text filters included; the order, cursor and paging fields are ignored.
CountLogsCappedAsync
A LogCount(int Count, bool Capped) for a pager. Without a text filter, the exact count CountLogsAsync gives, never capped. With MessageContains or CategoryContains, counted only up to OperationsService.LogCountCap (10,000): more matches than that give Count = 10000, Capped = true, to show as "10,000+"; exactly 10,000 is exact. The same on Postgres, Sqlite and InMemory. A custom implementation that does not override it returns the exact count, uncapped.
GetWorkQueueEntryDetailAsync
A WorkQueueEntryDetail, or null when no entry has the id: the entry's fields, its input masked as below, and for a queued entry with a subject, what it waits on. SubjectHeldBy is the dispatched entry for the same subject whose run is still Pending or InProgress; when nothing holds the subject, SubjectQueuedBehind is the confirmed queued sibling dispatch would offer first, by the filters dispatch applies (due, its manifest group enabled) and then priority and age. The dashboard's work queue entry page and the API's workQueue.detail both read it.
LookupAsync
LookupAsync(string text, CancellationToken ct) finds what an id, key or name names when the caller does not know its kind, and returns a list of LookupMatch(LookupKind Kind, string Id, string Label, string MatchedOn). Kind is Run, Manifest, WorkQueueEntry, DeadLetter, MachineInstance or Train; Id is the id the kind's page takes (a number, an instance's GUID, or a train's full name); Label is the train, manifest or machine with its state; MatchedOn is the field the text matched, id, externalId, name or subjectKey. A MachineInstance match also carries Machine, OwnerKind and RowId, which GetMachineInstanceAsync takes for a user's draft, and never the instance's context or whose a draft is.
The text is trimmed and matched exactly, never by prefix or contents. A positive number is looked up as the id of a run, manifest, work queue entry and dead letter; a GUID as an instance id; any text as a run's or manifest's or entry's external id, a manifest's name, an entry's subject key (the entries still queued or dispatched under it) and a registered train's full or friendly name. Every read is an exact match on an indexed column, at most OperationsService.LookupMatchCap (10) of each kind. Empty text, or text longer than OperationsService.LookupTextLimit (512), finds nothing. The API's lookup calls it. A custom implementation that does not override it throws NotSupportedException.
GetRecordedDecisionsAsync
A RecordedDecisionPage of the run's rows in trax.decision, in the order they were recorded, with answers withheld as described below. afterId is the previous page's NextCursor; take is clamped to 1 through 500. A run with no decisions gets an empty page.
GetDashboardMetricsAsync
The numbers behind the dashboard's home page and the API's metrics query. range sets the
granularity of the executions-over-time chart only; hideAdminTrains leaves the framework's admin
trains (AdminTrains.FullNames) out of every series, as the dashboard's "Hide admin trains" toggle
does. The fields are listed under the GraphQL queries.
GetMetricSeriesAsync
Runs over time for one train (MetricSeriesScope.Train, by name, its interface FullName), manifest (Manifest, by id) or group (Group, by id, every manifest in it). A MetricSeries with every bucket of the MetricSeriesRange, oldest first: LastHour in 60 one-minute buckets, Last24Hours in 96 of 15 minutes, Last7Days in 168 of an hour, Last30Days in 120 of 6 hours. Each MetricSeriesBucket counts the runs that started in it and completed, failed or were cancelled, and gives P50DurationMs and P95DurationMs of the completed ones, read from a duration histogram grouped in the database (bins about 13% wide; never outside the durations recorded; null when none completed). A train scope without a name, a manifest or group scope without an id, and a range OperationsService.MetricSeriesRefusal(scope, range) refuses throw ArgumentException. MetricSeriesRefusal refuses Last30Days for every scope: over a million runs in one train, manifest or group, the 30-day series took 560 to 660 ms on the Trax.Api stress fixture, over the 450 ms dashboard metrics budget, while the 7-day one (about 230,000 runs) took 170 to 340 ms. The cost follows the number of runs in the window, so a scope with far more runs a week than that fixture reads proportionally slower. The fields are listed under the GraphQL metrics queries.
GetAttentionAsync
GetAttentionAsync(CancellationToken ct) returns an Attention: what needs an operator's attention right now, in one read, for the console's Overview. Its lists hold at most OperationsService.AttentionListCap (50) entries each, largest or worst first, and every entry carries Link, the console path and query that lists what it counts.
| Property | What it holds |
|---|---|
Scheduler | An AttentionScheduler: ManifestManagerEnabled and JobDispatcherEnabled from the live scheduler settings, and Paused, true when either is off. Link is /system/scheduler. |
DeadLetters | AttentionDeadLetterGroup rows: the dead letters awaiting intervention, grouped by the TrainName their manifest runs and their Reason, with Count and OldestDeadLetteredAt. Link is /runs/dead-letters?status=AWAITING_INTERVENTION&train=…&reason=…. |
FailingTrains | AttentionFailingTrain rows: each train that failed at least once among the runs started in the last hour and whose LastHourFailureRate (FailedLastHour over FinishedLastHour) is above its BaselineFailureRate, the rate of the runs it started in the seven days before that hour. A rate counts completed and failed runs, not cancelled ones; a train with no earlier finished run has a baseline of 0. Link is /runs?state=FAILED&train=…. |
StuckInstances | AttentionStuckInstances rows: the stuck state-machine instances (see GetMachineInstancesAsync for the rule), grouped by Machine and State with Count, system instances and users' drafts together and never naming either. Link is /machines/{machine}?state=…&stuck=true. |
Queue | An AttentionQueue: Queued, the entries queued, confirmed and due, and OldestQueuedAt and OldestAgeSeconds for the oldest of them (null when none). Link is /runs/queued?status=QUEUED. |
OfflineHosts | AttentionHost rows: each host in the heartbeat table that has missed three beats, with InstanceId, HostName, LastHeartbeatAt and RunsDispatcher, most recently seen first. Link is /system/hosts. |
AsOf | When the read was made (UTC). |
The reads run at once, each through an index: dead letters by status, the week of finished runs through the metrics index, stuck instances through the partial indexes on due time and invoke token, the dispatchable queue entries, and the heartbeat table. Enum values in a Link are spelled as the GraphQL schema spells them. The API's attention calls it. A custom implementation that does not override it throws NotSupportedException.
GetManifestGroupDependencyGraphAsync
The cross-group dependency graph around one group: the group and the groups its manifests depend on
or are depended on by, as nodes, and each cross-group dependency as a parent to dependent edge.
Each node carries IsEnabled, whether its group is enabled. null when no group has the id; a group
with no cross-group dependencies gives a graph of the group alone.
GetGlobalManifestGroupGraphAsync
The whole cross-group dependency graph: every manifest group as a node and every cross-group dependency as a parent to dependent edge, with nothing highlighted. Dependencies within a group are left out. It backs the global graph on the dashboard's manifest-groups page.
LogQuery
public record LogQuery(
long? MetadataId = null, LogLevel? MinimumLevel = null, string? Category = null,
long? AfterId = null, int Skip = 0, int Take = 25)
{
public string? MessageContains { get; init; }
public string? CategoryContains { get; init; }
public LogOrder Order { get; init; } = LogOrder.NewestFirst;
}
public enum LogOrder { NewestFirst, OldestFirst }
public record LogCount(int Count, bool Capped);| Filter | Matches |
|---|---|
MetadataId | The entries of one run |
MinimumLevel | Entries at this level or above |
Category | Entries whose category is exactly this |
MessageContains | Entries whose message contains this text, ignoring case. %, _ and \ match themselves. Null or empty matches everything |
CategoryContains | Entries whose category contains this text, matched the same way |
Case is folded by the database: on Postgres by its collation, on Sqlite for ASCII letters only.
On Postgres a trigram index over the lowered message and category
(migration 063) finds a term of
three characters or more. A page with a text filter, read by cursor or from the start, first
reads the 10,000 entries nearest where it starts in order, which fills it when the term is common
there, and finds the rest of its matches through the index and sorts them, so a term common only
among old entries does not make a newest-first page walk every newer one. Over 3,000,000 log rows a
page for a term no row carries, or one in a million carry, took about 10 ms; one for a term most rows
carry about 20 ms; one for a term in about 110,000 entries, all of them old, about 45 ms (500 ms when
the page was read in id order alone); and one for a term about one row in a hundred carries about
110 ms. A page read by offset (Skip) is still read in id order. An exact count reads every entry the
term matches, so CountLogsAsync for a term most rows carry took about 325 ms there;
CountLogsCappedAsync stops at 10,001 matches and took 13 ms for the same term, and at most about
85 ms for any term measured. On Sqlite, and for a
term shorter than three characters, nothing indexes text inside a message: a text filter reads
entries in id order until it fills a page, so a rare term reads the whole table.
An exact count of a large, unfiltered log table is a full scan, and the scheduler has no provider-neutral way to estimate one, so a pager that only needs an approximate size should estimate it itself.
LogRecord
One entry of a LogPage's Items.
| Property | Type | Description |
|---|---|---|
Id | long | The entry's id, the keyset AfterId pages by |
MetadataId | long | The run that wrote it |
EventId | int | The ILogger event id |
Level | LogLevel | The entry's level |
Category | string | The logger category |
Message | string | The formatted message |
Exception | string? | The exception, when one was logged |
StackTrace | string? | Its stack trace |
Recorded decisions
GetRecordedDecisionsAsync reads the rows AddDecisionRecording
writes, one RecordedDecisionRecord each, a page at a time in id order from
ix_decision_metadata_id_id (migration 064):
public record RecordedDecisionRecord(
long Id, long MetadataId, string? QuestionKey, int Occurrence, string? Kind, string? Question,
string? Answer, string? Refused, bool IsRefused, string? Fingerprint, string? Model,
string? Decider, bool Replayed, string? Shadows, string? Routes, string? StateHash,
DateTime DecidedAt, bool AnswerWithheld, bool TrackWithheld);
public record RecordedDecisionPage(IReadOnlyList<RecordedDecisionRecord> Items, int Take, long? NextCursor);Question, Answer, Shadows and Routes are the row's JSON as stored. The stored row keeps
every value, because a requeue replays from it, so the read applies the rules
junction events apply before anything
leaves the service:
| Case | Left out |
|---|---|
The question is about a type marked [TraxSensitive] (AnswerWithheld) | Answer, Refused, Shadows and Routes, since each states or gives away the answer. The key and the question stay, and IsRefused still says whether the run refused it |
The run took a track on such an answer before this decision (TrackWithheld) | Everything but Id, MetadataId, Occurrence, IsRefused, Replayed and DecidedAt, since which question came next gives the track away. A page that starts after the track is withheld too |
Whether a key names a marked type is worked out from the key, as junction events work it out, so a key two types share is withheld when either is marked.
The replay links are plain columns read with the run and the entry: Metadata.ReplayDecisionsOf,
Metadata.ReplayAbandoned and WorkQueue.ReplayDecisionsOf.
Masking a stored input
A work queue entry's input and a manifest's properties keep their
[TraxSensitive]
members in clear, because a run starts from them. TransportInputRedaction.Redact(discovery, json, inputTypeName), in the same namespace, is the one masking both surfaces apply to those copies:
GetWorkQueueEntryDetailAsync applies it to the entry's input, and the dashboard's dead letter
page and the API's manifestDetail to a manifest's properties. It reads the copy back as the input
type of a train registered on this host and writes it with every sensitive member, and every member
typed object or as a JSON node, as {"_redacted": true}. When no registered train takes that
type, or the copy does not read or write as it (the type's own code throwing included), the whole
value is {"_redacted": true}.
State-machine instances
The operator's read-only view of trax.snapshot_draft: the dashboard's
Machines pages and the API's
machineInstances, machineInstance and machineInstanceCounts
read through these four methods, and one instance's runs and its cancel through the two after them.
public record MachineInstanceQuery(
string? Machine = null, string? State = null, SnapshotOwnerKind? OwnerKind = null,
int Skip = 0, int Take = 25)
{
public bool? Stuck { get; init; }
}
public record MachineInstanceKey(string Machine, SnapshotOwnerKind OwnerKind, Guid Id, long? RowId = null);
public record MachineInstanceRecord(
long RowId, string Machine, SnapshotOwnerKind OwnerKind, Guid Id, string State, int Version,
DateTimeOffset? CreatedAt, DateTimeOffset UpdatedAt, bool HasLiveInvokedRun);
public record MachineInstancePage(IReadOnlyList<MachineInstanceRecord> Items, int Skip, int Take);
public record MachineInstanceTotal(int Count, bool Capped);
public record MachineInstanceStateCount(string Machine, string State, SnapshotOwnerKind OwnerKind, long Count)
{
public long Stuck { get; init; }
}
public record MachineInstanceWaitingOn(long? ExecutionId, TrainState? TrainState, long? WorkQueueEntryId);| Method | Returns |
|---|---|
GetMachineInstancesAsync(query, ct) | see below |
CountMachineInstancesAsync(query, ct) | see below |
GetMachineInstanceAsync(key, ct) | see below |
GetMachineInstanceStateCountsAsync(machine, ct) | see below |
WatchMachineInstancesAsync(machine, ct) | see below |
GetMachineInstancesAsync
A page of instances matching the query's machine, state and owner kind (each optional), newest first by UpdatedAt, then by row. Take is clamped to 1 through 500, a negative Skip reads from the start, and a Skip above MachineInstanceCountCap (10,000) throws ArgumentOutOfRangeException, as the API's machineInstances refuses it.
CountMachineInstancesAsync
How many match, counted up to OperationsService.MachineInstanceCountCap (10,000): more give Count = 10000, Capped = true.
GetMachineInstanceAsync
One instance, or null. The owner kind is part of every lookup, so a user's draft never answers for a system instance under the same id. A user's draft also needs RowId, because several users can hold a draft under one id: without it the method throws ArgumentException.
GetMachineInstanceStateCountsAsync
Exact counts by machine, state and owner kind, ordered by those three; machine narrows it to one machine. Kept for MachineInstanceCountCacheDuration (5 seconds) per host and per machine, with callers asking meanwhile sharing one read, so a count can be that old.
WatchMachineInstancesAsync
The live feed of instances: each instance once whenever any path on any host writes it, narrowed to one machine when machine is set, and once more with Deleted set when it is deleted or expires. A record with Reload set names no instance: writes may have been missed (the change listener started or reconnected, or the reader fell behind), so the reader reads what it shows again. A burst of writes to one instance may arrive as its latest revision only. It starts with what is written after it opens, ends when ct fires, when the host stops or when the reader falls too far behind, and ends at once on a host that registers no state machines. Every reader on a host shares the host's one change listener; the dashboard's instance page and the API's onMachineInstanceChanged read it here. A service constructed without an IServiceProvider throws InvalidOperationException on the first read.
MachineInstanceRecord also carries Revision, the instance's revision, as an init-only property: 1
when the row was created and one more for every write since, bumped by the database; and Deleted, set
only on the live feed, on the instance as it was last seen before it was deleted or expired.
The list and the lookup also say whether an instance needs attention (the live feed leaves these
unset): DueAt, when its state's timer fires; Overdue, when that is past by more than
StateMachineOptions.InvokeOutcomeSweepInterval, the reconciler's pace; WaitingOn, the run its
state waits on as a MachineInstanceWaitingOn (its ExecutionId, null while still queued, its
TrainState, Pending while queued and Cancelled for an entry cancelled before it started, and
its WorkQueueEntryId), never the invoke token; Stuck, when it is overdue or its run failed or was
cancelled and the outcome was never applied (a run that ended more than one sweep ago, or an entry
cancelled while queued); and InStateSince, which is UpdatedAt until a transition log is kept.
MachineInstanceQuery.Stuck lists only the stuck instances (true) or only the others (false),
and MachineInstanceStateCount.Stuck counts them per state, in a second grouped read of the stuck
rows alone, kept in the same five-second cache as the counts.
A record never carries the snapshot's context or the owning user's key. The context is an untyped
JSON object, so nothing could mask its sensitive parts; an operator sees where an instance is and
when it got there, never what it holds. HasLiveInvokedRun says whether the instance holds an invoke
token, not what the token is. CreatedAt is null for a row written before migration 071 (Sqlite
033) added it. On Postgres a page under one machine and state reads
ix_snapshot_draft_machine_state_updated in order, and any other page reads
ix_snapshot_draft_updated; at two million instances every page measured took under 10 ms and the
counts about 120 ms.
Two more methods serve one instance's page and its one operator action, the dashboard's and the
API's alike (machineInstance { invokedRuns } and
cancelMachineInstance):
public record MachineInstanceRun(
long Id, string ExternalId, string TrainName, TrainState TrainState, DateTime StartTime,
DateTime? EndTime, FailureClass FailureClass, bool CancellationRequested, bool IsLive);
public record MachineInstanceRuns(IReadOnlyList<MachineInstanceRun> Items, bool Capped, long? QueuedEntryId);
public enum MachineInstanceCancelOutcome
{
Moved, RunCancelled, CancelRequested, // the run was cancelled or its cancel requested
UserOwned, NotFound, NoLiveRun, RunEnded, // refused, nothing changed
}
public record MachineInstanceCancelResult(
MachineInstanceCancelOutcome Outcome, string Message, string? State = null)
{
public bool Success { get; }
}| Method | Returns |
|---|---|
GetMachineInstanceRunsAsync(key, ct) | see below |
CancelMachineInstanceAsync(key, ct) | see below |
GetMachineInstanceRunsAsync
The runs the instance invoked, newest first, at most OperationsService.MachineInstanceRunCap (50), with the one its state waits on marked IsLive; QueuedEntryId is that run's work queue entry while it is still queued. null when no row matches. A system instance lists every run linked to it (read through ix_metadata_invoking_instance); a user's draft lists only the run its own invoke token names, because a run does not record which user's draft queued it. Never a run's input or output.
GetMachineInstanceTransitionsAsync
One page of the instance's history, newest first: a MachineInstanceTransitionRecord for every snapshot write Trax
recorded (trax.snapshot_transition), or null when no row matches. Each carries the Revision, the Kind
(SnapshotTransitionKind: Start, Command, Save, Timer, Outcome), the Trigger, the Actor
(SnapshotTransitionActor: Owner, Host, Machine, Policy, never which user), FromState, ToState, At,
PreviousAt (the transition before it), and what it started: StartedRun (MachineInstanceStartedRun, with
ExecutionId, TrainState and WorkQueueEntryId, never the token) or StartedChild
(MachineInstanceStartedChild, with Machine, OwnerKind, Id and RowId). take is clamped to 1 through 500;
pass the page's NextCursor as beforeRevision for older ones, null on the last page. The history is the row's, so a
user's draft shows only its own. It is read by row id through uq_snapshot_transition_draft_revision, so a page costs
its rows whatever the history's length. The API serves it as
machineInstance { transitions }.
public record MachineInstanceTransitionPage(
IReadOnlyList<MachineInstanceTransitionRecord> Items, int Take, long? NextCursor);CancelMachineInstanceAsync
Cancels a system-owned instance's live run: a still-queued entry is marked Cancelled by one conditional statement, and a dispatched run is flagged as CancelExecutionsAsync flags one. The dispatcher claims an entry and writes its run in one transaction, so the run either never starts or is cancelled. When this host registers the machine (AddStateMachines), a queued run's Cancelled outcome is applied in the call (Moved); otherwise a host that does applies it (RunCancelled). Every delivery is the one conditional update on the token, so the outcome is applied once. A dispatched run's outcome is applied when it ends (CancelRequested). Refused, with a typed outcome and changing nothing: UserOwned, NotFound, NoLiveRun, RunEnded.
The messages are public members of OperationsService, and both surfaces show them unchanged:
| Outcome | Message |
|---|---|
Moved | MovedMessage(key, state): the queued run was cancelled and the instance moved to state in this call |
RunCancelled | RunCancelledMessage(key): the queued run was cancelled; a host that registers the machine moves it |
CancelRequested | CancelRequestedMessage(key, state): the dispatched run's cancel flag is set |
UserOwned | UserOwnedCancelRefusal |
NotFound | InstanceNotFoundMessage(key) |
NoLiveRun | NoLiveRunMessage(key, state) |
RunEnded | RunEndedMessage(key, state): the run ended before the cancel reached it, and its outcome is on its way |
A requeue of a run a machine invoked is refused with InvokedRunRequeueRefusal(metadataId, machine),
and an operator's cancel of a run a user's draft started with UserOwnedRunCancelRefusal.
State-machine definitions
The machines the host registers, described from what each declares rather than from its instances,
so a machine with none is listed. The console's Machines page and the API's
operations.machines read
through this method.
public record MachineDescription(string Name, int Version, bool SystemOwned)
{
public string? SchemaHash { get; init; }
public string? InitialState { get; init; }
public IReadOnlyList<MachineStateDescription> States { get; init; }
public IReadOnlyList<MachineTransitionDescription> Transitions { get; init; }
public IReadOnlyList<MachineTimerDescription> Timers { get; init; }
public string? GraphUnavailableReason { get; init; }
}
public record MachineStateDescription(string Name)
{
public bool IsInitial { get; init; }
public bool IsFinal { get; init; }
public bool IsCommitted { get; init; }
public string? InvokesTrain { get; init; }
public string? SpawnsMachine { get; init; }
}
public enum MachineTransitionKind { Command, Timer, Outcome }
public record MachineTransitionDescription(string From, string Command, string To, MachineTransitionKind Kind)
{
public string? Guard { get; init; }
public string? Effect { get; init; }
public string? After { get; init; }
}
public record MachineTimerDescription(string State, string Command, string Target)
{
public string? After { get; init; }
public string? Field { get; init; }
public string? Min { get; init; }
public string? Max { get; init; }
}| Method | Returns |
|---|---|
GetMachinesAsync(ct) | every registered machine, ordered by name |
GetMachineBoardAsync(machine, ct) | the event-storming board of a machine served from features, or null |
GetMachinesAsync
Every machine registered in the container, each type once, ordered by name. The graph is read from
the machine's IR (IMachine.ExportIr()) by the same reading
trax machine show draws it from, so the two list the same states and edges
in the same order: the commands, then the timers, then each invoking state's outcomes in the order
they are tried. IsFinal is true for a state no transition leaves. InvokesTrain is the invoked
train's interface full name and SpawnsMachine the spawned machine's id. A timer appears twice: as
a Timer transition, whose After is the duration on one line, and in Timers with the fixed
duration (After) or the context field and its bounds (Field, Min, Max), all ISO-8601. A
machine declared with C# delegates has no IR: it is listed with its name, version and
GraphUnavailableReason, and no states. Each description is built once per machine and kept,
since a machine's definition does not change while the host runs. It reads no table.
GetMachineBoardAsync
The event-storming board of a machine the host serves from features
(ServeFeatures), drawn from the host file it serves
and its machines' IRs: a MachineBoardDescription of its Machine, its Lanes (the asked-for machine first, then
each machine a policy or a view joins it to, by name), its Stickies and its Links. A MachineStickyDescription
has an Id, a Kind (actor, command, event, policy, aggregate or view), the Machine whose lane it is
in, a Label, and the Scenarios that produced it, each a MachineScenarioDescription (File, Line,
Scenario), none when the host file has no provenance. A MachineBoardLinkDescription goes From one sticky's id
To another's. A machine not served from features answers null. It reads no table. The console's Board page and
operations.machineBoard read through it.
Behavior
Registered services
AddScheduler(...) registers the default implementation, OperationsService, as scoped. An API-only host registers it itself with services.AddScoped<IOperationsService, OperationsService>() (see AddTraxGraphQL).
Authorization
Neither method decides authorization itself: both leave it to the mediator, QueueTrainAsync and RequeueExecutionAsync through QueueAsync, and RunTrainAsync through PrepareAsync. An ITrainAuthorizationService decides when one is registered (Trax.Api registers one). Without one, a call inside a trusted scope passes, and a [TraxAuthorize] train is refused unless the host called AllowMissingAuthorizationService(). The dashboard calls both inside the "dashboard" trusted scope, because it is gated as a whole by its host (see Authorization: The Operations Surface).
Effects and log levels
The effects page and the server log levels are per-process settings rather than rows, so they
have services of their own, registered by AddScheduler beside this one:
IEffectSettingsService and
ILogLevelService. So does requeue-all,
whose job state is this node's:
IDeadLetterRequeueJobs.
Implementing it yourself
RunTrainAsync and every method added after it have a default implementation that throws NotSupportedException, so an implementation written before they were added still compiles. OperationsService needs the constructor that takes an IServiceProvider, which dependency injection picks, to resolve a job submitter; built with the older constructor, its RunTrainAsync throws InvalidOperationException. Its RunTrainAsync also needs an ITrainExecutionService that implements PrepareAsync, as the mediator's own does; one written before that method existed throws NotSupportedException rather than skip authorization.
Example
using Trax.Scheduler.Services.Operations;
public class ReportQueue(IOperationsService operations)
{
public async Task<string> QueueReport(CancellationToken ct)
{
var result = await operations.QueueTrainAsync(
new QueueTrainInput("MyApp.Trains.IBuildReportTrain", """{"region":"eu"}""", 0, null),
ct);
return result.Success ? $"Queued entry {result.Id}" : result.Message ?? "Refused";
}
}See also
- ITraxScheduler, the scheduler API it builds on
- Dashboard and the GraphQL mutations, its two callers
- Authorization: The Operations Surface