Reference

operations: system queries

Package Trax.Api.GraphQLSurface GraphQL schemaSource OperationsQueries.cs

Fields of OperationsQueries, the type behind query { operations { ... } }, that describe the host and its registered trains. The namespace exists only when the host calls ExposeOperationQueries(), and every field answers to the operations gate.

health

Returns the current health status of the Trax scheduler system. This is the same data reported by the ASP.NET IHealthCheck at /trax/health, exposed as a structured GraphQL type.

It is the dashboard's home page read, behind the operations gate like every field here. A monitor, load balancer or scraper reads /trax/health instead (AddTraxHealthCheck and MapHealthChecks), which needs no GraphQL document and no operator credential.

query {
  operations {
    health {
      status
      description
      queueDepth
      inProgress
      failedLastHour
      deadLetters
    }
  }
}

Returns: HealthStatus!

HealthStatus fields

FieldTypeDescription
statusString!"Healthy" or "Degraded"
descriptionString!Human-readable summary
queueDepthInt!Work items with status Queued
inProgressInt!Executions with TrainState.InProgress
failedLastHourInt!Failed executions in the last hour
deadLettersInt!Dead letters with status AwaitingIntervention

Status is Degraded when deadLetters > 0 or failedLastHour > 10.

trains

Returns every train registered in the DI container, including a runtime-generated input schema describing each property on the input type. Pass hideAdminTrains: true to exclude the framework's internal scheduler trains (manifest manager, job dispatcher, dead letter cleanup, etc.) from the result; the dashboard uses this flag when its "Hide admin trains" toggle is on.

query {
  operations {
    trains {
      fullName
      serviceTypeName
      implementationTypeName
      inputTypeName
      outputTypeName
      lifetime
      inputSchema {
        name
        typeName
        isNullable
        enumValues
      }
    }
  }
}

Returns: [TrainInfo!]!

TrainInfo fields

FieldTypeDescription
fullNameString!The train's canonical name, its service interface's FullName (e.g. MyApp.Trains.IProcessOrderTrain). This is the name every field that takes a train accepts: queueTrain and runTrain, trainStats, and the trainName filters on executions and workQueues
serviceTypeNameString!Friendly name of the service interface (e.g. IServiceTrain<OrderInput, OrderResult>), for display. No field that takes a train accepts it; use fullName
implementationTypeNameString!Friendly name of the concrete class
inputTypeNameString!Friendly name of the input type
outputTypeNameString!Friendly name of the output type
lifetimeString!DI lifetime (Singleton, Scoped, Transient)
inputSchema[InputPropertySchema!]!Public readable properties on the input type
requiredPolicies[String!]!The policies the train's [TraxAuthorize] attributes name
requiredRoles[String!]!The roles the train's [TraxAuthorize] attributes name
isQueryBoolean!Whether the train carries [TraxQuery]
isMutationBoolean!Whether the train carries [TraxMutation]
graphQLNameStringThe Name set on [TraxQuery] or [TraxMutation]; null when the field name is derived
isBroadcastEnabledBoolean!Whether the train carries [TraxBroadcast]
hasQueueSubjectKeyBoolean!Whether the train overrides QueueSubjectKey, so its queued runs for one subject run one at a time (an override may still return no key for a given input). A run started with runTrain bypasses that serialization and may run alongside queued or in-flight work for the same subject; the dashboard's Run dialog warns about it, and a client offering Run should too
chainRunGraph!The train's declared chain, read without a run: the same nodes, in the same order and places, that runGraph draws a run of it on, every one NOT_REACHED, with no steps and metadataId 0. hasGraph is false when the host cannot read the train's chain. Read only when selected, from the chain the host reads once per train
asksDecisionsBoolean!True when the train declares a step that asks a decider and records the decision: a Decide, or a Switch, Gate or Scale that asks its own question. Only such a train's runs have decisions a requeue can replay or ask afresh. False when the host cannot read the train's chain
ParameterTypeDefaultDescription
hideAdminTrainsBooleanfalseWhen true, filters out framework-internal scheduler trains (matches AdminTrains.FullNames in Trax.Scheduler.Configuration)

InputPropertySchema fields

FieldTypeDescription
nameString!The JSON name the input reader expects: the property name under the system JSON naming policy (camelCase by default), or its [JsonPropertyName] when it has one. A property marked [JsonIgnore] is left out
typeNameString!Friendly type name (e.g. String, Int32, DateTime?)
isNullableBoolean!Whether the property is nullable
enumValues[String!]The accepted values when the property is an enum or a nullable enum, spelled as the reader expects them. Null for any other type

The names and enum spellings come from the same options queueTrain and runTrain deserialize input with, so a client that builds its JSON from this schema writes what the reader accepts.

adminTrainNames

Returns the canonical FullNames of the framework's internal scheduler trains (job dispatcher, manifest manager, job runner, cleanup). This is the same list hideAdminTrains filters against on the trains, manifests and executions queries and the metrics.dashboard query.

query {
  operations {
    adminTrainNames
  }
}

The subscription feed streams every train on an admin host, so the dashboard reads this list once and filters the live onTrainStateChanged events client-side by the same rule the server applies to the grid, keeping the feed and grid consistent when "Hide admin trains" is on.

Returns: [String!]!

hosts

The host processes that have beaten in the last day, from trax.host_heartbeat. Every host with a data provider rewrites its row every 15 seconds whether or not it has work (Host Tracking), so the list shows an idle host as online and a dead one as offline. Online hosts come first, then by name.

query {
  operations {
    hosts {
      instanceId
      name
      environment
      traxVersion
      startedAt
      lastHeartbeat
      online
      runsDispatcher
      runsManifestManager
      workerSlots
      workerSlotsInUse
      currentlyRunning
    }
  }
}

It reads the small heartbeat table and counts each listed host's runs in progress, so it is cheap at any history size. A host on a version from before the heartbeat writes no row and is not listed.

Returns: [HostInfo!]!

HostInfo fields

FieldTypeDescription
instanceIdString!The process's instance id, the one stamped on each run it executes
nameString!Machine or container host name
environmentString!Hosting environment, such as server, ecs, kubernetes or lambda
traxVersionString!The Trax version the process runs
startedAtDateTime!When the process began beating
lastHeartbeatDateTime!When the process last wrote its row
onlineBoolean!true while the last heartbeat is under 45 seconds old (three beats)
runsDispatcherBoolean!The process runs the job dispatcher, switched on
runsManifestManagerBoolean!The process runs the manifest manager, switched on
workerSlotsIntLocal workers the process runs, null with none
workerSlotsInUseIntWorkers running a job at the last heartbeat, null with none
currentlyRunningInt!Runs attributed to this host still InProgress

trainStats

Execution roll-up for a single train, keyed by its interface FullName (the value stored in metadata.Name). Backs the summary cards on the dashboard's per-train detail page. The state grouping and ix_metadata_* indexes keep it cheap against a large metadata table.

query {
  operations {
    trainStats(trainName: "MyApp.Trains.Reports.IBuildDailyReportTrain") {
      total
      completed
      failed
      inProgress
      averageMilliseconds
      lastRun
      lastSuccessfulRun
    }
  }
}

Returns: TrainExecutionStats!

TrainExecutionStats fields

FieldTypeDescription
trainNameString!The interface FullName the stats are scoped to
totalLong!Total executions of this train
completed / failed / inProgress / pending / cancelledLong!Counts by state
lastRunDateTimeStart time of the most recent execution. Null when there are none
lastSuccessfulRunDateTimeEnd time of the most recent Completed execution. Null when there are none
averageMillisecondsFloatMean duration of completed executions. Null when there are none

effects

Lists the observational effects registered in the API process, with their enabled and toggleable state and, for an effect whose factory exposes runtime settings, those settings: as JSON, and as fields a settings editor can be built from. It reads through IEffectSettingsService in Trax.Scheduler, the service the dashboard's effects page calls, so both show the same values. Empty when the host registers no effect registry.

The effect registry and each settings object are in-memory, per-process singletons with no persistence or cross-process broadcast, so this reflects the API host only, not the scheduler/worker processes where effects actually run. setEffectEnabled toggles an effect and configureEffect changes its settings in this same process. Changing effect state across a distributed deployment would need a shared store plus a change broadcast, which is not built.

query {
  operations {
    effects {
      name
      fullName
      enabled
      toggleable
      isConfigurable
      configurationTypeName
      configuration
      fields {
        name
        typeName
        kind
        nullable
        enumValues
        sensitive
        hasValue
        value
        hint
      }
    }
  }
}

Returns: [EffectInfo!]!, ordered by fullName.

EffectInfo fields

FieldTypeDescription
nameString!Effect factory type name (short)
fullNameString!Effect factory type FullName
enabledBoolean!Whether the effect is currently enabled in this process
toggleableBoolean!Whether the effect can be toggled (infrastructure effects are always on)
isConfigurableBoolean!Whether the effect's factory exposes runtime settings (implements IConfigurableProviderFactory)
configurationTypeNameStringFullName of the settings type. Null when not configurable
configurationStringThe factory's current settings as camelCase JSON, with each [TraxSensitive] member written as {"_redacted": true}. Null when not configurable, or when the settings type cannot be written as JSON
fields[EffectSettingInfo!]!One per public read-write property of the settings type, editable ones first in declaration order. Empty when not configurable

EffectSettingInfo fields

FieldTypeDescription
nameString!The property's name, which configureEffect takes
typeNameString!The property's type, without Nullable<>, such as Int32 or TimeSpan
kindEffectFieldKind!How it is edited: BOOLEAN (a switch), ENUM (one of enumValues), TEXT (text read as the type: a number, string, date, time, duration, GUID or character) or SET_IN_CODE (a delegate, collection or object, which cannot be written here)
nullableBoolean!Whether it accepts no value; a null or blank value writes null
enumValues[String!]The member names, for an enum. Null otherwise
sensitiveBoolean!true for a property marked [TraxSensitive]: value is always null. It can still be written
hasValueBoolean!Whether the setting holds a value. true for a sensitive setting that holds one, so a client can show it is set without showing it
valueStringThe current value as text, in the form configureEffect reads back. Null when there is none, when the setting is sensitive, and for SET_IN_CODE
hintString!What to type, such as yyyy-MM-dd HH:mm:ss (UTC)

Settings can hold credentials. Like an execution's input, they are reachable only under the operations namespace, so the gate you put on it (GateOperations or RequireAuthorization) decides who reads them. See Train inputs and the operations gate.

lookup

Finds what an id, key or name names when you do not know its kind, the read behind the console's search. It calls IOperationsService.LookupAsync.

query {
  operations {
    lookup(text: "7f3c0e2a9b1d4c5e8f6a7b8c9d0e1f2a") {
      kind
      id
      label
      matchedOn
      machine
      ownerKind
      rowId
    }
  }
}

Returns: [LookupMatch!]!, empty when nothing matches.

FieldTypeDescription
kindLookupKind!RUN, MANIFEST, WORK_QUEUE_ENTRY, DEAD_LETTER, MACHINE_INSTANCE or TRAIN
idString!The id the kind's read takes: a run, manifest, entry or dead letter id, an instance's GUID, or a train's fullName
labelString!The train, manifest or machine, with its state
matchedOnString!The field the text matched: id, externalId, name or subjectKey
machineStringFor an instance, its machine; null otherwise
ownerKindSnapshotOwnerKindFor an instance, who owns it; null otherwise
rowIdLongFor an instance, the row machineInstance takes for a user's draft; null otherwise

Matches are exact, after trimming: runs by id and external id, manifests by id, external id and name, work queue entries by id, external id and subject key (those still queued or dispatched), dead letters by id, state-machine instances by id, and registered trains by full or friendly name. Each is one read of an indexed column, at most 10 matches of each kind. An instance is named the way machineInstances names it, never by its context or whose a user's draft is. Like every field here it answers to the operations gate.

attention

What needs an operator's attention, the read behind the console's Overview. It calls IOperationsService.GetAttentionAsync.

query {
  operations {
    attention {
      asOf
      scheduler { manifestManagerEnabled jobDispatcherEnabled paused link }
      deadLetters { trainName reason count oldestDeadLetteredAt link }
      failingTrains { trainName failedLastHour finishedLastHour lastHourFailureRate baselineFailureRate link }
      stuckInstances { machine state count link }
      queue { queued oldestQueuedAt oldestAgeSeconds link }
      offlineHosts { instanceId hostName lastHeartbeatAt runsDispatcher link }
    }
  }
}

Returns: Attention!.

FieldTypeDescription
schedulerAttentionScheduler!Whether the manifest manager and the job dispatcher are switched on; paused is true when either is off
deadLetters[AttentionDeadLetterGroup!]!Dead letters awaiting intervention, one group per train and reason, with count and oldestDeadLetteredAt, largest first
failingTrains[AttentionFailingTrain!]!Trains whose failure rate among runs started in the last hour is above their rate over the seven days before it, highest first
stuckInstances[AttentionStuckInstances!]!Stuck state-machine instances, one group per machine and state, largest first
queueAttentionQueue!Queued entries that are confirmed and due, and how long the oldest has waited
offlineHosts[AttentionHost!]!Hosts that have missed three heartbeats, most recently seen first
asOfDateTime!When the read was made

Each list holds at most 50 entries. Every entry has a link: the console path and query that lists what it counts, such as /runs?state=FAILED&train=Shop.ICheckoutTrain or /machines/order?state=Waiting&stuck=true. A stuck-instance group names a machine and a state, never an instance's owner or context. Like every field here it answers to the operations gate.

See also