Queries

Queries are organized into two groups under the root Query type:

type Query {
  discover: DiscoverQueries!
  operations: OperationsQueries!  # only when ExposeOperationQueries() is set
}
  • discover: auto-generated typed query fields for trains annotated with [TraxQuery]
  • operations: predefined operational queries: health status, registered trains, manifests, manifest groups, execution history, and the nested deadLetters namespace. Off by default, opt in with ExposeOperationQueries() on the builder.

Discover Queries (Auto-Generated)

Trax auto-generates strongly-typed query fields for trains that opt in with [TraxQuery]. Only trains with this attribute appear under discover.

Each whitelisted query train gets a single field named after the train (no prefix). The field accepts a strongly-typed input argument and returns the train's output type directly. Trains with Namespace set are grouped under a sub-namespace (e.g. discover { players { lookupPlayer } }).

Naming Convention

The query field names are derived from the train's service interface name (or overridden via [TraxQuery(Name = "...")]):

  1. Strip the I prefix
  2. Strip the Train suffix
  3. Use the result as the field name (lowercase first letter)

For example, ILookupPlayerTrain produces lookupPlayer.

Example

Given a train annotated with [TraxQuery]:

public record LookupPlayerInput
{
    public required string PlayerId { get; init; }
}
 
public record LookupPlayerOutput
{
    public required string PlayerId { get; init; }
    public required int Rank { get; init; }
}

The schema exposes:

query {
  discover {
    lookupPlayer(input: { playerId: "player-42" }) {
      playerId
      rank
    }
  }
}

Query trains with typed output

When a query train has a non-Unit output type, the output type is returned directly (not wrapped in a response type):

type DiscoverQueries {
  lookupPlayer(input: LookupPlayerInput!): LookupPlayerOutput!
}

Query trains with Unit output

When a query train has Unit output, it returns a response with the execution metadata:

FieldTypeDescription
metadataIdLong!Metadata ID of the completed execution

Operations Queries

health

Returns the current health status of the Trax scheduler system. This is the same data reported by the ASP.NET IHealthCheck at /trax/health, exposed as a structured GraphQL type.

query {
  operations {
    health {
      status
      description
      queueDepth
      inProgress
      failedLastHour
      deadLetters
    }
  }
}

Returns: HealthStatus!

HealthStatus fields

FieldTypeDescription
statusString!"Healthy" or "Degraded"
descriptionString!Human-readable summary
queueDepthInt!Work items with status Queued
inProgressInt!Executions with TrainState.InProgress
failedLastHourInt!Failed executions in the last hour
deadLettersInt!Dead letters with status AwaitingIntervention

Status is Degraded when deadLetters > 0 or failedLastHour > 10.


trains

Returns every train registered in the DI container, including a runtime-generated input schema describing each property on the input type. Pass hideAdminTrains: true to exclude the framework's internal scheduler trains (manifest manager, job dispatcher, dead letter cleanup, etc.) from the result; the dashboard uses this flag when its "Hide admin trains" toggle is on.

query {
  operations {
    trains {
      fullName
      serviceTypeName
      implementationTypeName
      inputTypeName
      outputTypeName
      lifetime
      inputSchema {
        name
        typeName
        isNullable
        enumValues
      }
    }
  }
}

Returns: [TrainInfo!]!

TrainInfo fields

FieldTypeDescription
fullNameString!The train's canonical name, its service interface's FullName (e.g. MyApp.Trains.IProcessOrderTrain). This is the name every field that takes a train accepts: queueTrain and runTrain, trainStats, and the trainName filters on executions and workQueues
serviceTypeNameString!Friendly name of the service interface (e.g. IServiceTrain<OrderInput, OrderResult>), for display. No field that takes a train accepts it; use fullName
implementationTypeNameString!Friendly name of the concrete class
inputTypeNameString!Friendly name of the input type
outputTypeNameString!Friendly name of the output type
lifetimeString!DI lifetime (Singleton, Scoped, Transient)
inputSchema[InputPropertySchema!]!Public readable properties on the input type
ParameterTypeDefaultDescription
hideAdminTrainsBooleanfalseWhen true, filters out framework-internal scheduler trains (matches AdminTrains.FullNames in Trax.Scheduler.Configuration)

InputPropertySchema fields

FieldTypeDescription
nameString!The JSON name the input reader expects: the property name under the system JSON naming policy (camelCase by default), or its [JsonPropertyName] when it has one. A property marked [JsonIgnore] is left out
typeNameString!Friendly type name (e.g. String, Int32, DateTime?)
isNullableBoolean!Whether the property is nullable
enumValues[String!]The accepted values when the property is an enum or a nullable enum, spelled as the reader expects them. Null for any other type

The names and enum spellings come from the same options queueTrain and runTrain deserialize input with, so a client that builds its JSON from this schema writes what the reader accepts.


adminTrainNames

Returns the canonical FullNames of the framework's internal scheduler trains (job dispatcher, manifest manager, job runner, cleanup). This is the same list hideAdminTrains filters against on the trains and executions queries and the metrics.dashboard query.

query {
  operations {
    adminTrainNames
  }
}

The subscription feed streams every train on an admin host, so the dashboard reads this list once and filters the live onTrainStateChanged events client-side by the same rule the server applies to the grid, keeping the feed and grid consistent when "Hide admin trains" is on.

Returns: [String!]!


hosts

Rolls up the processes that have executed trains, grouped by HostInstanceId from the metadata table. Backs the dashboard's cluster view across a split API + scheduler + worker deployment. Rows without a stamped host instance are ignored.

query {
  operations {
    hosts {
      instanceId
      name
      environment
      lastSeen
      totalExecutions
      currentlyRunning
    }
  }
}

This is a full aggregation over the metadata table (like metrics.dashboard), so treat it as an occasional-refresh read, not a hot poll. lastSeen is the most recent execution start on the host, the freshness signal the dashboard renders; there is no separate heartbeat.

Returns: [HostInfo!]!

HostInfo fields

FieldTypeDescription
instanceIdString!Stable per-process id (lives for the process's lifetime)
nameStringMachine / container host name, if stamped
environmentStringHosting environment (e.g. Production), if stamped
lastSeenDateTime!Most recent execution start on this host
totalExecutionsLong!Total executions attributed to this host
currentlyRunningInt!Executions on this host still InProgress

trainStats

Execution roll-up for a single train, keyed by its interface FullName (the value stored in metadata.Name). Backs the summary cards on the dashboard's per-train detail page. The state grouping and ix_metadata_* indexes keep it cheap against a large metadata table.

query {
  operations {
    trainStats(trainName: "Trax.Samples.GameServer.Trains.IRecalculateLeaderboardTrain") {
      total
      completed
      failed
      inProgress
      averageMilliseconds
      lastRun
      lastSuccessfulRun
    }
  }
}

Returns: TrainExecutionStats!

TrainExecutionStats fields

FieldTypeDescription
trainNameString!The interface FullName the stats are scoped to
totalLong!Total executions of this train
completed / failed / inProgress / pending / cancelledLong!Counts by state
lastRunDateTimeStart time of the most recent execution. Null when there are none
lastSuccessfulRunDateTimeEnd time of the most recent Completed execution. Null when there are none
averageMillisecondsFloatMean duration of completed executions. Null when there are none

effects

Lists the observational effects registered in the API process, with their enabled and toggleable state and, for an effect whose factory exposes runtime settings, those settings. Backs the dashboard's effects list.

The effect registry is an in-memory, per-process singleton with no persistence or cross-process broadcast, so this reflects the API host only, not the scheduler/worker processes where effects actually run. setEffectEnabled toggles an effect in this same process. Changing effect state across a distributed deployment would need a shared store plus a change broadcast, which is not built.

query {
  operations {
    effects {
      name
      fullName
      enabled
      toggleable
      isConfigurable
      configurationTypeName
      configuration
    }
  }
}

Returns: [EffectInfo!]!, ordered by fullName.

EffectInfo fields

FieldTypeDescription
nameString!Effect factory type name (short)
fullNameString!Effect factory type FullName
enabledBoolean!Whether the effect is currently enabled in this process
toggleableBoolean!Whether the effect can be toggled (infrastructure effects are always on)
isConfigurableBoolean!Whether the effect's factory exposes runtime settings (implements IConfigurableProviderFactory)
configurationTypeNameStringFullName of the settings type. Null when not configurable
configurationStringThe factory's current settings as camelCase JSON. Null when not configurable

Settings can hold credentials. Like an execution's input, they are reachable only under the operations namespace, so the gate you put on it (GateOperations or RequireAuthorization) decides who reads them. See Train inputs and the operations gate.


manifests

Returns a paginated list of scheduler manifests, ordered by ID descending (newest first). Supports both offset-based and keyset cursor pagination.

query {
  operations {
    manifests(skip: 0, take: 10) {
      items {
        id
        externalId
        name
        isEnabled
        scheduleType
        cronExpression
        intervalSeconds
        maxRetries
        timeoutSeconds
        lastSuccessfulRun
        manifestGroupId
        dependsOnManifestId
        priority
      }
      totalCount
      isEstimatedCount
      skip
      take
      nextCursor
    }
  }
}
ParameterTypeDefaultDescription
skipInt0Number of records to skip (offset pagination). A negative value reads as 0
takeInt25Number of records to return, from 1 to 500. See Page size
isEnabledBooleannullFilter by enabled/disabled
scheduleTypeScheduleTypenullFilter by schedule type (NONE, CRON, INTERVAL, ON_DEMAND, DEPENDENT, DORMANT_DEPENDENT, ONCE)
nameContainsStringnullCase-sensitive substring match on the train name
afterIdLongnullKeyset cursor. Returns records with id < afterId. When provided, skip is ignored. See Pagination
manifestGroupIdLongnullOnly manifests belonging to this group. The dashboard uses it to list a group's manifests

Returns: PagedResult<ManifestSummary>

ManifestSummary fields

FieldTypeDescription
idLong!Database ID
externalIdString!Unique external identifier (used for upsert/trigger)
nameString!Train type name
isEnabledBoolean!Whether the manifest is active
scheduleTypeScheduleType!Cron or Interval
cronExpressionStringCron expression (when scheduleType is Cron)
intervalSecondsIntInterval in seconds (when scheduleType is Interval)
maxRetriesInt!Maximum retry count on failure
timeoutSecondsIntExecution timeout
lastSuccessfulRunDateTimeTimestamp of last successful execution
manifestGroupIdLong!Parent group ID
dependsOnManifestIdLongID of the manifest this one depends on
priorityInt!Dispatch priority (0-31, higher runs first)
manifestGroupNameStringName of the parent group

A manifest's properties (the train input it runs with) are not on this type. Read them from manifestDetail, one manifest at a time.


manifest

Returns a single manifest by database ID.

query {
  operations {
    manifest(id: 42) {
      id
      externalId
      name
      isEnabled
      scheduleType
      cronExpression
      priority
    }
  }
}
ParameterTypeRequiredDescription
idLong!YesThe manifest's database ID

Returns: ManifestSummary (nullable, returns null if the ID does not exist)


manifestDetail

Returns everything manifest does plus the train input the manifest runs with and the rest of its scheduling settings. Use it for a manifest detail page.

query {
  operations {
    manifestDetail(id: 42) {
      id
      name
      manifestGroupName
      propertyTypeName
      properties
      misfirePolicy
      misfireThresholdSeconds
      scheduledAt
      nextScheduledRun
      varianceSeconds
    }
  }
}
ParameterTypeRequiredDescription
idLong!YesThe manifest's database ID

Returns: ManifestDetail (nullable, returns null if the ID does not exist). It carries every ManifestSummary field (with manifestGroupName non-null) and:

FieldTypeDescription
propertyTypeNameStringFully qualified type name of the train input
propertiesStringThe train input as stored JSON, with each [TraxSensitive] member masked. Can hold credentials; see Train inputs and the operations gate
misfirePolicyMisfirePolicy!What the manifest manager does with a missed run
misfireThresholdSecondsIntHow late a run can be before it counts as missed
scheduledAtDateTimeThe one-off run time, for a Once manifest
nextScheduledRunDateTimeWhen the manifest manager next plans to run it
varianceSecondsIntRandom jitter added to each run

manifestStats

Execution roll-up for a single manifest: run counts by state plus the most recent run and most recent successful run. Backs the summary cards on the dashboard's manifest detail page, through the same IOperationsService.GetManifestExecutionStatsAsync call. A manifest with no runs, or an id with no manifest, gets zeros and nulls. Served index-only by ix_metadata_manifest_state, so it stays fast on a manifest with a long history.

query {
  operations {
    manifestStats(manifestId: 42) {
      manifestId
      total
      completed
      failed
      inProgress
      pending
      cancelled
      lastRun
      lastSuccessfulRun
    }
  }
}
ParameterTypeRequiredDescription
manifestIdLong!YesThe manifest's database ID

Returns: ManifestExecutionStats! (never null; a manifest with no runs returns all-zero counts and null timestamps).

ManifestExecutionStats fields

FieldTypeDescription
manifestIdLong!The manifest these stats are for (echoes the argument)
totalLong!Total executions across all states
completedLong!Executions in Completed
failedLong!Executions in Failed
inProgressLong!Executions in InProgress
pendingLong!Executions in Pending
cancelledLong!Executions in Cancelled
lastRunDateTimeStart time of the most recent execution. Null when there are none
lastSuccessfulRunDateTimeEnd time of the most recent Completed execution. Null when there are none

manifestExclusions

The schedule exclusion windows configured on a manifest: the days, dates, ranges, or daily time windows during which it is intentionally skipped (not treated as a misfire). Backs the exclusions panel on the manifest detail page. The model is flat and discriminated: type selects which of the other fields apply.

query {
  operations {
    manifestExclusions(manifestId: 42) {
      type
      daysOfWeek
      dates
      startDate
      endDate
      startTime
      endTime
    }
  }
}
ParameterTypeRequiredDescription
manifestIdLong!YesThe manifest's database ID

Returns: [ManifestExclusion!]! (empty when the manifest has no exclusions or does not exist).

ManifestExclusion fields

FieldTypeApplies when type isDescription
typeExclusionType!alwaysDAYS_OF_WEEK, DATES, DATE_RANGE, or TIME_WINDOW
daysOfWeek[DayOfWeek!]DAYS_OF_WEEKWeekdays to skip (SUNDAY..SATURDAY)
dates[Date!]DATESSpecific calendar dates to skip
startDateDateDATE_RANGEFirst day of an inclusive skipped range
endDateDateDATE_RANGELast day of an inclusive skipped range
startTimeLocalTimeTIME_WINDOWDaily window start (supports midnight crossover)
endTimeLocalTimeTIME_WINDOWDaily window end

manifestGroups

Manifest group queries live under the operations.manifestGroups namespace, not at the top level. The namespace holds the paged list (groups), single-group lookup (group), and cross-group dependency graph (graph). See manifestGroups (nested under operations).


executions

Returns a paginated list of train executions (metadata records), ordered by ID descending (newest first). Supports both offset-based and keyset cursor pagination.

query {
  operations {
    executions(skip: 0, take: 10) {
      items {
        id
        externalId
        name
        trainState
        startTime
        endTime
        failureJunction
        failureReason
        manifestId
        cancellationRequested
      }
      totalCount
      isEstimatedCount
      skip
      take
      nextCursor
    }
  }
}
ParameterTypeDefaultDescription
skipInt0Number of records to skip (offset pagination). A negative value reads as 0
takeInt25Number of records to return, from 1 to 500. See Page size
trainStateTrainStatenullFilter by state (PENDING, IN_PROGRESS, COMPLETED, FAILED, CANCELLED)
trainNameStringnullExact-match filter on the train interface FullName
startedAfterDateTimenullOnly executions with startTime >= startedAfter
startedBeforeDateTimenullOnly executions with startTime <= startedBefore
orderSortOrderNEWESTNEWEST (id descending) or OLDEST (id ascending). Both stay keyset-safe
afterIdLongnullKeyset cursor. Returns records with id < afterId (or id > afterId when order: OLDEST). See Pagination
manifestIdLongnullOnly executions of this manifest. The dashboard uses it for a manifest's execution history
manifestGroupIdLongnullOnly executions of any manifest in this group (resolved through manifest.manifest_group_id). The dashboard uses it for a group's recent executions
hideAdminTrainsBooleanfalseWhen true, excludes the framework's internal scheduler trains (matches AdminTrains.FullNames against metadata.Name, which stores the interface FullName). The dashboard sets this from its "Hide admin trains" toggle
failureClassFailureClassnullOnly executions recorded with this failure class: UNCLASSIFIED, TRANSIENT, CONFLICT, or PERMANENT. Every run that did not fail records UNCLASSIFIED, so failureClass: UNCLASSIFIED on its own also matches every completed, pending, in-progress and cancelled run; combine it with trainState: FAILED for unclassified failures only

When any filter is supplied the count is exact (isEstimatedCount: false); an unfiltered list may use the database's row estimate, on every page alike (see Estimated counts). startedAfter/startedBefore use the ix_metadata_start_time_desc index so they stay fast at scale. manifestId and manifestGroupId are served by the covering index ix_metadata_manifest_state, so a manifest's or group's history stays index-only even against millions of rows. failureClass is served by ix_metadata_failure_class on (failure_class, id DESC) (Postgres). It covers every row rather than only classified ones: the class arrives as a query parameter, and a generic plan cannot prove a parameter satisfies a partial index's predicate, so a partial index would go unused. Arbitrary-column sorting is deliberately not offered: it is incompatible with keyset pagination over millions of rows (it forces OFFSET scans or a full sort). Filter to narrow the set instead.

Returns: PagedResult<ExecutionSummary>

ExecutionSummary fields

FieldTypeDescription
idLong!Metadata ID
externalIdString!External identifier
nameString!Train type name
trainStateTrainState!Current state (Pending, InProgress, Completed, Failed, Cancelled)
startTimeDateTime!When execution began
endTimeDateTimeWhen execution finished (null if still running)
failureJunctionStringName of the junction that failed (null if no failure)
failureReasonStringException message on failure
manifestIdLongAssociated manifest ID (null if not scheduler-initiated)
cancellationRequestedBoolean!Whether cancellation was requested
failureClassFailureClass!How the failure was classified: UNCLASSIFIED, TRANSIENT, CONFLICT, or PERMANENT. UNCLASSIFIED when the run did not fail, no failure classifier is registered, or it did not recognise the failure. Later releases may add values to FailureClass; a client generated from an older schema should treat a value it does not know as UNCLASSIFIED rather than failing to read the response

execution

Returns a single execution by metadata ID.

query {
  operations {
    execution(id: 100) {
      id
      externalId
      name
      trainState
      startTime
      endTime
      failureJunction
      failureReason
    }
  }
}
ParameterTypeRequiredDescription
idLong!YesThe execution's metadata ID

Returns: ExecutionSummary (nullable, returns null if the ID does not exist)


executionDetail

Returns the full detail for a single execution, including the input / output payloads and stackTrace that execution / executions omit to keep list reads lean. Use this for a detail page; use execution for a light single-row lookup.

query {
  operations {
    executionDetail(id: 100) {
      id
      externalId
      name
      trainState
      startTime
      endTime
      failureJunction
      failureReason
      failureException
      stackTrace
      input
      output
      manifestId
      cancellationRequested
      currentlyRunningJunction
      junctionStartedAt
      hostName
      hostEnvironment
      hostInstanceId
      hostLabels
      failureClass
      parentId
      scheduledTime
      executor
    }
  }
}
ParameterTypeRequiredDescription
idLong!YesThe execution's metadata ID

Returns: ExecutionDetail (nullable; null when the ID does not exist).

input and output are the raw JSON payloads as stored (Postgres returns them in jsonb canonical form). There is no separate junction table: junction context is the currentlyRunningJunction (while IN_PROGRESS) and failureJunction (on failure) fields. childCount is the number of sub-executions (metadata rows whose parentId is this execution), for rendering a parent/child tree, and parentId is this execution's own parent. Nothing in Trax sets parentId at present, a train dispatched from a junction included (see Nested Trains), so it is 0 unless something outside Trax writes the column. failureClass is the same FailureClass enum as on ExecutionSummary. scheduledTime is when a scheduled run was due (null for one that was not scheduled), executor is the project name of the process that ran it, and hostLabels is the host's user-supplied labels as a JSON object.

input and output can hold credentials. They are on this single-row read and on no list; see Train inputs and the operations gate.


executionChildren

Paginated child executions of a parent (metadata rows whose parentId matches the given id), newest first. Keyset-paginated on id like the top-level executions list. Backed by the partial index ix_metadata_parent_id, so it stays O(page size) even on the huge metadata table.

query {
  operations {
    executionChildren(parentId: 100, take: 25) {
      items { id name trainState startTime endTime }
      totalCount
      nextCursor
    }
  }
}
ParameterTypeDefaultDescription
parentIdLong!noneThe parent execution's metadata id
takeInt25Page size, from 1 to 500. See Page size
afterIdLongnullKeyset cursor (id < afterId)

Returns: PagedResult<ExecutionSummary> (count is always exact).


Train inputs and the operations gate

A train's input can carry credentials, so the admin surface reads it one row at a time. An execution's input, a manifest's properties and a work queue entry's input are on executionDetail, manifestDetail and workQueue.detail only, never on a list type, and an effect's settings are on effects. All of them sit under the operations namespace, so whatever gates it (RequireAuthorization() or GateOperations(...), see API security) is what decides who reads them. There is no separate field-level gate: a caller who can read one execution's input can read a manifest's properties too. The reasoning is recorded in Trax.Api's ADR api/0005.

A member marked [TraxSensitive] is masked in all three, written as {"_redacted": true}. An execution's input is recorded that way. A manifest's properties and a work queue entry's input are stored with the real value, because a run starts from them, so the read masks them: it reads the stored JSON back as the input type of the train registered on this host and writes it with the mask. When the host has no registered train whose input type the copy names, or the JSON does not read as that type, nothing shows the copy holds no sensitive member, so the whole value reads as {"_redacted": true}.


PagedResult

All paginated queries return the same wrapper type:

FieldTypeDescription
items[T!]!The page of results
totalCountInt!Total number of records matching the query's filters, the same on every page: neither skip nor afterId changes it
skipInt!The skip value that was applied, after a negative value was read as 0
takeInt!The take value that was applied, after clamping to 1 through 500
isEstimatedCountBoolean!true when totalCount is a fast estimate rather than an exact count. See Pagination
nextCursorLongID of the last item in the page. Pass as afterId to fetch the next page via keyset pagination. null when no items are returned

Pagination

Paginated queries support two strategies. Both can be used interchangeably. The dashboard uses offset pagination internally, while API consumers can opt into keyset cursors for better deep-page performance.

Page size

Every paged read in the operations namespace clamps take to 1 through 500 and reads a negative skip as 0, rather than refusing the request. A take above 500 returns 500 rows, and take: 0 or a negative take returns one row. The skip and take on the returned page are the values that were applied, so a client can see that its request was clamped. To read more than 500 rows, page with afterId.

Offset pagination (default)

Pass skip and take as before. This uses SQL OFFSET/LIMIT under the hood. Performance degrades on deep pages (high skip values) because the database must scan and discard rows up to the offset.

query {
  operations {
    executions(skip: 100, take: 25) { items { id } totalCount }
  }
}

Keyset cursor pagination

Pass afterId (the nextCursor from the previous page) instead of skip. This uses WHERE id < @afterId, which is constant-time regardless of how deep you paginate because it seeks directly to the cursor position via the primary key index.

# First page
query {
  operations {
    executions(take: 25) { items { id } totalCount nextCursor }
  }
}
 
# Next page: pass nextCursor as afterId
query {
  operations {
    executions(afterId: 4201, take: 25) { items { id } totalCount nextCursor }
  }
}

When afterId is provided, skip is ignored.

Estimated counts

totalCount is the size of the whole list the filters select, whichever page you are on: the cursor and skip never change it. This is the convention HotChocolate's own connections and GitHub's API follow.

For an unfiltered list of a large table (10,000 rows or more), totalCount is the database's own row estimate instead of an exact COUNT(*), which would scan every row on every page. The provider supplies it through ISqlDialect.EstimateRowCount: on PostgreSQL it is pg_class.reltuples, which ANALYZE, VACUUM and autovacuum keep current and which is typically within a few percent. When the estimate is used, isEstimatedCount is true, on the first page and every later one.

The count is exact, and isEstimatedCount is false, whenever a filter is supplied, when the table is smaller than that, when PostgreSQL has never analyzed the table, and on providers that keep no estimate (SQLite, and the in-memory provider).

Performance at scale

The operations queries are stress-tested against millions of rows (Trax.Api.Tests.Stress, run with dotnet test --filter TestCategory=Stress). At 3,000,000 metadata rows on laptop-class PostgreSQL:

  • Keyset pagination stays flat. A far-end page (an afterId near the end of the id sequence) returns in ~35ms no matter how deep it is, because it seeks through the primary key index rather than counting past skipped rows.
  • Deep offset pagination does not. A skip near the end of the table scans and discards every skipped row: ~430ms at a 3,000,000-row offset, more than 10x slower than the equivalent keyset page.

Build list views on keyset cursors: read the first page with take, then pass each response's nextCursor as the next request's afterId. Reserve skip for shallow, bounded jumps. Filtered reads (status, trainName, metadataId, minimumLevel, category) and their exact counts also stay under ~100ms at the same scale, so filter controls stay responsive.

The same suite times every operations mutation against those tables (each single-row or scoped write, including the manifest and group cancels that filter the metadata table, finishes in under ~50ms), the point reads behind the detail pages, the persisted-operations list, lookups and writes over a 100,000-operation catalog, and subscription fan-out to 1,000 subscribers. onDataChanged coalesces a storm of 200,000 change signals into one event per changed domain per subscriber, delivered to all of them in under half a second. onTrainStateChanged delivers each event to every subscriber when the rate is moderate, but at 1,000 subscribers and a sustained 80 or more state changes a second, a few subscribers miss some events: a live feed can lag behind the grid until its next refetch. requeueAllDeadLetters and acknowledgeAllDeadLetters are timed over every dead letter the seed leaves awaiting intervention, 500,000 of them: acknowledging all takes about 7.5 s and requeueing all, which also writes one work queue entry per manifest, about 36 s, so treat them as rare operator actions. The batch cancels and enable/disable mutations on a full 1,000-id selection, and runTrain, each finish in under 20 ms.

config (nested under operations)

The operations.config namespace returns the live scheduler runtime settings (the dashboard-editable subset of SchedulerConfiguration, LocalWorkerOptions, and MetadataCleanupConfiguration). The dashboard's Server Settings page and this query both read from the same in-memory singleton, so they agree. The page saves through the same operations call as updateScheduler, sending only the fields the operator changed, and reloads from this snapshot afterwards.

Persistence: settings written via operations.config.updateScheduler (or the dashboard) are stored in the singleton-row trax.scheduler_config table, only the ones a save names. Every running scheduler applies them to its in-memory singleton at startup and re-reads the row every few seconds, so they survive restarts and reach every scheduler host. A setting never saved keeps each host's configured value. See config for which fields a save stores.

scheduler

query {
  operations {
    config {
      scheduler {
        manifestManagerEnabled
        jobDispatcherEnabled
        manifestManagerPollingInterval
        jobDispatcherPollingInterval
        maxActiveJobs
        defaultMaxRetries
        failureCountWindow
        defaultRetryDelay
        retryBackoffMultiplier
        maxRetryDelay
        defaultJobTimeout
        stalePendingTimeout
        recoverStuckJobsOnStartup
        deadLetterRetentionPeriod
        autoPurgeDeadLetters
        localWorkerCount
        metadataCleanupInterval
        metadataCleanupRetention
      }
    }
  }
}

Returns: SchedulerConfigSnapshot.

environmentName and logLevels

The API host's environment and its Logging:LogLevel configuration, which the dashboard shows as its environment badge and on its server settings page.

query {
  operations {
    config {
      environmentName
      logLevels { category level }
    }
  }
}

environmentName is IHostEnvironment.EnvironmentName (String!). logLevels is [LogLevelSetting!]!, one { category, level } per key under Logging:LogLevel, Default first and the rest by category; empty when the host configures none. Only that section is read, so no other configuration value (a connection string, a secret) is reachable from here. Both describe the API process, not the scheduler or worker processes.

SchedulerConfigSnapshot fields

FieldTypeDescription
manifestManagerEnabledBoolean!Whether the manifest manager polling service runs
jobDispatcherEnabledBoolean!Whether the job dispatcher polling service runs
manifestManagerPollingIntervalTimeSpan!How often the manifest manager polls
jobDispatcherPollingIntervalTimeSpan!How often the job dispatcher polls
maxActiveJobsIntGlobal concurrency cap. Null means no cap
defaultMaxRetriesInt!Default retry budget for new manifests
failureCountWindowTimeSpan!How far back failed runs count toward retry backoff and MaxRetries
defaultRetryDelayTimeSpan!First-retry delay
retryBackoffMultiplierFloat!Exponential backoff factor
maxRetryDelayTimeSpan!Upper bound on backoff
defaultJobTimeoutTimeSpan!Default per-execution timeout
stalePendingTimeoutTimeSpan!When pending entries are reaped
recoverStuckJobsOnStartupBoolean!Whether stuck-job recovery runs on startup
deadLetterRetentionPeriodTimeSpan!How long resolved dead letters are kept before purging
autoPurgeDeadLettersBoolean!Whether the dead letter cleanup service runs
localWorkerCountIntIn-process worker thread count. Null when the scheduler runs no local worker pool (no database provider, or a replaced job submitter). Local workers are on by default with a database provider; ConfigureLocalWorkers tunes them
metadataCleanupIntervalTimeSpanMetadata cleanup poll interval. Null when cleanup is not configured
metadataCleanupRetentionTimeSpanHow long completed metadata is kept. Null when cleanup is not configured

metrics (nested under operations)

The operations.metrics namespace returns the data behind the dashboard's KPI cards, charts, and server health panel. Every field comes from the shared IOperationsService, so the GraphQL response and the dashboard render exactly the same numbers.

dashboard

query {
  operations {
    metrics {
      dashboard(range: LAST24_HOURS, hideAdminTrains: true) {
        kpis { executionsToday successRate currentlyRunning unresolvedDeadLetters }
        executionsOverTime { timestamp completed failed cancelled }
        topFailures { trainName count }
        topAverageDurations { trainName averageMilliseconds }
        throughputSeries {
          trainName
          buckets { timestamp count }
        }
      }
    }
  }
}
ParameterTypeDefaultDescription
rangeMetricsRangeLAST24_HOURSGranularity of the executions-over-time chart. LAST60_MINUTES returns 60 buckets (1 minute each); LAST24_HOURS returns 24 buckets (1 hour each). The other series are always over the last 7 days
hideAdminTrainsBooleanfalseWhen true, framework admin trains (matching AdminTrains.FullNames) are excluded from every series

Returns: DashboardMetrics.

DashboardMetrics fields

FieldTypeDescription
kpisDashboardKpis!Today's headline counts
executionsOverTime[ExecutionsBucket!]!Per-bucket counts at the requested granularity
topFailures[TrainFailureCount!]!Top 10 trains by failure count over the last 7 days
topAverageDurations[TrainAverageDuration!]!Top 10 trains by average duration over the last 7 days (root-level executions only)
throughputSeries[ThroughputSeries!]!Top-3 trains plus an "Other" series, 28 6-hour buckets covering 7 days. Empty series are dropped

DashboardKpis fields

FieldTypeDescription
executionsTodayInt!Total executions started today (UTC)
successRateFloat!Completed / (Completed + Failed) as a percentage. Zero when no terminal executions exist today
currentlyRunningInt!Executions currently in InProgress
unresolvedDeadLettersInt!Dead letters in AwaitingIntervention

ExecutionsBucket fields

FieldTypeDescription
timestampDateTime!UTC start of the bucket
completedInt!Completed executions in the bucket
failedInt!Failed executions
cancelledInt!Cancelled executions

TrainFailureCount fields

FieldTypeDescription
trainNameString!Train interface FullName
countInt!Failures over the last 7 days

TrainAverageDuration fields

FieldTypeDescription
trainNameString!Train interface FullName
averageMillisecondsFloat!Mean execution time over completed root-level runs in the last 7 days

ThroughputSeries fields

FieldTypeDescription
trainNameString!Train interface FullName, or the literal string "Other" for the aggregated remainder series
buckets[ThroughputBucket!]!28 6-hour buckets, oldest first

ThroughputBucket fields

FieldTypeDescription
timestampDateTime!UTC start of the bucket
countInt!Completed executions in the bucket

dashboard runs several aggregations over the last-24h and last-7-day windows on every call, and those windows hold hundreds of thousands to millions of rows at scale. The metadata table carries two covering indexes for them (ix_metadata_metrics_state_time and ix_metadata_metrics_window) so every aggregation is a heap-free index-only scan. At 3,000,000 metadata rows the whole block returns in ~400-525ms, against ~630-770ms without the covering indexes. It is the heaviest operations read, so poll it on an interval (a few seconds) rather than on every dashboard interaction.

server

Process-level snapshot. CPU% is not on ServerMetrics because it can only be measured as a delta between two samples; use the sibling serverCpuPercent field for that.

query {
  operations {
    metrics {
      server { processStartTimeUtc uptimeSeconds workingSetBytes gcHeapBytes }
    }
  }
}

Returns: ServerMetrics.

FieldTypeDescription
processStartTimeUtcDateTime!When the host process started
uptimeSecondsFloat!Seconds since process start
workingSetBytesLong!Process.WorkingSet64
gcHeapBytesLong!GC.GetTotalMemory(false)

serverCpuPercent

This API process's CPU utilisation since the previous poll, as a percentage of total capacity (normalised by core count, so 100 means every core fully busy). CPU% is a delta measurement, so the sampler holds the previous sample: the first poll after startup returns null while the baseline primes, and each subsequent poll reports usage since the last. Poll it on the same interval as the rest of the dashboard's server panel.

It is a sibling of server rather than a field on ServerMetrics because the sampling state is per-process and stateful, which is exactly why it is not part of the shared IOperationsService snapshot the scheduler and dashboard also read.

query {
  operations {
    metrics {
      serverCpuPercent
    }
  }
}

Returns: Float (nullable; null on the first poll or if no measurable time has elapsed since the last).


logs (nested under operations)

The operations.logs namespace returns paginated reads of trax.log, the framework's per-execution log table. The dashboard's Logs page is backed by this query. Logs are written by the framework, never by API consumers, so there are no log mutations.

query {
  operations {
    logs {
      logs(skip: 0, take: 50, minimumLevel: WARNING) {
        items { id metadataId eventId level category message exception stackTrace }
        totalCount
        isEstimatedCount
        nextCursor
      }
    }
  }
}
ParameterTypeDefaultDescription
skipInt0Number of records to skip (offset pagination). A negative value reads as 0. Ignored when afterId is provided.
takeInt25Number of records to return, from 1 to 500. See Page size
metadataIdLongnullFilter to logs for a single execution
minimumLevelLogLevelnullIncludes the supplied level and anything more severe. LogLevel follows Microsoft.Extensions.Logging: TRACE, DEBUG, INFORMATION, WARNING, ERROR, CRITICAL, NONE
categoryStringnullExact-match filter on the logger category (e.g. Trax.Samples.GameServer.Trains.Combat.ResolveCombatTrain)
afterIdLongnullKeyset cursor. Returns records with id < afterId

Returns: PagedResult<LogEntry>.

When any filter or afterId is supplied, the count is exact (isEstimatedCount: false). Unfiltered first-page reads use the same pg_class.reltuples estimator as the other large-table queries because the log table grows quickly.

LogEntry fields

FieldTypeDescription
idLong!Database ID (monotonic, used as the keyset cursor)
metadataIdLong!The execution this log line belongs to
eventIdInt!EventId from the ILogger call site
levelLogLevel!Severity
categoryString!Logger category, typically the originating type name
messageString!Truncated to 4000 chars at write time
exceptionStringException message if any (truncated to 2000 chars)
stackTraceStringStack trace if any (truncated to 4000 chars)

manifestGroups (nested under operations)

The operations.manifestGroups namespace exposes every read scoped to manifest groups: the paged list, single-group lookup, and the cross-group dependency graph the dashboard renders as a DAG. The list lives here (rather than as a sibling of manifests at the operations root) because both the namespace and a sibling manifestGroups field would camelCase to the same name in the schema, and HotChocolate would silently drop one.

groups

Returns a paginated list of manifest groups, ordered by ID descending. Supports both offset-based and keyset cursor pagination.

query {
  operations {
    manifestGroups {
      groups(skip: 0, take: 10) {
        items {
          id
          name
          maxActiveJobs
          priority
          isEnabled
          createdAt
          updatedAt
        }
        totalCount
        isEstimatedCount
        skip
        take
        nextCursor
      }
    }
  }
}
ParameterTypeDefaultDescription
skipInt0Number of records to skip (offset pagination). A negative value reads as 0
takeInt25Number of records to return, from 1 to 500. See Page size
nameContainsStringnullCase-sensitive substring match on the group name
afterIdLongnullKeyset cursor. Returns records with id < afterId. See Pagination

Returns: PagedResult<ManifestGroupSummary>

ManifestGroupSummary fields

FieldTypeDescription
idLong!Database ID
nameString!Group name
maxActiveJobsIntConcurrency limit for the group (null = unlimited)
priorityInt!Default priority for manifests in this group
isEnabledBoolean!Whether the group is active
createdAtDateTime!When the row was created
updatedAtDateTime!Last patch via updateManifestGroup

group

Single-group lookup by ID. Used by dashboards to pre-populate the group settings form before sending an updateManifestGroup patch.

query {
  operations {
    manifestGroups {
      group(id: 7) {
        id
        name
        maxActiveJobs
        priority
        isEnabled
      }
    }
  }
}
ParameterTypeRequiredDescription
idLong!YesManifest group database ID

Returns: ManifestGroupSummary (nullable; null when the group does not exist).

stats

Execution roll-up for a set of manifest groups in a single round-trip: manifest count, executions by state, and last run per group. Backs the per-group stat columns on the dashboard's manifest groups list, which calls it with just the ids of the visible page, through the same IOperationsService.GetManifestGroupExecutionStatsAsync call. Every requested id gets a row (zeros when the group has no manifests or executions), returned in the order requested so the caller can zip it to its rows. The metadata side is served by ix_metadata_manifest_state and the manifest side by ix_manifest_manifest_group_id.

query {
  operations {
    manifestGroups {
      stats(groupIds: [1, 2, 7]) {
        groupId
        manifestCount
        totalExecutions
        completed
        failed
        inProgress
        lastRun
      }
    }
  }
}
ParameterTypeRequiredDescription
groupIds[Long!]!YesThe group ids to roll up, at most 1000 distinct ones. Duplicates are collapsed; an empty list returns an empty result. More than 1000 distinct ids fails the field with a GraphQL error coded TRAX_TOO_MANY_IDS

Returns: [ManifestGroupStats!]!

ManifestGroupStats fields

FieldTypeDescription
groupIdLong!The group these stats are for
manifestCountLong!Manifests belonging to the group
totalExecutionsLong!Executions across all of the group's manifests
completedLong!Executions in Completed
failedLong!Executions in Failed
inProgressLong!Executions in InProgress
lastRunDateTimeStart time of the group's most recent execution. Null when there are none

graph

Returns the 1-hop cross-group dependency neighborhood for a manifest group: every group containing a manifest the focal group's manifests depend on (upstream), every group containing a manifest depending on the focal group's manifests (downstream), and the focal group itself. Edges are directed parent → dependent.

query {
  operations {
    manifestGroups {
      graph(groupId: 7) {
        nodes { id name isHighlighted }
        edges { fromId toId }
      }
    }
  }
}
ParameterTypeRequiredDescription
groupIdLong!YesDatabase ID of the focal manifest group

Returns: ManifestGroupDependencyGraph (nullable). Returns null only when the group does not exist. Empty groups still return a single-node graph (focal group, no edges) so the UI can render the focal node.

dependencyGraph

The whole cross-group dependency graph in one shot: every manifest group as a node and every cross-group dependency (a manifest in one group depending on a manifest in another) as a directed parent → dependent edge. Nothing is highlighted. Backs the global DAG the dashboard renders on the manifest-groups page; graph(groupId) is the same shape narrowed to one group's neighborhood.

query {
  operations {
    manifestGroups {
      dependencyGraph {
        nodes { id name isHighlighted }
        edges { fromId toId }
      }
    }
  }
}

Returns: ManifestGroupDependencyGraph! (never null; empty when there are no groups). Same-group dependencies are excluded, so a deployment whose groups never depend on each other returns nodes with no edges.

ManifestGroupDependencyGraph fields

FieldTypeDescription
nodes[DependencyGraphNode!]!All groups in the neighborhood plus the focal group
edges[DependencyGraphEdge!]!Cross-group edges only. Same-group dependencies are excluded

DependencyGraphNode fields

FieldTypeDescription
idLong!Manifest group ID
nameString!Group name
isHighlightedBoolean!true for the focal group; the UI uses this to render it differently

DependencyGraphEdge fields

FieldTypeDescription
fromIdLong!Parent group ID (the group whose manifests are depended on)
toIdLong!Dependent group ID

workQueue (nested under operations)

The operations.workQueue namespace exposes paginated reads of the work queue. The work queue is the intermediary between scheduling and dispatch: every queued execution (manifest triggers, dashboard re-runs, dead-letter requeues, GraphQL queueTrain calls) lands here as a Queued row that the JobDispatcher picks up.

query {
  operations {
    workQueue {
      workQueues(skip: 0, take: 25, status: QUEUED) {
        items {
          id
          externalId
          trainName
          status
          createdAt
          dispatchedAt
          scheduledAt
          priority
          dispatchAttempts
          manifestId
          metadataId
          deadLetterId
          inputTypeName
        }
        totalCount
        isEstimatedCount
        nextCursor
      }
    }
  }
}

workQueues

ParameterTypeDefaultDescription
skipInt0Number of records to skip (offset pagination). A negative value reads as 0. Ignored when afterId is provided.
takeInt25Number of records to return, from 1 to 500. See Page size
statusWorkQueueStatusnullFilter by lifecycle state (QUEUED, DISPATCHED, CANCELLED)
trainNameStringnullExact-match filter on the interface FullName (e.g. Trax.Samples.GameServer.Trains.Combat.IResolveCombatTrain)
afterIdLongnullKeyset cursor. Returns records with id < afterId. See Pagination

Returns: PagedResult<WorkQueueSummary>

When any filter or afterId is supplied, the count is exact and isEstimatedCount is false. Unfiltered first-page reads use the same fast estimator as the other large-table queries.

WorkQueueSummary fields

FieldTypeDescription
idLong!Database ID
externalIdString!GUID assigned at creation
trainNameString!Train interface FullName
statusWorkQueueStatus!QUEUED, DISPATCHED, or CANCELLED
createdAtDateTime!When the entry was queued
dispatchedAtDateTimeWhen the dispatcher picked it up (null while queued or if cancelled before dispatch)
scheduledAtDateTimeEarliest dispatch time. Null means dispatch immediately
priorityInt!Dispatch priority 0-31
dispatchAttemptsInt!Number of times dispatch was attempted and failed
manifestIdLongSource manifest ID, if scheduled
metadataIdLongMetadata ID created at dispatch, if dispatched
deadLetterIdLongDead letter that triggered this requeue, if applicable
inputTypeNameStringFully qualified type name of the input, for deserialization
confirmedAtDateTimeWhen the entry became eligible for dispatch. Null while it is still being staged, and stays null on a staged entry that was cancelled (by an operator, or by the stale staged entry sweep); the dispatcher never claims an unconfirmed entry. Also null on an entry dispatched more than a day before the database was migrated to the version that added it, which the migration does not backfill
subjectKeyStringThe subject the entry is serialized against, from the train's QueueSubjectKey. Null when the train does not set one. The value is computed by the consumer's train, so it may carry record identifiers

workQueue (single)

Returns a single entry by database ID.

query {
  operations {
    workQueue {
      workQueue(id: 42) { id status priority dispatchAttempts }
    }
  }
}
ParameterTypeRequiredDescription
idLong!YesThe work queue entry's database ID

Returns: WorkQueueSummary (nullable, returns null if the ID does not exist).

detail

Returns one entry with the train input it was queued with and, for a queued entry with a subject, what it is waiting on. Backs a work queue detail page.

query {
  operations {
    workQueue {
      detail(id: 42) { id status subjectKey input subjectHeldBy subjectQueuedBehind }
    }
  }
}
ParameterTypeRequiredDescription
idLong!YesThe work queue entry's database ID

Returns: WorkQueueDetail (nullable, returns null if the ID does not exist). It carries every WorkQueueSummary field and:

FieldTypeDescription
inputStringThe train input as stored JSON, with each [TraxSensitive] member masked. Can hold credentials; see Train inputs and the operations gate
subjectHeldByLongFor a queued entry with a subject: the dispatched entry for the same subject whose run is still pending or in progress. Dispatch skips the subject until that run finishes
subjectQueuedBehindLongFor a queued entry with a subject that nothing holds: the queued entry for the same subject that dispatch offers first (confirmed, due, in an enabled group, then higher priority, then older). Dispatch offers one entry per subject each cycle

Both are null for an entry that is not queued or has no subject. The Blazor dashboard's work queue detail page reports the same two answers from the same predicate.


deadLetters (nested under operations)

The operations.deadLetters namespace exposes paginated dead-letter reads (deadLetters, deadLetter). See scheduler/dead-letters-and-cleanup for the full surface and examples.