Reference

operations: execution queries

Package Trax.Api.GraphQL·Surface GraphQL schema·Source OperationsQueries.cs

Fields of OperationsQueries that read train executions (metadata rows) and what a run recorded.

executions

Returns a paginated list of train executions (metadata records), ordered by ID descending (newest first). Supports both offset-based and keyset cursor pagination.

query {
  operations {
    executions(skip: 0, take: 10) {
      items {
        id
        externalId
        name
        trainState
        startTime
        endTime
        failureJunction
        failureReason
        manifestId
        cancellationRequested
        parentId
        currentlyRunningJunction
      }
      totalCount
      isEstimatedCount
      skip
      take
      nextCursor
    }
  }
}

When any filter is supplied the count is exact (isEstimatedCount: false), except under failureReasonContains; an unfiltered list may use the database's row estimate, on every page alike (see Estimated counts). With failureReasonContains the count stops at 10,000 matches: past that totalCount is 10000 and isCountCapped is true, so a client shows "10,000+" rather than waiting on a count of every failed run. A term matching 10,000 or fewer is counted exactly. startedAfter/startedBefore use the ix_metadata_start_time_desc index so they stay fast at scale. manifestId and manifestGroupId are served by the covering index ix_metadata_manifest_state, so a manifest's or group's history stays index-only even against millions of rows. failureClass is served by ix_metadata_failure_class on (failure_class, id DESC) (Postgres). It covers every row rather than only classified ones: the class arrives as a query parameter, and a generic plan cannot prove a parameter satisfies a partial index's predicate, so a partial index would go unused. Arbitrary-column sorting is deliberately not offered: it is incompatible with keyset pagination over millions of rows (it forces OFFSET scans or a full sort). Filter to narrow the set instead.

On Postgres failureReasonContains is served by a trigram index on lower(failure_reason), ix_metadata_failure_reason_trgm (migration 066). A first page or a cursor page with a text filter is read in two steps: the 10,000 ids nearest where it starts, in order, which fills the page when the term is common there, and then the rest of the matches through the index, sorted. So a term found only in old runs does not make a newest-first page walk every newer run. At 3,000,000 runs, two in nine of them failed, a page with its count takes under 10 ms for a term that matches nothing, a few dozen runs or two runs in nine, and under 100 ms for a term found in 67,000 old runs only, alone or with hostName. A term shorter than three characters has no trigram to look up and reads the table. SQLite has no trigram index, so there both filters read the table.

Returns: PagedResult<ExecutionSummary>

ExecutionSummary fields

executionDetail

Returns the full detail for a single execution, including the input / output payloads and stackTrace that executions omits to keep list reads lean.

query {
  operations {
    executionDetail(id: 100) {
      id
      externalId
      name
      trainState
      startTime
      endTime
      failureJunction
      failureReason
      failureException
      stackTrace
      input
      output
      manifestId
      cancellationRequested
      currentlyRunningJunction
      junctionStartedAt
      hostName
      hostEnvironment
      hostInstanceId
      hostLabels
      failureClass
      parentId
      scheduledTime
      executor
      replayDecisionsOf
      childCount
      invokedBy { machine instanceId ownerKind rowId }
      workQueueEntryId
      deadLetterId
    }
  }
}

Returns: ExecutionDetail (nullable; null when the ID does not exist).

input and output are the raw JSON payloads as stored (Postgres returns them in jsonb canonical form). There is no separate junction table: junction context is the currentlyRunningJunction (while IN_PROGRESS) and failureJunction (on failure) fields. childCount is the number of sub-executions (metadata rows whose parentId is this execution), for rendering a parent/child tree, and parentId is this execution's own parent. Nothing in Trax sets parentId at present, a train dispatched from a junction included (see Nested Trains), so it is 0 unless something outside Trax writes the column. failureClass is the same FailureClass enum as on ExecutionSummary. scheduledTime is when a scheduled run was due (null for one that was not scheduled), executor is the project name of the process that ran it, and hostLabels is the host's user-supplied labels as a JSON object. replayDecisionsOf (Long) is the execution whose recorded decisions this run was queued to replay, by a requeue or a manifest's retry that replays; it is null when the run replays nothing, a run queued to ask afresh included. The dashboard shows it as Replays Decisions Of. replayAbandoned (Boolean!) is true when the run was queued to replay replayDecisionsOf and asked its deciders afresh instead, because that replay could not be honoured; false for a run that replayed or was never queued to. The decisions themselves are on decisions. resumeFrom (Long) is the failed or cancelled execution this run resumed from a checkpoint, shown on the dashboard as Resumes, and resumeAt (String) the step it resumed at, null when it resumed after the latest checkpoint; both are null for a run that ran from the top.

What the run is linked to, so a client can go from it to what started it:

Each is one indexed read: ix_work_queue_metadata_id, ix_dead_letter_retry_metadata_id, and the instance's unique invoke token. The RunInvoker type is Trax.Api.DTOs.RunInvoker.

input and output can hold credentials. They are on this single-row read and on no list; see Train inputs and the operations gate.

executionChildren

Paginated child executions of a parent (metadata rows whose parentId matches the given id), newest first. Keyset-paginated on id like the top-level executions list. Backed by the partial index ix_metadata_parent_id, so it stays O(page size) even on the huge metadata table.

query {
  operations {
    executionChildren(parentId: 100, take: 25) {
      items { id name trainState startTime endTime }
      totalCount
      nextCursor
    }
  }
}

Returns: PagedResult<ExecutionSummary> (count is always exact).

junctionRuns

The steps of one execution, in the order it reached them, as AddJunctionEvents() recorded them: each junction that ran, each question a routing step asked and the track it took. Empty for an execution with none recorded, and for an id with no execution. Read through JunctionRunQueries.ForRun, the query the dashboard's timeline reads too.

query {
  operations {
    junctionRuns(metadataId: 100) {
      position
      kind
      name
      state
      startedAt
      durationMs
      failureClass
      questionKey
      answer
      confidence
      replayed
      answerWithheld
      nameWithheld
      trackPosition
      attempt
    }
  }
}

Returns: [JunctionStep!]!, the same type the onJunctionEvent subscription carries. A step carries no input, output or failure message, and an answer to a question about a [TraxSensitive] type is never present. Once a routing step's answer is withheld, every later step of the run is withheld too, whatever its kind: a junction, a question or a further route is named (withheld), with nameWithheld true, and a question or route has questionKey, answer and confidence null and answerWithheld true. trackPosition is the position of the latest routing step before a step, of any kind, and null before the first. Trax cannot tell where a track rejoins the chain, so every step after a route counts as on its track. The recorded decider is not kept, so decider is always null here.

A withheld step still records its kind, position, state and timing, and for a failed junction its exception type and failure class. The run's own failureJunction, the train's failed event, and a train started from a junction on the track are recorded as for any run. See Junction Events.

A junction whose end event was dropped stays IN_PROGRESS in these rows after its run has ended, so read a step's state together with the execution's.

It answers to the operations gate, as executionDetail does, so a caller refused one is refused the other. The rows trail the live subscription by moments: a client following a running execution subscribes first, then reads this, and keeps for each position whichever is further along.

nodeId is the id of the declared node the step ran for, as runGraph names it: a junction carries its own node's id, a question the id of the Decide step that asked it, a route the id of its routing step. It is null for a step recorded before node ids were, and wherever the step's name is withheld, because the id names the track the step sits on.

branchPath is the path of the Parallel branch the step ran in, as in Parallel#0/cocitation, or null outside any branch. Branches run side by side, so their steps interleave by position; group them by branchPath to read one branch on its own. It is withheld wherever the step's name is, for the same reason as nodeId: a branch inside a track names the track.

runGraph

One execution drawn on its train's declared chain: each node with the steps the run recorded for it and where it stands, the track each routing step took, each Parallel step's branches, and the steps that match no node. It reads the run's first 500 steps through JunctionRunQueries.ForRun, as junctionRuns does, and places them by nodeId through RunGraphs.Match, which the dashboard's run graph calls too.

query {
  operations {
    runGraph(metadataId: 100) {
      train
      hasGraph
      hash
      chainChanged
      moreSteps
      canResume
      nodes {
        id
        kind
        state
        replayed
        trackTaken
        canResume
        checkpointed
        steps { position state failureClass }
        tracks { name taken nodes { id state } }
      }
      unmatchedSteps { position name nodeId state }
    }
  }
}

Reading a graph at any depth. nodes and each track's nodes nest, so a query that follows them repeats tracks { nodes { ... } } once per level, and the server refuses a query that repeats the same fields past a few levels (HotChocolate's field-cycle limit). A chain can nest Parallel and routing steps deeper than that. allNodes lists every node of the graph at any depth in one list instead: depth first, each node before the nodes on its tracks, and each track's nodes in declared order. Each node carries parentId (the routing or Parallel step whose track it sits on, null at the top level), track (that track's or branch's name) and depth (0 at the top level), so a client rebuilds the tree by grouping the list on parentId and track, in list order. Select only a node's tracks' own fields there (name, taken), not their nodes:

query {
  operations {
    runGraph(metadataId: 100) {
      hasGraph
      allNodes {
        id kind state parentId track depth
        tracks { name taken }
      }
    }
  }
}

Returns: RunGraph, or null for an id with no execution. In C# its nodes are RunGraphNodes, their tracks RunGraphTracks, and a node's state a RunNodeState (all in Trax.Api.DTOs). A node's id names the step and the routing step and track it sits in, as in Switch<Lane>#0/Fast/Ship#0, and is what a recorded step's nodeId refers to. kind is a ChainStepKind (CHAIN, I_CHAIN, SHORT_CIRCUIT, EXTRACT, RESOLVE, SEED, DECIDE, SWITCH, GATE, SCALE, PARALLEL, CHECKPOINT), and opaque is true for a step whose junction is decided only at run time. A PARALLEL step's tracks are its branches, one per Branch in declared order, and a step inside a branch has the branch in its id, as in Parallel#0/cocitation/ScoreCoCitation#0. To read a train's declared chain without a run, use ChainGraph in C#. A node's state is one of:

A CHECKPOINT node records no step of its own: it is COMPLETED once the run stored it, unless a withheld route came before it.

A junction whose end event was dropped keeps the IN_PROGRESS row its start wrote (see junctionRuns). The graph reads each step with the run's own state, so once the execution has completed, failed or been cancelled, such a node is INTERRUPTED rather than IN_PROGRESS: nothing is still running in a run that has ended. While the run runs, it stays IN_PROGRESS.

trackTaken is the route's recorded answer, or the one track holding recorded steps when the answer is missing.

A PARALLEL node runs every branch, so its tracks are its branches, none is SKIPPED for another having run, and each branch's nodes stand as the run recorded them. Every branch is taken once the step has started, and trackTaken is always null. The step records nothing of its own, so its state is read from its branches: FAILED when any branch failed, CANCELLED when one was stopped (a sibling's failure cancels the others, as does cancelling the run), IN_PROGRESS while any branch has a step running or still to reach, COMPLETED once every branch has finished, and NOT_REACHED when no branch has started. A branch has finished only when every node on it has, at any depth along the tracks its routing steps took: a branch that ends in a routing step holds the Parallel step open until the track the route took has run. Once the run has ended, a step that would still be IN_PROGRESS is INTERRUPTED. A route whose answer is withheld inside one branch withholds that branch's later nodes and every node after the join, but not the other branches, which ran on their own. The graph is the chain as the host declares it now, so a step recorded before node ids were, after a withheld route, or for a node the chain no longer declares, is listed in unmatchedSteps rather than dropped. hasGraph is false when the host has no graph for the train; nodes is then empty and every step is unmatched. moreSteps says the run recorded more than 500 steps, so a later node can show as not reached when it ran.

When the chain has changed. A run records the hash of its train's chain when it starts (see Run chain hash), and chainChanged is true when that hash differs from hash, the chain the host declares now. The nodes are still the current chain's, so a run whose node ids match a changed chain is placed on nodes it may not have run as drawn: a step that moved, a track whose contents changed. Old chains are not stored, so the graph cannot be drawn as it was. chainChanged is false when the run recorded no hash (a run from before it was recorded, or from a host that skips the startup chain check) and when the host has no graph for the train.

Where a run can resume. For a failed or cancelled execution, and for one that itself resumed, the read also asks the resume check once, over the run's checkpoints and those of the runs it resumed, rather than once per node. canResume on a node is true when resumeExecution(id, from) can resume the run there, and canResume on the graph when it can resume after the latest checkpoint, with from omitted. Both are false for a run that did not fail or was not cancelled, and for one a state machine's step started. The mutation can still refuse a run the graph offers, for its saved input or a resume already queued, with the reason. checkpointed is true on a node holding a checkpoint the run can resume from. The graph says only that a checkpoint exists and where: what it holds, and the tracks stored with it, are on no operator surface.

It answers to the operations gate, as junctionRuns does.

decisions

One run's recorded decisions, in the order it made them: each question it asked a decider, the answer it acted on (or refused), and the tracks routing steps took on it. These are the rows decision recording writes to trax.decision and a requeue replays. It reads through IOperationsService.GetRecordedDecisionsAsync, the read the dashboard's run page makes.

query {
  operations {
    decisions(metadataId: 1234, take: 50) {
      items {
        id
        questionKey
        occurrence
        kind
        answer
        refused
        isRefused
        model
        decider
        replayed
        replayRefused
        routes
        stateHash
        decidedAt
        answerWithheld
        trackWithheld
      }
      nextCursor
    }
    executionDetail(id: 1234) { replayDecisionsOf replayAbandoned }
  }
}

Returns: DecisionPage! with items: [DecisionRecord!]!, take: Int! (after clamping) and nextCursor: Long (the last decision's id, null on an empty page). A run that recorded none, and an id with no run, return an empty page. At a million recorded decisions, a run with a few reads in about a millisecond and a page of 500 deep into a 2,000-decision run in under 10 ms.

DecisionRecord fields

The stored rows keep every value either way, because a requeue replays from them; only this read leaves them out, by the rule junctionRuns follows. It answers to the operations gate like every field here.

See also

• Queries
• operations: execution mutations
• PagedResult
• ChainGraph