Reference

operations: bulk operations

Package Trax.Api.GraphQL·Surface GraphQL schema·Source OperationsMutations.BulkOperations.cs, OperationsQueries.BulkOperations.cs, BulkOperation.cs, BulkOperationFilter.cs

A bulk operation applies one action to every run, dead letter or manifest a list read's filter matches, however many that is. The id-list mutations (cancelExecutions, requeueDeadLetters, acknowledgeDeadLetters, triggerManifests, setManifestsEnabled) take at most 1,000 ids and answer in the request; a bulk operation takes the filter instead, runs in the background on the node that received it, and records what it did. Both sit under operations, behind the host's operations gate. The design is Trax.Api/docs/adr/0045-a-bulk-operation-applies-to-a-confirmed-snapshot-of-a-filter.md.

startBulkOperation

OperationsMutations.StartBulkOperation.

mutation {
  operations {
    startBulkOperation(
      kind: CANCEL
      filter: { executions: { trainName: "MyApp.IImportTrain", trainState: IN_PROGRESS } }
      expectedCount: 312
    ) {
      success message matched
      operation { id status total }
    }
  }
}

The server reads the matching ids once, now, in pages of 10,000. When there are not exactly expectedCount of them it starts nothing: success is false and matched is the number there are now, to show the operator and confirm again. A list's totalCount can be an estimate (unfiltered runs or manifests) or capped at 10,000 (runs filtered by failure text); the refusal's matched is always exact. Otherwise the operation starts and is returned at once with status: RUNNING.

Each BulkOperationKind applies to one list, and a filter for another is refused:

Requeueing runs (rather than dead letters) is not a bulk kind: requeueExecution checks the caller's per-train authorization, and a background batch has no caller.

A batch is at most 500 ids, in its own short transaction, so the console's reads are served between batches and an operation that stops part-way leaves its finished batches applied. A requeue batches dead letters by manifest, because the shared requeue folds one manifest's dead letters into a single queued run only within one call; a manifest with more than 500 awaiting gets a batch of up to 1,000. After each batch the server reads its rows back: an id is done when it is in the state the operation asks for (cancellation requested, requeued, acknowledged, triggered, enabled, disabled), and failed otherwise, with the reason its row gives ("It had already ended (completed).", "It no longer exists.").

One bulk operation runs per node at a time. While one runs, startBulkOperation returns success: false with that one as operation.

The caller is recorded as actor. When the host audits the API (AddAudit), the request that started the operation is audited as usual, and its outcome is written as a second entry with operationName: "bulkOperation", the caller's principal, and the operation's id, kind, status and counts in its metadata. The entry never carries the filter or the note.

StartBulkOperationResponse

BulkOperationFilterInput

BulkOperationFilter as input; give exactly one of executions, deadLetters, manifests. An omitted field filters nothing, as on the list.

bulkOperations and bulkOperation

OperationsQueries.GetBulkOperations and OperationsQueries.GetBulkOperation.

{
  operations {
    bulkOperation(id: "0f8e2c1a-6d1b-4c3e-9a57-1b2d3c4e5f60") {
      status total done failed message
      failureReasons { reason count }
      failures(take: 100) { id reason }
    }
  }
}

bulkOperations lists every operation this node knows, newest first; bulkOperation(id: UUID!) reads one, or null. A node knows the one running and those that finished in the last 24 hours, at most 100. The state is in the node's memory, as the requeue-all job's is: another node does not know the operation, and a restart forgets it. A forgotten operation is started again over what is left, since the ids it finished no longer match a filter such as "in progress" or "awaiting intervention".

BulkOperation