Debug a failed remote run

Start from the run's id, which the dashboard's run page and the operations.executions query show.

Read the run's row. The failure fields are filled from the runner's structured response:
```
SELECT train_state, failure_exception, failure_junction, failure_reason, failure_class
FROM trax.metadata WHERE id = <id>;
```
failure_junction names the junction that threw, and failure_exception its type. The stack trace is not on this row for a /trax/run failure; it is in the runner's log.
Read its log. If the runner registers AddDataContextLogging(), junction-level logs are persisted:
```
SELECT * FROM trax.log WHERE metadata_id = <id> ORDER BY id;
```
Check whether the runner ever started it. failure_exception = 'StalePendingTimeout' means the run stayed Pending until the stale-pending reaper failed it: the runner never began. Check the runner's health, the network path between the two, and whether the runner is deployed.
Check whether the dispatch was requeued. failure_exception = 'DispatchRequeued' means that attempt never reached a runner and the work queue entry was queued again; failure_reason holds the submitter's error. Such a run does not count against the manifest's retries. A run whose failure_junction is DispatchJobsJunction with any other exception is the attempt that used up MaxDispatchAttempts, and counts once.
Check whether the runner refused it. A refused request never reaches a train. Over HTTP the runner answers 401 (a signature or policy mismatch) or 413 (a body over its limit), so the dispatch fails and step 4 applies. A refused Lambda Execute invocation or SQS message throws on the runner after the scheduler has handed it over, so the row stays Pending until step 3's reaper fails it. The runner logs refusals as a warning, at most once a minute with a count. See Runner authorization posture.

The fields of the response are in Remote error responses, and the dispatcher's handling of a failed submit in JobDispatcher: Dispatch failures.

SDK Reference

AddTraxJobRunner | UseRemoteWorkers | StalePendingTimeout / MaxDispatchAttempts