14.5 Kill the worker mid-pack, and resume

Place another order, and kill the worker while it packs, as in chapter 11:

pkill -9 -f Shopfront.Worker

A minute and a half later the hub has failed the run, as before. Start the worker again. As lin, who is staff, find the run and ask where it can resume:

LIN=$(curl -s 'http://localhost:5080/dev/token/lin?staff=true')
curl -s $G -H "Authorization: Bearer $LIN" -H 'Content-Type: application/json' \
  -d '{"query":"{ operations { executions(take: 1, trainName: \"Shopfront.Trains.FulfilOrder.IFulfilOrderTrain\", trainState: FAILED) { items { id trainState failureReason } } } }"}'

{"data":{"operations":{"executions":{"items":[{"id":32,"trainState":"FAILED","failureReason":"Job was stuck InProgress beyond the configured stale in-progress timeout"}]}}}}

RUN=<the id>
curl -s $G -H "Authorization: Bearer $LIN" -H 'Content-Type: application/json' -d '{
  "query": "query($id: Long!) { operations { runGraph(metadataId: $id) { canResume allNodes { id kind state checkpointed } } } }",
  "variables": { "id": '$RUN' } }'

{"data":{"operations":{"runGraph":{"canResume":true,"allNodes":[
  {"id":"Parallel#0","kind":"PARALLEL","state":"COMPLETED","checkpointed":false},
  {"id":"Parallel#0/box/ChooseBoxJunction#0","kind":"CHAIN","state":"COMPLETED","checkpointed":false},
  {"id":"Parallel#0/courier/GetCourierRatesJunction#0","kind":"CHAIN","state":"COMPLETED","checkpointed":false},
  {"id":"BuyLabelJunction#0","kind":"CHAIN","state":"COMPLETED","checkpointed":false},
  {"id":"Checkpoint<ShippingLabel>#0","kind":"CHECKPOINT","state":"COMPLETED","checkpointed":true},
  {"id":"PackOrderJunction#0","kind":"CHAIN","state":"INTERRUPTED","checkpointed":false},
  {"id":"Resolve#0","kind":"RESOLVE","state":"NOT_RECORDED","checkpointed":false}]}}}}

runGraph is the run drawn on its train's chain, each step with where it stands. The branches and the label are done, the checkpoint holds the label, packing was interrupted, and canResume says a resume after the checkpoint is allowed. Resume it:

curl -s $G -H "Authorization: Bearer $LIN" -H 'Content-Type: application/json' -d '{
  "query": "mutation($id: Long!) { operations { resumeExecution(id: $id) { success message id } } }",
  "variables": { "id": '$RUN' } }'

{"data":{"operations":{"resumeExecution":{"success":true,"message":"Work queue entry 5 created.","id":5}}}}

The resume is queued like any run, and the worker picks it up:

info: Shopfront.Trains.FulfilOrder.Junctions.PackOrderJunction[0]
      Packing order b1ce6f68-5f80-4b63-abcb-644536aea37e
info: Shopfront.Trains.FulfilOrder.Junctions.PackOrderJunction[0]
      Packed order b1ce6f68-5f80-4b63-abcb-644536aea37e as TRK-B1CE6F68

It neither chose a box, asked the rates nor bought a label: the run started after the checkpoint, with the label it stored, and packed. The failed run stays failed; the resume is a run of its own, and it completed. In the console, the failed run's page has the same as buttons: Resume, after its latest checkpoint, and Resume from here on a step.

A few things decide whether a run can resume:

It has to be the same chain. The checkpoint stores a hash of the chain and the shape of the state. A deploy that changed either refuses the resume, with the reason, and staff requeue it instead, from the top.
Steps after the checkpoint run again on every resume, as on any retry. PackOrderJunction packs again; a real warehouse that had half-packed a box would check for it.
The work before the checkpoint must have committed. A checkpoint refuses to be stored while the run's Trax data context holds unsaved changes. A junction that writes, writes and saves before the checkpoint, so a resumed run never skips work that never happened.
A retry resumes too. A manifest run that fails, the restock say, resumes after its latest checkpoint on its next retry, and so does a dead letter that is requeued. requeueExecution never does: it is the way to run something again from the top.

SDK Reference

Parallel | Checkpoint | Checkpoints and branches | resumeExecution | runGraph

Next: 14.6 Test it