Reference

UseSqsWorkers

Package Trax.Scheduler.SqsNamespaces Trax.Scheduler.Sqs.Extensions, Trax.Scheduler.Sqs.Configuration, Trax.Scheduler.Sqs.LambdaSource SqsSchedulerExtensions.cs, SqsWorkerOptions.cs, SqsJobRunnerHandler.cs, SqsJobSubmitter.cs

Routes chosen trains' queued runs to an Amazon SQS queue. A consumer, normally an AWS Lambda function using SqsJobRunnerHandler, takes the messages and runs the jobs. Trains not routed here keep running on the local workers.

Signatures

public static SchedulerConfigurationBuilder UseSqsWorkers(
    this SchedulerConfigurationBuilder builder,
    Action<SqsWorkerOptions> configure,
    Action<SubmitterRouting>? routing = null
)
 
public class SqsJobRunnerHandler
{
    public SqsJobRunnerHandler(IServiceProvider serviceProvider);
    public Task<SQSBatchResponse> HandleBatchAsync(SQSEvent sqsEvent, CancellationToken cancellationToken = default);
    public Task HandleAsync(SQSEvent sqsEvent, CancellationToken cancellationToken = default);
}

Parameters

ParameterTypeDescription
configureAction<SqsWorkerOptions>Sets the queue URL and client options
routingAction<SubmitterRouting>?Names the trains sent to this queue with ForTrain<TTrain>(), which returns the routing for chaining. When omitted, no train is routed here by name; [TraxRemote] trains are, if this is the first UseRemoteWorkers, UseSqsWorkers or UseLambdaWorkers call. With none of the three, a [TraxRemote] train fails the scheduler's build

SqsWorkerOptions

PropertyTypeDefaultDescription
QueueUrlstringrequiredThe queue URL, for example https://sqs.us-east-1.amazonaws.com/123456789/trax-jobs
ConfigureSqsClientAction<AmazonSQSConfig>?nullConfigures the SQS client: region, a LocalStack ServiceURL, and so on
MessageGroupIdstring?nullFor a FIFO queue, a fixed message group id. Null gives each message its own group (no ordering). Ignored for a standard queue
SigningKeybyte[]?nullThe key shared with the consumer's AddTraxJobRunner(runner => runner.SigningKey = ...), at least 32 bytes. When set, each message carries a Trax-Signature message attribute over its body. See Authorization Posture

SqsJobRunnerHandler

MemberReturnsOn a record that must be delivered again
SqsJobRunnerHandler(IServiceProvider)Takes the consumer's built service provider, which has AddTraxJobRunner registered
HandleBatchAsync(SQSEvent, CancellationToken)SQSBatchResponseReports that record only, as a batch item failure
HandleAsync(SQSEvent, CancellationToken)TaskRuns every record, then throws the first such record's exception, so Lambda retries the whole batch

Returns

UseSqsWorkers returns the SchedulerConfigurationBuilder, for chaining.

Behavior

Sending

A routed entry is serialized as a RemoteJobRequest (the metadata id and the input, if any) and sent as one message to QueueUrl; on a FIFO queue with MessageGroupId and a MessageDeduplicationId. The job id is "sqs-{messageId}". A message is at most 256 KB, so a send with a larger input fails. SQS returns nothing to the sender, so a run that needs its output uses UseRemoteRun or UseLambdaRun alongside.

A train can be routed to one submitter only; routing it to two throws InvalidOperationException when the scheduler is built. UseSqsWorkers() may be called once per queue, each call with its own options and client, sending only the trains it routes.

Consuming

The handler:

  1. Refuses the batch when the runner options have no posture (a SigningKey, or AllowUnsignedRequests() for a queue only the scheduler can write to). A message's age and repeats are not checked, because SQS redelivers by design; the job's Pending metadata row stops a second run.
  2. Runs every record, each in its own DI scope: verifies its Trax-Signature attribute, reads it as a RemoteJobRequest (refusing a body that repeats a property), and calls ITraxRequestHandler.ExecuteJobAsync.
  3. Reports as failures only the records that did not reach their train: a refused signature or body, or a failure before the run started (its row still Pending). SQS redelivers those, and the queue's dead-letter policy applies.
  4. Acknowledges every other record: a train that ran, whether it succeeded or failed (a failure is recorded on the run's row and the manifest's retries act on it), and a redelivery of a run another delivery already started or finished.

Lambda reads the SQSBatchResponse only when the event source mapping's FunctionResponseTypes includes ReportBatchItemFailures; without it the whole batch counts as succeeded.

A cancel reaches the consumer through the database: it sets the run's cancel flag, which the train sees at its next junction boundary when the consumer registers CancellationCheckProvider (through AddJunctionProgress()).

Registered services

ServiceLifetimeDescription
IAmazonSQS (keyed)SingletonThe call's own client, configured by its ConfigureSqsClient
SQS job submitterCreated per dispatchAn internal IJobSubmitter the JobDispatcher creates for each train routed to this call
SqsWorkerOptions, IAmazonSQSSingletonThe first call's options and client, also registered by type

The default IJobSubmitter is not replaced: local workers keep running the trains not routed here.

Examples

using Trax.Scheduler.Sqs.Extensions;
 
services.AddTrax(trax => trax
    .AddEffects(effects => effects.UsePostgres(connectionString))
    .AddMediator(assembly)
    .AddScheduler(scheduler => scheduler
        .UseSqsWorkers(
            sqs =>
            {
                sqs.QueueUrl = "https://sqs.us-east-1.amazonaws.com/123456789/trax-jobs.fifo";
                sqs.MessageGroupId = "trax-jobs";
            },
            routing => routing.ForTrain<IBatchProcessTrain>())
        .Schedule<ISyncTrain>("sync", new SyncInput(), Every.Minutes(5))));

A Lambda consumer:

using Amazon.Lambda.Core;
using Amazon.Lambda.SQSEvents;
using Trax.Scheduler.Sqs.Lambda;
 
public class Function
{
    private static readonly IServiceProvider Services = BuildServiceProvider();
    private readonly SqsJobRunnerHandler _handler = new(Services);
 
    public async Task<SQSBatchResponse> FunctionHandler(SQSEvent sqsEvent, ILambdaContext context)
    {
        // Stop a few seconds before the function's timeout, so runs can record their outcome.
        using var timeout = new CancellationTokenSource(context.RemainingTime - TimeSpan.FromSeconds(5));
        return await _handler.HandleBatchAsync(sqsEvent, timeout.Token);
    }
 
    private static IServiceProvider BuildServiceProvider()
    {
        var services = new ServiceCollection();
        services.AddTrax(trax => trax
            .AddEffects(effects => effects.UsePostgres(Environment.GetEnvironmentVariable("TRAX_DB")!))
            .AddMediator(typeof(Function).Assembly));
        var signingKey = Convert.FromBase64String(
            Environment.GetEnvironmentVariable("TRAX_RUNNER_SIGNING_KEY")!);
        services.AddTraxJobRunner(runner => runner.SigningKey = signingKey);
        return services.BuildServiceProvider();
    }
}

See also