AddMetadataCleanup

Enables automatic purging of old metadata records for high-frequency system trains. The internal scheduler trains (JobDispatcher, ManifestManager, MetadataCleanup, DeadLetterCleanup, JobRunner) are always cleaned up while this is enabled. Additional consumer train types can be added via the configure callback.

Signature

public SchedulerConfigurationBuilder AddMetadataCleanup(
    Action<MetadataCleanupConfiguration>? configure = null
)

Parameters

Returns

SchedulerConfigurationBuilder, for continued fluent chaining.

MetadataCleanupConfiguration

Methods

The two-argument overloads throw:

• ArgumentOutOfRangeException when retention is zero or negative.
• InvalidOperationException when the train is an internal scheduler train, which is always swept at RetentionPeriod.
• InvalidOperationException when the same name was already added with a different retention. The same retention twice is accepted and is not counted twice.

RetentionPeriod and every per-train retention must be between one second and ten years, CleanupInterval between one second and 30 days, and DeleteBatchSize at least 1 when set. AddScheduler checks them when the scheduler is built and fails with an InvalidOperationException naming each value out of range (see Value Ranges). To keep a train's metadata as long as possible, give it ten years.

A train declared once under its interface name and again under its class name with different retentions cannot be caught here, because those are unrelated strings until the train registry relates them. The host refuses to start instead.

Examples

Default Configuration

.AddScheduler(scheduler => scheduler
    .AddMetadataCleanup()  // Cleans all internal scheduler train metadata
)

Custom Configuration

.AddScheduler(scheduler => scheduler
    .AddMetadataCleanup(cleanup =>
    {
        cleanup.RetentionPeriod = TimeSpan.FromHours(2);
        cleanup.CleanupInterval = TimeSpan.FromMinutes(5);
        cleanup.AddTrainType<MyHighFrequencyTrain>();
        cleanup.AddTrainType("MyNamespace.AnotherTrain");
    })
)

A Retention Per Train

.AddScheduler(scheduler => scheduler
    .AddMetadataCleanup(cleanup =>
    {
        cleanup.RetentionPeriod = TimeSpan.FromMinutes(30);
        cleanup.AddTrainType<IDeltaImportAllTrain>();                      // on the default
        cleanup.AddTrainType<IPatchCustomerTrain>(TimeSpan.FromDays(30));  // its own
    })
)

Trains sharing a cutoff are swept together, so the batched delete runs once per distinct retention. Use this when one train's metadata is worth keeping for a different reason than the rest, for example because its input is replayable and the others are noise. Pair it with ShouldSaveInputs to decide whose input is stored in the first place.

Remarks

• Only metadata in a terminal state (Completed, Failed, or Cancelled) older than RetentionPeriod is deleted. Pending and InProgress metadata is never cleaned up.
• A run is kept, whatever its age, while a Queued work queue entry or any metadata row names it in replay_decisions_of, so a re-queue that replays its decisions can still read them. When the linking run has expired too, it is deleted first and the run it links to goes in a later batch or sweep.
• The cleanup service runs as an IHostedService on the configured CleanupInterval.
• The internal scheduler trains (JobDispatcher, ManifestManager, MetadataCleanup, DeadLetterCleanup, JobRunner) are always pruned while cleanup is enabled. You don't need to add them manually, and a consumer can never accidentally leave one out.
• A cleanup batch that hits an unexpected foreign-key reference or other error is bisected to isolate the offending row, which is logged and skipped, so one bad row can't abort the whole sweep.
• Train type names are matched against the name column in the metadata table (which stores the interface FullName, the canonical train name).
• The persisted runtime override replaces RetentionPeriod only. Editing the retention from the dashboard's Server Settings page, or through the updateScheduler mutation, moves the cutoff for every train on the default and leaves a retention passed to AddTrainType untouched. A per-train value is normally set because of what the row holds rather than as an operational preference, so shortening one is a code change. See scheduler ADR 0003.