Introduction: Scheduled Tasks Aren’t What You Think

If you’re using .NET background services to schedule jobs—maybe with IHostedService, Timer, or even Hangfire—you probably feel like you’ve got it under control. You’ve got CRON expressions set up. Your billing job runs every day at 2 AM. Logs look good. Life is good.

Until your server reboots at 1:58 AM. Or you deploy a new version of your app at 1:59 AM. Or your cloud instance silently pauses and resumes.

And then… nothing happens. Or worse—it happens twice.

1. The Lie: “It Runs at 2 AM”

Let’s start with the illusion: you wrote this Hangfire schedule:

RecurringJob.AddOrUpdate(() => RunBillingJob(), Cron.Daily);

This looks deterministic, right? Every day, one run. But here’s the problem:

• CRON relies on the app being alive at the right time
• There’s no exact-once guarantee
• Time zones and DST are rarely handled right
• A slow or blocked queue can delay or batch jobs
• Deployments and container restarts can skip executions

TL;DR: Your job might not run, might run late, or might run twice.

2. Lifecycle Pitfalls in BackgroundService and IHostedService

Let’s zoom in on .NET’s native background services:

public class MyJob : BackgroundService
{
    protected override async Task ExecuteAsync(CancellationToken stoppingToken)
    {
        while (!stoppingToken.IsCancellationRequested)
        {
            await DoWork();
            await Task.Delay(TimeSpan.FromHours(24), stoppingToken);
        }
    }
}

On paper, it looks good. But:

• What if the server clock jumps?
• What if the app restarts mid-delay?
• What if DoWork() throws?

Pro tip: Task.Delay isn’t scheduling—it’s just sleeping. You’re not scheduling anything; you’re delaying execution and hoping the app survives.

3. Stop Scheduling—Start Validating

The modern fix? Decouple execution from timing.

Instead of trying to run a job at 2 AM, run a validation job every 15 minutes that checks what should have run.

Example:

var dueJobs = _db.Jobs
    .Where(j => j.ScheduledTime <= _clock.UtcNow && !j.HasRun);

foreach (var job in dueJobs)
{
    await RunJob(job);
}

This way:

• You can retry missed jobs
• You control exactly-when logic in code
• You’re no longer hostage to CRON, uptime, or server clocks

4. The Hard Stuff: Clustering and Concurrency

If your app runs in multiple instances (e.g. in Kubernetes, Azure App Service, etc.), you need to ensure:

• The job runs only once
• Two instances don’t race to execute the same thing

Use distributed locks:

• Redis-based locks (RedLock)
• SQL Server advisory locks
• PostgreSQL pg_advisory_lock()

using var lockHandle = await _lockProvider.AcquireAsync("billing-job");

if (lockHandle != null)
{
    await RunBillingJob();
}

Bonus: Add a fallback if lock acquisition fails—log, alert, or retry.

5. Observability is Not Optional

If your scheduled task fails silently, you’ve already lost.

You need:

• Structured logging (Serilog, Seq)
• Telemetry (OpenTelemetry, App Insights, Grafana)
• Dashboards to show job state (last run, status, next run)

Even better? Expose a health check endpoint that verifies your jobs are running as expected.

Conclusion: Scheduled Tasks Are a Lie You Can’t Afford

The lesson here isn’t to avoid background jobs—it’s to design them like real, fault-tolerant systems:

• Make them resilient to restarts
• Audit what should run, not just what did
• Handle clocks and concurrency
• Don’t blindly trust your scheduler

Modern apps need reliability by design, not by luck.