{"id":118,"date":"2025-04-05T12:11:27","date_gmt":"2025-04-05T10:11:27","guid":{"rendered":"https:\/\/www.insync.co.za\/blog\/?p=118"},"modified":"2025-04-05T12:17:46","modified_gmt":"2025-04-05T10:17:46","slug":"your-net-background-services-are-lying-to-you-the-hidden-flaws-in-scheduled-tasks-nobody-talks-about","status":"publish","type":"post","link":"https:\/\/www.insync.co.za\/blog\/2025\/04\/05\/your-net-background-services-are-lying-to-you-the-hidden-flaws-in-scheduled-tasks-nobody-talks-about\/","title":{"rendered":"Your .NET Background Services Are Lying to You: The Hidden Flaws in Scheduled Tasks Nobody Talks About"},"content":{"rendered":"\n

Introduction: Scheduled Tasks Aren\u2019t What You Think<\/h1>\n\n\n\n

If you’re using .NET background services to schedule jobs\u2014maybe with IHostedService, Timer, or even Hangfire\u2014you probably feel like you’ve got it under control. You\u2019ve got CRON expressions set up. Your billing job runs every day at 2 AM. Logs look good. Life is good.<\/p>\n\n\n\n

Until your server reboots at 1:58 AM. Or you deploy a new version of your app at 1:59 AM. Or your cloud instance silently pauses and resumes.<\/p>\n\n\n\n

And then\u2026 nothing happens.<\/strong> Or worse\u2014it happens twice.<\/strong><\/p>\n\n\n\n

1. The Lie: \u201cIt Runs at 2 AM\u201d<\/h2>\n\n\n\n

Let\u2019s start with the illusion: you wrote this Hangfire schedule:<\/p>\n\n\n\n

RecurringJob.AddOrUpdate(() => RunBillingJob(), Cron.Daily);<\/code><\/pre>\n\n\n\n
This looks deterministic, right? Every day, one run. But here’s the problem:<\/p>\n\n\n\n
    \n
  • CRON relies on the app being alive at the right time<\/li>\n\n\n\n
  • There\u2019s no exact-once guarantee<\/li>\n\n\n\n
  • Time zones and DST are rarely handled right<\/li>\n\n\n\n
  • A slow or blocked queue can delay or batch jobs<\/li>\n\n\n\n
  • Deployments and container restarts can skip executions<\/li>\n<\/ul>\n\n\n\n
    TL;DR: Your job might not run, might run late, or might run twice.<\/strong><\/p>\n\n\n\n
    2. Lifecycle Pitfalls in BackgroundService and IHostedService<\/h2>\n\n\n\n
    Let\u2019s zoom in on .NET\u2019s native background services:<\/p>\n\n\n\n
    public class MyJob : BackgroundService\n{\n    protected override async Task ExecuteAsync(CancellationToken stoppingToken)\n    {\n        while (!stoppingToken.IsCancellationRequested)\n        {\n            await DoWork();\n            await Task.Delay(TimeSpan.FromHours(24), stoppingToken);\n        }\n    }\n}<\/code><\/pre>\n\n\n\n
    On paper, it looks good. But:<\/p>\n\n\n\n
      \n
    • What if the server clock jumps?<\/li>\n\n\n\n
    • What if the app restarts mid-delay?<\/li>\n\n\n\n
    • What if DoWork() throws?<\/li>\n<\/ul>\n\n\n\n
      Pro tip:<\/strong> Task.Delay isn\u2019t scheduling\u2014it’s just sleeping. You’re not scheduling anything; you’re delaying execution and hoping the app survives.<\/p>\n\n\n\n
      3. Stop Scheduling\u2014Start Validating<\/h2>\n\n\n\n
      The modern fix? Decouple execution from timing<\/strong>.<\/p>\n\n\n\n
      Instead of trying to run a job at<\/em> 2 AM, run a validation job every 15 minutes that checks what should have run.<\/p>\n\n\n\n
      Example:<\/p>\n\n\n\n
      var dueJobs = _db.Jobs\n    .Where(j => j.ScheduledTime <= _clock.UtcNow && !j.HasRun);\n\nforeach (var job in dueJobs)\n{\n    await RunJob(job);\n}<\/code><\/pre>\n\n\n\n
      This way:<\/p>\n\n\n\n
        \n
      • You can retry missed jobs<\/li>\n\n\n\n
      • You control exactly-when logic in code<\/li>\n\n\n\n
      • You\u2019re no longer hostage to CRON, uptime, or server clocks<\/li>\n<\/ul>\n\n\n\n
        4. The Hard Stuff: Clustering and Concurrency<\/h2>\n\n\n\n
        If your app runs in multiple instances (e.g. in Kubernetes, Azure App Service, etc.), you need to ensure:<\/p>\n\n\n\n
          \n
        • The job runs only once<\/li>\n\n\n\n
        • Two instances don\u2019t race to execute the same thing<\/li>\n<\/ul>\n\n\n\n
          Use distributed locks:<\/p>\n\n\n\n
            \n
          • Redis-based locks (RedLock)<\/li>\n\n\n\n
          • SQL Server advisory locks<\/li>\n\n\n\n
          • PostgreSQL pg_advisory_lock()<\/li>\n<\/ul>\n\n\n\n
            using var lockHandle = await _lockProvider.AcquireAsync(\"billing-job\");\n\nif (lockHandle != null)\n{\n    await RunBillingJob();\n}<\/code><\/pre>\n\n\n\n
            Bonus: Add a fallback if lock acquisition fails\u2014log, alert, or retry.<\/p>\n\n\n\n
            5. Observability is Not Optional<\/h2>\n\n\n\n
            If your scheduled task fails silently, you\u2019ve already lost.<\/p>\n\n\n\n
            You need:<\/strong><\/p>\n\n\n\n
              \n
            • Structured logging (Serilog, Seq)<\/li>\n\n\n\n
            • Telemetry (OpenTelemetry, App Insights, Grafana)<\/li>\n\n\n\n
            • Dashboards to show job state (last run, status, next run)<\/li>\n<\/ul>\n\n\n\n
              Even better? Expose a health check endpoint<\/strong> that verifies your jobs are running as expected.<\/p>\n\n\n\n
              Conclusion: Scheduled Tasks Are a Lie You Can\u2019t Afford<\/h3>\n\n\n\n
              The lesson here isn\u2019t to avoid background jobs\u2014it\u2019s to design them like real, fault-tolerant systems:<\/p>\n\n\n\n
                \n
              • Make them resilient to restarts<\/li>\n\n\n\n
              • Audit what should run, not just what did<\/li>\n\n\n\n
              • Handle clocks and concurrency<\/li>\n\n\n\n
              • Don\u2019t blindly trust your scheduler<\/li>\n<\/ul>\n\n\n\n
                Modern apps need reliability by design<\/strong>, not by luck.<\/p>\n","protected":false},"excerpt":{"rendered":"
                Introduction: Scheduled Tasks Aren\u2019t What You Think If you’re using .NET background services to schedule jobs\u2014maybe with IHostedService, Timer, or even Hangfire\u2014you probably feel like you’ve got it under control. You\u2019ve got CRON expressions set up. Your billing job runs every day at 2 AM. Logs look good. Life is good. Until your server reboots […]<\/p>\n","protected":false},"author":1,"featured_media":121,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"pagelayer_contact_templates":[],"_pagelayer_content":"","footnotes":""},"categories":[18],"tags":[20,19,28],"class_list":["post-118","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-software-development","tag-coding","tag-software","tag-software-development"],"_links":{"self":[{"href":"https:\/\/www.insync.co.za\/blog\/wp-json\/wp\/v2\/posts\/118","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.insync.co.za\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.insync.co.za\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.insync.co.za\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.insync.co.za\/blog\/wp-json\/wp\/v2\/comments?post=118"}],"version-history":[{"count":2,"href":"https:\/\/www.insync.co.za\/blog\/wp-json\/wp\/v2\/posts\/118\/revisions"}],"predecessor-version":[{"id":124,"href":"https:\/\/www.insync.co.za\/blog\/wp-json\/wp\/v2\/posts\/118\/revisions\/124"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.insync.co.za\/blog\/wp-json\/wp\/v2\/media\/121"}],"wp:attachment":[{"href":"https:\/\/www.insync.co.za\/blog\/wp-json\/wp\/v2\/media?parent=118"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.insync.co.za\/blog\/wp-json\/wp\/v2\/categories?post=118"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.insync.co.za\/blog\/wp-json\/wp\/v2\/tags?post=118"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}