Design Problems/Job Scheduler (like Cron)

Job Scheduler (like Cron)

Medium
QueueDistributed SystemsReliability

High-Level Architecture

Rendering diagram...

Design a distributed job scheduling system that allows users to schedule one-time or recurring tasks. The system must guarantee exactly-once execution, handle failures gracefully, and scale to millions of scheduled jobs. Think of it as a distributed, fault-tolerant cron system.

Clarifying Questions

  • How many jobs? (10M scheduled jobs, 1M executions/hour at peak)
  • Job types? (HTTP callbacks, function invocations, message publishing)
  • Timing precision? (Within 5 seconds of scheduled time)
  • Retry policy? (Configurable retries with exponential backoff)
  • Do we need exactly-once execution? (Yes, at-least-once with deduplication)