The Cron Job That Worked Fine Until Daylight Saving Time
Had a cron job scheduled at 2:30 AM that ran a data sync between two databases. Worked perfectly for eight months straight. Then DST hit, and 2:30 AM didn't exist that night.
The job just... didn't run. No error, no log entry, nothing. Cron skipped the nonexistent time slot and moved on with its life.
Why this is sneaky
Most cron implementations handle DST differently. Some skip jobs when clocks spring forward, others run them twice when clocks fall back. The behavior depends on your OS, your cron daemon version, and sometimes the phase of the moon (kidding, but barely).
If you're running anything important at 2 AM — and let's be honest, that's when most maintenance jobs run — you're exposed to this twice a year.
The quick fix
Move critical jobs away from the 2-3 AM window. 4 AM works. 1 AM works. Just avoid the DST danger zone entirely.
But that only solves one problem. The bigger issue is that you probably have no way to know when a cron job silently doesn't run. Your monitoring checks if the server is up, if the app responds, if the database is reachable. It doesn't check whether that data sync job actually fired at 2:30 AM.
What we do now
Every scheduled job we care about pings a monitoring endpoint when it completes. If the ping doesn't arrive within the expected window, we get an alert. Dead simple — add one curl command to the end of your script and you're covered.
The DST job? We caught it immediately the next year because the monitoring alert fired at 3:01 AM telling us the 2:30 job hadn't reported in. Fixed it before anyone noticed. That felt good.
Jobs that bite you
It's never the jobs you think about. The critical payment processing job has monitoring, logging, alerts, and a runbook. It's the little stuff that gets you:
- Log rotation scripts
- Temp file cleanup
- Certificate renewal checks
- Database statistics refresh
- Cache warmup after deploys
If it runs on a schedule and you'd eventually notice it stopped — put a heartbeat on it. Takes 30 seconds per job. Saves you a mystery debugging session at the worst possible time.