The backup that missed a Sunday in March

A school ran a Python backup every night. The programme checked the clock every minute and started a backup at 02:30.

It worked for months. Then, on a Monday in late March, Sunday’s backup was missing. There were no errors, and the server was on.

In October the reverse happened. One Sunday, two backups appeared, both marked 02:30.

Why did it skip the backup in March and run it twice in October?

How to solve the cases
A clock showing half past two next to a server and calendar pages for March and October

Hints

Reveal them one at a time, only when you need them.

Solution

The clocks changed. In March they jumped from 02:00 to 03:00, so 02:30 never occurred. In October they went back from 03:00 to 02:00, so 02:30 occurred twice and both times triggered the backup.

What happened?

In mainland Spain, summer time begins on the last Sunday in March at 02:00 and ends on the last Sunday in October at 03:00. The change is one hour earlier in the Canary Islands. The server was set to Europe/Madrid, so its clock changed automatically.

The programme compared the local clock with a fixed string. It did not measure elapsed time.

  1. Every minute it read local time with datetime.now() and checked for 02:30.
  2. In March the clock jumped from 01:59 to 03:00. The condition never matched, so there was no backup or error.
  3. In October, after 02:59 it returned to 02:00. The condition matched twice, an hour apart.
  4. The log also used local time, so both October backups showed 02:30.

Python’s PEP 495 calls a missing local time a gap and a repeated one a fold. Naive timestamps from datetime.now() cannot distinguish those two October times.

Can this still happen? Yes. Some schedulers handle clock changes differently. Cronie, for example, runs certain skipped fixed-time jobs just after the spring jump and avoids running them twice in autumn. Store and compare instants in UTC (Coordinated Universal Time). If a job must follow local time, define what it should do when that time is missing or repeated.

Technical explanation

The programme loop

A simplified version of the programme:

import time
from datetime import datetime

while True:
    if datetime.now().strftime("%H:%M") == "02:30":
        run_backup()
    time.sleep(60 - datetime.now().second)  # wait until the next minute

The condition requires the clock to show exactly 02:30. That happens zero times in March and twice in October.

How could it be fixed and prevented?

To fix it, run a backup promptly and check whether any data from the gap was left unprotected. Verify both October backups before deleting the spare.

To prevent it, combine these approaches:

  • Schedule outside the transition. In mainland Spain, times from 02:00 up to 03:00 can be skipped or repeated. 04:00 occurs once each day.
  • Use UTC, which has no daylight saving time. Python offers datetime.now(timezone.utc), and systemd timers support UTC calendars such as OnCalendar=--* 01:30:00 UTC.
  • Avoid relying on an exact match. Store the last successful backup date and run once the scheduled time has passed if the day’s backup is still missing.
  • Monitor the result. An alert after more than 24 hours without a successful backup would have caught the problem on Sunday.

Further reading

Spain sets the dates and times in its Spain’s Royal Decree 236/2002 on daylight saving time — BOE.

Python’s handling of gaps and folds is described in PEP 495 — Local Time Disambiguation.

Cronie documents clock changes and fixed-time jobs in cron(8) — Cronie manual.

The UTC calendar syntax is documented in systemd.time(7) — Calendar events and time zones.

Have a story worth investigating?

Suggest a case

A project by Eniun

How do you solve the cases?

Each case presents a mystery that seems to defy explanation.

A moderator who knows the solution reads the mystery to the group. The group works out what happened by asking questions, discussing hypotheses and using what they know.

You can project the case on a screen so everyone can read it and refer back to it during the discussion.

The moderator can guide the discussion without giving away the answer. If the group gets stuck, each case offers three graded hints under “Hints”, which can be revealed one at a time.

Once the moderator confirms that the group has found the answer, select “Show answer” to open the solution.

You can also solve the cases on your own: read the story, try to come up with an explanation and use the hints only when you need them.