Solution
The clocks changed. In March they jumped from 02:00 to 03:00, so 02:30 never occurred. In October they went back from 03:00 to 02:00, so 02:30 occurred twice and both times triggered the backup.
What happened?
In mainland Spain, summer time begins on the last Sunday in March at 02:00 and ends on the last Sunday in October at 03:00. The change is one hour earlier in the Canary Islands. The server was set to Europe/Madrid, so its clock changed automatically.
The programme compared the local clock with a fixed string. It did not measure elapsed time.
- Every minute it read local time with
datetime.now() and checked for 02:30.
- In March the clock jumped from 01:59 to 03:00. The condition never matched, so there was no backup or error.
- In October, after 02:59 it returned to 02:00. The condition matched twice, an hour apart.
- The log also used local time, so both October backups showed 02:30.
Python’s PEP 495 calls a missing local time a gap and a repeated one a fold. Naive timestamps from datetime.now() cannot distinguish those two October times.
Can this still happen? Yes. Some schedulers handle clock changes differently. Cronie, for example, runs certain skipped fixed-time jobs just after the spring jump and avoids running them twice in autumn. Store and compare instants in UTC (Coordinated Universal Time). If a job must follow local time, define what it should do when that time is missing or repeated.
Technical explanation
The programme loop
A simplified version of the programme:
import time
from datetime import datetime
while True:
if datetime.now().strftime("%H:%M") == "02:30":
run_backup()
time.sleep(60 - datetime.now().second) # wait until the next minute
The condition requires the clock to show exactly 02:30. That happens zero times in March and twice in October.
How could it be fixed and prevented?
To fix it, run a backup promptly and check whether any data from the gap was left unprotected. Verify both October backups before deleting the spare.
To prevent it, combine these approaches:
- Schedule outside the transition. In mainland Spain, times from 02:00 up to 03:00 can be skipped or repeated. 04:00 occurs once each day.
- Use UTC, which has no daylight saving time. Python offers
datetime.now(timezone.utc), and systemd timers support UTC calendars such as OnCalendar=--* 01:30:00 UTC.
- Avoid relying on an exact match. Store the last successful backup date and run once the scheduled time has passed if the day’s backup is still missing.
- Monitor the result. An alert after more than 24 hours without a successful backup would have caught the problem on Sunday.
A project by Eniun