The files that kept corrupting themselves

A computer in an office started producing faulty backups. Some archives would not open, and some copied photos showed coloured stripes.

It did not happen every time. Repeating the same backup could succeed or fail, and different files were affected. The computer also froze from time to time without an obvious cause.

The technician replaced the backup drive, the cable and the backup software and ran a virus scan. The files kept getting corrupted, even when they were compressed on the computer itself without being copied to any external drive.

What was damaging the files?

How to solve the cases
A monitor showing a striped photo and a damaged archive beside a new external drive

Hints

Reveal them one at a time, only when you need them.

Solution

A RAM (random-access memory) module was faulty. Some cells occasionally changed value, so data passing through them during copying or compression was already corrupted by the time it reached the disk.

What happened?

To copy or compress a file, the computer reads it, loads it into RAM, processes it and then writes it. A faulty cell can flip a bit from 0 to 1 or vice versa without warning.

A Google study of thousands of servers found errors in over 8% of the modules studied per year, with errors tied to persistent faults predominating.

  1. The fault affected only parts of memory. A file loaded there might have one of its bits flipped.
  2. Memory allocation changes between operations, so the affected files and timing varied.
  3. One flipped bit can make an archive fail an integrity check or produce stripes in a photo.
  4. The same fault caused freezes when it hit data or instructions used by the operating system.
  5. Replacing the drive, cable or software did not help because the damage occurred before the data reached the disk.

The symptoms looked like a storage fault because the corruption became visible in stored files, although it happened in memory.

How could it be fixed and prevented?

To fix it, test memory using a tool such as MemTest86 from a bootable USB drive or Windows Memory Diagnostic (mdsched.exe). If errors appear, test the modules individually, replace the faulty one and remake the affected backups from the originals.

To prevent it from recurring:

  • Verify backups. Compare the hash of each original and copy, for example with PowerShell’s Get-FileHash, or use backup software that verifies its writes.
  • Check RAM when failures are random. Unpredictable freezes, varied blue screens and random corruption are typical clues. Run a memory test before changing other components.
  • Test memory after building or upgrading a computer before trusting it with important data.
  • Use error-correcting memory (ECC) in critical systems, such as servers, to detect and correct many memory errors.

Further reading

PassMark describes the symptoms and how to test memory from a USB drive on the site for MemTest86 — PassMark Software.

Memory error rates were measured in DRAM Errors in the Wild: A Large-Scale Field Study — Google Research, by Schroeder, Pinheiro and Weber (2009).

To compare the hash of an original and its copy on Windows, see Get-FileHash — Microsoft Learn.

Have a story worth investigating?

Suggest a case

A project by Eniun

How do you solve the cases?

Each case presents a mystery that seems to defy explanation.

A moderator who knows the solution reads the mystery to the group. The group works out what happened by asking questions, discussing hypotheses and using what they know.

You can project the case on a screen so everyone can read it and refer back to it during the discussion.

The moderator can guide the discussion without giving away the answer. If the group gets stuck, each case offers three graded hints under “Hints”, which can be revealed one at a time.

Once the moderator confirms that the group has found the answer, select “Show answer” to open the solution.

You can also solve the cases on your own: read the story, try to come up with an explanation and use the hints only when you need them.