Designing a batch image pipeline that won’t corrupt your originals

A batch image pipeline is trivial to demo and hard to make safe. The demo processes ten clean JPEGs. Production hands you a broken TIFF, a HEIC your library can’t decode, a file whose embedded metadata is larger than its pixels, and a disk that fills at image 9,000 of 10,000. A pipeline that hasn’t planned for those corrupts the batch instead of skipping the bad file.

Never mutate the original in place

The first rule is non-destructive: read the source, write a new output, and only swap it in once the new file is verified complete. Editing the original directly means a crash mid-write leaves you with neither the old file nor a valid new one. Keep the source until the replacement has proven itself.

Verify, then replace

Don’t trust that the write succeeded — confirm it. Re-open the output, check the dimensions decode and the metadata block is present, and only then replace the live file. This single discipline turns “the optimizer corrupted my library” into “the optimizer skipped three files and told me which.”

Isolate failure to one image

Each image should succeed or fail on its own. One undecodable file must not abort the other 9,999 — it should be logged, skipped, and surfaced in a report. Treat the batch as thousands of independent jobs, not one big transaction.

• Non-destructive: original stays until the output is verified.
• Verify-then-replace: re-read the result before trusting it.
• Per-image isolation: a bad file is skipped, never fatal.

Takeaway: the hard part of batch processing isn’t speed, it’s safety. Design for the broken file and the full disk, and a large batch becomes boring — which is exactly what you want from something touching your originals.