27 February 2025 / Data systems / 8 chapters

Designing resumable large-data pipelines

Gold contact pads and fine traces on a dark printed circuit board.
Photo: Vishnu Mohanan (opens in a new tab)
Read the Introduction

This guide begins with a job stopped halfway through an import. It develops the state and restart rules needed to identify accepted work, isolate failed records and continue without rebuilding the entire run.

Chapter 1: A stopped run exposes the missing contract

All articles