Choose a stable unit of work

A resumable pipeline needs a work item that remains identifiable across attempts. A row number can be enough for an immutable source file, provided the file itself has a durable identity. It is weaker when a user may upload a corrected file under the same name, when blank lines change positions, or when parsing rules can turn one row into several destination records.

A practical item identity usually combines source identity with a stable locator or business identifier. Examples include a file hash plus row number, an export ID plus source record ID, or a feed name plus upstream sequence number. Keep the original locator even when the source contains a useful business key. Duplicate business keys are often part of the data problem, and collapsing them too early removes evidence.

The item boundary should also match the point where independent recovery is possible. If three source rows must be validated and committed together, treating each row as independently complete creates a false checkpoint. If one row produces ten child records inside one database transaction, the row may still be a suitable item because the transaction makes the group atomic.

Record enough source context to investigate without retaining more sensitive material than necessary. The item can hold:

• a source reference and locator;
• a checksum of the normalised input;
• the parser or mapping version;
• the current state;
• attempt count and last attempt time;
• a bounded error code and message;
• destination identifiers created or matched;
• an idempotency key for external calls.

Do not put the whole source payload into every status row by default. That multiplies personal or confidential data across logs and operational tables. Keep the source in its controlled store, then refer back to it through a durable identifier.