Picking a unit of work that stays the same between attempts

A resumable pipeline needs a work item you can still recognise on the second and third attempt. A row number can be enough if the source file never changes and the file itself has a durable identity. It gets shaky when someone can upload a corrected file under the same name, when blank lines move rows around, or when the parser can turn one row into several destination records.

In practice I'd build the item identity from the source identity plus a stable locator or business identifier. That could be a file hash plus row number, an export ID plus the source record ID, or a feed name plus the upstream sequence number. Hang on to the original locator even when the source has a handy business key. Duplicate business keys are often part of the data problem you're dealing with, and if you collapse them too early you lose the evidence.

The item boundary also needs to sit where you can actually recover one item independently. If three source rows have to be validated and committed together, marking each row complete on its own gives you a checkpoint that isn't true. On the other hand, if one row produces ten child records inside a single database transaction, the row can still be a perfectly good item, because the transaction makes that group all-or-nothing.

Keep enough source context to investigate a problem, without copying around more sensitive material than you need. An item record could hold:

• a source reference and locator
• a checksum of the normalised input
• the parser or mapping version
• the current state
• the attempt count and last attempt time
• a short error code and message (capped in length)
• any destination identifiers it created or matched
• an idempotency key for external calls

I wouldn't put the whole source payload in every status row by default. That spreads personal or confidential data across logs and operational tables. Leave the source in its controlled store and point back to it with a durable identifier.