Setting failed records aside so the run keeps going

One malformed record shouldn't force you to restart a big import. Skipping it without writing anything down is just as bad, because then the run can't be reconciled. The pipeline needs to tell a data problem that will fail the same way every time apart from an operational failure that might work later.

Validation errors are things like missing required fields, invalid references, unsupported values and rule conflicts. Give them stable error codes and keep the source locator. The message an operator sees can explain the field and what was expected, without dumping secrets into a general log.

Retryable faults are things like a dropped connection, rate limiting, a dependency that's temporarily down, or a worker's lease expiring. Tie the retry policy to the operation. A brief database blip might be fine to handle with automatic backoff. An external action where you don't know if it went through needs reconciling before you try again.

Set an attempt limit, but don't make it the only rule for quarantine. Running into the same schema mismatch five times just wastes time, because waiting won't change the input. Going the other way, if you know a service is down, it can make sense to hold items in retryable until it's back, without burning through their attempts.

The quarantine queue should offer a handful of explicit choices:

• fix the source and create a new version
• change a mapping under a new configuration version
• mark the item as deliberately skipped
• release it for another attempt

Keep the original attempt history whichever one you pick. If you keep editing a failed status row until it looks clean, you wreck the record you'd need to understand what happened in the run.

Group repeated errors for whoever's operating it. Thousands of rows failing the same validation should show up as one main fault, with affected counts and some sample records picked under your privacy rules. The operator can look at the pattern first and then open individual records to fix them.