Validation is not enough
Most data quality tools focus on the wrong moment. They validate data before it moves: profiling source data, checking schemas, running pre-flight checks. This catches structural problems like wrong data types or missing required fields. But it misses the failures that happen during transformation and loading.
A record passes pre-validation with clean source data. During transformation, a date conversion produces an invalid result because the source uses a non-standard format for that specific record. A join condition matches the wrong record because of a duplicate key that did not exist in the sample data. A calculated field overflows because the source values are larger than the test data suggested.
These failures happen at execution time, inside the pipeline, after pre-validation has already given the green light. The question is not whether bad records will appear during a pipeline run. The question is what happens when they do.
The cost of silent failures
The default behavior in most integration tools is to either stop the entire pipeline on the first error or silently skip bad records. Both are wrong.
Stopping on the first error means a single bad record out of 100,000 blocks the entire run. The 99,999 good records wait while someone investigates and fixes the one problem. In time-sensitive pipelines like nightly data loads, real-time syncs, and regulatory reporting, this delay is unacceptable.
Silently skipping bad records is worse. The pipeline reports success. The destination receives data. But the record count is short. Somewhere between source and destination, records vanished without a trace. Nobody knows how many, which ones, or why. The data looks complete until someone runs a reconciliation and discovers the gap.
Neither approach treats bad records as something that requires a deliberate, configurable response. Quality enforcement means defining what should happen when a record fails, and having that definition execute consistently every time.
Three enforcement actions
When a quality rule fails during a pipeline run, one of three actions can execute. Each serves a different purpose, and choosing the right one depends on the severity of the failure and the downstream impact of the bad record.
Quarantine
Quarantined records are removed from the output and stored separately for review. They are preserved in their entirety, in a dedicated quarantine store, with full context about the failure: the specific rule that triggered the quarantine, the field values that caused it, and a timestamp.
When to use it: When the record might be fixable, and you want to review and potentially reprocess it. A transaction record with an unrecognized currency code might have a typo that can be corrected. A customer record with a duplicate ID might represent a legitimate new record that needs a different ID assignment. A record with an invalid email format is wrong, but the rest of the fields are fine and the record should not be lost.
What it looks like in practice: A quality rule checks that currency_code matches a known ISO currency code. A record with currency_code: "UDS" (likely a typo for “USD”) fails the rule. The record is quarantined: removed from the output and stored separately with the failure details. After the pipeline completes, a data steward reviews quarantined records, corrects the typo, and resubmits the fixed record.
The tradeoff: Quarantine requires someone to review and act on the stored records. If quarantined records pile up without review, the quarantine becomes a data graveyard. Use quarantine when you have a defined process for reviewing and reprocessing failed records.
Flag
Flagged records are included in the output, but they carry a marker indicating that a quality rule failed. The destination receives the record, but the flag signals that the data may need attention.
When to use it: When the record is usable despite the quality issue, and downstream systems or users need to know about the problem without being blocked by it. A customer record with a phone number in the wrong format is still a valid customer record. The phone number is suspect, but the rest of the fields are fine.
What it looks like in practice: A quality rule checks that phone_number matches the expected format for the customer’s country. A record has a phone number with an extra digit. The record loads into the destination with a flag indicating the phone number format violation. A downstream team can filter for flagged records and clean them up without holding up the pipeline.
The tradeoff: Flagged records reach the destination with potentially bad data. If downstream systems process flagged fields without checking the flag, the bad data propagates. Flagging works when there is a human or automated review step after loading.
Stop Job
Stop job halts the pipeline entirely. No more records are processed. The output produced so far may be rolled back or marked as incomplete, depending on the configuration.
When to use it: When a single failure indicates a systemic problem that affects the entire dataset. If a quality rule checks that the source_system field matches an expected value and it does not, the source data might be from the wrong system entirely. Processing more records will not help; every record has the same problem.
What it looks like in practice: A quality rule checks that the schema_version field matches the expected version. The first record has an unrecognized schema version, indicating that the source format has changed without notice. The pipeline stops immediately. The operations team is alerted. No partial data reaches the destination.
The tradeoff: Stopping the job is the nuclear option. It blocks all data movement, including the records that might be perfectly fine. Use it only when the failure suggests that continuing would produce unreliable output across the board.
Choosing the right action for each rule
The three actions are not interchangeable. Choosing the right one requires thinking about two dimensions: the severity of the failure and the tolerance of the destination.
| Destination tolerates missing records | Destination requires all records | |
|---|---|---|
| Minor quality issue | Quarantine (save for review) | Flag (include with marker) |
| Potentially fixable issue | Quarantine (save for review) | Flag + quarantine (include flagged, also save copy for review) |
| Systemic data problem | Stop job (stop everything) | Stop job (stop everything) |
A single pipeline typically uses multiple actions across different quality rules. Format validation might use flag. Referential integrity checks might use quarantine. Schema version checks might use stop job. The combination of actions defines the pipeline’s quality posture.
Test runs and violation tracking
Quality rules should not be tested in production for the first time. Running the pipeline against sample data in test mode executes every quality rule and reports what would happen without actually loading data or taking action.
A test run might reveal that 3% of records would be quarantined by a phone format rule. This is useful information before production. Maybe the rule is too strict and should flag instead of quarantine. Maybe the source data has a known subset with phone numbers in a different format, and the rule needs a transformation condition to handle it.
In datathere, test runs execute the full pipeline (including quality rules, transformations, and joins) against sample data. The output shows exactly which records would be quarantined, flagged, or stopped, with the specific rule and field values for each violation.
After production runs, violation tracking provides a persistent record of quality failures across runs. Trends emerge: a particular field fails the same rule consistently, or a specific source produces more quarantined records than others. These trends inform decisions about whether to tighten rules, fix source data quality, or adjust transformation logic.
Quality enforcement as a system
Individual quality rules catch individual problems. The value of enforcement comes from treating quality as a system — a coherent set of rules with deliberately chosen actions, tested before production, tracked over time, and reviewed as part of the certification workflow.
A quality rule without the right action is just a check. A check without tracking is just a one-time observation. Tracking without review is just data nobody reads. The full system (rules, actions, testing, tracking, and review) is what turns data quality from an aspiration into an operational discipline.
The difference between a pipeline that “usually works” and a pipeline you trust is not the number of quality rules. It is the precision of the response when those rules fail.