Data Quality Gates: Treating Bad Data Like a Failed Build

Contributed by Animesh Pathak, Harness

Every test was green, the deployment went out clean, and by Monday morning, the revenue dashboard was wrong, the weekly report went to the leadership team with numbers nobody could explain, and a machine learning model had quietly learned from garbage.

No code broke. What broke was the data. A source system sent a file with half the usual rows. A column that was never empty suddenly arrived full of blanks. And the pipeline did what most pipelines do: it shrugged and loaded everything anyway.

Here is the strange part. In software delivery, we solved this exact problem years ago. If a developer pushes code and the tests fail, the build turns red and nothing ships. Nobody debates this. Yet, in most data pipelines, there is no equivalent. Data flows from system to system with no checkpoint, no red light, no “this batch did not pass”.

This post is about adding that red light. The idea is called a data quality gate: a checkpoint in the pipeline where data must pass a set of checks before it is allowed to move on. If the checks fail, the pipeline stops – exactly like a failed build. Everything below uses open source tools and works on any CI/CD system: Tekton, GitHub Actions, GitLab CI, or anything else that can run a container.

First, what do we mean by “data”?

It is worth pinning this down, because “data” gets used loosely.

In this post, data means the records a business runs on: orders, customers, payments, sign-ups, sensor readings, clicks. These records are born in source systems (a shop, an app, a payment provider), travel through pipelines into a warehouse, and end up feeding dashboards, reports, and models. Each record is rows and columns: an order has a customer, an amount, a status, a date.

But here is the part most pipelines skip: a value is only “right” or “wrong” relative to an agreed meaning. Is an order with the amount “zero” an error, or a legitimate free item? Is cancelled a valid status, or was it retired last quarter? Does created_at mean when the customer clicked, or when our system received it? None of these questions can be answered by looking at the data. Someone has to decide, and the decision has to be written down so that somewhere in the pipeline can enforce it.

Most organizations do write these decisions down – in a wiki, a glossary, a governance document that was accurate the day it was written. The problem is that documents cannot stop a pipeline. This post is about moving those definitions into a place that can.

The problem: pipelines trust everything

A typical data pipeline moves data through stages. A source system exports records. A job loads them into a warehouse. Dashboards read from the warehouse. Reports and models read from the dashboards’ tables. Each stage trusts the one before it, completely.

So when a bad batch arrives – fewer rows than usual, empty fields that should never be empty, dates from the future – nothing stops it. The damage travels. And data damage has a nasty property that code damage does not: it spreads silently. A crashed service page someone at 2 a.m. A wrong number on a dashboard just sits there, looking exactly like a right number, until a human happens to squint at it, if they even notice 😀. By then, three teams downstream have already made decisions based on it.

The cost of bad data is not the bad batch itself. It is every decision, report, and model built on top of it before anyone notices.

The idea: a failed build, but for data

Software delivery already has the answer. We just have not applied it to data.

When code changes, tests run automatically, and failure blocks the deployment. A data quality gate does the same thing for a batch of data: when new data arrives, checks run automatically, and failure blocks the data from moving downstream.

A “check” here is a simple, machine-readable rule about what healthy data looks like:

• The orders table should never receive fewer than 1,000 rows a day.
• The customer_id column must never be empty.
• The newest record should be less than 24 hours old.
• Every order_status value must be one of four known words.

Individually, each rule is almost embarrassingly simple. That is the point. A factory does not inspect parts with philosophy; it inspects them with a checklist. The power is not in any single rule – it is in the rule running automatically, on every batch, forever.

How a gate works: write, audit, publish

The safest way to build a gate uses a pattern called write-audit-publish. It has three steps:

1. Write. The new batch is loaded into a staging table – a holding area that downstream users never read from.
2. Audit. Every check runs against the staging table. This is the gate.
3. Publish. If all checks pass, the data is swapped into the production table in one atomic step. If any check fails, the batch stays in staging, the pipeline turns red, and an alert goes out. The failed data is preserved exactly as it arrived, which makes debugging simple: the evidence is sitting right there.

The key property: downstream users only ever see data that passed the audit. A failed batch inconveniences one pipeline run. It does not poison a single dashboard.

If you have ever deployed an application to a staging environment, run smoke tests, and then flipped traffic over – this is the same move. Data engineering did not need a new idea. It needed to borrow an old one.

A working example

Here is a gate built with Soda Core, an open source data quality tool, running as a step in Tekton, the CI/CD framework. The checks file is the heart of it – and it reads almost like English:

# checks/orders_checks.yml
checks for orders_staging:
  # Volume: a half-empty batch is a broken batch
  - row_count > 1000

  # Completeness: these columns must never be empty
  - missing_count(customer_id) = 0
  - missing_count(order_status) = 0

  # Freshness: stale data is quietly wrong data
  - freshness(created_at) < 24h

  # Validity: only known values allowed
  - invalid_count(order_status) = 0:
      valid values: [pending, paid, shipped, cancelled]

  # Sanity: money should not be negative
  - min(amount_cents) >= 0

And the Tekton Task that runs the audit step against the staging table:

apiVersion: tekton.dev/v1
kind: Task
metadata:
  name: data-quality-gate
spec:
  description: >-
    Audits a freshly loaded staging table. Fails the pipeline
    if the data does not pass every check.
  workspaces:
    - name: source          # the repo with checks/ and config
  params:
    - name: soda-image
      default: sodadata/soda-core:latest
  steps:
    - name: audit-staging-table
      image: $(params.soda-image)
      workingDir: $(workspaces.source.path)
      script: |
        #!/usr/bin/env bash
        set -euo pipefail
        soda scan -d warehouse \
          -c config/soda_configuration.yml \
          checks/orders_checks.yml
        echo "PASS: batch is safe to publish."

soda scan exits with a non-zero code when any check fails, so the Task fails, the PipelineRun fails, and – critically – the publish step that comes after it never runs. The batch stays in staging. That ordering is the whole gate: audit before publishing.

Here is how that Task sits inside a full Tekton Pipeline:

And here is exactly what happens on each outcome. This is not a policy someone has to remember to follow – it is how Tekton’s runAfter dependency works. A failed Task simply never lets the next one start:

The same step drops into any CI system. In GitHub Actions or GitLab CI it is one job with the same soda scan command: 

If your team already uses dbt, dbt test plays the same role; if you prefer Great Expectations, its checkpoints do too. The tool matters far less than the position: between load and publish, with the power to block.

What checks should you write?

Six families cover almost everything. Start with one check from each:

<tdstyle=”text-align: left;”=””>refunds ≤ total orders
</tdstyle=”text-align:>

Do not try to write fifty checks on day one. Six good checks that block the pipeline beat sixty aspirational ones in a document nobody reads.

Not every failure should stop the world

A build analogy again: some code issues fail the build, others are warnings. Data checks deserve the same two levels:

Most teams get this backwards at first – everything blocks, alerts fire constantly, and within a month everyone ignores them. That’s worse than no gate at all, because now the red light means nothing. Start strict on the checks that protect correctness (IDs, validity) and lenient on the ones that track trends (volume, freshness). Promote a warning to a block only after it has proven itself worth trusting.

The bigger picture: checks are governance you can execute

Step back from the mechanics for a moment, because the checks file is quietly doing a second job.

Data governance is the practice of deciding, for the data a business runs on: who owns it, what each field means, what counts as valid, and who is allowed to change those answers. Every organization above a certain size has governance in some form. The trouble is where it usually lives – policy documents, glossary wikis, and quarterly council meetings. Well-intentioned, and completely invisible to the pipeline. A rule that exists only in a document is a rule the data never has to obey.

A quality gate changes that. Look again at one line from the checks file:

- invalid_count(order_status) = 0:
    valid values: [pending, paid, shipped, cancelled]

That is not just a test. It is a definition – “these four words are what an order status means here” – written in a file, owned by a team, changed only through a reviewed pull request, and enforced on every batch automatically. The glossary entry and the enforcement are now the same artifact. Nobody has to remember to check the wiki, because the wiki, in effect, runs in CI.

Here is how the pieces of traditional governance map onto the gate:

This is the same shift that infrastructure went through with infrastructure-as-code: rules moved out of documents and into version-controlled files that systems actually obey. Call it governance-as-code. Nothing about the council or the glossary was wrong – they just had no way to say no. The gate gives them one.

There is a practical payoff too. When someone asks “why did the pipeline reject this batch?“. The answer is not “policy” – it is a specific line, in a specific file, with a git history showing who wrote it and what discussion led to it. Rejections stop being mysterious, and definitions stop being tribal knowledge.

Hurdles you will hit (I did)

“What are the right thresholds?” 

Nobody knows their normal row count off the top of their head. 

Fix: run the checks in warn-only mode for two weeks first. The logs tell you what normal looks like; then set thresholds and turn on blocking. Measure before you enforce.

The gate slows the pipeline

Auditing a huge table on every run can take longer than loading it. 

Fix: check the new batch in staging, not the whole table. Full-table checks can run on a schedule instead of on every load.

Alert fatigue

Covered above, but it deserves repeating because it is the number one reason gates get removed. A gate that cries wolf gets deleted. Tune ruthlessly.

Fix: Alert only on failures that need action. Set clear severity levels also tune thresholds to reduce false positives, and assign an owner to each alert. Review noisy checks regularly and remove alerts that don’t lead to meaningful action.

“Whose check is it anyway?”

The team that produces the data knows what it should look like; the team that consumes it feels the pain when it is wrong. 

Fix: checks live in the repository, in version control, changed through pull requests – so both teams can propose, review, and see every rule. This is the governance model from the previous section doing real work: a quality rule nobody can find is a quality rule nobody trusts.

The upstream team keeps breaking the same field

A gate protects you, but it does not fix them. 

Fix: share the failing check with the producing team as a data contract – “this is what downstream depends on”. The same file that blocks your pipeline becomes the specification for theirs.

Why this matters: you cannot roll back a decision

One last borrow from software thinking – and one place the analogy breaks in an important way.

A bad deployment can be rolled back. A bad batch of data can also, technically, be rolled back: delete it, reload it. But the decisions made on top of it cannot. The discount campaign launched because the dashboard undercounted revenue. The stock ordered because the forecast trained on duplicated rows. The report that leadership already read. Data has consumers, and consumers act. There is no git revert for these consumer actions.

Which means the same conclusion we keep arriving at in delivery engineering applies here with extra force: the only cheap place to catch bad data is before anyone uses it. A quality gate is exactly that place – automatic, on every batch, in the one position where a red light still costs nothing.

Conclusion

Data quality gates are not a platform migration or a brand new process. They are a habit borrowed from CI/CD and applied one step earlier: load into staging, audit with simple declared checks, and publish only on green.

Start with six checks in warn mode, measure for two weeks, then let them block. The point is not to make every minor flaw stop the world, but to ensure that data with known issues never quietly poisons downstream systems.

And notice what you get for free along the way. The moment your definitions of “valid” live in a version-controlled checks file that can stop a pipeline, you have working data governance – not the document kind, the executable kind. Ownership, definitions, change history, and audit trail all fall out of the same small file that turns the build red.

Data integrity is the goal and data validation is the check. A data quality gate is where you enforce it. That is the whole move: taking a familiar software delivery practice and using it to catch bad data before it ever becomes a bad decision.