Skip to content

Case study · AI Infrastructure & RAG

Cutting a data pipeline by about half

The company assumed they had a scaling problem. The real question was how much of the pipeline needed to run in real time at all.

The system, as found
  1. Sources
  2. Stream ingest
  3. Real-time processing
  4. Live tables and models
  5. Consumers
Client
a data engineering and AI infrastructure company
Problem
A data pipeline growing more complex and costly, read as a scaling problem.
Approach
Re-architected around what each consumer truly needed, not real time by default.
Verified result
About half the pipeline removed

The problem

The company's pipeline was getting harder to run and more expensive to keep up. The instinct, the common one, was that this was a scaling problem: more volume, more streaming, more infrastructure to hold it together.

That diagnosis leads to spending more on the same design. Before adding capacity, Stallwart asked which stages genuinely needed to be real time, and who was reading them at that freshness.

The reality behind the problem

The scaling instinct is expensive precisely because it looks responsible. Adding capacity to a pipeline that is slow or costly treats the symptom and preserves the design, so the bill grows while the real question goes unasked: does this stage need to run in real time at all?

Most of the pipeline's cost was in keeping data fresh that no consumer read at that freshness. Real time had become the default rather than a decision, and the default was paid for continuously, in infrastructure to run it and complexity to keep it reliable.

What was assumed

We need more scale.

What Stallwart asked

Does this actually need to be real time?

The desired outcome

The goal was not a faster pipeline. It was a smaller one that still produced everything the business actually used.

  • Every stage matched to the consumer that reads its output and the freshness it truly needs.
  • Latency removed wherever no consumer depended on it.
  • Every output the business relied on preserved, unchanged.
  • Streaming kept only on the paths that genuinely require it.
  • A system that is cheaper to run, simpler to reason about, and easier to recover.

The system we designed

Stallwart mapped every stage to the system or person that read its output, and how fresh that reader actually needed it. Stages feeding consumers who were fine with hourly or daily data moved to batch, and the streaming infrastructure behind them came out.

How it works

  1. 01

    Trace each stage to its consumer

    Every output matched to who reads it and the freshness they truly need. A stage with no real-time consumer is cost with no return.

  2. 02

    Move latency-insensitive work to batch

    Stages whose consumers were fine with periodic data became scheduled jobs. Batch is cheaper to run, simpler to reason about, and easier to recover.

  3. 03

    Keep real time where it earns its place

    Streaming stayed only on the path a consumer genuinely depends on, which made that path easier to keep reliable.

The architecture

Real time, only where it earns its place.

Before

  1. SourcesSource
  2. Stream ingestReal-time
  3. Real-time processingReal-time
  4. Live tables and modelsReal-time
  5. ConsumersConsumer

Everything streamed, whether or not anyone read it live.

After

  1. SourcesSource
  2. Batch ingestBatch
  3. Batch processingBatch
  4. Real-time path, where requiredReal-time
  5. ConsumersConsumer

Batch by default. Streaming kept only where a consumer depends on it.

Architecture

What sits underneath.

The re-architecture reorganized the pipeline around who reads each output, not around a real-time default.

01

Sources and ingestion

The same sources feed the pipeline, but ingestion is batch by default and streams only where a downstream consumer depends on sub-minute freshness.

02

Batch processing

Stages whose consumers are fine with hourly or daily data run as scheduled jobs, which are cheaper to run, simpler to reason about, and easier to recover after a failure.

03

Real-time path, where required

Streaming is kept only on the one path a consumer genuinely depends on, which makes that path smaller and easier to keep reliable.

04

Consumers

Every output the business relied on is preserved. What changed is how often a stage runs, matched to how fresh its consumer needs it, not what it produces.

The hard parts

Where the engineering judgment was.

01

Question the requirement, not the capacity

A scaling spend assumes the design is right. The cheaper win was seeing that much of it should not exist.

The trade-off

Slower to a fix than buying capacity, but it removes the cost for good.

02

Batch by default, streaming on purpose

Real time is an operational tax, paid only where a consumer depends on it.

The trade-off

Some data is now hourly or daily by design, where nobody needed it sooner.

03

Treat a smaller system as the goal

Less pipeline is less to run, less to break, and less to pay for.

The trade-off

Removing infrastructure takes nerve. The payoff is durability, not a bigger footprint.

Reliability & guardrails

How it avoids the wrong call.

A smaller pipeline is a more reliable one: less to run, less to break, less to watch.

Batch recovers cleanly

A failed scheduled job can be re-run against its window. Batch stages are easier to recover than a streaming path, where recovery means replaying and reconciling live state.

Streaming scoped to where it earns its place

Keeping real time only on the path that needs it shrinks the surface that has to stay reliable under load, instead of spreading that cost across the whole pipeline.

No silent data loss

Outputs consumers depend on are preserved exactly. The change is cadence, matched to each consumer, not the data itself.

Simpler to reason about

A pipeline you can hold in your head fails in ways you can predict. Removing the stages nobody read removed whole categories of failure with them.

Productionization

What turns it from a demo into software.

What kept the smaller pipeline dependable in daily operation.

Observability

Each stage reports whether it ran, on what window, and whether it succeeded, so a missed or late job is visible rather than discovered downstream.

Backfill and replay

Batch stages can be re-run over a past window to correct or recover data, a recovery path far simpler than reconstructing streamed state.

Cost visibility

With real time scoped to where it is needed, the cost of the pipeline tracks the work that matters, and the spend that was going to unused freshness is gone.

Staged cutover

The re-architecture was rolled out so each change could be validated against the outputs the business relied on before the old path was removed.

The result

About halfthe pipelineremoved

Cutting the real-time work that no consumer needed took out about half of the company's data pipeline. The outputs the business relied on stayed. What went with it was the cost and complexity of keeping that half running.

In practice

Data Scientist, a data engineering and AI infrastructure company

The team's data scientist described the engagement as a shift from treating the problem as a scaling exercise to questioning whether the real-time architecture was necessary at all. Within two calls, the team had concluded that roughly half the pipeline did not need to exist.

Ownership & handover

What you receive, and keep.

The re-architected pipeline is yours to run, with nothing held back.

  • Source code for the batch and streaming stages.
  • Infrastructure as code for the scheduling, processing, and storage.
  • Runbooks for operating, backfilling, and recovering each stage.
  • Documentation of which stages are real time and why.
  • The mapping of every stage to its consumer and required freshness.

What we learned

The insight that generalizes: when a system grows expensive, the first question is not how to scale it but whether it should exist at the size it does. Matching each stage to the freshness its consumer actually needs is how you find the half of a pipeline that was cost with no reader. A smaller system is the goal, not a consolation.

This work connects to

  • AI Infrastructure & RAG
  • Data pipeline architecture
  • Production AI systems

Frequently asked

What is data pipeline re-architecture?

Data pipeline re-architecture is redesigning how data moves and is processed through a system, rather than just adding capacity to the existing design. It often means questioning which stages need real time, which can move to batch, and which do not need to exist at all.

How did Stallwart cut the company's pipeline in half?

By matching every stage to the consumer reading its output and the freshness that consumer actually needed. Stages that ran in real time but fed consumers who did not need real-time data moved to batch or were removed, which took out about half the pipeline.

When should a pipeline run in real time versus batch?

Run a stage in real time only when a downstream consumer depends on sub-minute freshness. If the consumer is fine with hourly or daily data, batch is cheaper, simpler, and more reliable. Real time applied by default is usually wasted cost.

Does removing real-time processing lose data or accuracy?

No. The data and its outputs are preserved. What changes is how often a stage runs, matched to how fresh its consumer needs it. Nothing a consumer used was removed, only latency no consumer depended on.

Have a system that grew more complex than the problem it solves?

Bring the workflow. We work backwards from what actually needs to exist.

Last updated: October 6, 2026