Case study · AI Infrastructure & RAG
Cutting a data pipeline by about half
The company assumed they had a scaling problem. The real question was how much of the pipeline needed to run in real time at all.
- Sources
- Stream ingest
- Real-time processing
- Live tables and models
- Consumers
- Client
- a data engineering and AI infrastructure company
- Problem
- A data pipeline growing more complex and costly, read as a scaling problem.
- Approach
- Re-architected around what each consumer truly needed, not real time by default.
- Verified result
- About half the pipeline removed
The problem
The company's pipeline was getting harder to run and more expensive to keep up. The instinct, the common one, was that this was a scaling problem: more volume, more streaming, more infrastructure to hold it together.
That diagnosis leads to spending more on the same design. Before adding capacity, Stallwart asked which stages genuinely needed to be real time, and who was reading them at that freshness.
The reality behind the problem
The scaling instinct is expensive precisely because it looks responsible. Adding capacity to a pipeline that is slow or costly treats the symptom and preserves the design, so the bill grows while the real question goes unasked: does this stage need to run in real time at all?
Most of the pipeline's cost was in keeping data fresh that no consumer read at that freshness. Real time had become the default rather than a decision, and the default was paid for continuously, in infrastructure to run it and complexity to keep it reliable.
What was assumed
We need more scale.
What Stallwart asked
Does this actually need to be real time?
The desired outcome
The goal was not a faster pipeline. It was a smaller one that still produced everything the business actually used.
- Every stage matched to the consumer that reads its output and the freshness it truly needs.
- Latency removed wherever no consumer depended on it.
- Every output the business relied on preserved, unchanged.
- Streaming kept only on the paths that genuinely require it.
- A system that is cheaper to run, simpler to reason about, and easier to recover.
The system we designed
Stallwart mapped every stage to the system or person that read its output, and how fresh that reader actually needed it. Stages feeding consumers who were fine with hourly or daily data moved to batch, and the streaming infrastructure behind them came out.
How it works
- 01
Trace each stage to its consumer
Every output matched to who reads it and the freshness they truly need. A stage with no real-time consumer is cost with no return.
- 02
Move latency-insensitive work to batch
Stages whose consumers were fine with periodic data became scheduled jobs. Batch is cheaper to run, simpler to reason about, and easier to recover.
- 03
Keep real time where it earns its place
Streaming stayed only on the path a consumer genuinely depends on, which made that path easier to keep reliable.
The architecture
Real time, only where it earns its place.
Before
- SourcesSource
- Stream ingestReal-time
- Real-time processingReal-time
- Live tables and modelsReal-time
- ConsumersConsumer
Everything streamed, whether or not anyone read it live.
After
- SourcesSource
- Batch ingestBatch
- Batch processingBatch
- Real-time path, where requiredReal-time
- ConsumersConsumer
Batch by default. Streaming kept only where a consumer depends on it.
Architecture
What sits underneath.
The re-architecture reorganized the pipeline around who reads each output, not around a real-time default.
Sources and ingestion
The same sources feed the pipeline, but ingestion is batch by default and streams only where a downstream consumer depends on sub-minute freshness.
Batch processing
Stages whose consumers are fine with hourly or daily data run as scheduled jobs, which are cheaper to run, simpler to reason about, and easier to recover after a failure.
Real-time path, where required
Streaming is kept only on the one path a consumer genuinely depends on, which makes that path smaller and easier to keep reliable.
Consumers
Every output the business relied on is preserved. What changed is how often a stage runs, matched to how fresh its consumer needs it, not what it produces.
The hard parts
Where the engineering judgment was.
Question the requirement, not the capacity
A scaling spend assumes the design is right. The cheaper win was seeing that much of it should not exist.
The trade-off
Slower to a fix than buying capacity, but it removes the cost for good.
Batch by default, streaming on purpose
Real time is an operational tax, paid only where a consumer depends on it.
The trade-off
Some data is now hourly or daily by design, where nobody needed it sooner.
Treat a smaller system as the goal
Less pipeline is less to run, less to break, and less to pay for.
The trade-off
Removing infrastructure takes nerve. The payoff is durability, not a bigger footprint.
Reliability & guardrails
How it avoids the wrong call.
A smaller pipeline is a more reliable one: less to run, less to break, less to watch.
Batch recovers cleanly
A failed scheduled job can be re-run against its window. Batch stages are easier to recover than a streaming path, where recovery means replaying and reconciling live state.
Streaming scoped to where it earns its place
Keeping real time only on the path that needs it shrinks the surface that has to stay reliable under load, instead of spreading that cost across the whole pipeline.
No silent data loss
Outputs consumers depend on are preserved exactly. The change is cadence, matched to each consumer, not the data itself.
Simpler to reason about
A pipeline you can hold in your head fails in ways you can predict. Removing the stages nobody read removed whole categories of failure with them.
Productionization
What turns it from a demo into software.
What kept the smaller pipeline dependable in daily operation.
Observability
Each stage reports whether it ran, on what window, and whether it succeeded, so a missed or late job is visible rather than discovered downstream.
Backfill and replay
Batch stages can be re-run over a past window to correct or recover data, a recovery path far simpler than reconstructing streamed state.
Cost visibility
With real time scoped to where it is needed, the cost of the pipeline tracks the work that matters, and the spend that was going to unused freshness is gone.
Staged cutover
The re-architecture was rolled out so each change could be validated against the outputs the business relied on before the old path was removed.
The result
About halfthe pipelineremoved
Cutting the real-time work that no consumer needed took out about half of the company's data pipeline. The outputs the business relied on stayed. What went with it was the cost and complexity of keeping that half running.
In practice
Data Scientist, a data engineering and AI infrastructure company
The team's data scientist described the engagement as a shift from treating the problem as a scaling exercise to questioning whether the real-time architecture was necessary at all. Within two calls, the team had concluded that roughly half the pipeline did not need to exist.
Ownership & handover
What you receive, and keep.
The re-architected pipeline is yours to run, with nothing held back.
- Source code for the batch and streaming stages.
- Infrastructure as code for the scheduling, processing, and storage.
- Runbooks for operating, backfilling, and recovering each stage.
- Documentation of which stages are real time and why.
- The mapping of every stage to its consumer and required freshness.
What we learned
The insight that generalizes: when a system grows expensive, the first question is not how to scale it but whether it should exist at the size it does. Matching each stage to the freshness its consumer actually needs is how you find the half of a pipeline that was cost with no reader. A smaller system is the goal, not a consolation.
This work connects to
- AI Infrastructure & RAG
- Data pipeline architecture
- Production AI systems
Frequently asked
What is data pipeline re-architecture?
Data pipeline re-architecture is redesigning how data moves and is processed through a system, rather than just adding capacity to the existing design. It often means questioning which stages need real time, which can move to batch, and which do not need to exist at all.
How did Stallwart cut the company's pipeline in half?
By matching every stage to the consumer reading its output and the freshness that consumer actually needed. Stages that ran in real time but fed consumers who did not need real-time data moved to batch or were removed, which took out about half the pipeline.
When should a pipeline run in real time versus batch?
Run a stage in real time only when a downstream consumer depends on sub-minute freshness. If the consumer is fine with hourly or daily data, batch is cheaper, simpler, and more reliable. Real time applied by default is usually wasted cost.
Does removing real-time processing lose data or accuracy?
No. The data and its outputs are preserved. What changes is how often a stage runs, matched to how fresh its consumer needs it. Nothing a consumer used was removed, only latency no consumer depended on.
Have a system that grew more complex than the problem it solves?
Bring the workflow. We work backwards from what actually needs to exist.
Last updated: October 6, 2026