Case study · AI Infrastructure & RAG
Cutting a data pipeline by about half
NautilusPrinciple assumed they had a scaling problem. The real question was how much of the pipeline needed to run in real time at all.
- Sources
- Stream ingest
- Real-time processing
- Live tables and models
- Consumers
- Client
- NautilusPrinciple
- Problem
- A data pipeline growing more complex and costly, read as a scaling problem.
- Approach
- Re-architected around what each consumer truly needed, not real time by default.
- Verified result
- About half the pipeline removed
The problem
NautilusPrinciple's pipeline was getting harder to run and more expensive to keep up. The instinct, the common one, was that this was a scaling problem: more volume, more streaming, more infrastructure to hold it together.
That diagnosis leads to spending more on the same design. Before adding capacity, Stallwart asked which stages genuinely needed to be real time, and who was reading them at that freshness.
What was assumed
We need more scale.
What Stallwart asked
Does this actually need to be real time?
What Stallwart built
Stallwart mapped every stage to the system or person that read its output, and how fresh that reader actually needed it. Stages feeding consumers who were fine with hourly or daily data moved to batch, and the streaming infrastructure behind them came out.
- 01
Trace each stage to its consumer
Every output matched to who reads it and the freshness they truly need. A stage with no real-time consumer is cost with no return.
- 02
Move latency-insensitive work to batch
Stages whose consumers were fine with periodic data became scheduled jobs. Batch is cheaper to run, simpler to reason about, and easier to recover.
- 03
Keep real time where it earns its place
Streaming stayed only on the path a consumer genuinely depends on, which made that path easier to keep reliable.
The architecture
Real time, only where it earns its place.
Before
- SourcesSource
- Stream ingestReal-time
- Real-time processingReal-time
- Live tables and modelsReal-time
- ConsumersConsumer
Everything streamed, whether or not anyone read it live.
After
- SourcesSource
- Batch ingestBatch
- Batch processingBatch
- Real-time path, where requiredReal-time
- ConsumersConsumer
Batch by default. Streaming kept only where a consumer depends on it.
Engineering decisions
Question the requirement, not the capacity
A scaling spend assumes the design is right. The cheaper win was seeing that much of it should not exist.
The trade-off
Slower to a fix than buying capacity, but it removes the cost for good.
Batch by default, streaming on purpose
Real time is an operational tax, paid only where a consumer depends on it.
The trade-off
Some data is now hourly or daily by design, where nobody needed it sooner.
Treat a smaller system as the goal
Less pipeline is less to run, less to break, and less to pay for.
The trade-off
Removing infrastructure takes nerve. The payoff is durability, not a bigger footprint.
The result
About halfthe pipelineremoved
Cutting the real-time work that no consumer needed took out about half of NautilusPrinciple's data pipeline. The outputs the business relied on stayed. What went with it was the cost and complexity of keeping that half running.
In practice
Sukanthen
Data Scientist, NautilusPrinciple
NautilusPrinciple's data scientist described the engagement as a shift from treating the problem as a scaling exercise to questioning whether the real-time architecture was necessary at all. Within two calls, the team had concluded that roughly half the pipeline did not need to exist.
This work connects to
- AI Infrastructure & RAG
- Data pipeline architecture
- Production AI systems
Frequently asked
What is data pipeline re-architecture?
Data pipeline re-architecture is redesigning how data moves and is processed through a system, rather than just adding capacity to the existing design. It often means questioning which stages need real time, which can move to batch, and which do not need to exist at all.
How did Stallwart cut NautilusPrinciple's pipeline in half?
By matching every stage to the consumer reading its output and the freshness that consumer actually needed. Stages that ran in real time but fed consumers who did not need real-time data moved to batch or were removed, which took out about half the pipeline.
When should a pipeline run in real time versus batch?
Run a stage in real time only when a downstream consumer depends on sub-minute freshness. If the consumer is fine with hourly or daily data, batch is cheaper, simpler, and more reliable. Real time applied by default is usually wasted cost.
Does removing real-time processing lose data or accuracy?
No. The data and its outputs are preserved. What changes is how often a stage runs, matched to how fresh its consumer needs it. Nothing a consumer used was removed, only latency no consumer depended on.
Have a system that grew more complex than the problem it solves?
Bring the workflow. We work backwards from what actually needs to exist.
Last updated: October 6, 2026