Pipelines

Definition

Pipelines coverage in this archive spans 3 posts from Apr 2017 to May 2025 and centers on data correctness and operability under real production constraints. The strongest adjacent threads are ai, data, and infrastructure. Recurring title motifs include ai, pipeline, extra, and steps.

Key claims

The common theme is that schema, ownership, and query shape drive most downstream outcomes.
The consistent theme from 2017 to 2025 is disciplined execution over hype cycles.
This topic repeatedly intersects with ai, data, and infrastructure, so design choices here rarely stand alone.

Practical checklist

Define freshness, correctness, and latency targets before choosing storage or pipeline patterns.
Start with the newest post to calibrate current constraints, then backtrack to older entries for first principles.
When boundary questions appear, cross-read ai and data before committing implementation details.

Failure modes

Scaling pipelines before locking down source-of-truth and reconciliation behavior.
Optimizing single queries while ignoring data model drift and access patterns.
Applying guidance from 2017 to 2025 without revisiting assumptions as context changed.

References

Your AI Pipeline Is Just ETL With Extra Steps (And That's Fine)

May 2025

AI data pipelines aren't some new paradigm. They're ETL with a retrieval layer bolted on. The discipline that makes them work is the same discipline that has always made pipelines work: detect change, chunk intelligently, keep indexes fresh.

Video Understanding AI: What Actually Works

Feb 2025

I pointed a video understanding pipeline at 200 hours of meeting recordings. The results taught me more about pipeline design than about meetings.

How I Build Data Pipelines That Actually Survive Production

Apr 2017

Every pipeline I've built at the fintech startup broke at some point. Here's the design approach that made them recoverable instead of catastrophic.