Data Engineering Patterns: Batch vs. CDC vs. Streaming
A comparison of data ingestion patterns from building the fintech startup's financial data pipelines, plus when each one actually makes sense.
Data Engineering
Definition
Data Engineering coverage in this archive spans 3 posts from Apr 2017 to May 2021 and centers on data correctness and operability under real production constraints. The strongest adjacent threads are analytics, data pipelines, and streaming. Recurring title motifs include data, engineering, patterns, and batch.
Working claims
- The common theme is that schema, ownership, and query shape drive most downstream outcomes.
- The consistent theme from 2017 to 2021 is disciplined execution over hype cycles.
- This topic repeatedly intersects with analytics, data pipelines, and streaming, so design choices here rarely stand alone.
How to apply this
- Define freshness, correctness, and latency targets before choosing storage or pipeline patterns.
- Start with the newest post to calibrate current constraints, then backtrack to older entries for first principles.
- When boundary questions appear, cross-read analytics and data pipelines before committing implementation details.
Where teams get burned
- Scaling pipelines before locking down source-of-truth and reconciliation behavior.
- Optimizing single queries while ignoring data model drift and access patterns.
- Applying guidance from 2017 to 2021 without revisiting assumptions as context changed.
Suggested reading path
- Start here (current state): Data Engineering Patterns: Batch vs. CDC vs. Streaming
- Then read (operating middle): GDPR for Engineers: What We Actually Built at a Fintech Startup
- Finish with (foundational context): How I Build Data Pipelines That Actually Survive Production
Related posts
- Data Engineering Patterns: Batch vs. CDC vs. Streaming
- GDPR for Engineers: What We Actually Built at a Fintech Startup
- How I Build Data Pipelines That Actually Survive Production
References
GDPR for Engineers: What We Actually Built at a Fintech Startup
Eleven days before the GDPR deadline, here's the technical implementation work we did at the fintech startup — data mapping, consent storage, erasure pipelines, and the backup problem nobody warns you about.
How I Build Data Pipelines That Actually Survive Production
Every pipeline I've built at the fintech startup broke at some point. Here's the design approach that made them recoverable instead of catastrophic.