Quick take
Most ML in production is data plumbing and ops. Master those and the model is the easy part.
I’ve been building NLP and sentiment analysis systems at the fintech startup for a while now. We process financial news at scale – classifying articles, extracting sentiment, figuring out which stories actually move markets. And the thing that surprises most backend engineers when they first touch ML? The model is maybe 10% of the work. The rest is everything you already know how to do, just with new failure modes.
This post is for backend engineers getting pulled into ML projects. Not data scientists. Not researchers. People who build services, own uptime, and wonder why the model team keeps asking for “just one more pipeline.”
The Three Kinds of ML You’ll Actually See
Supervised learning dominates production. Labeled data in, predictions out. Classification or regression. At the fintech startup, most of our models are supervised – we feed in financial articles with known sentiment labels, and the model learns to score new ones. Simple concept, messy execution.
Unsupervised learning shows up for clustering and anomaly detection. We use it for grouping related news stories. Useful, but less common in typical backend work.
Reinforcement learning? You’ll probably never touch it. Skip it.
The Lifecycle Is Boring (That’s the Point)
Collect data. Clean it. Build features. Train a model. Evaluate. Deploy. Monitor. Repeat forever.
Backend engineers own the first step, the last two, and large chunks of everything in between. If you think your job ends at “expose the model behind an API,” you’re going to have a bad time. Training-serving skew alone has cost us weeks of debugging at the fintech startup. The model works perfectly in the notebook. Performs terribly in production. Every. Single. Time. Until you get disciplined about it.
Data Is the Whole Game
I can’t stress this enough. The model is a commodity. The data is the product.
At the fintech startup we ingest thousands of financial articles per day. Each one needs to be cleaned, normalized, deduped, and tagged before a model ever sees it. A sentiment model trained on messy data doesn’t produce “noisy predictions.” It produces wrong predictions that look confident. That’s worse.
Here’s what a basic feature builder looks like:
# Example: build features for a single user
def build_user_features(user_id, db, now):
user = db.query(User).get(user_id)
orders = db.query(Order).filter_by(user_id=user_id).all()
if not user:
return None
total_orders = len(orders)
total_spent = sum(o.total for o in orders)
last_order_at = max((o.created_at for o in orders), default=None)
return {
"user_id": user_id,
"account_age_days": (now - user.created_at).days,
"total_orders": total_orders,
"total_spent": total_spent,
"avg_order_value": total_spent / total_orders if total_orders else 0,
"days_since_last_order": (now - last_order_at).days if last_order_at else None,
}
Straightforward. But now imagine this runs during training with one set of data transformations and during inference with a slightly different one. Your model is now subtly broken and nobody knows. We learned this the hard way – our sentiment scores drifted for two weeks before anyone noticed the feature computation differed between batch training and real-time serving.
Feature stores exist to solve exactly this. Compute features once, store them with point-in-time correctness, serve them identically to training and inference. Whether you build your own or use an off-the-shelf solution doesn’t matter. What matters is that both paths see the same numbers.
Serving: Pick Your Pain
Two patterns. Batch and real-time.
Batch works when latency doesn’t matter. Pre-compute predictions nightly, store them, serve from a cache. We batch-score article relevance this way at the fintech startup – run it overnight, have results ready for morning markets. Simple and reliable.
Real-time is for interactive features. User types a query, you need a prediction in 50ms. This is where things get operationally interesting. You’re now running model inference on the hot path. Latency budgets are real. The model is large. Feature computation has external dependencies. Any of those can spike your p99 and ruin someone’s day.
Embedding the model in your application process is the fastest path to production and the fastest path to regret. Model updates now require app deploys. Memory usage goes up. You can’t scale them independently.
A dedicated model service is more work upfront but lets you version, deploy, and scale the model separately. We went this route for our real-time NLP scoring and haven’t looked back.
Managed services? Fine if you accept the vendor lock-in and latency constraints. Just go in with your eyes open.
For latency, the playbook is boring but effective: cache aggressively, precompute what you can, use lighter models for hot paths, and always have a fallback. If the sentiment model is down, we return a neutral score and flag it. Users get a degraded experience instead of an error.
Deploy Like It’s a Service, Because It’s
Version your models. models/sentiment/v7 is fine. What’s not fine is deploying a model without knowing what data it was trained on or what its evaluation metrics looked like.
Treat model deploys like service deploys. Canary first. A/B test if you can. Feature flags if you can’t. We roll out new sentiment models to 5% of traffic, compare prediction distributions against the old model, and only promote if the numbers look sane. No heroics.
Rollback needs to be trivial. If v7 starts producing garbage, switching back to v6 should take seconds, not a meeting.
Monitoring Is Where Backend Engineers Shine
This is your wheelhouse. Latency, error rates, throughput – you already track these. For ML you add a few more:
Prediction distribution. If your sentiment model suddenly thinks everything is positive, something broke. Track the histogram.
Confidence scores. If average confidence drops, the model is seeing data it wasn’t trained for. At the fintech startup, a confidence drop on our article classifier was our first signal that a new type of financial instrument was showing up in the news. The model hadn’t seen crypto coverage before. Neither had we, honestly.
Feature drift. If the input distributions shift from what the model saw during training, predictions will degrade. You won’t always have ground truth labels right away – in financial sentiment, the “correct” label might not be clear for days or weeks. So you need proxy signals.
Working With Data Scientists
I’ll be direct. The handoff between data scientists and backend engineers is where most ML projects die.
Data scientists need clean, documented data and a clear path to production. Backend engineers need a contract: what goes in, what comes out, how fast, and what “wrong” looks like. If you don’t agree on these things explicitly, you’ll agree on them implicitly through production incidents.
The best setup I’ve seen is shared ownership. The data scientist owns the model logic. The backend engineer owns the serving infrastructure. Both own the pipeline. Nobody gets to throw something over a wall and walk away.
In Short
ML in production is mostly engineering. Data pipelines, feature consistency, serving infrastructure, monitoring, and deployment discipline. The model itself is usually the part that changes least. If you get the plumbing right – consistent features, clear latency budgets, solid monitoring for drift – you’ll ship more reliable ML than most teams running exotic architectures on shaky foundations.
That’s the job. It’s not glamorous. It’s useful.