Machine Learning Engineer Interview Questions

Machine learning engineer interviews are hard to prepare for because the job title covers two different jobs. One is modelling — framing a problem, choosing an approach, evaluating it honestly. The other is engineering — making that model survive contact with real traffic, real latency budgets, real data pipelines that break at 3am. Most candidates over-prepare for the first and get rejected on the second. The questions below are written the way strong interviewers actually ask them: open-ended production scenarios where the model is already built and something is going wrong, and where the gap between a competent answer and an exceptional one is visible within the first thirty seconds.

What Machine Learning Engineer Interviews Test

Problem framing under ambiguity. Turning a vague business ask into a learning problem with a defined label, a horizon, and a metric someone will actually be held to. Candidates who skip this and jump to model choice lose the round before they start.

Production judgment. Understanding how systems degrade after launch — drift, feedback loops, delayed labels, stale features — and knowing which failure you are looking at before you reach for a fix.

Engineering depth. The coding and system-design bar for ML roles has moved close to the software engineering bar. Serving architecture, feature pipelines, caching, and CI/CD for models are all fair game.

Trade-off reasoning under constraints. Accuracy against latency, cost, interpretability, and time-to-ship. Interviewers are watching how you reason at each fork, not whether you land on one blessed architecture.

Communication with non-ML stakeholders. Explaining why a metric moved, why a model should be killed, or why a rule beats a model — to people who do not care about AUC.

Question 1: The offline win that lost money

You shipped a new ranking model for an e-commerce homepage. Offline it beat the incumbent on every metric you tracked — NDCG up, offline CTR estimate up 12%. Two weeks into a 50/50 A/B test, click-through rate is genuinely up 9% in the treatment arm, but revenue per session is down 3% and average order value is down more than that. Your PM wants to ship it because "engagement is up." Your director wants to know whether the offline evaluation was broken. Walk me through how you find out what happened, what you tell both of them, and what you do with the model.

Why interviewers ask this

This is the single most common real failure in applied ML and it tests whether a candidate can separate three distinct things: a broken evaluation, a correct evaluation of the wrong objective, and a real trade-off the business has to decide on. Weak candidates immediately assume the offline pipeline had a leak and start auditing code. Strong candidates recognise the offline evaluation may be perfectly correct — the model optimised exactly what it was told to optimise, and the objective was wrong.

Example strong answer

"My first move is to establish which of three worlds we're in, because the fix is different in each.

World one: the offline evaluation was wrong. I'd check for the usual suspects — position bias in the logged data, since if I trained on clicks without correcting for where an item was displayed, the model learned to predict slots rather than preference. I'd also check for temporal leakage in the train/test split, and whether the offline CTR estimate used an off-policy correction or just naively replayed logged impressions. But I'd note that this world is unlikely here, because the online CTR did go up 9%. The offline metric predicted the online metric correctly. That's evidence the pipeline worked.

World two — the one I think we're actually in: the model optimised CTR and CTR is a proxy that has come uncoupled from revenue. The specific mechanism I'd look for first is a shift in the price distribution of ranked items. Cheap, high-engagement items — accessories, low-cost impulse buys — have structurally higher click rates than the high-consideration items that carry margin. A pure CTR objective will promote them. I'd segment the treatment arm's top-10 ranked slots by price decile and category and compare against control. If the treatment is systematically surfacing cheaper items, that's the whole story, and it's not a bug.

World three: a distributional or interaction effect — the new model cannibalised search or category browse, so sessions that used to convert through a different path now end on the homepage. I'd check conversion rate by entry path, not just by arm.

On the diagnosis I'd run a per-decile revenue decomposition: sessions × click rate × conversion rate × AOV, treatment versus control. That tells you exactly which term is dragging.

What I'd tell the PM: engagement is up and that's real, but we're trading margin for clicks at roughly [ratio from the decomposition], and CTR was never the thing we're paid on. What I'd tell the director: the offline evaluation was directionally correct, so I don't want to rebuild it — I want to change the objective.

Then the fix. I would not ship as-is and I would not kill the model. I'd retrain with a revenue-weighted objective, or more precisely a multi-objective setup where the label is weighted by expected margin, with a guardrail on the engagement metric so we don't overcorrect into showing only expensive items nobody clicks. Then re-run the test with revenue per session as the primary metric and CTR as a guardrail — the opposite of how we set it up the first time. That reversal is the real lesson, and I'd want it written into how we define success for the next model before we start it."

Follow-up questions

Suppose the revenue drop is only 0.4% and not statistically significant at two weeks, but the CTR gain is significant. Your PM says ship now and monitor. What's your recommendation and what would change your mind?
Now assume you retrain on a revenue-weighted objective and revenue per session goes up 2% but the number of distinct sellers appearing on the homepage drops by 40%. How do you handle that?

Question 2: Degradation with no ground truth

You own a fraud model at a payments company. It scored 0.94 AUC offline and has been live for two weeks. Chargebacks are up 18% month-over-month, but your model dashboard shows precision and recall unchanged — because those are computed on confirmed fraud, and confirmation takes about 45 days. Your risk lead is asking whether the model is broken. You have no labels for the period in question and won't for six weeks. What do you do this week?

Why interviewers ask this

Delayed and partial labels are the normal state in fraud, credit, churn, and healthcare — and this question separates people who have only worked with clean benchmark datasets from people who have operated a model. It also tests whether the candidate understands that a monitoring dashboard computed on stale labels is worse than no dashboard, because it manufactures false confidence.

Example strong answer

"The dashboard isn't wrong, it's answering a question about six weeks ago. So the first thing I'd say to the risk lead is that we currently have no live signal, and fixing that is part of the work.

I'd separate three hypotheses, because they need different responses.

First, input drift from a broken pipeline. An upstream schema change, a null-rate spike, a currency or timezone field that silently changed — these look identical to a behavioural shift in the output but are cheap to rule out. I'd compare live feature distributions against the training distribution for the top 20 features by importance, using population stability index or a KS test, plus a plain null-rate and cardinality check on every feature. I'd do this first because it takes an hour and resolves a meaningful share of incidents.

Second, benign covariate shift — new merchant categories onboarded, a marketing push bringing in a different customer mix. Real drift, but the model may still be well-calibrated on the new population.

Third, adversarial adaptation. Fraudsters found a gap and are exploiting it. This is the dangerous one because the standard reflex — retrain on recent data — actively makes it worse. The recent data is labelled by the model's own decisions, so anything it approved looks legitimate. You retrain on your own blind spot.

To tell them apart without labels, I'd use proxies with short feedback loops. Score distribution shift: if the model is now confidently approving a cluster of transactions that sit in a region of feature space it barely saw in training, that's the adversarial signature. Manual review queue outcomes — the analysts have opinions within days, not 45. Dispute and customer-complaint rates by merchant category and BIN range. And I'd check whether the chargeback increase is concentrated or spread; concentration in one merchant category or one geography points to a specific exploit rather than general decay.

Operationally, I'd route a small holdout — 1 to 2% of traffic — to full manual review starting immediately. That's expensive, but it buys us unbiased labels on a continuous basis and it's the thing I'd argue hardest for, because without it we're structurally blind every time this happens.

Short term, if the drift is concentrated, I'd ship a rules-based tripwire on the affected segment this week — lower the approval threshold or add a velocity rule. Rules are worse than models on average and much better than models when you need to act in two days.

Medium term: retrain with sampling that accounts for the feedback loop, add the score-distribution and PSI checks as automated alerts, and change the monitoring so that label-lagged metrics are clearly marked as lagged on the dashboard. The failure here was as much a monitoring design failure as a model failure."

Follow-up questions

Your holdout is expensive — the risk lead approves 0.5% instead of 2%. How does that change what you can detect, and how long until you'd have a usable signal?
Six weeks later the labels arrive and the model's AUC on that period was actually fine — 0.93. The chargebacks came from transactions the model correctly flagged but the ops team auto-approved due to a queue backlog. What do you change?

Question 3: The latency budget

Your team's ranking model is a gradient-boosted ensemble plus a transformer-based text encoder. p99 latency for the full request is 340ms. Product has committed to a 100ms p99 budget for the endpoint because the feature is moving above the fold. You measured that a naive distillation to a smaller model costs you 4 points of recall@10, which product says is unacceptable — they'll accept at most 1 point. Design your approach.

Why interviewers ask this

This is a systems question wearing an ML costume. Candidates who only know modelling reach for quantisation and distillation and stop. Candidates with production experience know most of the 340ms is usually not the model, and that the biggest wins come from restructuring when computation happens, not from making the model smaller.

Example strong answer

"Before I touch the model I want the latency broken down, because I'd bet a meaningful chunk of the 340ms isn't inference. I'd profile feature fetch, network hops, deserialisation, the encoder forward pass, the GBDT scoring, and post-processing separately. In systems I've seen, feature retrieval from an online store is frequently the largest single term, and it's much easier to fix than model architecture.

Assuming the breakdown is roughly: 120ms feature fetch, 150ms text encoder, 40ms GBDT, 30ms overhead — here's the plan in order of return on effort.

The text encoder is the obvious target, and the key insight is that most of what it encodes doesn't change per request. Item text embeddings are static between catalogue updates. I'd precompute them offline in a batch job and serve them from a cache or the feature store, which removes the encoder from the request path entirely for the item side. If there's a query-side encoder that genuinely needs to run live, that's a much smaller input and a candidate for a distilled or quantised version where the accuracy cost is far lower than distilling the whole system.

That suggests a two-stage architecture: a cheap retrieval stage using precomputed embeddings and approximate nearest neighbour to get from the full catalogue down to a few hundred candidates, then the expensive ranker applied only to that shortlist. Going from scoring thousands of items to scoring 200 is usually a bigger latency win than any amount of model compression, and the recall cost is controllable — you can tune the candidate set size against the recall target directly.

On feature fetch: batch the calls, colocate the online store with the serving layer, and cache user-level features with a short TTL. Many user features are stable over a session.

Only after that would I look at model-level compression, and I'd go in this order: INT8 quantisation of the encoder, which is usually close to lossless; then structured pruning; then distillation as a last resort, since that's the one that cost 4 points. I'd also check whether the 4-point loss came from distilling on the original training set rather than on the teacher's soft outputs over live traffic distribution — that's a common and fixable mistake.

How I'd validate: measure recall@10 at each step against the current production model on a held-out slice, and set a stop rule — if cumulative recall loss hits 0.8 points I stop compressing and go find latency elsewhere. And I'd hold a shadow deployment running both for a week before the switch so we're comparing p99 under real traffic, not a load test.

If after all that we're at 130ms and 1.5 points down, I'd bring product the actual curve rather than a yes or no — here's recall as a function of latency budget, pick a point. That conversation goes much better with a curve than with an opinion."

Follow-up questions

Precomputing item embeddings means the catalogue is stale between batch runs. A merchant updates a product title and it doesn't rank correctly for six hours. How do you handle that?
The two-stage approach passes the latency test but you notice long-tail items almost never make the candidate set anymore. What's the diagnosis and what do you do?

Question 4: Training-serving skew

A churn model performs well in offline evaluation and in a shadow deployment reading from the same offline pipeline. When it goes live reading from the online feature store, its precision drops by roughly a third. Nobody changed the model. Find the bug — walk me through your reasoning, not just a list of possibilities.

Why interviewers ask this

Training-serving skew is the most common silent failure in production ML and it almost never announces itself as an error. The question tests systematic debugging: whether a candidate can narrow the space efficiently rather than listing every possible cause, and whether they understand the specific ways offline and online feature computation diverge.

Example strong answer

"The shadow deployment detail is the most useful thing in the question. The model behaves correctly when fed offline-computed features and incorrectly when fed online-computed features. That isolates the fault to the feature layer — the model, the label definition, and the evaluation code are all cleared. So I'd stop considering them.

My first concrete step is a direct comparison. I'd take a few thousand live requests, log the exact feature vector the online store served, then recompute those same features offline for the same entities at the same timestamps, and diff them feature by feature. That single artefact usually names the bug in under an hour. I'd rank the diff by feature importance so I'm looking at the ones that matter.

The categories I'd expect to find, roughly in order of frequency:

Time-window semantics. Offline, a feature like 'logins in the last 30 days' is computed as of the label date with the full window available. Online, it's computed as of now, possibly against a window that's still filling, or against an aggregation job that ran six hours ago. A partially-filled window looks like a low-activity user, which for a churn model is a strong signal in exactly the wrong direction.

Label leakage that only exists offline. If any feature was computed after the event it's meant to predict — a support ticket count that includes the cancellation ticket, for example — the offline model has been reading the answer. Online it can't, and performance collapses. The signature is a model that was suspiciously good offline. Given the drop is a third of precision, I'd take this seriously.

Default and missing-value handling. The offline job fills nulls with a column mean; the online store returns a literal zero or a sentinel. For tree models this silently reroutes examples down a different branch.

Encoding drift. Category vocabularies built at training time versus a live encoder that hashes unseen categories into an unknown bucket. If new categories appeared since training, a growing share of traffic falls into that bucket.

Entity resolution. Joining on a user ID that means something slightly different in the two systems — device ID versus account ID, or anonymous sessions that get merged offline and not online.

What I'd do after finding it: fix the immediate bug, then fix the class of bug. The structural answer is a single feature definition used by both paths — a shared transformation library or a feature store that computes point-in-time-correct training data from the same code that serves online. Plus an automated skew check in CI that samples live serving vectors and diffs them against offline recomputation, alerting when any feature's distribution diverges beyond a threshold. If we only fix the one feature, we'll be back here next quarter with a different one."

Follow-up questions

The diff shows every feature matching except one aggregate that's off by a small amount on 8% of rows. Would you consider that the cause of a one-third precision drop? How would you confirm?
How would you design point-in-time-correct training data generation for a feature that depends on a slowly-changing dimension, like a customer's plan tier?

Question 5: The segment the model fails

Your loan pre-approval model is 91% accurate overall. During a pre-launch review, a data scientist on another team finds it's 68% accurate for self-employed applicants, who are 7% of volume. The model doesn't use employment type as a feature. Launch is in nine days and the business case assumed this quarter. What's your recommendation?

Why interviewers ask this

This tests three things simultaneously: technical understanding of why removing a protected or sensitive attribute doesn't remove the disparity, the judgment to know that an aggregate metric can conceal a segment-level failure serious enough to block a launch, and the ability to hold a position under commercial pressure. It's also a domain where the answer has legal weight, which good candidates flag without being asked.

Example strong answer

"My recommendation is that we don't launch to self-employed applicants in nine days. We can launch to everyone else if the segment analysis holds up. Let me explain how I get there.

First, the fact that employment type isn't a feature is not a defence — it's a red flag that we didn't check. The model has proxies: income variance across months, deposit irregularity, business-category merchant codes, gaps in payroll deposits. Any of those reconstruct self-employment. Removing the label while keeping the proxies gives you the same disparity with less ability to measure it.

Second, I'd want to know what kind of error it is before deciding anything. 68% accuracy could be false rejections of creditworthy applicants — a fairness and revenue problem — or false approvals of applicants who default, which is a credit-loss problem. These have opposite fixes. So the immediate task is a confusion matrix by segment, plus calibration curves: is the model's predicted probability of default actually the observed rate for this group at each score band? A model can be well-calibrated within a group and still have lower accuracy there, and that's a very different situation from a model that is systematically overconfident on the segment.

Third, why is it happening? Almost always representation. If self-employed applicants are 7% of volume they may be 3% of the training set after filtering, and their features have genuinely different structure — income is lumpy, not a stable monthly figure. The model has learned a pattern that assumes salaried cash flow. That's a modelling problem with real fixes: segment-specific features that measure income stability over a longer horizon, reweighting or oversampling in training, or a separate model for that population with a routing layer.

On the commercial conversation: I would not frame this as ML wanting more time. I'd frame it as a launch scoping decision with a quantified downside. If this is false rejections, we're declining a group we should be approving, and in a regulated lending context that carries disparate impact exposure — I'd want compliance in the room before launch, not after. If it's false approvals, here's the expected credit loss on 7% of volume at the observed error rate. Both of those are numbers a business leader can act on. 'The model is unfair' is not.

The path I'd propose: launch on the 93% of volume where we've validated performance, keep self-employed applicants on the existing manual process for one more cycle, and commit to a specific fix with a date. That preserves most of the business case and doesn't ship a known defect into a regulated decision.

And separately, a process change: segment-level performance reporting should be part of the standard model review, not something a colleague catches nine days before launch. I'd propose a required slice analysis — by volume-weighted segments and any regulated attribute — as a launch gate."

Follow-up questions

The business says self-employed applicants are the fastest-growing segment and excluding them defeats the purpose. Does that change your recommendation, and how?
You build a separate model for self-employed applicants and it reaches 88%. Now the two models disagree on borderline applicants who could be classified either way. How do you handle routing?

Question 6: Defending the GPU bill

Your team runs a deep model for on-site personalisation. Inference costs about $38,000 a month in GPU spend. The last clean A/B test showed it lifts conversion by 0.4% relative to a much cheaper gradient-boosted baseline. Finance has flagged the line item. Your VP asks you directly: do we keep it?

Why interviewers ask this

Senior ML engineers are expected to reason about their own cost centre, and many candidates have never been asked to. The question also tests intellectual honesty — whether someone will argue to keep their own model alive regardless of the numbers, or actually do the arithmetic and be willing to recommend killing it.

Example strong answer

"I'd want to answer this with a number, not a position, so let me define what I need. 0.4% relative lift on conversion is meaningless until I know the base. If we do $50M a quarter through this surface, 0.4% relative is roughly $200K a quarter, which is $67K a month against a $38K cost — worth keeping, though not by as much as anyone would like. If we do $8M a quarter, it's $32K a quarter against $114K of cost and we should turn it off today. So my first answer to the VP is: give me a day, and I'll bring you the contribution margin, not an opinion.

There are three things I'd check that usually change the answer.

Is the lift real and durable? A 0.4% relative lift is small. I'd check the test's power — was it run long enough to detect 0.4% reliably, or is the confidence interval straddling zero? I'd also check whether it's decayed since launch; novelty effects on personalisation are common and a lift measured at launch often isn't there six months later. If we haven't re-tested, that's the cheapest thing to do first: a holdback of 5% of traffic on the baseline model, run for a month. That costs almost nothing and it settles the question.

Is the lift uniform? Often it's concentrated — new users, or one high-value segment, or one surface. If 80% of the lift comes from 20% of traffic, the right answer isn't keep-or-kill, it's route the deep model to the traffic where it pays and serve the baseline everywhere else. That can cut cost by more than half while retaining most of the benefit, and it's usually the highest-value move available.

Is the $38K actually irreducible? I'd look at utilisation before I look at architecture. In my experience GPU spend on inference is frequently 30-50% idle capacity from over-provisioning for peak. Batching, autoscaling, moving to smaller instance types, caching predictions for users whose features haven't changed since the last request, or precomputing scores offline for the head of the distribution and serving deep inference only for the tail — any of these can take a large bite out of the bill without touching model quality. I'd also check whether we're running fp32 where INT8 would do.

What I'd tell the VP: here's the monthly contribution, here's my confidence in it, here are three cost reductions I can land in a sprint, and here's the traffic-routing option. If after all that it's still negative, I'd recommend we shut it off and I'd say so plainly. A model that doesn't pay for itself is a liability regardless of who built it — and being the person who says that is how you get trusted the next time you ask for infrastructure budget."

Follow-up questions

You run the holdback and the lift is now 0.1%, not significant. But the deep model powers three other surfaces that were never tested independently. How do you proceed?
Finance asks you to commit to a cost-per-prediction target for next year. What would you need to know before agreeing to one?

Question 7: Fine-tune or prompt

You need to classify inbound customer support tickets into 40 categories to route them. You have 900 hand-labelled examples, unevenly distributed — the top 5 categories cover 60% of them and 12 categories have fewer than 10 examples each. Volume is about 4,000 tickets a day and expected to triple. An off-the-shelf hosted model with a well-written prompt gets roughly 82% accuracy in your quick test. What do you build, and how do you decide?

Why interviewers ask this

This is now a routine build decision and it tests whether a candidate reasons from constraints — data volume, class imbalance, cost per call, latency, maintenance burden — or from fashion. It also surfaces whether they think about the long-tail classes, which is where these systems actually fail.

Example strong answer

"With 900 examples across 40 classes I don't have enough data to fine-tune a classifier from scratch and expect it to handle the tail. Twelve classes with under 10 examples will not be learned reliably by any supervised approach at that volume. So the honest starting point is that the prompted approach is the right first system, and the interesting question is what I build around it.

Here's how I'd frame the decision. 82% accuracy over 40 classes is a reasonable baseline, but aggregate accuracy is hiding everything that matters. I'd first break it down per class and, more importantly, ask what each error costs. Misrouting a billing question to the general queue costs a few minutes. Misrouting a security incident or a regulatory complaint costs a lot more. So I'd want per-class precision and recall, weighted by business cost, before I optimise anything.

My proposed build: start with the prompted model plus retrieval — few-shot examples selected per ticket by nearest-neighbour over an embedding of the labelled set. That directly addresses the tail problem, because a rare-class ticket retrieves its own rare examples rather than competing with the head classes for space in a static prompt. In practice this is usually a several-point improvement over static few-shot and it costs almost nothing to build.

Second, a confidence-gated fallback. Rather than forcing a decision on every ticket, route low-confidence predictions to a human queue and log the human's label. At 4,000 tickets a day, even a 10% deferral rate generates 400 labelled examples daily. Within a month I have 12,000 labels, which is a completely different data situation — at that point fine-tuning a small model becomes viable and the cost economics flip. So I'd design the first system explicitly as a labelling engine for the second one.

On cost and latency: at 4,000/day scaling to 12,000, per-call API cost matters but isn't dominant. What I'd watch is the vendor dependency — pricing changes, model deprecations, and silent behaviour changes on version updates. I'd pin the model version, keep a frozen evaluation set of a few hundred tickets, and run it on every version change so we detect regressions rather than discovering them through complaint volume.

When I'd switch to a fine-tuned small model: once I have roughly 100+ examples per class for the classes that matter, or once per-call cost crosses the engineering cost of maintaining our own. A fine-tuned encoder for 40-class classification is cheap to serve and much faster, and it removes the vendor risk. I'd likely end up with a hybrid — the small model handles the head classes it's confident on, and the larger prompted model handles the tail and the ambiguous cases.

What I would not do is spend three weeks fine-tuning on 900 examples to try to beat 82%. That's the version of this project that ships late and worse."

Follow-up questions

After a month your deferral queue has produced 12,000 labels — but they're biased toward cases the first system found hard. What problem does that create for the fine-tuned model, and how do you correct for it?
Support leadership wants a single accuracy number for a QBR. What number do you give them and what caveat comes with it?

Question 8: Launching with no data

Your company is expanding a delivery ETA prediction system into a new country. In the existing market the model uses two years of completed-trip data. In the new market you have zero completed trips on launch day, a partner courier network you don't control, and a business that has publicly committed to a launch date. Design the approach for the first 90 days.

Why interviewers ask this

Cold start is where ML meets commercial reality, and it tests whether a candidate can sequence a system over time rather than designing one static architecture. It also tests whether they know when not to use ML — the correct answer for day one is usually not a model.

Example strong answer

"I'd think of this as three phases with explicit graduation criteria between them, and I'd write those criteria down before launch so we're not arguing about readiness later.

Phase one, day zero to roughly week three. No model. A deterministic estimate: routing-engine travel time plus a fixed handling buffer, calibrated using whatever ground truth we can buy or borrow — a mapping provider's traffic estimates, and if possible a two-week pre-launch pilot with a small number of couriers doing real routes. I'd deliberately bias the estimate long. An ETA that's 8 minutes pessimistic annoys people less than one that's 4 minutes optimistic, and in a new market where trust hasn't been established yet, that asymmetry is worth real accuracy. I'd also widen the displayed ETA into a range rather than a point estimate, which buys tolerance while we're genuinely uncertain.

The important engineering work in phase one is logging. Every field the mature model will eventually need — courier location traces, pickup timestamps, handoff delays, address geocoding confidence, venue prep time — instrumented from the first order. The single most common failure in a market launch is discovering in month four that you have three months of data missing the two features that mattered.

Phase two, roughly weeks three to eight. Transfer from the existing market. I'd take the mature model and evaluate which features transfer. Structural relationships — distance to duration, time-of-day effects, the shape of restaurant prep delays — often transfer well. Absolute levels rarely do, because road networks, traffic patterns, and building access differ. So I'd fine-tune the existing model on incoming local data with the source-market model as an initialisation, and I'd add a market indicator plus interaction terms. I'd also consider a simple residual model: use the existing model's prediction as a feature and learn the local correction, which needs far less data than learning the whole function.

Graduation criterion into phase two: enough completed trips to have coverage across the main city zones and time-of-day buckets, not just a raw count. I'd define it as something like 200 completed trips per zone-hour bucket for the top 80% of volume.

Phase three, week eight onward. Local model trained primarily on local data, with the transferred model as a fallback for zones that are still thin — new suburbs, newly onboarded venue types. This routing-by-data-density approach means we never have a cliff where a zone gets a bad prediction because it's underrepresented.

Throughout, I'd hold one thing fixed: a live accuracy dashboard segmented by zone and time-of-day, with an automatic rollback to the deterministic estimate if any segment's error exceeds a threshold. In a new market the failure mode I'm most worried about isn't a slightly wrong model — it's a confidently wrong model in one neighbourhood destroying trust in a city we're trying to establish."

Follow-up questions

Six weeks in, the local data is dominated by one dense central district because that's where demand started. How do you avoid a model that's excellent downtown and poor everywhere else?
The business wants to display a tighter ETA range because a competitor does. What evidence would you need before agreeing?

Question 9: Shipping a model like software

Walk me through your CI/CD pipeline for a model that retrains weekly and serves 50 million predictions a day. Specifically: what runs automatically, what requires a human, how a new version gets to production, and what happens at 2am when it's wrong.

Why interviewers ask this

This separates candidates who have shipped models from candidates who have trained them. There's no clever insight to find — it's a question about whether the person has operational habits. Interviewers listen for rollback, for automated gates, for what is deliberately not automated, and for whether the candidate has thought about the difference between a bad deploy and a bad model.

Example strong answer

"I'd describe it as four gates, and the design principle is that a model version is an artefact that gets promoted, never edited in place.

Gate one, data validation, runs before training starts. Schema checks, null rates, cardinality, range checks per feature, and a distribution comparison against the previous training window. If the incoming data fails, training doesn't run and someone gets paged. This is the cheapest gate and it catches the most incidents — upstream pipeline breakages are far more common than model problems.

Gate two, training and offline evaluation. Fully automated. Trains on a fixed window, evaluates on a held-out temporal slice — never a random split, because random splits leak future information into the past. The gate isn't just aggregate metric versus the incumbent; it's a slice-level comparison. A new version has to be no worse than the incumbent on every defined segment beyond a tolerance, not just better on average. Aggregate-only gates are how you ship a model that's better overall and much worse for one country. I'd also run a fixed set of behavioural tests — known inputs with expected directional outputs, the ML equivalent of unit tests.

Gate three, shadow deployment. The new version scores live traffic in parallel without its predictions being used, for at least 24 hours. Compare prediction distributions against the incumbent, check p99 latency under real load, and diff feature vectors between the training pipeline and the serving path to catch skew. Automated pass/fail on latency and distribution divergence.

Gate four, staged rollout. 1% of traffic, then 10%, then 50%, then full, with automatic rollback triggers at each stage on business metrics and technical metrics — not just error rate. The rollback has to be a config change that takes effect in seconds, not a redeploy. I'd keep the previous two versions warm and routable at all times.

What requires a human: promotion past the first traffic stage, any change to the feature set or the label definition, and any override of a failed gate. Weekly automatic retrains on an unchanged pipeline can go through without a human up to 1%. Anything structural gets review.

At 2am when it's wrong: the on-call runbook has one first action, which is roll back to the last known-good version. Diagnose afterwards. The most common mistake I've seen is an engineer trying to understand the problem while it's still serving traffic. Every model version, its training data snapshot ID, its evaluation report, and its config are stored in a registry so the rollback target is unambiguous and the post-mortem has something to read.

The other thing I'd insist on: the alert has to distinguish 'the deploy is broken' from 'the model is degrading'. Those page different people and have different responses, and conflating them wastes the first twenty minutes of every incident."

Follow-up questions

Your weekly retrain passes every gate for eleven weeks, then week twelve fails the slice check on one small segment. Do you ship the previous version for another week or investigate first? What's the cost of each choice?
How do you handle a rollback when the new model has already influenced the data — for example, a recommender whose outputs shape what gets clicked and therefore what you train on next week?

Question 10: Arguing against your own model

A senior stakeholder wants an ML model to decide which customers get a retention discount. You've looked at the data and believe a three-rule heuristic would capture most of the value, be live in a week instead of a quarter, and be explainable to the support team who'll have to apply it. The stakeholder has told the leadership team an AI project is underway. How do you handle it?

Why interviewers ask this

Seniority in ML is partly the judgment to not build things. This question tests whether a candidate can hold a technical position in a political situation without either capitulating or being difficult about it. Interviewers are listening for whether the candidate makes the stakeholder's problem their own rather than treating them as an obstacle.

Example strong answer

"I'd start by separating what they want from what they said. They said 'ML model.' What they want is a defensible, effective way to allocate retention spend, and probably to be seen leading a modern initiative. Those aren't in conflict with my recommendation if I frame it right — but they will be if I open with 'we don't need ML for this.'

So I wouldn't open with that. I'd open by agreeing with the goal and proposing a way to get there faster, with the model as phase two rather than the thing we're not doing.

Concretely, my pitch: give me a week to ship the heuristic as a baseline, instrumented and A/B tested against no-discount and against the current manual process. That does three things. It puts retention value on the board this month instead of next quarter. It gives us a measured baseline, which we need anyway — without it, we'll never be able to say what the ML model was worth, and a model that can't demonstrate incremental value over a rule is a model that gets killed in the next cost review. And it generates the treatment-assignment data a causal model would need, because the real question here isn't who will churn, it's who will change their behaviour because of the discount. Those are different problems and the second one needs experimental variation to solve.

That last point is the technical substance of my position and I'd lead with it in the room. A churn-prediction model tells you who's leaving. Some of those people are leaving no matter what, and discounting them is pure margin loss. Some would have stayed anyway, and discounting them is also margin loss. The model you actually want is an uplift model, and you cannot train one without randomised treatment data. So the rule-based phase isn't a detour — it's how we generate the data the sophisticated version requires. That reframes the sequence as necessary rather than as me pushing back.

On the explainability point, which I'd raise second: support agents have to apply this and customers will ask why they didn't get an offer. Three rules can be explained in a sentence. A gradient-boosted model's output cannot, and in a discount context that becomes a fairness complaint eventually.

If the stakeholder still insists on shipping a model first, I'd make sure my recommendation is written down with the reasoning, agree to the plan, and build it well. Being right in a meeting is worth much less than being trusted over a year, and the fastest way to lose that is to be the engineer who won't build what was asked. But I'd fight hard for the baseline test running in parallel, because that's the thing that tells us the truth either way."

Follow-up questions

The heuristic ships and retains 80% of the value the stakeholder projected for the model. They now want to cancel the ML phase. Do you argue for it, and on what grounds?
How would you design the randomisation for the discount experiment when finance is unwilling to give discounts to a random control group?

Preparation tip

The habit that separates candidates who get machine learning engineer offers from candidates who get to the final round and stop there is this: they diagnose before they solve. Every question above is designed so that the obvious fix — retrain, distill, add features, build the model — is available in the first ten seconds, and taking it is the wrong move. Strong candidates spend the first third of their answer narrowing down which failure they're actually looking at, and they say out loud what would change their mind. Practise talking through the diagnosis step. It feels slower and it is the thing interviewers are scoring.