What do AI engineer interviews ask most?

How you build features on language models: retrieval and RAG, agents and tools, evaluation, cost and speed, and safety. Coding rounds are usually ordinary Python.

Do I need to know LangChain or LangGraph?

Knowing one helps, and some ads name them. Interviews test whether you understand what such a framework does underneath: the steps, the limits, the search.

How deep does the machine learning theory go?

For product AI engineer roles, the basics: embeddings, attention in plain words, test data, simple measures. Research and platform roles go deeper.

How should I prepare in a week?

Build or revisit one small retrieval project with a test set, say the two worked answers out loud, and prepare one failure story for each section.

Are these answers tied to one provider?

No. Where a detail depends on a provider or a version it is marked, so you can check the current documentation.

AI engineer interview questions, with worked answers

Eighteen questions that AI engineer interviews ask again and again. For each: what the interviewer is checking, a short answer, a full answer in plain words, and a likely follow-up.

The cvscored teamPublished 25 Oct 2026 · Updated 25 Oct 2026

Contents

How to use these questions
What an AI engineer is expected to know
Retrieval and RAG
Agents and tools
Evaluation
Models, cost and speed
Safety, privacy and the law
Basics and system design
Your CV for these questions
Key takeaways
Frequently asked questions
More on the AI engineer role

How to use these questions

Read the short answer and say it out loud. That is your first thirty seconds. Then read the full answer and pick two or three points to add. Interviewers want trade-offs, limits and how you would measure the result, not a list of tools.

A note that says “depends on the provider” or “depends on the release” marks a detail that can change with a provider or a release. Check the current documentation before you quote it.

What an AI engineer is expected to know

Python for production: clean, tested code and a small web service, for example with FastAPI.
TypeScript: many AI engineers also build the part of the feature people see in the browser.
Language model APIs (LLM): sending requests, letting the model call tools, getting answers in a fixed format, and keeping an eye on tokens (the units the model reads and bills by). depends on the provider
RAG (retrieval-augmented generation): the model answers from passages you look up first, not from memory. That covers splitting documents, search, ranking and citations, often with a vector database.
Agents: a model that decides which tool to use next, step by step, within limits.
Evals: a fixed set of test questions, scored the same way every time, so you can tell whether a change made things better or worse.
Shipping: Git, CI/CD (tests run on every change), Docker, one cloud (Azure or AWS), sometimes Kubernetes.
Models: when a good prompt is enough, when to look things up, and when to fine-tune (train a model further on your own examples).
Safety and law: prompt injection (text that tries to give the model new orders), GDPR, and the basics of the EU AI Act.
Basics: machine learning terms such as training and test data, and enough statistics to tell a real improvement from luck.

Retrieval and RAG

How do you set up agentic search and retrieval in a chatbot?

What they check: that you start simple, know each part of search and retrieval, put hard limits on the agent, keep access rights out of the model's hands, and can measure the result.

Short answer: "I start with plain retrieval: make the question clear, search by keywords and by meaning at the same time, merge the results, put the best ones first, and answer with citations. I only let an agent plan several searches when one search is not enough. The agent works under a step limit, a token budget and a time limit. Every claim must point to a passage we found, and we check that. If something breaks, there is a fallback, and the bot says what it could not find instead of guessing."

The full answer, in eight steps:

Do I need an agent at all? Most questions need one lookup, like "what is the notice period?". That is fast, cheap and easy to test. An agent (the model planning several searches) is slower and dearer. It pays off for questions with several parts, questions that need several sources, or questions that need a follow-up such as "which contract do you mean?". A small, cheap model in front can send each question the right way.
Make the question clear. Turn "and the second one?" into a full question using the chat history. Split a question with two parts into two searches. Pull out filters such as product, date or language, and pass them to the search as filters, not as words.
Give the model a few clear tools. For example "search the help pages", "open one section" and "look up an order". Each tool says when to use it and returns short results with an id. Access rights are checked inside each tool, from who is signed in. The model never decides who may see what.
Split the documents well. Chunk them (cut them into pieces) along their headings, not every few hundred characters. Put the heading at the top of each chunk, so it is found more easily. Keep tables whole. Languages with long compound words, such as German, need care: the words must be split for keyword search.
Search two ways at once: hybrid search. Keyword search (BM25, which scores passages by the words they share with the question) finds exact names, codes and rare words. Vector search (which compares meanings) finds the same idea in other words. Merge the two lists so that a passage near the top of either list ends up near the top. A common way to do this is reciprocal rank fusion, which only looks at the positions, so you never have to compare two different kinds of score.
Rerank. A reranker is a second, more careful model that reads the question and each of the top results together and puts them in a better order. Keep the best five to eight. If even the best one scores low, treat it as "nothing found". That is what lets the bot say "I don't know" honestly.
Cite and check. Number the passages and ask for a citation after every fact. Afterwards, check that each citation points to a real passage and that the passage really says it. Remove or flag sentences that are not backed up. Some APIs have built-in citations; the check stays. depends on the provider
Plan for failure. If vector search is down, use keyword search alone. If the reranker is down, keep the merged order. If the model fails, retry after a short wait, then use a second model. If nothing is found, try one broader search, then say so. Text found in a document is never an order for the model, and tools that change anything need the person's OK.

The limits, in plain words: a step limit (at most five or six tool calls), a token budget for the whole run, a time limit, and a stop if the agent repeats the same search. When a limit is reached, the bot answers with what it has and says what is missing. Every step is logged, so you can see later what happened.

How you would measure it: how often the right passage is in the top five (recall), whether every claim is backed by a passage (faithfulness), whether the bot declines when there is no answer, how long answers take, and what each answer costs.

A weak answer: "I'd use LangChain and a vector database." No limits, no access rights, no measurement, no plan for "not found".

Follow-up: "The right passage is only in the top five for six questions in ten. What do you do?" Sort the misses first. Is the passage missing from the index (a splitting problem)? Is it found but ranked too low (a ranking problem)? Or was the question rewritten badly? Fix the biggest group and measure again.

How do you choose how to split documents into chunks?

What they check: that you measure instead of guessing, and that you follow the document's structure.

Short answer: "I cut along headings and paragraphs, keep the heading with each chunk, and try two or three sizes on my test questions. I pick the size that finds the right passage most often. If small chunks lose context, I give the model the whole section around the chunk it found."

The full answer: Different documents want different chunks: one question and answer per chunk for an FAQ, one clause for a contract, a whole table for a table.

A weak answer: "512 tokens with some overlap", with no reason. Follow-up: "Answers miss a condition that comes two paragraphs later." Give the model the whole section, or cut by clause.

Keyword search, vector search, or both? When do you add a reranker?

What they check: that you know what each kind of search is good at.

Short answer: "Both. Keyword search for exact names and codes, vector search for the same meaning in other words, merged into one list. I add a reranker when the right passage is in the top fifty but not in the top five. My test questions show me that."

The full answer: Vector search alone misses order numbers and product codes. Keyword search alone misses "cancel" when the text says "terminate". A reranker costs a little time on every question, so only rerank the top thirty to fifty.

A weak answer: "Vector search is always better." Follow-up: "Which embedding model?" One that handles all the languages of your content, chosen by testing on your own questions. An embedding model turns text into numbers that capture its meaning. depends on the provider

How do you make the model cite its sources, and check that the citations are real?

What they check: that you do not trust a citation just because it is there.

Short answer: "I number the passages, ask for a citation after every fact, and check afterwards that each cited passage really says it. Sentences without support are removed or flagged."

The full answer: First check that the cited number exists at all. A short exact quote is the easiest to check. For the rest, a small second model can judge whether the passage supports the sentence. Measure that checker on examples too.

A weak answer: "The prompt tells it to cite sources." Follow-up: "The check removes too much." Often the passage is right but too short. Give the checker the whole section.

Agents and tools

How do you design the tools an agent can call?

What they check: that you know the model only sees each tool's name, description and result, so those are what you design.

Short answer: "A few tools that each do one thing, with names and descriptions that say when to use them, checked inputs, short results, and error messages the model can act on."

The full answer:

Names that cannot be confused: "search orders" and "get one order", not "search" and "find".
A bad input gets a clear message back, such as "an order number has 8 digits", so the model can try again.
Tools that only read are kept apart from tools that change things. A change needs the person's OK.
Test each tool on its own like any API, with unit tests, before the model uses it.

A weak answer: "I give the model access to our database." Follow-up: "The model picks the wrong tool." Read the logs: usually two descriptions sound alike. Merge the tools or sharpen the wording, and add those cases to the test questions.

How do you stop an agent from looping, and cap its cost and time?

What they check: hard limits, and a sensible answer when a limit is reached.

Short answer: "A step limit, a token budget, a time limit and a timeout on every tool, plus a stop when it repeats itself. When a limit is reached, it answers with what it has and says what is missing."

The full answer: Tell the agent how much budget is left, so it plans within it. Log the cost of every run and get an alert when a run costs far more than usual. If many runs hit the limit, fix the tools or the plan rather than raising the limit.

A weak answer: "We set a high limit and hope." Follow-up: "What does the person see when the limit is hit?" A useful partial answer and an offer to hand over to a person.

How do you build a harness that is robust and covers every step?

What they check: that the code around the model is solid, and that you can test every step, not only the final answer.

Short answer: "The harness is the code that runs the model: it prepares the request, calls the model and the tools, keeps track of the conversation and enforces the limits. I make each step small and testable, give every outside call a timeout and a retry, log every step, and test each step on its own as well as the whole run."

The full answer:

Small steps: rewrite the question, search, rank, act, write, check. Each can be tested and replaced on its own.
Outside calls: a timeout and a retry after a short wait for every model and tool call. Anything that changes data is protected so a retry cannot do it twice.
Answers in a fixed format: check each reply against the format you expect. If it is wrong, ask once more with the error, then fall back.
Saved state: a half-finished task can be picked up again after a crash.
Tests for every step: saved model replies make tests fast, free and the same every time. Also test what happens when a tool is slow, broken or returns rubbish.

A weak answer: "We check the final answers by hand." Follow-up: "A tool changes the format of its replies." The format check in the tests fails at once, before any real user sees a broken answer.

Evaluation

What steps does evaluating an LLM feature need, like probes for evals?

What they check: that you test an AI feature like any other software: test sets, checks you can repeat, a gate before release and budgets. And that you know how an AI judge can be wrong.

Short answer: "I decide what good means for each step and for the whole answer, including cost and speed. I build a set of real test questions, with groups for the hard cases, and small probes that each test one behaviour. Every change runs against them. I use exact checks where I can and an AI judge only where I must, and I check the judge against people. A release stops if quality drops by more than chance, if any critical case breaks, or if cost or speed goes over budget."

The full answer, in eight steps:

Decide what good means. For search: is the right passage near the top? For the answer: is every claim backed by a passage, does it answer the question, is it in the right language, does it decline when it should? For the whole feature: does the task get done, how long does it take, and what does it cost?
Build the eval set. Start with one to three hundred real questions: from support tickets, from experts, and some generated from the documents and checked by a person. For each, write down where the answer is, what it must say, what it must not say, and whether there is an answer at all. Group them: easy, several parts, no answer, tricky, and each language you support. Keep some aside that you never tune on.
Add probes. A probe is a small test for one behaviour, so a failure tells you exactly what broke. For example: the answer is not in the documents (does it decline?), a document contains a hidden order (is it ignored?), two documents disagree (is the newer one used?), a tool fails (does the agent recover?), personal data is in the context (does it leak?).
Test every step, not only the end. Run each step on its own with saved inputs, then the whole chain, so a failure points to one step. Saved model replies keep the everyday test runs cheap and repeatable. Run against the live model on a schedule, or when the model or the prompt changes, with a spending cap. Note the versions of everything with every result.
Score with exact checks first. Use a judge (a second model that grades answers) only for open questions such as "does it answer the question?", with simple pass-or-fail points rather than a score out of ten.
Guard against a flaky judge. A judge can give a different grade each time. Fix the judge model's version. Run it three times and take the majority. Have people grade fifty to a hundred answers, and trust the judge only if it mostly agrees with them. Swap the order when it compares two answers, because judges tend to prefer the first one. Prefer a judge from another model family, because models tend to like their own writing. depends on the provider
Regression checks before release. A regression is something that used to work and now does not. Compare the new results with the last release, question by question. Count what went from pass to fail and from fail to pass: nine against three can be luck, twenty against three is not. Critical questions must all pass, and the no-answer group may never get worse, even when the average goes up.
Budgets. Agree a cost per hundred answers and a time that most answers must stay under. Measure both on every run and stop a change that breaks them, even if quality improves. After release, watch feedback, handovers to people, "not found" answers, speed and cost, and add real failures to the eval set every week.

A weak answer: "We look at some answers and they seem fine." Or one judge score with nothing to check it against.

Follow-up: "The judge says quality went up, but people complain more. Why?" The judge may like long or confident answers, the eval set may lack the new kinds of questions, or the group that got worse is small in the test but large in real use. Check the judge against people again, and compare the test groups with what people really ask.

How do you use an LLM as a judge without being misled by it?

What they check: that you know the judge can be wrong in steady ways.

Short answer: "Simple pass-or-fail questions, the passage and a good answer to compare with, a fixed judge model, several runs, and a regular check against people."

The full answer: Judges are poor at checking facts they cannot see, so always give them the passage. In everyday test runs, judge a sample; before a release, judge everything.

A weak answer: "The judge gives each answer a score out of ten." Follow-up: "The judge and people disagree on one group of questions." Look at those cases, fix the grading points, and keep people grading that group until they agree.

What do you test before release, and what do you watch in production?

What they check: that you think about the feature after launch too.

Short answer: "Before release: the eval gate, the probes, and a load test for speed and cost. After release: feedback, handovers to people, 'not found' answers, speed, cost and errors, with alerts, and a person reading a sample of conversations every week."

The full answer: Release to a small share of users first and roll back on a bad signal. Log enough to replay a case without keeping personal data you do not need. Model providers update models, so fix the version where you can. depends on the provider

A weak answer: "We release and wait for complaints." Follow-up: "Answers got slower overnight with no change on our side." Check whether the provider changed the model or is under load, and have a fallback model ready.

Models, cost and speed

Prompting, retrieval or fine-tuning: when do you use each?

What they check: that you pick the simplest thing that works.

Short answer: "A good prompt first. Retrieval when the answer depends on facts the model does not know or that change. Fine-tuning when I need a fixed style or format, or one narrow task done cheaply, and I have good examples to train on."

The full answer: Retrieval adds knowledge and keeps it current. Fine-tuning teaches behaviour, and is a poor way to add facts. LoRA is a cheap way to fine-tune: instead of changing the whole model, you train a small add-on. A small fine-tuned model can replace a large one for a narrow task, if your test questions show the quality holds. Tools: PyTorch and Hugging Face, or the provider's own service. depends on the provider

A weak answer: "We fine-tune it on our documents so it knows them." Follow-up: "The documents change every week." Then retrieval, because a fine-tuned model would be out of date.

How do you cut cost and waiting time?

What they check: that you measure first and do not trade quality for speed without knowing.

Short answer: "I measure where the time and tokens go, then send less text, reuse answers, use a small model for easy questions, and show the answer as it is written."

The full answer:

Shorter prompts and fewer, shorter passages.
Caching: reuse answers to repeated questions, and the provider's prompt caching for a long fixed start of the prompt. depends on the provider
Routing: a small model for easy questions and a large one for hard ones.
Streaming: people see the first words at once.
Batch work nobody waits for, often at a lower price. depends on the provider

Every change runs the eval set, because a cheaper answer that is worse is not a saving.

A weak answer: "We switch to the cheapest model." Follow-up: "How do you know quality held?" The eval set, run before and after.

How do you get answers in an exact format every time?

What they check: that you do not trust the model's output blindly.

Short answer: "I use the API's structured output where it exists, check every answer against the format in my own code, and ask once more with the error if it is wrong."

The full answer: Keep the format small and simple, with fixed choices where possible. Count the failures, so you know if they rise. depends on the provider

A weak answer: "I ask nicely for JSON in the prompt." Follow-up: "It fails one time in a hundred." The check catches it, the retry fixes most, and the rest fall back to a safe answer.

You run an open model on your own servers. What decides the hardware and the speed?

What they check: that you can reason about memory, speed and cost.

Short answer: "Memory first: the model itself, plus the memory each running conversation needs. Then a serving tool that handles many requests at once. Then I compare the cost with an API at our real traffic."

The full answer: Quantisation stores the model in less precise numbers to save memory and money, but it can cost quality, so test it. A serving tool such as vLLM handles many requests together, which makes it much faster than one at a time. depends on the release Run it in Docker on Kubernetes with graphics cards, and scale with the waiting requests.

A weak answer: "Self-hosting is always cheaper." Follow-up: "Traffic is low at night." Scale down, or use an API for the quiet hours.

Safety, privacy and the law

How do you defend against prompt injection in documents and tool results?

What they check: that you limit what the model can do, instead of hoping it obeys.

Short answer: "Text the model reads is data, never orders. The model gets read-only tools by default, anything that changes things needs the person's OK, and access rights are checked in the tools, not by the model."

The full answer: Do not let the model send data out, for example in a link, unless the address is on an allowlist. Add injected text to your probes. The OWASP Top 10 for LLM applications is a good checklist.

A weak answer: "We tell the model to ignore bad instructions." Follow-up: "A document asks the model to email a customer list." It cannot: it has no tool that sends email.

Personal data in an AI feature: what do you do about GDPR?

What they check: that you know where the data goes and how long it stays.

Short answer: "Send only what the task needs, know where the provider processes and keeps it, keep logs short and locked down, and make deletion work everywhere, including the search index."

The full answer: The EU AI Act sorts uses by risk. Most chatbots must say they are an AI. Uses such as hiring or credit are high risk and need much more. Know which group your feature is in. depends on the provider

A weak answer: "The provider handles GDPR." Follow-up: "A customer asks to be deleted." Their data goes from the records, the logs and the search index.

Basics and system design

What is an embedding, and what does attention do, in plain words?

What they check: that you understand the basics well enough to explain them.

Short answer: "An embedding turns text into a list of numbers, so that texts with similar meaning end up close together. Attention lets each word in a model look at the other words and take in what matters, which is how the model uses context."

The full answer: Expect a follow-up on machine learning basics: keep training and test data apart, watch for data that would not exist at the time of a real prediction, and judge rare cases by how many you catch, not by overall accuracy.

A weak answer: a long list of model names with no explanation. Follow-up: "Why compare embeddings by angle, not by distance?" The direction carries the meaning; the length mostly does not.

Design a support chatbot for an insurer, in two languages.

What they check: that you can put the parts together, with numbers, limits and a plan for failure.

Short answer: "I would start with plain retrieval over the policy documents, add a careful agent for questions about the person's own contract, check every answer, hand over to staff when unsure, and measure it all with a test set in both languages."

The full answer:

Ask first: how many questions a day, which data, what a wrong answer costs, what must go to a person.
Data: policy documents searched both ways, with each language's long words handled; customer data only through read-only tools that check who is asking.
Answers: with citations, checked, in the person's language, with a handover when nothing backs up an answer.
Tests: a set of questions per language, with groups for policy, own contract, no answer and injected text; a gate before each release.
Numbers and law: a cost per answer and a waiting time agreed up front; say it is an AI; keep personal data in the EU.

A weak answer: starting with the agent and the newest framework. Follow-up: "What do you build first?" Plain retrieval and the test set; the agent and any fine-tuning come later.

Your CV for these questions

Every answer above is easier when your CV already says it, because the interviewer asks about what is on it. Put one line under each job for each of these, in the words the ads use:

Skill	Weak line	Strong line, with proof
RAG	Built RAG pipelines.	Hybrid retrieval over 40,000 pages; right page in the top five for 83 of 100 test questions, up from 61.
Evals	LLM evaluation.	300-question eval set with probes and a release gate; caught 3 regressions before release.
Agents	Agentic workflows.	Order agent with 3 tools and a step limit; handovers to staff down by a third.
Cost	Optimised costs.	Routing and caching cut the cost per answer by 45%; test scores held.
Kubernetes	Docker, Kubernetes.	Ran an open model on Kubernetes that scales with demand; answers under 2 seconds at peak.

Then score your CV against a real job: you will see which of these the job needs and which your CV does not prove yet.

Key takeaways

Start simple; let an agent plan several searches only where one search is not enough.
Hybrid search, reranking, chunking along headings and checked citations are the heart of retrieval answers.
Every agent needs a step limit, a token budget, a time limit and a fallback.
Evals are a release gate: eval sets, probes, guards against a flaky judge, regression checks and budgets.
Back every answer with a CV line that has a number in it.

Frequently asked questions

What do AI engineer interviews ask most?: How you build features on language models: retrieval and RAG, agents and tools, evaluation, cost and speed, and safety. Coding rounds are usually ordinary Python.
Do I need to know LangChain or LangGraph?: Knowing one helps, and some ads name them. Interviews test whether you understand what such a framework does underneath: the steps, the limits, the search.
How deep does the machine learning theory go?: For product AI engineer roles, the basics: embeddings, attention in plain words, test data, simple measures. Research and platform roles go deeper.
How should I prepare in a week?: Build or revisit one small retrieval project with a test set, say the two worked answers out loud, and prepare one failure story for each section.
Are these answers tied to one provider?: No. Where a detail depends on a provider or a version it is marked, so you can check the current documentation.

How to use these questions

What an AI engineer is expected to know

Retrieval and RAG

How do you set up agentic search and retrieval in a chatbot?

How do you choose how to split documents into chunks?

Keyword search, vector search, or both? When do you add a reranker?

How do you make the model cite its sources, and check that the citations are real?

Agents and tools

How do you design the tools an agent can call?

How do you stop an agent from looping, and cap its cost and time?

How do you build a harness that is robust and covers every step?

Evaluation

What steps does evaluating an LLM feature need, like probes for evals?

How do you use an LLM as a judge without being misled by it?

What do you test before release, and what do you watch in production?

Models, cost and speed

Prompting, retrieval or fine-tuning: when do you use each?

How do you cut cost and waiting time?

How do you get answers in an exact format every time?

You run an open model on your own servers. What decides the hardware and the speed?

Safety, privacy and the law

How do you defend against prompt injection in documents and tool results?

Personal data in an AI feature: what do you do about GDPR?

Basics and system design

What is an embedding, and what does attention do, in plain words?

Design a support chatbot for an insurer, in two languages.

Your CV for these questions

Key takeaways

Frequently asked questions

More on the AI engineer role