How we assess
Every applicant goes through 5 gates: application, an AI-assisted screen, a live technical assessment scored by a practice lead, a client-fit interview, and reference and credential checks. The questions below come from the live assessment. We score answers on reasoning and trade-offs, not on recalling definitions.
20 technical questions with model-answer hints
Questions 1–10 (role-specific)
Question 1: How do you choose a chunking strategy for long policy documents?
Model-answer hint
structure-aware splits, overlap and metadata, validated by retrieval recall per strategy.
Question 2: When does hybrid search beat pure vector search?
Model-answer hint
exact terms, IDs and rare entities; combine BM25 with embeddings and tune the weighting.
Question 3: What does a re-ranker add, and what does it cost?
Model-answer hint
better precision at top-k against extra latency; measure the gain before keeping it.
Question 4: How do you get reliable structured output from an LLM?
Model-answer hint
JSON schema or function calling, validation, bounded retries, constrained decoding where available.
Question 5: Which RAG evaluation metrics matter most for a customer-facing assistant?
Model-answer hint
faithfulness, answer relevance, context precision and recall, tied to user harm.
Question 6: How do you use LLM-as-judge without fooling yourself?
Model-answer hint
calibrate against human labels, control position and length bias, use a rubric, spot-check.
Question 7: How would you cut token cost by 40% without hurting quality?
Model-answer hint
caching, model routing, prompt compression, tighter retrieval, measured against the eval set.
Question 8: How do you stop indirect prompt injection from retrieved content?
Model-answer hint
treat retrieved text as data, separate instructions, restrict tools, filter outputs.
Question 9: How do you enforce document permissions in retrieval?
Model-answer hint
ACL metadata filters at query time, identity propagation, leakage tests.
Question 10: What do you log for each LLM request, and what must you never log?
Model-answer hint
prompt version, retrieval IDs, latency, cost, feedback; redact personal data and secrets.
Questions 11–20 (AI & Data core)
Question 11: How do you decide between rules, classic ML and an LLM for a problem?
Model-answer hint
start from decision value and error cost; choose the simplest option that meets the metric.
Question 12: How do you detect training-serving skew?
Model-answer hint
shared feature code, logged inference inputs, distribution checks between training and live data.
Question 13: How do you build an evaluation set that predicts production results?
Model-answer hint
representative sampling, slice coverage, leakage checks, labelled with measured agreement, refreshed on a schedule.
Question 14: What goes on a model-health dashboard, and who acts on each signal?
Model-answer hint
quality, drift, latency, cost and the business KPI, each with an owner and threshold.
Question 15: How do you handle personal data in a training pipeline under PDPL or GDPR?
Model-answer hint
minimisation, lawful basis, pseudonymisation, residency, retention limits, DPIA.
Question 16: How do you roll back a model or prompt change that degrades results?
Model-answer hint
versioned registry, canary or shadow release, automated gates, one-step rollback.
Question 17: When does fine-tuning beat retrieval or better prompting?
Model-answer hint
behaviour, format and domain style favour tuning; fresh facts favour retrieval; weigh data and cost.
Question 18: Describe a data-quality incident you caught.
Model-answer hint
how it was detected, root cause, the contract or test added, the alert that now exists.
Question 19: How do you forecast and cap the monthly cost of an AI workload?
Model-answer hint
tokens or GPU hours, caching, batching, model routing, budgets with alerts.
Question 20: Explain a modelling trade-off you made to a non-technical executive.
Model-answer hint
plain language, business impact, the option rejected and why.
Interviewer tip: Ask for a real example after every answer. Strong candidates describe what they built, what broke and what they changed.
Practical task
Given 200 bilingual policy PDFs and 30 sample questions, build a minimal RAG service that returns cited answers, then deliver a short evaluation report covering faithfulness, citation accuracy and the one improvement you would ship next.
How to run it Share the brief 24 hours ahead or run it live. Ask the candidate to talk through trade-offs as they work; reasoning matters more than a finished artefact.
Red flags our practice leads watch for
- Cannot say how quality was measured on a system they shipped.
- Treats the vector database as the whole solution and ignores retrieval tuning.
- Has no answer for prompt injection beyond trusting the model.
- Has never looked at cost per request or latency percentiles.
- Universal flag: Vague answers that never reach a concrete system, number or decision the candidate personally owned.
The scorecard we use
Score each dimension 1–4 (1 = no evidence, 2 = partial, 3 = solid, 4 = exceptional):
| Dimension | Weight | What a 4 looks like |
|---|---|---|
| Technical depth, production judgement and security | 40% | Explains how and why, designs for failure and sensitive data, and knows each tool’s limits |
| Practical task and problem solving | 30% | Delivers a working solution with tests and documented decisions |
| Communication and documentation | 20% | Explains clearly to technical and non-technical audiences, in English or Arabic |
| References and regional context | 10% | References confirm the claimed scope; knows the frameworks and data-residency rules that apply |
Pass bar: weighted average of 3.0 out of 4 or higher, with no dimension below 2. The practice lead approves the final result.