AI/LLM Testing
Build & own an LLM eval harness in CI (Part A) + AI-assisted testing is now table-stakes (Part B). Capability-first vocabulary. STOP before training/fine-tuning, transformer internals, MLOps.
- Semantic similarity (embeddings + cosine) vs BLEU/ROUGE/exact-match
Compare outputs by meaning via embedding vectors and cosine distance instead of surface string overlap.
'How would you assert two differently-worded answers mean the same thing?'
- Why traditional pass/fail assertions break for AI
Exact-match assertions reject valid output variants and accept lucky-but-wrong answers, so AI needs scored/threshold evaluation instead.
'How do you test a feature whose output is different every run?'
- AI failure taxonomy (hallucination, drift, prompt sensitivity, bias, toxicity, safety, task-failure)
The catalog of ways an AI feature fails that a human tester must design tests around.
'What kinds of failures do AI features have that normal software doesn't?'
- Golden set / evaluation dataset creation, curation, freshness
A curated labeled reference set (input, expected/reference, context, label) that is the foundation every eval scores against.
'How do you build a golden dataset? How big? How do you keep it from going stale?'
- Annotation strategy + inter-annotator agreement
How reference labels are produced and how you measure whether human annotators agree.
'Who decides the expected output, and how do you trust those labels?'
- Synthetic data generation to scale eval sets
Use an LLM to generate additional labeled eval cases when hand-labeled data is scarce.
'How do you scale an eval set beyond what you can label by hand?'
- Judge failure modes (position, verbosity, self-preference, inconsistency)
Systematic biases that make an LLM judge unreliable — order preference, longer=better, favoring its own family, run-to-run variance.
'LLM-as-a-judge — how do you know the judge itself is right?'
- LLM-as-judge basics: single-output vs pairwise, reference-based vs referenceless, rubrics
Using an LLM with a written rubric to score outputs that have no exact correct string.
'How do you score an answer when there's no single correct string?'
- Validating the judge against human labels (Cohen's kappa, cross-family check)
Proving the judge is trustworthy by measuring its agreement with a small human-labeled set before relying on it.
'How would you prove your automated judge agrees with a human?'
- Precision/recall/F1 + confusion matrix for classifier/guardrail quality
Measuring a binary/multi-class AI component (or guardrail) with a confusion matrix instead of pass/fail.
'How do you measure a guardrail's quality?'
- G-Eval, DAG metrics, deterministic metrics, strict_mode hard gates
The spectrum from LLM-scored custom criteria (G-Eval) to deterministic checks, and when to bolt on a binary pass gate.
'When would you use G-Eval vs a deterministic metric?'
- Confidence-threshold / calibration testing
Verifying the model emits the right decision AT the right confidence level, and that low-confidence cases route correctly.
'How do you test that the model is confident when it should be?'
- RAG core four: faithfulness, answer relevancy, context precision, context recall
Scoring a retrieval-augmented feature by separating retrieval quality from generation quality.
'How would you detect hallucination in a RAG chatbot?'
- HallucinationMetric (ground-truth context) vs FaithfulnessMetric (retrieval_context)
Two distinct hallucination checks — one against the truth you provide, one against what the system actually retrieved.
'What's the difference between faithfulness and hallucination scoring?'
- Vector DB / embedding retrieval testing (dedup, top-k, relevancy)
Testing the retrieval layer directly — embedding similarity, top-k relevance, and dedup correctness.
'How do you test the retrieval half of a RAG system?'
- Input vs output guardrails; placement, fail-open vs fail-closed
Where safety checks sit (before the model vs after) and the default behavior when a guardrail itself errors.
'Where do you put guardrails and what happens when one fails?'
- Prompt-injection, jailbreak, PII, toxicity, secret-leakage, format/JSON-schema tests
The concrete adversarial and safety test categories every LLM-app QA must exercise.
'What adversarial cases do you throw at an LLM feature?'
- Explainability, citations, audit trails, human-in-the-loop validation
Testing that AI decisions are traceable, cite sources, and route to humans where required (regulated/enterprise need).
'How do you test an explainable/auditable AI decision system?'
- Red-teaming: adversarial corpora, attack mutation, attack-success-rate as gated metric
Systematically attacking the model with mutated attacks and tracking attack-success-rate as a release gate.
'How would you red-team an LLM feature and gate on the result?'
- Semantic regression / consistency gating on a pinned corpus
Blocking a merge when semantic scores on a fixed golden corpus regress vs the last known-good baseline.
'How do you stop a prompt/model change from silently degrading quality?'
- Non-determinism handling: thresholds, tolerances, distributions, semantic equivalence
Testing techniques that accept a band of valid outputs instead of one exact string.
'How do you make a non-deterministic test stable and meaningful?'
- Model/prompt drift detection over time (offline + online)
Detecting that quality has degraded after deployment due to model/prompt/data changes.
'How do you catch model/prompt drift after deployment?'
- DeepEval hands-on (pytest-native, assert_test, CI gate)
The pytest-native eval framework an SDET runs as unit tests in CI — the primary vehicle for his moat.
'Walk me through your LLM eval suite in CI.'
- Awareness of LangSmith, TruLens, HELM, MLflow, W&B
Naming the tracking/observability/benchmark tools in the ecosystem without deep use.
'What else is in the LLM eval/observability ecosystem?'
- Ragas hands-on for RAG (complementary to DeepEval)
The RAG-metrics standard, best in a notebook/data-pipeline model vs DeepEval's pytest/CI model.
'What do you use for RAG evaluation specifically?'
- promptfoo for prompt regression + red-teaming (YAML/CLI, multi-model)
Prompt-level A/B and regression across providers plus built-in adversarial red-teaming.
'How do you regression-test a prompt change across models?'
- Structured-output / schema-validity testing (Pydantic, Instructor, JSON schema)
Asserting the model returns well-formed, schema-conformant structured output every time.
'How do you guarantee the model returns valid JSON your app can parse?'
- Framework selection by integration model (DeepEval=pytest/CI, Ragas=notebook, promptfoo=prompt A/B+red-team)
Choosing the eval tool by how it fits the team's workflow, not by metric-count — a senior-sounding decision frame.
'Ragas vs DeepEval vs promptfoo — how do you choose?'
- Offline eval gate in CI (block release on regression)
Running evals against the golden set on every change and failing the build on quality regression — the single most-asked capability.
'Design an eval gate for an LLM feature going to prod.'
- Online production monitoring: live-trace sampling, referenceless metrics, A/B
Measuring quality on live traffic (no ground truth) via referenceless metrics + A/B, not just offline.
'How do you know quality in production, not just in CI?'
- Agentic workflow testing: tool-call correctness, task success, multi-step trajectories
Validating multi-step agents — did it call the right tools, in the right order, and complete the task.
'How do you test an AI agent that plans and calls tools?'
- Cross-tenant isolation / multi-tenant AI safety testing
Ensuring one tenant's data/context never leaks into another tenant's AI responses.
'How do you test tenant isolation in a multi-tenant AI product?'
- Known limits of AI-assisted testing (hallucinated assertions, test explosion, human review)
Where AI-generated tests go wrong and why a human must still own final judgment.
'What are the risks of letting AI write your tests?'
- Using Claude / Cursor / Copilot to generate + maintain test suites
Daily 'AI-first' workflow using coding assistants to write and refactor Playwright/Pytest tests faster — now near table-stakes.
'How do you use AI in your day-to-day testing?'
- Self-healing / AI test-gen tools awareness (Healenium, Mabl, Testim, ZeroStep, AgentQL, Diffblue)
Naming the commercial/OSS AI test-gen and self-heal tools and their tradeoffs.
'What AI QA tools have you evaluated?'
- MCP (Model Context Protocol) as the LLM-to-tool interface
The standard that lets an LLM drive a browser/tools via structured accessibility-tree snapshots (browser_click, browser_snapshot...).
'What is MCP and how does it fit AI-driven testing?'
- Playwright Agents: Planner, Generator, Healer (on Playwright MCP)
Three cooperating agents that plan, generate, and self-heal Playwright tests via MCP + accessibility tree.
'Have you used Playwright's AI agents / self-healing?'
- Prompt engineering for test artifacts (rubrics, test data, structured output)
Writing effective prompts to produce eval rubrics, edge-case test data, and schema-bound output.
'How do you prompt an LLM to help build tests?'
- Statistical rigor for eval (metric choice on skewed data, rank correlation, judge-human agreement)
Choosing statistically sound metrics and agreement measures rather than eyeballing averages.
'How do you know your eval numbers are statistically meaningful?'
- Eval-as-a-service: shared gates + quality dashboards as infra
Thinking of evals as reusable platform infrastructure (gates + dashboards) for the whole org, not one-off scripts.
'How would you make eval a capability the whole team relies on?'
- ML / computer-vision model output validation (confidence thresholds, regression across model versions)
Validating classic ML/CV model outputs (detection/classification/STT-TTS) across versions — adjacent to LLM eval, real in our pool.
'How would you test a CV/ML model's outputs and version upgrades?'
- LLM cost / latency / token-usage testing
Treating token cost and response latency as testable, gate-able quality attributes of an LLM feature.
'How do you keep an LLM feature fast and cheap enough?'
- Training/fine-tuning models, LoRA/PEFT, RLHF mechanics
Building/adapting models — an ML-engineer job, not an SDET's.
(not asked of SDET-2; deflect to 'I validate model behavior, I don't train them')
- Transformer internals / attention math / writing architectures
The math of how models work internally.
(not asked)
- Deriving new academic eval metrics / implementing benchmark papers / deep MLOps serving / vector-DB internals tuning
Research-grade metric design and model-serving/infra optimization.
(not asked)
