Hallway thumbs-up is not a scorecard.
Teams that shipped RAG pilots on hallway demos are rebuilding around Google’s Gen AI evaluation service on Vertex AI, per Google Cloud documentation and a production-readiness codelab. The shift is measurement-first: retrieval quality scored before generation faithfulness, named rubrics replacing subjective thumbs-up reviews, and repeatable experiment IDs that tie a corpus version to a metric snapshot.
Google documents a two-step client flow: generate responses for a fixed prompt set, then score those responses: plus an EvalTask helper that bundles a table of prompts and answers with a metric list and an experiment name. Large jobs move to asynchronous batch evaluation with results written to Cloud Storage.
This Field Note describes what operators do when they follow that guidance. It rests on Google Cloud evaluation documentation and the Evaluate RAG Systems with Vertex AI codelab listed in Sources. Named customer citation-rate improvements are UNKNOWN.
Job the system was hired to do
Score whether a RAG pipeline retrieves the right context, uses it in generation, and produces answers that are grounded, safe, and instruction-following: before executives see a chat UI.
Enterprise buyers who signed 2024–2025 RAG pilots often framed the job as natural-language Q&A over approved documents. Google’s codelab narrows that contract: a RAG system combines three failure points: retrieval quality, context utilization, and generation quality: and a response can fail at any layer. A model may ignore correct context, or write fluently from wrong chunks.
Google’s evaluation service maps metrics to that decomposition. Predefined rubrics cover question-answering quality, groundedness, instruction following, and safety. The codelab recommends combining those with a small number of custom pointwise rubrics: short evaluator prompts that score relevance or helpfulness on a 1–5 scale. Reference-based evaluation adds semantic similarity when golden answers exist; reference-free evaluation judges groundedness against the prompt’s embedded context.
The system was not hired to win a demo with three SQuAD excerpts. It was hired to produce a scorecard that survives procurement, security review, and the first production incident.
First week
Enable the Vertex AI API, install the evaluation extras for the Vertex AI Python SDK, and authenticate Application Default Credentials. Google documents a Rapid Eval service account provisioned on the first evaluation request, with permission to call prediction endpoints for model-based metrics.
Stand up a Workbench notebook or local Python environment. Build a golden set as a table with two required columns for RAG eval: the full prompt the generator saw (user question plus retrieved context), and the model answer. Google recommends roughly one hundred examples for statistical reliability; the codelab uses a few SQuAD-style rows to teach the pattern. Operators we infer from Google guidance typically start with fifty domain questions where subject-matter owners can name the authoritative passage.
Week one keeps a retrieve-first mindset even when the SDK scores end-to-end prompt–response pairs. Operators log which chunk configuration produced each prompt’s context: chunk size, embedder version, top-k: because groundedness scores on stale or wrong context mislead tuning.
SDK usage belongs in a notebook cell, not in the scorecard narrative. The codelab’s interactive path looks like this:
import vertexai
from vertexai.preview.evaluation import EvalTask
vertexai.init(project="YOUR_PROJECT", location="us-central1")
# DataFrame with columns: prompt, response
eval_dataset = rag_eval_rows
eval_task = EvalTask(
dataset=eval_dataset,
metrics=[
"question_answering_quality",
"groundedness",
"safety",
"instruction_following",
],
experiment="rag-policy-corpus-v3",
)
result = eval_task.evaluate()
Google states evaluation jobs run on the Vertex AI backend and may take several minutes. Results include aggregate scores and per-row evaluator explanations.
Parallel work: wire a Cloud Storage destination if the golden set will exceed interactive limits. Batch evaluation is for when immediate results are not required: poll the long-running operation, then read outputs from the bucket. First-week teams often skip batch until the interactive pipeline works on ten rows.
To compare two RAG configurations, duplicate the dataset for each chunk or retriever setup, run the same metric list twice, and read radar or bar plots plus per-row explanations. The visualization helpers live in the codelab’s notebook utilities; the decision still rests on reading why groundedness failed on a given row.
What broke or was routed around
Prompt column formatting broke metric interpretation. Evaluator models treat the prompt column as the full input the generator saw. Teams that logged only the user question: without the retrieved passage: got groundedness scores that blamed the LLM for retrieval errors. Operators standardized on question, then a clear context block, then the retrieved passage, matching the codelab template.
Fixed chunking broke tables and procedures before eval ever ran. Naive splits produced contexts that could not support faithful answers; quality and groundedness looked like model failures. Teams retuned chunk boundaries, re-embedded, regenerated the prompt column, and re-ran the task. Exact chunk strategies vary by corpus and are not one-size in Google docs.
Model-based metric variance broke executive trust in single-digit score deltas. Rubric metrics use an evaluator model; scores shift with evaluator version and temperature. Google documents pre-building adaptive rubrics and reusing named rubric groups across runs. Operators who skipped rubric versioning could not explain why groundedness rose without a corpus change.
Interactive runs timed out on large golden sets. Teams with hundreds of rows moved to batch evaluation after saving inference outputs to Cloud Storage. Parameters align between interactive and batch paths; the operational difference is polling and storage layout.
Third-party model comparison broke IAM assumptions. Google documents evaluating external providers through an OpenAI-compatible path with an API key in the environment: useful for bake-offs, outside the default Rapid Eval service account. Security review of outbound keys became a gate; timing per tenant is UNKNOWN.
Teams without experiment discipline lost reproducibility. EvalTask logs to a Vertex AI Experiment name; operators who reused generic strings could not tie a scorecard to an embedder hash. They switched to experiment-per-corpus-version naming before sharing numbers outside engineering.
What a person still does
Writes golden questions and confirms reference contexts match authoritative sources. Google’s codelab uses SQuAD passages; enterprise teams need legal-approved policy text. No rubric replaces a domain owner saying this PDF version is current.
Authors custom rubric prompts when predefined metrics miss business criteria: tone for support, regulatory hedging for finance. Google ships example metric names as a catalog; the rubric language is still human-written.
Reads per-row explanations for low-scoring answers. Aggregate radar plots hide single-row catastrophes: the wrong brain region named in the codelab’s Model B example. Qualitative review remains operator work.
Sets pass thresholds and decides whether a two-point groundedness gap blocks launch. Google supplies metrics; the customer sets gates. Published enterprise thresholds are UNKNOWN.
Owns corpus sync and re-embed triggers. Evaluation scores a snapshot; when SharePoint or Cloud Storage sources update, someone must regenerate prompts and re-run interactive or batch evaluation.
Cost / time in operator units
Google’s codelab estimates completing the lab under one dollar in Cloud resources for SQuAD-scale examples. Production golden sets at one hundred prompts across several model-based metrics multiply evaluator inference cost. Google doesn't publish a universal dollars-per-eval figure; cost scales with dataset size, metric count, and evaluator model choice.
Operator time shifts from playground prompt tinkering to table curation and experiment logging. A first ten-row run is hours including SDK setup; a defensible fifty-question golden set from domain experts is often days to weeks. Batch evaluation adds polling overhead but saves notebook babysitting on large jobs.
Infrastructure splits three ways: RAG serving (retrieval plus generation), Vertex AI Experiments storage, and evaluation inference. The Rapid Eval service account adds an IAM review item on first request: usually minutes for admins familiar with Vertex.
Iteration cost is lower than shipping a policy bot whose groundedness was never measured. Exact engineer-hours per groundedness point improvement are UNKNOWN without customer exports.
What they would do next time
Follow Google’s decomposition: fix prompt formatting (question plus context), score groundedness before debating generator model upgrades, and log every run to a named Vertex AI Experiment tied to corpus and embedder versions.
Build the golden set before the executive demo. The codelab’s Model A versus Model B comparison only works because both datasets exist upfront; hallway demos without exported scorecards create commitments the first real run disproves.
Use predefined metrics plus one or two custom rubrics in business language: not ten custom metrics on day one. Metric sprawl inflates evaluator cost without improving decisions.
Plan batch evaluation when the golden set exceeds comfortable interactive runtime. Save frozen inference outputs to Cloud Storage so batch jobs reuse the same answers.
Version rubric groups so evaluator instructions don't drift between runs.
Re-run the same EvalTask after every material chunk, top-k, or reranker change. Generation metrics on a moving retrieval baseline waste comparison history.
When choosing between two generators on identical prompts, run them side by side on the same dataset rather than comparing unrelated notebooks.
Treat third-party model eval as a separate security review path when outbound API keys leave the project.
Named customer before/after groundedness scores are UNKNOWN. Google docs describe method, not a published enterprise case study with raw counts.