by Format-3

Share article

{
"@graph": [
{
"@type": "Article",
"image": {
"url": "https://media.babylovegrowth.ai/blog-images/organization-13731/1792888444765_Evaluation-sheets-arranged-for-answer-review.jpeg",
"@type": "ImageObject",
"caption": "Evaluation sheets arranged for answer review"
},
"author": {
"url": "https://format-3.co",
"name": "Format-3",
"@type": "Organization"
},
"headline": "LLM Evaluation Metrics: What to Measure and What to Distrust",
"publisher": {
"url": "https://format-3.co",
"name": "Format-3",
"@type": "Organization"
},
"inLanguage": "en-GB",
"description": "A practical guide to LLM evaluation metrics: which ones measure quality, where BLEU and LLM judges mislead, and how to build a test set that reflects real use.",
"dateModified": "2026-10-26T00:00:00.000Z",
"datePublished": "2026-10-26T00:00:00.000Z"
},
{
"@type": "BreadcrumbList",
"itemListElement": [
{
"item": "https://format-3.co",
"name": "Format-3",
"@type": "ListItem",
"position": 1
},
{
"item": "https://format-3.co/llm-evaluation-metrics",
"name": "LLM Evaluation Metrics: What to Measure and What to Distrust",
"@type": "ListItem",
"position": 2
}
]
}
],
"@context": "https://schema.org"
}

LLM Evaluation Metrics: What to Measure and What to Distrust

LLM evaluation metrics are structured tests that score a model’s output against the thing you actually need from it: a correct answer, a grounded summary, a safe reply, a response fast enough to feel instant. The hard part is not finding metrics. There are dozens. The hard part is choosing the three or four that describe your product, and resisting the comfort of a single number that looks scientific and tells you very little.

This guide covers the families of metrics, where each one breaks, how to build a benchmark that survives contact with real users, and where human judgement still has to sit.

Format-3

Build better AI-powered digital products

Format-3 brings strategy, design, and engineering together to create and scale digital products, including experiences shaped by AI and emerging technologies.

Explore Format-3

TL;DR

• Metrics fall into a few families: statistical overlap, embedding or entailment-based similarity, model-based (LLM-as-a-judge), and operational measures such as latency and cost. No single family is enough.
• Generic scores like BLEU or perplexity say little about whether your application works. Write task-specific tests that mirror production traffic.
• LLM judges scale well but carry position, verbosity and self-enhancement bias. Calibrate them against humans.
• A trustworthy benchmark has a representative dataset, documented prompts and parameters, contamination checks and a refresh schedule.
• Treat evaluation as a continuous practice, not a launch gate.

Table of Contents

• Why a leaderboard score is not a product decision
• The main families of LLM evaluation metrics
• A quick map: which metric for which question
• Where the familiar metrics mislead
• How to evaluate an LLM, step by step
• Making LLM benchmarking reliable
• Where humans still belong
• Our perspective: the metric is a design decision
• Putting this to work with Format-3
• FAQ
• Sources

Why a leaderboard score is not a product decision

Imagine two teams shipping the same support assistant. One picks the model at the top of a public leaderboard. The other writes sixty real customer questions, defines what a good answer looks like, and tests three models against them. The second team often ships a “lower-ranked” model, because it was measured on the work that matters.

That is the quiet provocation at the centre of this topic: public benchmarks answer the question “how capable is this model in general?” Your users are asking something narrower and harder: “does this work for me, today, on my problem?”

OpenAI’s own guidance frames it the same way. Evaluations are described as structured tests that help you assess accuracy, performance and reliability despite the nondeterminism of generative systems, and the same guide recommends task-specific tests that mirror production traffic rather than treating evals as a way to compare models in isolation.

Stakes also vary by sector. A 2024 ACM survey of LLM evaluation stresses that safety and reliability are paramount when models serve sensitive fields such as healthcare and finance. A typo-prone marketing draft and a clinical summary should not share an evaluation standard.

The main families of LLM evaluation metrics

It helps to sort metrics by how they compute a score, because the computation determines what they can and cannot see.

Statistical and overlap metrics

These compare the model’s output with a reference text using word or n-gram overlap. BLEU and ROUGE are the classics, and exact match and F1 sit in the same family for short factual answers. They are cheap, deterministic and fast, which makes them useful as regression checks. They are also blind to meaning: a correct answer phrased differently scores badly, and a fluent wrong answer that borrows the reference’s vocabulary scores well.

The ACM survey groups automatic metrics by what they measure: accuracy (such as exact match and F1), calibration, fairness and robustness. That grouping is a useful reminder that “quality” is several questions, not one.

Semantic and entailment-based metrics

Embedding similarity (BERTScore is the best-known example) compares meaning rather than surface words. Entailment-based metrics go further for factual consistency: methods such as SummaC and FactCC treat consistency as a classification task, asking whether the output entails or contradicts a source premise. They are well suited to summarisation and retrieval-grounded answers, where the source text is available to check against.

Model-based scorers (LLM-as-a-judge)

Here a second model grades the first against a rubric. Microsoft’s generative AI playbook distinguishes reference-free evaluators (fluency, coherence) from reference-based ones (quality, similarity) and notes both their scalability and their potential for positional or verbosity bias. Judges can grade nuance that overlap metrics cannot, which is why most modern evaluation stacks lean on them. The price is a new thing to validate.

Operational metrics

Quality is not only about words. Platform documentation such as DataRobot’s metrics reference lists operational measures alongside quality and safety ones: latency in seconds, cost based on token pricing, and token counts for inputs, outputs and retrieved text. A model that is 3% better on accuracy but twice as slow and three times as costly may be the worse product choice.

A quick map: which metric for which question

What you want to know: Does the answer match a known correct one? | Metric family: Statistical | Typical examples: Exact match, F1 | Main weakness: Penalises valid rephrasing

What you want to know: Is the summary close in meaning to a reference? | Metric family: Semantic | Typical examples: Embedding similarity, BERTScore | Main weakness: Can reward plausible but wrong text

What you want to know: Is the output faithful to the source? | Metric family: Entailment | Typical examples: SummaC, FactCC | Main weakness: Needs a clear source document

What you want to know: Is it coherent, helpful, on-tone? | Metric family: Model-based | Typical examples: Rubric-scored LLM judge | Main weakness: Judge bias, needs calibration

What you want to know: Is it safe and fair? | Metric family: Safety and fairness | Typical examples: Toxicity checks, demographic parity | Main weakness: Needs curated adversarial cases

What you want to know: Is it usable in production? | Metric family: Operational | Typical examples: Latency, token cost | Main weakness: Says nothing about correctness

Notice what the table implies: your metric list is a portfolio. The research behind a 2025 practical guide to evaluating LLM-reliant systems cautions against relying on single metrics and argues for a balanced scorecard, such as term-overlap measures like ROUGE alongside semantic similarity and natural language inference measures.

Where the familiar metrics mislead

BLEU, ROUGE and perplexity

Perplexity measures how well a model predicts the next word. It is meaningful for language modelling research and close to meaningless for “is this answer useful?” OpenAI cautions against relying solely on generic metrics such as perplexity or BLEU because metric-based evaluations can miss nuance. If your dashboard leads with these, ask what decision they would change.

LLM judges

A judge model is not neutral. It may prefer the first answer it sees, prefer longer answers, or favour text that resembles its own style. Microsoft points out that self-enhancement bias and limited mathematical reasoning make calibration, including human-in-the-loop calibration, necessary. The AI Evaluation Playbook adds that judge methods such as direct prompting or chain-of-thought are flexible but can perform poorly in low-resource language settings. If your users write in more than English, test the judge in those languages too.

A practical routine: take 50 to 100 outputs, have two people grade them, run the judge on the same set, and look at where it disagrees with humans. Fix the rubric until the disagreement is explainable. Only then let the judge run unattended.

Narrow automated checks

Some checks are valuable precisely because they are narrow: does the answer contain a required disclaimer, is the JSON valid, does a response cite the right source. Even visibility-style checks can be automated, such as using an to see whether AI models cite a given page. These are useful signals, but they answer one question each. They belong inside a scorecard, never in place of one.

How to evaluate an LLM, step by step

The 2025 practical guide mentioned above names three pillars: curating representative datasets, choosing diverse quantitative and qualitative metrics, and designing methods that handle non-determinism and prompt sensitivity. Here is how that turns into a working sequence.

1. Define success before you test. Write down what a good answer is for your use case, in plain language a colleague outside engineering would accept. OpenAI recommends defining success criteria early as part of evaluation-driven development.
2. Build a representative dataset. Start from real inputs: support tickets, search queries, documents your users upload. Include the easy cases, the awkward ones and the ones you hope never happen. A hundred well-chosen examples beat ten thousand scraped ones.
3. Pick two or three metrics per quality dimension. One cheap deterministic check, one semantic or judge-based score, and one human spot-check is a sensible default.
4. Run repeatedly, not once. Because outputs vary, run each test several times and look at the spread, not just the average. Document prompts and parameters so a result can be reproduced.
5. Log production data. Real traffic shows you failure modes your test set never imagined. Feed the interesting ones back into the dataset.
6. Re-evaluate whenever something changes. A new model version, a prompt edit or a retrieval tweak is a reason to rerun the suite. The guidance calls for continuous evaluation as systems evolve.

Evaluating the system, not just the model

Most products are not a bare model. They are a model plus retrieval, prompts, tools and guardrails. Measure each layer. For a retrieval-based assistant, check whether the right passages were retrieved before blaming the generator for a wrong answer. The metrics for those layers differ: retrieval is judged on relevance and ranking, generation on faithfulness and helpfulness, and the whole on task completion.

Making LLM benchmarking reliable

Benchmarking is the most searched part of this topic, and the most misunderstood. A public benchmark is a shared exam. Once the questions leak into training data, the exam stops measuring ability and starts measuring memory.

A 2024 systematic survey of LLM evaluation recommends datasets that vary by capability, task and language complexity, and stresses contamination tests so that the evaluation measures unseen capabilities. It also points to documenting prompts and parameters and refreshing evaluations periodically. For your own benchmark, that translates into four habits:

• Keep a private holdout set that never appears in prompts, examples or fine-tuning data.
• Vary the difficulty. If every item is easy, every model looks good.
• Version everything: dataset, prompt, model, temperature, scoring rubric.
• Retire and replace items as models saturate them.

There is a deeper caveat. The ACM survey observes that LLMs may show emergent abilities that surpass current evaluation protocols, which means the measurement itself needs to keep evolving. A benchmark is a snapshot of what you knew to ask.

Where humans still belong

Automation is tempting because human review is slow and costly. But OpenAI explicitly advises against “vibe-based” evaluation, meaning informal impressions, and recommends structured processes that combine human feedback with automated metrics. The AI Evaluation Playbook goes further: human judgement remains the gold standard for high-leverage tasks such as rubric creation, alignment and final safety checks, even though it is slower and more expensive.

A workable split: automate the high-volume, low-risk checks; reserve people for rubric design, calibration of the judge, and anything where a wrong answer hurts someone, such as medical, legal or financial output. In those settings, name the reviewer: a clinician for health content, a compliance specialist for regulated finance language.

Our perspective: the metric is a design decision

We build digital products for a living, and the pattern we see most often is that evaluation gets treated as an engineering afterthought, handed over once the interface is already fixed. That is backwards. Deciding what counts as a good answer is a product and design decision. It determines the tone of a reply, how uncertainty is shown, when the system should say “I don’t know” and what a user sees when it fails.

There is a habit we would gently challenge: choosing a model, then looking for metrics that flatter it. Reverse the order. Describe the experience you want a person to have, translate that into three or four observable behaviours, and let those behaviours pick the metrics. If a score goes up and the experience does not improve, the score was never the point.

We also think teams over-invest in the model and under-invest in the system around it. Our view, expanded in our piece on LLM integration patterns, is that reliability usually comes from how retrieval, prompts, fallbacks and interface choices fit together, not from any single component. Evaluation should mirror that: test the whole journey, including the moment something goes wrong.

Finally, be wary of the tidy dashboard. A clean line trending upward is persuasive, and persuasion is exactly when you should read ten raw outputs by hand.

— Martin

Putting this to work with Format-3

If you are planning an AI feature and want evaluation designed in from the first sprint rather than bolted on, this is the work we do. We combine product discovery, design and full-stack engineering with AI implementations, and we can scope a design sprint or discovery engagement to define success criteria, test sets and the user experience around them before you commit to a build. You can also see how we approached a live, real-time product in our StreamLayer case study.

FAQ

What are the most important LLM evaluation metrics?

There is no universal shortlist. For most applications, start with task accuracy or correctness, faithfulness to source material, relevance, safety, and operational measures such as latency and cost. Choose the ones tied to your success criteria, and pair every automated metric with some human review.

How do I evaluate an LLM for my own use case?

Write down what a good answer looks like, collect a representative set of real inputs, and score outputs with a mix of deterministic checks, model-based scoring and human review. Rerun the suite whenever the model, prompt or retrieval setup changes. A small, carefully chosen dataset is better than a large generic one.

Can I trust an LLM to grade another LLM?

Partly. Judge models scale well, but they can favour longer answers, answer position or their own style, and they can struggle in lower-resource languages. Calibrate the judge against human grades on a sample before relying on it, and recheck whenever you change the judge or the rubric.

Are public LLM benchmarks useful?

They are useful for a rough sense of general capability and for shortlisting models. They are weak evidence for how a model will perform on your tasks, partly because of possible data contamination. Treat them as a starting filter, then test on your own private dataset.

Sources

• Evaluation best practices (OpenAI)
• A Survey on Evaluation of Large Language Models (ACM)
• Evaluation metrics (Microsoft Learn)
• LLM metrics reference (DataRobot docs)
• A Practical Guide for Evaluating LLMs and LLM-Reliant Systems
• A Systematic Survey and Critical Review on Evaluating Large Language Models
• Decide on metrics (AI Evaluation Playbook)

By Format-3

Press & Media

More thoughts

Thought leadership creates value, builds knowledge and takes a stand, bridging the gap between traditional and digital platforms