> For the complete documentation index, see [llms.txt](https://karini-ai.gitbook.io/karini-ai-documentation/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://karini-ai.gitbook.io/karini-ai-documentation/prompt-management/prompt-and-agent-evaluation/evaluation-metrics-and-scoring.md).

# Evaluation Metrics & Scoring

This section describes the available metrics, their scoring semantics, and how to interpret these signals when validating and optimizing your AI workflows.

### **Default Evaluation Metrics**

Karini AI’s default evaluation uses built-in metrics to measure how well your prompts and agents perform. Each metric is scored on a 0–1 scale and includes additional information such as confidence and reasoning.

#### Standard Prompt Metrics

These metrics apply whenever the system generates a text response (for both prompts and agents).

<table><thead><tr><th>Metric</th><th>Applies To</th><th>What It Measures</th><th width="131.4000244140625">Score Range</th><th>Typical Use Case</th></tr></thead><tbody><tr><td><strong>Answer Relevancy</strong></td><td>Prompts &#x26; Agents</td><td>How well the response addresses the user’s query and stays on-topic.</td><td>0 = irrelevant, 1 = fully relevant</td><td>Detecting off-topic or incomplete answers</td></tr><tr><td><strong>Answer Faithfulness</strong></td><td>Prompts &#x26; Agents</td><td>Whether the response is factually correct and grounded in the given context.</td><td>0 = unfaithful, 1 = fully faithful</td><td>Reducing hallucinations and unsupported statements</td></tr><tr><td><strong>Answer Similarity</strong></td><td>Prompts &#x26; Agents</td><td>Semantic similarity between the response and the ground-truth answer.</td><td>0 = very different, 1 = semantically identical</td><td>Comparing against reference answers</td></tr><tr><td><strong>Context Sufficiency</strong></td><td>Prompts &#x26; Agents</td><td>If the provided context contains enough information to answer the question.</td><td>0 = insufficient, 1 = fully sufficient</td><td>Debugging retrieval / RAG or missing context</td></tr><tr><td><strong>Guideline Adherence</strong></td><td>Prompts &#x26; Agents</td><td>How well the response follows instructions, tone, and formatting guidelines.</td><td>0 = does not follow, 1 = fully adheres</td><td>Enforcing style guides, formatting, or policy rules</td></tr></tbody></table>

#### Agent-Specific Metrics

These metrics are only used for agentic workflows involving tools and multi-step reasoning.

| Metric                   | Applies To  | What It Measures                                                            | Score Range                                | Typical Use Case                                  |
| ------------------------ | ----------- | --------------------------------------------------------------------------- | ------------------------------------------ | ------------------------------------------------- |
| **Tool Accuracy**        | Agents only | Correctness of tool selection and parameter usage.                          | 0 = wrong/poor usage, 1 = perfect usage    | Evaluating how well agents use tools/APIs         |
| **Goal Achievement**     | Agents only | How well the agent accomplishes the intended objective or task.             | 0 = goal not met, 1 = goal fully achieved  | Validating multi-step workflow success            |
| **Execution Efficiency** | Agents only | Efficiency of the agent’s execution path (steps, redundancy, resource use). | 0 = very inefficient, 1 = highly efficient | Optimizing cost, latency, and unnecessary actions |

#### **Metric Output Fields**

Each metric returns structured information for every test case.

| Field          | Description                                           | Range / Type |
| -------------- | ----------------------------------------------------- | ------------ |
| **Score**      | Numerical rating assigned by the metric.              | 0.0 – 1.0    |
| **Confidence** | How confident the evaluator is in the assigned score. | 0.0 – 1.0    |
| **Reasoning**  | Text explanation describing why the score was given.  | Free text    |

#### Confidence Levels

| Confidence Range | Level  | Interpretation                                   |
| ---------------- | ------ | ------------------------------------------------ |
| **0.8 – 1.0**    | High   | Strong, reliable signal; can usually be trusted. |
| **0.5 – 0.79**   | Medium | Useful guidance; review edge or critical cases.  |
| **0 – 0.49**     | Low    | Uncertain; manual review recommended.            |

#### &#x20;**Aggregate Metrics**

| Metric                 | Description                                        | Purpose                                             |
| ---------------------- | -------------------------------------------------- | --------------------------------------------------- |
| **Mean**               | Average score across all test cases.               | Overall performance level.                          |
| **Median**             | Middle score value across all test cases.          | Robust central tendency, less affected by outliers. |
| **Standard Deviation** | Variation of scores around the mean.               | Measures consistency (lower = more stable).         |
| **Min / Max**          | Lowest and highest scores observed in the dataset. | Shows best-case and worst-case behavior.            |

### **Creating Custom Metrics**

Custom metrics enable you to extend the default evaluation framework with criteria tailored to your specific domain and quality requirements. This allows you to formalize what “good” output means for your organization and have an LLM systematically apply those rules during evaluation.

* **Custom Prompt**: Define an evaluation prompt that receives the model or agent output and ground truth, and returns a structured score.
* **Domain-Specific**: Encode industry- or use case–specific guidelines (e.g., regulatory constraints, brand voice, safety rules).
* **Model Selection**: Choose which LLM will act as the evaluator, balancing accuracy, latency, and cost.
* **Flexible Scoring**: Design a custom scoring rubric and criteria to align with internal review or compliance processes.