Exact Match

Probably the simplest ways to evaluate an LLM or runnable's string output against a reference label is by a simple string equivalence.

This can be accessed using the exact_match evaluator.

from langchain.evaluation import ExactMatchStringEvaluator

evaluator = ExactMatchStringEvaluator()

Alternatively via the loader:

from langchain.evaluation import load_evaluator

evaluator = load_evaluator("exact_match")

evaluator.evaluate_strings(
    prediction="1 LLM.",
    reference="2 llm",
)

{'score': 0}

evaluator.evaluate_strings(
    prediction="LangChain",
    reference="langchain",
)

{'score': 0}

Configure the ExactMatchStringEvaluator

You can relax the "exactness" when comparing strings.

evaluator = ExactMatchStringEvaluator(
    ignore_case=True,
    ignore_numbers=True,
    ignore_punctuation=True,
)

# Alternatively
# evaluator = load_evaluator("exact_match", ignore_case=True, ignore_numbers=True, ignore_punctuation=True)

evaluator.evaluate_strings(
    prediction="1 LLM.",
    reference="2 llm",
)

{'score': 1}