Eval Engine

Contents

Eval Engine#

Overview#

The Eval Engine is used to evaluate the performance of an LLM’s output against a given input. The engine returns a score between 0 and 1, along with a reason for the score.

Eval Engines require either validate*criteria(*, value) or validate*evaluation_steps(*, value) to be set. If criteria is set, Gen AI Builder will generate evaluation_steps for you. This is useful for getting started, but you may to explicitly set evaluation_steps for more complex evaluations. Either criteria or evaluation_steps must be set, but not both.

import json

from griptape.engines import EvalEngine

engine = EvalEngine(
    criteria="Determine whether the actual output is factually correct based on the expected output.",
)

score, reason = engine.evaluate(
    input="If you have a red house made of red bricks, a blue house made of blue bricks, what is a greenhouse made of?",
    expected_output="Glass",
    actual_output="Glass",
)

print("Eval Steps", json.dumps(engine.evaluation_steps, indent=2))
print(f"Score: {score}")
print(f"Reason: {reason}")
Contents