OpenAI Agents

OpenAI Agents is OpenAI's SDK for building agents that reason, call tools, and hand off to other agents.

The deepeval integration plugs into the agents SDK's tracing pipeline as a TracingProcessor. Every agent run, LLM call, and tool call becomes a span you can inspect — without rewriting your agent code.

deepeval's OpenAI Agents integration enables you to:

• Trace every agent run — each Runner.run(...) produces a trace, and each LLM, tool, and sub-agent call becomes a component span.
• Evaluate the complete ordered agent trajectory — score the sequence across all applicable agent, LLM, tool, and handoff spans.
• Evaluate traces or individual components with any deepeval metric.
• Run evals from scripts or CI/CD — same agent, different surfaces.
• Customize trace and span data through agent/tool config (Python) or next_*_span staging (both).

Getting Started

pip install -U deepeval openai-agents

from agents import Runner, add_trace_processor
from deepeval.openai_agents import Agent, DeepEvalTracingProcessor, function_tool
from deepeval.dataset import EvaluationDataset, Golden
from deepeval.metrics import TaskCompletionMetric

add_trace_processor(DeepEvalTracingProcessor())

@function_tool
def get_weather(city: str) -> str:
    """Return the weather in a city."""
    return f"It's always sunny in {city}!"

agent = Agent(
    name="weather_agent",
    instructions="Answer weather questions concisely.",
    tools=[get_weather],
    agent_metrics=[TaskCompletionMetric()],
)

# Goldens are the inputs you want to evaluate.
dataset = EvaluationDataset(goldens=[Golden(input="What's the weather in Paris?")])

for golden in dataset.evals_iterator():
    Runner.run_sync(agent, golden.input)

What gets traced

Each Runner.run(...) call produces a trace — the end-to-end unit your user observes. Inside that trace are component spans for every step the agent took:

• Agent spans — one per Agent invocation, including handoffs to other agents.
• LLM spans — model calls (Responses API and Chat Completions).
• Tool spans — function tools, MCP tools, and other agents-SDK tool calls.

Trace                          ← what the user observes
└── Agent: weather_agent       ← one Runner.run(...) call
    ├── LLM: gpt-4o            ← component span: model plans
    ├── Tool: get_weather      ← component span: tool input + output
    └── LLM: gpt-4o            ← component span: final answer

The trace and its component spans are independently evaluable.

Running evals

There are two surfaces for running evals against an OpenAI Agents app. Pick by where you want results to surface — your terminal during development, or your CI pipeline as a pass/fail gate.

In CI/CD

Use the deepeval pytest integration. Each parametrized test invocation becomes one agent run; failing metrics fail the test, which fails the build.

import pytest
from agents import Runner, add_trace_processor
from deepeval.openai_agents import Agent, DeepEvalTracingProcessor, function_tool
from deepeval.dataset import EvaluationDataset, Golden
from deepeval.metrics import TaskCompletionMetric
from deepeval import assert_test

add_trace_processor(DeepEvalTracingProcessor())

@function_tool
def get_weather(city: str) -> str:
    """Return the weather in a city."""
    return f"It's always sunny in {city}!"

agent = Agent(
    name="weather_agent",
    instructions="Answer weather questions concisely.",
    tools=[get_weather],
)

dataset = EvaluationDataset(goldens=[
    Golden(input="What's the weather in Paris?"),
    Golden(input="What's the weather in London?"),
])

@pytest.mark.parametrize("golden", dataset.goldens)
def test_openai_agents_app(golden: Golden):
    Runner.run_sync(agent, golden.input)
    assert_test(golden=golden, metrics=[TaskCompletionMetric()])

Run it with:

deepeval test run test_openai_agents_app.py

In a script

Use EvaluationDataset + evals_iterator. Each Golden becomes one agent run; metrics score the resulting trace.

import asyncio

from deepeval.dataset import EvaluationDataset, Golden
from deepeval.evaluate.configs import AsyncConfig
from deepeval.metrics import TaskCompletionMetric
...

dataset = EvaluationDataset(goldens=[
    Golden(input="What's the weather in Paris?"),
    Golden(input="What's the weather in London?"),
])

for golden in dataset.evals_iterator(
    async_config=AsyncConfig(run_async=True),
    metrics=[TaskCompletionMetric()],
):
    task = asyncio.create_task(Runner.run(agent, golden.input))
    dataset.evaluate(task)

Sync (Runner.run_sync) and async (Runner.run) execution both work; pick whichever matches your code.

Evaluate agent trajectories

Trajectory evaluation scores the complete ordered trace: the agent's decisions, model calls, tool use, handoffs, and other applicable spans in the sequence they occurred. This is distinct from evaluating only the final output or attaching a metric to an individual component span.

Pass trajectory metrics to evals_iterator so each instrumented OpenAI Agents run is evaluated at trace scope:

from deepeval.metrics import TaskCompletionMetric, StepEfficiencyMetric, PlanAdherenceMetric
from deepeval.openai_agents import DeepEvalTracingProcessor
from deepeval.dataset import EvaluationDataset, Golden
from agents import Runner, add_trace_processor
...

add_trace_processor(DeepEvalTracingProcessor())
dataset = EvaluationDataset(goldens=[Golden(input="What's the weather in Paris?")])
metrics = [TaskCompletionMetric(), StepEfficiencyMetric(), PlanAdherenceMetric()]

for golden in dataset.evals_iterator(metrics=metrics):
    Runner.run_sync(agent, golden.input)

See the trajectory evaluation guide for metric selection and interpretation. To score one step instead of the ordered run, apply metrics to components next.

Applying metrics to components

Passing metrics to evals_iterator evaluates the trace. To evaluate a component — a specific agent, LLM call, or tool — attach metrics where that component is created.

Agent spans

Use agent_metrics=[...] on deepeval.openai_agents.Agent. The metric is applied to that agent's span on every run, including when it's invoked as a sub-agent through a handoff.

from deepeval.metrics import TaskCompletionMetric
from deepeval.openai_agents import Agent
...

agent = Agent(
    name="weather_agent",
    instructions="Answer weather questions concisely.",
    tools=[get_weather],
    agent_metrics=[TaskCompletionMetric()],
)

LLM calls

Use llm_metrics=[...] on Agent. The metric is applied to the LLM span produced for that agent's model calls. Useful when you want to score the model's reasoning step in isolation.

from deepeval.metrics import AnswerRelevancyMetric
from deepeval.openai_agents import Agent
...

agent = Agent(
    name="weather_agent",
    instructions="Answer weather questions concisely.",
    tools=[get_weather],
    llm_metrics=[AnswerRelevancyMetric()],
)

Tool calls

Pass metrics=[...] to function_tool to evaluate a specific tool's behavior. Useful for tools that return non-deterministic content (e.g. retrieval, summarization tools).

from deepeval.openai_agents import function_tool
from deepeval.test_case import LLMTestCaseParams
from deepeval.metrics import GEval

@function_tool(metrics=[GEval(
    name="Helpful Weather Lookup",
    criteria="The output must be a clear weather summary for the requested city.",
    evaluation_params=[LLMTestCaseParams.INPUT, LLMTestCaseParams.ACTUAL_OUTPUT],
)])
def get_weather(city: str) -> str:
    """Return the weather in a city."""
    return f"It's always sunny in {city}!"

For deterministic tool calls, prefer update_current_span(...) to add metadata, inputs, and outputs instead of attaching metrics to the tool span.

Customizing trace and span data

The integration captures inputs, outputs, model names, and tool calls automatically. For anything dynamic, the right API depends on where your code runs.

• Use with trace(...) for trace-level fields (name, tags, metadata, thread_id, user_id).
• Use Agent/function_tool kwargs (agent_metrics, llm_metrics, metrics=, confident_prompt) for component-level defaults.
• Use update_current_trace(...) and update_current_span(...) from inside a tool body to mutate fields the framework can't see.

from deepeval.tracing import update_current_trace, update_current_span
from deepeval.openai_agents import function_tool

@function_tool
def get_weather(city: str) -> str:
    """Return the weather in a city."""
    update_current_trace(metadata={"city": city})
    update_current_span(metadata={"source": "static-table"})
    return f"It's always sunny in {city}!"

Advanced patterns

The primitives above compose around one boundary: the agents SDK owns the run lifecycle, and your code attaches metrics where they make sense.

Evaluate a sub-agent through handoff

When a parent agent hands off to a sub-agent, the sub-agent's span runs as a child of the parent's.

Attaching agent_metrics to the sub-agent scores that hand-off step in isolation.

from deepeval.metrics import TaskCompletionMetric, AnswerRelevancyMetric
from deepeval.openai_agents import Agent
...

triage_agent = Agent(
    name="triage",
    instructions="Route the question to the right specialist.",
    handoffs=[
        Agent(
            name="weather_specialist",
            instructions="Answer weather questions.",
            tools=[get_weather],
            agent_metrics=[TaskCompletionMetric()],
        ),
    ],
    agent_metrics=[AnswerRelevancyMetric()],
)

No trace-level metrics required

Trace-level metrics are end-to-end metrics: they score the whole trace. They are not strictly necessary here because the metrics are already attached to the triage and specialist agents, so CI/CD and scripts only need to run the agent.

This is how you'd run it:

import pytest
from agents import Runner
from deepeval import assert_test
...

@pytest.mark.parametrize("golden", dataset.goldens)
def test_triage_agent(golden: Golden):
    Runner.run_sync(triage_agent, golden.input)
    assert_test(golden=golden)

deepeval test run test_openai_agents_app.py

import asyncio
...

for golden in dataset.evals_iterator(async_config=AsyncConfig(run_async=True)):
    task = asyncio.create_task(Runner.run(triage_agent, golden.input))
    dataset.evaluate(task)

Wrap an agent run in observe

When the agent run is part of a larger operation, wrap the outer function with @observe. OpenAI Agents spans nest under your observed span when a current trace already exists.

from agents import Runner
from deepeval.tracing import observe
...

@observe(name="respond_to_user")
async def respond_to_user(prompt: str) -> str:
    result = await Runner.run(agent, prompt)
    return result.final_output.strip()

Bind a Confident AI prompt

Pass confident_prompt= to attach a Confident AI Prompt to every LLM span produced by that agent. Prompt analytics (commit hash, version, label) flow with the trace.

from deepeval.openai_agents import Agent
from deepeval.prompt import Prompt

prompt = Prompt(alias="weather-system")
prompt.pull(version="latest")

agent = Agent(
    name="weather_agent",
    instructions=prompt.interpolate(),
    tools=[get_weather],
    confident_prompt=prompt,
)

API reference

deepeval.openai_agents.Agent(...) accepts the SDK's standard Agent kwargs plus the following deepeval-specific ones:

function_tool(..., metrics=[...]) accepts the SDK's standard kwargs plus metrics, applied to that tool's span on every call.

Register with add_trace_processor(DeepEvalTracingProcessor()).

For runtime helpers (update_current_trace, update_current_span) and the test surface (@observe, assert_test, trace(...)), see the tracing reference.

FAQs