ag2-testing

Testing agents and tools

When to use

Writing tests for code that builds AG2 beta Agents, custom @tool functions, middleware, or response schemas — anywhere you don't want to make real LLM API calls.

60-second recipe — mock an LLM response

import pytest
from autogen.beta import Agent
from autogen.beta.testing import TestConfig

@pytest.mark.asyncio
async def test_mocked_response():
    agent = Agent("test_agent")
    reply = await agent.ask("Hi!", config=TestConfig("This is a mocked response."))
    assert reply.body == "This is a mocked response."

TestConfig(*responses) replaces the model client. Each positional arg is the mocked response for the next sequential turn — strings for text replies, ToolCallEvent for tool dispatches.

Simulate a successful tool call

Pass a ToolCallEvent first (the model "decides" to call the tool), then the final answer:

import pytest
from autogen.beta import Agent
from autogen.beta.events import ToolCallEvent
from autogen.beta.testing import TestConfig

@pytest.mark.asyncio
async def test_tool_success():
    def my_tool() -> str:
        return "tool execution result"

    agent = Agent("test_agent", tools=[my_tool])
    config = TestConfig(
        ToolCallEvent(name="my_tool"),
        "final result",
    )
    reply = await agent.ask("Please use my_tool", config=config)
    assert reply.body == "final result"

Test tool error paths

If a tool raises, the exception propagates to ask():

@pytest.mark.asyncio
async def test_tool_raises():
    def failing_tool() -> str:
        raise ValueError("Something went wrong")

    config = TestConfig(
        ToolCallEvent(name="failing_tool"),
        "result",
    )
    agent = Agent("test_agent", config=config, tools=[failing_tool])

    with pytest.raises(ValueError, match="Something went wrong"):
        await agent.ask("Hi!")

Tool not found

If the LLM calls a tool the agent doesn't have, the framework raises ToolNotFoundError:

from autogen.beta.exceptions import ToolNotFoundError

@pytest.mark.asyncio
async def test_tool_not_found():
    config = TestConfig(ToolCallEvent(name="unregistered_tool"))
    agent = Agent("test_agent", config=config)
    with pytest.raises(ToolNotFoundError, match="Tool `unregistered_tool` not found"):
        await agent.ask("Hi!")

Useful test patterns

Override Depends dependencies

def get_production_db():
    raise Exception("Do not call in tests!")

@tool
def read_data(db: Annotated[object, Depends(get_production_db)]) -> str:
    return "Data"

agent = Agent("test", tools=[read_data])
agent.dependency_provider.override(get_production_db, lambda: "mock_db")

Override Inject dependencies

Just pass dependencies={...} to agent.ask(...):

await agent.ask("Read", dependencies={"database_pool": fake_pool})

Capture stream events

from autogen.beta import MemoryStream
from autogen.beta.events import ToolCallEvent

stream = MemoryStream()
collected: list[ToolCallEvent] = []
stream.where(ToolCallEvent).subscribe(lambda e: collected.append(e))

await agent.ask("Test", stream=stream)
assert collected[0].name == "expected_tool"

Multi-turn mock

Each positional arg in TestConfig(...) corresponds to one model response. For a multi-turn test, supply enough responses for each turn the test exercises.

Going deeper

• Source doc: website/docs/beta/testing.mdx.
• Test markers / async config — repo pyproject.toml. Use @pytest.mark.asyncio (the project uses pytest-asyncio).
• Streams (for asserting events): website/docs/beta/advanced/stream.mdx.

Common pitfalls

• Forgetting @pytest.mark.asyncio — the test will skip or fail oddly.
• Mismatched response count — TestConfig runs out of responses if the agent makes more LLM calls than you expect (e.g. tool error → another LLM call). Add more positional args or assert that the call sequence is what you intended.
• Mocking the LLM but not the tool — your tool function still runs (and may hit real APIs / disk). Mock the tool if you're isolating LLM behaviour, or override its Depends to inject test doubles.
• Asserting on reply.body when you set a response_schema — body is the raw text. Use await reply.content() for the validated value.
• Sharing Agent instances across async tests — agents carry mutable state (variables, dependencies). Construct fresh agents per test for isolation.
• Using real provider clients in CI — wrap the provider config with TestConfig per-test or via a fixture; never rely on OPENAI_API_KEY etc. being available in test environments.