Give Botie a Typed Job with Pydantic AI

A technician asks Botie what to inspect when a pump vibrates. The response includes a sensible sentence and a manual reference. Unfortunately, that reference was never in the material we gave him. How should the program notice before displaying it?

We have already built a tool loop and an output contract. Now we can give those responsibilities to a framework while keeping our own checks. Pydantic AI connects Python functions, typed results and the data belonging to a particular request.

Wire the field check before the citation check. Then change the proposed answer or the excerpts available to this run.

accept for content review

Suppose the interface has an answer panel and a source panel. An arbitrary paragraph forces the interface to guess where one ends and the other begins. A record with answer and citations fields gives each panel something named to read.

We use a Pydantic BaseModel to declare those fields. Constraints can require a nonempty list and reject unexpected keys. A declared type is a contract about representation. It does not establish that the answer is useful or true.

Keep the distinction visible when testing. A correctly shaped invented citation passes a type check and fails an evidence check.

The excerpts belong to this request. Passing them into the run makes that relationship visible and lets a test substitute a small fixture. In Pydantic AI, deps_type describes the dependency type, while deps supplies the actual instance when the agent runs.

RunContext gives a tool access to those application-provided dependencies. The model chooses tool arguments; it does not construct the authenticated user or database connection. A dependency might hold a scoped database client instead of our small dictionary. Enforce access inside that client or tool.

Do not paste a service credential into the prompt to make it available. Server code can use a credential without returning it as tool output.

The read_manual tool returns the excerpts allowed for the request. Its Python signature tells the framework what inputs exist, and its docstring describes the operation to the model. Our tool has no model-chosen arguments, which prevents the model from supplying another customer’s identity.

The returned dictionary still becomes information the model can use. If an excerpt contains text that tells Botie to ignore instructions, treat it as manual content. A tool result does not acquire authority over the application because it arrived through a Python function.

The tool’s job is reading evidence. Creating an inspection job will require a separate permission boundary later.

Agent holds the model configuration and the permitted tool definitions. run_sync executes a run from an ordinary synchronous script; use await agent.run in an asynchronous server. The returned result has an output value matching the declared type when the run succeeds.

This recipe sends an invented excerpt to your chosen provider. Its output validator checks citation membership after the structural check. ModelRetry asks for a corrected response within the framework’s retry limits. Exhausted retries and provider failures still need an application error path.

Follow the names through the code: Manuals is request data, Answer is the result contract, and read_manual is an operation. They have different jobs.

import os
from dataclasses import dataclass
from pydantic import BaseModel, ConfigDict, Field
from pydantic_ai import Agent, RunContext, ModelRetry

@dataclass
class Manuals:
    excerpts: dict[str, str]

class Answer(BaseModel):
    model_config = ConfigDict(extra="forbid", strict=True)
    answer: str = Field(min_length=1)
    citations: list[str] = Field(min_length=1)

agent = Agent(
    os.environ["PERCEPTRON_LAB_AGENT_MODEL"],
    deps_type=Manuals,
    output_type=Answer,
    instructions="Read the supplied manual with read_manual. Cite its IDs. "
                 "Describe the manual; do not schedule work.",
)

@agent.tool
def read_manual(ctx: RunContext[Manuals]) -> dict[str, str]:
    """Read the manual excerpts available to this request."""
    return ctx.deps.excerpts

@agent.output_validator
def check_citations(ctx: RunContext[Manuals], answer: Answer) -> Answer:
    if not answer.answer.strip():
        raise ModelRetry("Answer text cannot be blank.")
    if any(key not in ctx.deps.excerpts for key in answer.citations):
        raise ModelRetry("Cite only IDs returned by read_manual.")
    return answer

result = agent.run_sync(
    "What should a technician check when pump P7 vibrates?",
    deps=Manuals({"manual-7": "Inspect the mounting and coupling."}),
)
print(result.output.model_dump())

An answer can cite manual-7 correctly while saying the opposite of its text. Membership asks whether we have that source. Entailment asks whether the source supports the claim. Those are different questions, so keep different tests for them.

A useful fixture can contain two excerpts with different pump types. Then a response that cites the available but wrong pump manual should fail the task evaluation even though it passes our membership validator. Preserve the original excerpt and its version so a reviewer can understand the disagreement.

The workshop output says accept for content review. That wording names exactly what passed.

Start with deterministic unit tests of the tool and validators. Give the citation check an absent ID, an empty list, and an extra field. A live model is unnecessary for those cases.

Then run provider-backed evaluation separately to investigate whether the model reads the right excerpt and handles missing evidence. Repeating those runs tells us something different from repeatedly checking a Python dictionary. Mocking the provider helps test control flow, but it cannot establish answer quality.

Our offline exercise implements the two boundary checks directly so you can see what the framework is organising.

available = {"manual-2", "manual-7"}
candidate_ids = ["manual-7", "manual-9"]
missing = [key for key in candidate_ids if key not in available]
print(missing)

['manual-9']

If validation cannot produce a usable result, return a clear unsuccessful state to the application. Do not replace the answer with an empty success object, and do not keep retrying indefinitely. Record the failure category without logging credentials or unnecessary manual content.

A technician can supply another excerpt or ask for review. That is a useful next step even when the agent has no supported answer. Later evaluation lessons will distinguish those justified refusals from failures to answer questions the manual could resolve.

A framework reduces repeated plumbing. The acceptance policy still belongs to the application.