Can We Find the Run Behind This Score?

Botie’s new prompt seems better at answering maintenance questions. The average quality score has gone up. Before we celebrate, which requests received scores? Were both prompt versions tested on comparable questions? Did a tool-level score get mistaken for a judgement of the final answer?

Langfuse gives us somewhere to connect execution records, prompt versions and evaluations. We will work through the records first, then see how the SDK records those relationships. A useful dashboard depends on those relationships being correct.

Attach one rubric version’s scores by trace ID, select the prompt version, and report the available-score mean beside its coverage.

Use one trace for a maintenance request whose outcome we want to inspect. Its observations describe operations within that request, such as retrieving a manual and producing an answer. A session can group several requests from the same conversation.

Keep these meanings stable. If one deployment records a whole conversation as one trace and another records each turn separately, a trace-level latency comparison mixes different units of work.

The trace boundary should answer a question someone actually asks about the service.

A general span can enclose application work. A generation describes a model interaction and can carry the model identifier, input, output and usage. A retrieval operation should keep the retrieved document identities attached to that operation.

The resulting tree lets us inspect the evidence available before an answer. It cannot reveal private model reasoning, and it should not be presented as a complete explanation of why the model chose particular words.

Instrument actual operations. A recorded answer fixture should not be labelled as a live model call.

Start with a harmless local lookup and a surrounding request observation. The context manager keeps the active parent while the child runs. When the script ends, flush queued telemetry so a short-lived process does not exit before export.

Configure credentials and the destination in your own environment. The browser workshop has neither, so its records stay local. A successful export checks the instrumentation path, not the correctness of a maintenance assistant.

This example deliberately records a span for a dictionary lookup. It does not claim to run a model.

from langfuse import get_client

client = get_client()
manuals = {"P7": {"revision": "r4", "procedure": "inspection-required"}}
with client.start_as_current_observation(as_type="span", name="maintenance-request") as request:
    with client.start_as_current_observation(as_type="span", name="manual-lookup") as lookup:
        result = manuals.get("P7")
        lookup.update(output={"found": result is not None, "revision": result["revision"]})
    request.update(output={"lookup_complete": result is not None})
client.flush()

A label such as production can point to a different prompt tomorrow. A version identifies a particular saved revision. Record the resolved version with the run so we can distinguish changes to instructions from changes in questions.

The compiled text also depends on its variables. Keep the manual revision, the retrieval configuration and the model configuration identifiable. A prompt version alone cannot reproduce an answer whose evidence changed.

Resolved versions describe what ran. A movable label describes how the application chose it.

Passing a managed prompt object to a Langfuse generation links the observation to that prompt. The helper below accepts a real text-generation callable from your application; the lesson does not replace it with a canned model answer.

Create a text prompt named maintenance-answer with a question variable before using the helper. Decide separately whether retaining its compiled input is appropriate for the data your service handles.

This records the input and output of the supplied callable. Its result and cost depend on the provider and implementation you supply.

from langfuse import get_client

def traced_answer(question, generate, model_name):
    client = get_client()
    prompt = client.get_prompt("maintenance-answer")
    compiled = prompt.compile(question=question)
    with client.start_as_current_observation(
        as_type="generation", name="maintenance-answer",
        model=model_name, prompt=prompt, input=compiled,
    ) as generation:
        answer = generate(compiled)
        generation.update(output=answer)
        return answer

An outcome score might describe the whole request. A retrieval score might describe whether a particular result contains the answer. Attach each score at the level it evaluates, using the trace ID and, for an observation-level judgement, its observation ID.

Name the criterion and identify the evaluator or rubric version. A score named quality is hard to compare if one evaluator measures citation support and another measures writing style. A human judgement and a model judgement can disagree without either being a transmission error.

The numeric scale is part of the contract. Our workshop uses zero to one; other evaluations can use categories or different ranges.

The scoring worker may finish requests in a different order. It may also score yesterday’s request while today’s run is still waiting. Use trace IDs to connect scores to runs, then filter the desired prompt version.

For this workshop, select one rubric revision first. Each run has at most one resulting score. Duplicate scores for that selection are an error that requires a policy, not an invitation to keep whichever arrived last.

An identity join protects the meaning of a row when execution order changes.

scores = {"visit-B": 1.0, "visit-A": 0.0}
for trace_id in ["visit-A", "visit-B", "visit-C"]:
    print(trace_id, scores.get(trace_id))

visit-A 0.0
visit-B 1.0
visit-C None

If the evaluator fails on malformed answers, discarding those rows can make a broken prompt look excellent. Keep the run in the denominator for coverage even when its score is absent. Report evaluation errors separately so someone can fix them.

Our mean uses only present scores and is explicitly labelled that way. If none are present, its value is missing rather than zero. Deciding that an evaluation failure should fail a release is a separate policy we will define later.

Available-score mean = sum of present scores / number of present scores
Evaluation coverage = number of scored runs / number of selected runs

Neither replacing every missing score with zero nor silently dropping it answers all questions. Preserve its missing state first.

A dataset item describes one task and its expected behaviour. An experiment runs a candidate on items and connects its outputs to the corresponding cases. The experiment identity should include the application and prompt revisions being compared.

A production conversation is useful evidence, but copying it into an evaluation dataset requires review. Remove sensitive fields, preserve the information boundary and record why the case matters. Do not expose a hidden expected answer to the agent under test.

The next lesson builds these cases so an experiment can measure more than a few favourable conversations.

When a score falls, open the underlying run. Did retrieval return the right manual revision? Did the answer omit a required warning? Did the evaluator judge against evidence that the agent never received?

Then compare like cases across versions. A chart filtered to prompt 3 is useful, but a higher score on an easier set of requests does not establish that prompt 3 improved the same task.

Observability helps us find evidence. Evaluation design determines what comparison that evidence supports.

A telemetry timeout should have a defined effect on the application. Decide whether export is buffered, retried, sampled or allowed to fail without blocking the maintenance response. Monitor dropped observations so an empty dashboard cannot quietly appear healthy.

Separate optional diagnostic export from the durable evidence required to authorise a write. If the application requires an approval record before dispatch, losing that record should prevent the write under that policy. A temporarily unavailable dashboard does not need to stop an otherwise permitted read-only manual lookup.

Treat exporting input and output as a data flow with access controls. The presence of a tracing SDK does not make every field appropriate to collect. Keep organisation boundaries in the tracing project’s access policy too, and test that a dashboard user cannot inspect another organisation’s requests merely by knowing a trace ID.

Keep a stable set of non-sensitive identifiers even when you reduce payload detail.

Langfuse combines tracing with prompt and evaluation records. Phoenix also provides trace inspection and evaluations, using OpenTelemetry and OpenInference conventions for model-related data. Existing teams may already have an OpenTelemetry collector and a tracing backend.

Compare how the tools preserve your required IDs, where the data is stored, and how evaluators attach their results. Installing several exporters does not by itself produce better evidence, and can duplicate records if instrumentation overlaps.

The portable skill is following a judgement back to the exact execution. The named tools provide different places to do that.