← All workshopsRead Evaluating Generative Models

Adapting and Evaluating Modern Models

Can a good average hide repetition?

Compare repetition with varied outputs

Botie's generator can receive a high average quality score by repeating the same output. The evaluation must make that repetition visible.

BOTIE

Which conclusion can these two measurements support when the varied outputs have a lower quality average?
The inputs we are comparing

Quality scores

Example 1

  1. 00.9
  2. 10.9
  3. 20.9

Example 2

  1. 00.8
  2. 10.9
  3. 20.7

Generated outputs

Example 1

  1. 0red cat
  2. 1red cat
  3. 2red cat

Example 2

  1. 0red cat
  2. 1blue dog
  3. 2green bird
Result for example 1
  1. 00.9
  2. 10.3333
Choose an explanation to test
Four answers repeat two explanations

This check uses the pieces from “Report quality beside distinct-output fraction”.

BOTIE

Botie has generated four answers. Report their mean supplied quality score and the fraction of distinct output strings, in that order. Treat exact text equality as the repetition check used by this small exercise.
Quality scores
  1. 00.8
  2. 10.6
  3. 20.9
  4. 30.7
Generated outputs
  1. 0Inspect the belt.
  2. 1Check the sensor.
  3. 2Inspect the belt.
  4. 3Check the sensor.

Use a number, text, or an array matching the requested result. For a matrix, put each row in brackets. Saving your prediction runs your current construction on these inputs and checks the earlier examples too.

Build a construction in the workspace above first. This case will test those pieces.

What task-specific evidence would help choose between these two output sets?

Keep complementary evaluation criteria visible before deciding how to trade them off.

These are illustrative quality and diversity checks. Distinct text alone does not establish useful diversity, and evaluating a real generative model requires task-specific evidence.

Work through the Python challenges →
Continue the courseWhat Would We Predict Before Fitting Anything? →
← Retrieval-Augmented Generation