Making a Fair Comparison by Assignment

The previous records mixed treatment with the gardener’s judgement. This time Botie can assign a schedule by a random procedure before observing outcomes.

That procedure makes the groups comparable in expectation. It does not guarantee identical groups in one small experiment, so we still need to calculate an estimate and describe its sampling uncertainty.

Treatment mean = (6 + 8 + 7) / 3 = 7
Control mean = (4 + 5 + 6) / 3 = 5

Mean difference = 7 − 5 = 2
Individual counterfactual outcomes: unobserved

The comparison uses outcomes from the stated assignment design. Change one group's outcomes and inspect the difference, while keeping in mind which units were independently assigned.

2

What receives the random assignment: a plant, a bed, or a greenhouse? Choose a unit that fits how the watering is actually applied. If water runs from one bed into another, treating one bed can affect the other's outcome, so the no-interference assumption may fail.

Record the assignment probabilities and measure outcomes consistently. If whole greenhouses received the assignments, treating every plant as an independent experiment would overstate how much independent evidence we have.

The examples assume independent units and no spillovers.

For an intention-to-treat comparison, group units by assigned treatment, even if some do not fully follow the schedule. Changing groups after seeing adherence can undo the benefit of randomisation.

Take the treatment mean minus the control mean. Under the experiment’s assumptions, random assignment supports interpreting this as an estimate of the average assignment effect.

Treatment outcomes = [6,8,7]; mean = 7
Control outcomes = [4,5,6]; mean = 5
Estimated difference = 2

This is a group average, not the effect on every individual bed.

Different random assignments can give different estimates. For independent groups, a common standard-error estimate combines each sample variance divided by its sample size, then takes a square root.

Use sample variance with the stated degrees-of-freedom correction. Small samples and unusual designs may need randomisation-based or other design-appropriate inference rather than a casual normal approximation.

Estimated SE = sqrt(s_treated²/n_treated + s_control²/n_control)

The exercise calculates the standard error but does not claim a precise interval from three units per group.

import numpy as np
assigned_treatment = np.array([5., 7.])
assigned_control = np.array([4., 4.])
means = np.array([assigned_treatment.mean(), assigned_control.mean()])
print(means.tolist())
print(float(means[0] - means[1]))

[6.0, 4.0]
2.0

Specify the outcome and analysis before inspecting results. Repeatedly testing many outcomes or stopping at the first favourable result changes error rates unless accounted for.

Missing outcomes, nonadherence, spillovers, and changes in measurement can undermine interpretation. Report them alongside the estimated effect instead of letting randomisation stand in for every design check.

The workshop compares group means; the second coding task adds uncertainty.

If neighbouring beds share irrigation, assigning each plant independently may not create independent treatments. Water delivered to one can affect another. Randomise beds or larger groups when that matches how treatment is delivered, and analyse uncertainty at the appropriate unit.

Random assignment balances starting conditions in expectation, not perfectly in every small run. Record the assignment procedure before outcomes arrive and inspect whether the intended procedure was followed.

A bed assigned extra watering may miss a scheduled visit. Comparing beds by what they actually received can reintroduce selection: the missed visits may happen on particular kinds of beds.

An intention-to-treat comparison groups outcomes by original assignment. It estimates the effect of the assignment policy under the experiment's conditions, including its compliance behaviour. It is not automatically the effect of receiving the treatment exactly as intended.

import numpy as np
assigned = np.array([0,0,1,1])
received = np.array([0,0,1,0])
growth = np.array([4.,6.,9.,5.])
itt = growth[assigned == 1].mean()-growth[assigned == 0].mean()
received_difference = growth[received == 1].mean()-growth[received == 0].mean()
print(float(itt))
print(float(received_difference))

2.0
4.0

Four beds are enough to demonstrate the calculation, not enough to establish a stable agronomic effect. Variation among units makes the estimated difference change across possible assignments and samples.

Choose a sample-size plan, outcome definition and analysis before inspecting results. A randomisation test can compare the observed statistic with assignments allowed by the actual design. Its permutations must respect blocks or clusters when those were part of assignment.

Stopping the first time an ordinary significance threshold is crossed does not have the same error properties as one preplanned analysis. Nor does trying many outcomes and reporting only the most favourable one.

Use a fixed analysis plan or a suitable sequential design. Report effect size and uncertainty alongside the practical cost of the intervention. A small p-value alone does not establish that the effect is large enough to matter.