Trying an Action Before Knowing Its Value

Two watering schedules are available. The first produced a good result yesterday, while the second has barely been tried. Repeating the first might work well today, but it leaves Botie uncertain about whether the second is better.

This is a decision that also collects information. We will first treat each trial as independent of later conditions, then learn how to update an action's estimated reward and occasionally explore another action.

B's first reward: 7
B's next reward: 3
Updated mean = (7 + 3) / 2 = 5

The new reward belongs to one selected action. Change its previous count and see why the same surprise has a smaller effect after many observations.

5

An action is one available choice. A reward is the numerical feedback after taking it. An action-value estimate is our current estimate of its expected reward. We observe feedback for the action selected, not the unrealised alternatives.

A reward must describe the objective we intend to optimise. A signal based only on immediate water savings could reward starving plants. In this bandit lesson, rewards are supplied synthetic measurements rather than a recommendation for greenhouse control.

The simplified environment has no state transition affecting future trials.

Keep a separate count and reward estimate for each schedule. When Botie tries one schedule, only that schedule has gained a new observation. Increase its count and move its mean toward the new reward by the reciprocal of the updated count.

For the first trial, that sets the estimate to the observed reward. Updating the untried schedule too would give it credit or blame for an outcome we never observed under that choice.

Old estimate = 7; previous count = 1; new reward = 3
New count = 2
New estimate = 7 + (3 - 7) / 2 = 5

This update equals the arithmetic mean of that action's observed rewards.

A greedy policy chooses an action with a highest estimated value. An epsilon-greedy policy explores with probability epsilon and otherwise chooses greedily. Here exploration samples uniformly from all actions, including a greedy one.

With one unique greedy action, it can therefore be selected through either branch. Define tie handling and seed the generator when comparing runs.

Two actions; epsilon = 0.2
Greedy action probability = 0.8 + 0.2 / 2 = 0.9
Other action probability = 0.2 / 2 = 0.1

Exploration probability is not the same as the probability of choosing a non-greedy action.

import numpy as np
old_count = 3
old_mean = 4.0
reward = 8.0
new_count = old_count + 1
correction = (reward - old_mean) / new_count
print(new_count)
print(old_mean + correction)

4
5.0

A lucky first observation can mislead a greedy learner. Compare policies over repeated seeded simulations and inspect both collected reward and action counts. Keep test conditions separate from the random sequence used to choose settings.

Sample averages assume a stable reward distribution for each action. In a changing environment, a fixed update rate may track recent rewards better, but it no longer equals the full historical mean.

Botie's construction performs one correctly attributed update. The next lesson adds a state so actions can affect what happens afterward.

A bandit learner chooses one schedule and sees its result. It cannot update the unchosen schedule with the reward it imagines that schedule would have produced. In a simulation, the evaluator may know both reward distributions, but the agent still receives only its chosen outcome.

Keep counts and value estimates for every action. An epsilon-greedy choice sometimes samples an action to gather information; otherwise it chooses an action with the largest current estimate. Define a tie rule so initial behaviour is understandable.

The incremental mean needs the selected action's own count. A global step count gives the wrong denominator when actions have been tried different numbers of times.

This trace supplies the chosen actions and rewards, isolating the estimate update from the selection policy. The complete experiment adds a policy before each feedback event.

import numpy as np
count = np.zeros(2, dtype=int)
value = np.zeros(2)
for action, reward in [(0,1.), (1,4.), (0,3.)]:
    count[action] += 1
    value[action] += (reward-value[action])/count[action]
print(count.tolist())
print(value.tolist())

[2, 1]
[2.0, 4.0]

Compare policies over complete sequences and repeated seeded environments. Keep the reward process stationary if that is the assumption being tested; a changing process requires a learner that can forget outdated evidence.

In a simulator with known expected rewards, expected regret compares the chosen action's expectation with the best available expectation. Realised reward remains noisy. Do not give the agent the simulator's hidden best action merely because the evaluator can use it.