Planning One Step, Then the Remaining Journey

Each watering choice can lead to several future states, each with its own choices. Listing every complete future quickly becomes unwieldy. But many different paths can lead back to the same state.

We can keep one estimate for the remaining value of each state. Compare actions using their immediate rewards and those estimates, then update the state values and repeat. This reuses the same subproblem wherever it occurs.

One action can pay more now but leave less value for later. Change the supplied continuation values and check whether the preferred action changes.

0

For each action, combine expected immediate reward with the discounted expectation of the next state's value. A transition row averages over next states; the discount then scales the continuation.

Here rewards are already expected immediate rewards for each state-action pair. If a reward depends on the next state, its expectation must also be calculated under that transition row.

Q(s,a) = expected_reward(s,a) + gamma × sum_next P(next|s,a) V(next)

The value vector belongs to the next-state axis, not the action axis.

Suppose we already have a policy. To estimate how it behaves, we average action values using the probabilities with which that policy chooses actions. Value iteration asks a different question: which action currently has the greatest estimated value in this state?

That maximum is the optimality update we will use. Early estimates can be inaccurate, so choosing their current best action does not mean we have already found the optimal policy. We repeat the updates and examine convergence under the model's assumptions.

Fixed policy: V_new(s) = sum_a policy(a|s) Q(s,a)
Optimality update: V_new(s) = max_a Q(s,a)

Do not substitute a maximum into a calculation claimed to evaluate a fixed non-greedy policy.

In synchronous value iteration, calculate every new state value from the same old vector. Only replace the vector after the complete sweep. Reusing already updated entries within a sweep is a different update schedule.

Terminal states have no continuation and stay at their defined terminal value, typically zero after the terminal reward is counted. Stop using a chosen change tolerance or iteration budget.

V_old → all action scores → V_new
change = max(abs(V_new - V_old))

A finite discounted problem with bounded rewards has useful convergence guarantees, but floating-point tolerance is still a practical stopping decision.

import numpy as np
immediate = np.array([2., 0.])
continuation = np.array([1., 8.])
gamma = 0.5
scores = immediate + gamma * continuation
print(scores.tolist())
print(int(np.argmax(scores)))

[2.5, 4.0]
1

After updating values, choose a highest-scoring action in each state. Check the resulting actions, terminal handling, and reward accounting in simple states with known answers.

A perfectly solved inaccurate model can still produce a poor real policy. Botie's workshop calculates action scores for one state; the Python example repeats that principle in a complete two-state model.

Planning assumes transitions and rewards are known. The next lesson replaces a model expectation with sampled experience.

For each state, calculate the expected reward plus discounted continuation for every action, then retain the best action value. A synchronous sweep reads all continuations from the old table and writes a new table.

Updating entries in place creates a different update schedule. It can still be a useful algorithm, but its intermediate values should not be compared as though it performed the same synchronous sweep.

This two-state model gives Botie an action that stays where he is and an action that moves to the other state. Rewards belong to state-action pairs, and transitions are deterministic here.

Repeat until the largest table change is small or an explicit iteration limit is reached. Then extract the greedy action from the converged action values.

import numpy as np
reward = np.array([[0.,1.],[2.,0.]])
next_state = np.array([[0,1],[1,0]])
value = np.zeros(2)
for _ in range(1000):
    action_value = reward + .5*value[next_state]
    updated = action_value.max(axis=1)
    change = np.max(np.abs(updated-value))
    value = updated
    if change < 1e-10:
        break
policy = (reward + .5*value[next_state]).argmax(axis=1)
print(np.round(value, 4).tolist())
print(policy.tolist())

[3.0, 4.0]
[1, 0]

For a finite discounted model with valid probabilities and discount below one, Bellman optimality updates contract differences between value tables. This supports convergence of the planning calculation.

It does not establish that the model accurately describes the greenhouse. A missing action cost or incorrect transition probability can produce a precisely computed but poor real policy. Compare simulated policy behaviour with observed transitions and test sensitivity to uncertain model entries.