Describing What an Action Changes

A schedule's value depends on the soil. Watering a dry tray may help, while watering an already wet tray may waste water. The same action cannot be evaluated sensibly without describing the situation in which it is taken.

Let's describe the situations the greenhouse can be in. For each state, we need to know which actions Botie can take and what might happen after each action. The next state may be uncertain, so we will record transition probabilities. Along with the reward rule, these give us a small Markov decision process to investigate.

Dry: probability 0.1, value 0
Moist: probability 0.9, value 10
Expected later value = (0.1 × 0) + (0.9 × 10) = 9

An action can have several possible consequences. Change its transition probabilities while keeping each row a valid distribution, then compare the expected values of the next states.

A state is the information used to predict the consequences of an action. The Markov assumption says that, given the current state and action, earlier history adds no further information about the next transition and reward distribution.

This is a modelling assumption, not a property guaranteed by giving a variable the name state. If two trays have the same moisture label but different root conditions that affect outcomes, our state may omit important information.

Partial observations may require remembering history or maintaining a belief over hidden states.

An action is a choice available in a state. A transition row gives probabilities for the possible next states after that choice. Entries must be non-negative and sum to one. A reward describes immediate feedback, which may depend on the transition as well.

Keep the next-state order fixed across transition rows and value vectors. Multiplying a probability by the wrong state's value can produce a plausible but meaningless result.

Next states: [dry, moist]
P(next | dry, water) = [0.1,0.9]
P(next | dry, wait) = [0.8,0.2]

The transition probabilities are supplied assumptions in this lesson, not estimated greenhouse measurements.

A deterministic policy selects one action in each state. A stochastic policy assigns action probabilities. Either can behave differently in dry and moist states.

A transition model describes what the environment does after an action; a policy describes which action the agent chooses. Do not confuse the two probability tables.

Example policy:
if dry: water
if moist: wait

The policy can be evaluated under the transition model even before learning a policy from experience.

import numpy as np
transition = np.array([0.75, 0.25])
state_values = np.array([0., 8.])
contributions = transition * state_values
print(contributions.tolist())
print(float(contributions.sum()))

[0.0, 2.0]
2.0

Multiply each possible next state's supplied value by its transition probability, then add. This is an expectation over possible outcomes, not a prediction that one trial will return the average outcome.

A complete action comparison adds immediate expected reward and an appropriate future discount. We will introduce that accumulation next; Botie's construction here isolates the expectation step.

Expected next value after watering
= 0.1 × 0 + 0.9 × 10
= 9

A single transition still lands in one state. Repeated trials need not equal the expectation exactly.

A dry-soil reading can occur immediately after a sensor fault or after several days without water. If those situations have different next outcomes under the same action, the reading alone may not supply a useful Markov state.

Add relevant history or maintain a belief over hidden conditions when needed. The Markov assumption says the chosen state contains the information needed for predicting the next transition. Naming an array state does not make that assumption true.

For each current state and action, the next-state probabilities should be nonnegative and sum to one. Their positions must refer to the same state vocabulary throughout the model.

An expected-value calculation averages across those outcomes. A simulated transition instead draws one outcome from that distribution. Confusing the two can produce a learner that experiences averaged states that do not actually exist.

import numpy as np
probability = np.array([.2,.8])
next_value = np.array([-2.,3.])
print(bool((probability >= 0).all() and np.isclose(probability.sum(), 1)))
print(round(float(probability@next_value), 3))

True
2.0