← All workshopsRead Valuing Rewards That Arrive Later

Reinforcement Learning

Combine the rewards from one episode

BOTIE

Create one discount weight per reward, starting at one, then take the dot product of rewards and weights.
Ready to test
0 passed · 0 failed · 2 not run

An empty floor. Start with the information your machine needs.

Received result

Your answer will arrive here.

Examples, experiments and workspace controls
Live values

Connect a tool's inputs to see what it produces. You can try a branch before connecting it to Result.

Try your own inputs

Use a copy of the example to try your own data. Keep the problem's assumptions in mind, and predict the effect of a change before checking the output. You can also add another branch to compare two ways of doing the calculation.

Start at an output arrow, then click or drag to an input. Drag empty space to pan. Delete removes a selection. Ctrl/Cmd+Z undoes a change.
Context for this challenge

Create one discount weight per reward, starting at one, then take the dot product of rewards and weights.

Botie has a completed reward sequence. Earlier and later rewards need weights that reflect the declared per-step discount.

Create weights beginning at one and progressing by gamma, then take their dot product with the ordered episode rewards.

Examples and results
Rewards after successive actions
  1. 0-1
  2. 10
  3. 24
Discount per step
0.5
Required result
0
Rewards after successive actions
  1. 02
  2. 13
  3. 29
Discount per step
0
Required result
2
Try a fresh construction

This clears this draft, including its undo history. Earlier successful attempts stay recorded. Build again without opening hints to record an independent attempt.

Stuck? Inspect a worked construction

The tools below form one possible solution. You can assemble it, follow its signals, and undo that change to return to your own construction.

Assumptions and test precision

Rewards and episode termination are supplied. A return from one realised episode is not a policy's expected value. Numerical tests allow a difference of 0.001. Lists and structures must also have the right shape.

The cost arrives before the benefit

This check uses the pieces from “Attach a discount weight to every reward”.

BOTIE

Botie's route costs 3 reward units immediately, earns 2 at the next step, then earns 8 one step later. Calculate its return with discount 0.5, starting the first reward at discount power zero.
Rewards after successive actions
  1. 0-3
  2. 12
  3. 28
Discount per step
0.5

Use a number, text, or an array matching the requested result. For a matrix, put each row in brackets. Saving your prediction runs your current construction on these inputs and checks the earlier examples too.

Build a construction in the workspace above first. This case will test those pieces.

Why is this realised return different from the expected value of a policy?

One episode supplies a trajectory's return; a policy value must account for other possible trajectories.

Rewards and episode termination are supplied. A return from one realised episode is not a policy's expected value.

Work through the Python challenges →
Continue the coursePlanning One Step, Then the Remaining Journey →
← Describing What an Action Changes