Picture a small simulated creature foraging on a grid. It has learned that one kind of target is worth collecting and another is worth avoiding. Halfway through its life, without warning, the two trade values. To keep doing well it has to notice that its favorite has gone bad and, which is harder, find out that the thing it learned to avoid is now worth having.
We study agents like this at FriendMachine, in a research project called Embodied Field Intelligence. Our agent learned target values with a crude rule, so we replaced it with a standard textbook one and expected a small improvement. The agent did get better at noticing that a reward had changed. But it collected less reward when targets moved, and it often never rediscovered a target that had turned good.
What we took from it is that learning what something is worth and deciding how strongly to pursue it are different jobs. When we separated them, and gave the agent a bounded reason to revisit things it had learned to avoid, average return (the total reward the agent collects over a lifetime) rose by about 40% on fresh tests. The rest of this post explains how we got there and where the result falls short.
The setup
The project explores agents whose memory and planning are built from fields. A field here is a layer of values laid over the agent’s internal map, where each cell’s value is updated from its neighbors, so information about rewards and costs spreads across the map and the agent can move toward promising areas.
The agent wanders a small grid world, seeing only a 5×5 patch around itself. Two kinds of target, A and B, are scattered around. Picking one up pays a reward, and every move costs a little. The agent keeps one learned weight per kind of target, a single number for how good it believes that kind is, and that weight shapes the fields it uses to decide where to go.
We tested four kinds of world: static rewards, targets that grow back, targets that move, and a reward swap. Most of this post is about the swap. Target A starts at +1 and target B at −0.8, and at action 300 of a 600-action lifetime they trade values. A good learner should stop collecting A and discover that B is now worth having.
A rule that piles up, and a rule that tracks
Everything depends on how the weight responds when a target’s reward changes. The existing rule adds each reward to the weight and caps the result:
weight ← weight + 0.25 × reward, kept within −1.5 to +1.5.
After twenty +1 rewards, the weight sits at the +1.5 cap. When that target starts paying −0.8, the agent needs eight bad pickups before the weight stops being positive.
The replacement is the standard way to track a changing average, from Sutton and Barto’s Reinforcement Learning: An Introduction (chapter 2). It moves the weight a fixed fraction of the way toward each new reward:
weight ← weight + 0.25 × (reward − weight)
Now the weight estimates what the target actually pays. Twenty +1 rewards leave it at +1, and only three −0.8 pickups flip its sign.
What happened instead
We fixed every setting in advance, so the results could not be tuned to the test, and then compared the two rules on 24 fresh blocks of runs, 96,000 actions in all. The runs were paired, so both rules faced the same randomly generated worlds. Each result below comes with a 95% interval, a range that reflects how much the difference varied from block to block. An interval that sits entirely above zero means the gain is unlikely to be an accident of which worlds were drawn.
The intended effect held up. After the swap, the agent collected fewer bad A targets, 3.75 on average, down from 8.17. Return after the swap improved by 2.53 points, with a 95% interval of [+0.08, +5.41].
Two other things got worse. The agent often stopped finding the newly good target. Pickups of B after the swap fell from 3.08 to 2.08, and in 19 of 24 runs the new learner never found B again before its lifetime ended, against 15 of 24 for the old rule. Probably it had learned early that B was bad and kept steering around it, so it never saw that B had changed. It also lost return when targets moved, by 3.25 points, with an interval of [−5.91, −0.64].
One likely contributor is that an accurate weight of +1 pulls less strongly than the old rule’s capped +1.5, so the more honest number may have produced a weaker attraction. The tests did not isolate the cause.
Separating belief from action
Both failures suggested that a single number was doing two jobs, recording what a target is worth and setting how hard the agent chases it, and that making it better at the first may have made it worse at the second.
The next round of work pulled the two jobs apart. The agent keeps an estimate of what each target is worth, learned only from rewards it actually receives. A separate step turns that estimate into how strongly the agent pursues the target. We call the whole package of rules that turns what the agent knows into moves a controller.
That split made room for two additions. First, a readout gain, a fixed multiplier, doubles the estimate before it shapes behavior, restoring the pull the honest estimate had lost. Second, as time passes since the agent last picked up a target type, it becomes slightly more willing to try that type again, up to a limit.
A constant willingness to revisit helped after the swap but kept sending the agent back to targets that were still bad, costing return elsewhere. What worked was making it conditional. The agent carries a reward-change signal. When a target pays something different from last time, the signal jumps by the size of the change, up to 1, and then fades slowly. The revisit bonus is scaled by that signal, so the agent reconsiders what it has been avoiding mainly when rewards have recently changed.
pursuit strength = 2 × (estimate + age bonus × change signal)
The age bonus grows from 0 toward 1 with time since the last pickup, and the estimate now moves three-quarters of the way toward each new reward.
The result on fresh tests
We committed to one controller and its settings before evaluating it, then ran 40 new blocks, 480 lifetimes and 240,000 actions in all.
Average return rose from 15.95 to 22.31, a gain of 6.36 points (39.9%), with a paired 95% interval of [+5.56, +7.13]. In the swap world, pickups of the newly good B after the swap rose from 2.05 to 16.83, and pickups of the newly bad A fell from 8.30 to 2.05. The old rule never found B again in 33 of 40 runs, and the new controller missed it in 2. The static and regrowth worlds did not get worse (+0.085 and +3.87 points). A separate stress test with larger rooms, longer lifetimes and a later swap also improved in every condition.
Caveats
The controller did not pass every check we set. Before testing, we required that it do no meaningful harm when targets move, meaning the bottom of its 95% interval had to stay above −0.50. The average there was positive (+0.92), but the interval reached −1.29, and −0.70 in the stress test. By our own rule that is a failure, so the new controller is an option that has to be switched on, and the agent’s default behavior is unchanged.
We can’t say which part did the work. The readout gain, the faster learning rate and the change-gated bonus changed together.
The revisit rule is a hand-designed heuristic. One change signal is shared by both target types, so a change in either prompts revisiting both. That fits the swap world, where both change at once, but it is not learned evidence that targets change together. Every test world had noiseless rewards, so random variation could trigger the signal falsely, and a change to a target the agent never revisits goes unnoticed. None of these cases was tested.
The field-based design did not make the difference. These ingredients are classical, so we tested them in other designs too. A later comparison gave the same rules, under the same computing limit, to the field agent and to two conventional agents, one that stores values in a table and one that plans by searching its map. Over 1,152 held-out lifetimes in eight kinds of world, the rules improved all three. The table version matched the field version while using about 14% less computing time. No advantage for the field design was established, though the comparison could not show the two are equivalent either. It used a pared-down version of the agent, not the full controller described here. Credit the result to the learning rules, not to the field-based design.
Finally, these are small, synthetic worlds. The 40% figure averages four particular kinds of world and is not a general measure of capability.
In most runs our agent never went back to B after the swap, so it had no chance to learn that B had changed. The same split between belief and behavior may matter beyond foraging, especially when an agent’s caution keeps it from collecting the evidence that would correct it.