We are FriendMachine, and for the last few years we have been giving AI agents research problems in science and mathematics that are genuinely hard, often beyond what we expected them to solve. The agents propose ideas, write code, run experiments and report results, with people choosing the problems and setting direction.
We do this for two reasons. Some of the problems are worth solving in their own right. And watching agents attempt them is one of the best ways we know to measure what the models can actually do: where they make real progress and where they produce convincing nonsense instead. Both change as the models improve.
That second purpose means we expect a lot of wrong answers. An agent pushed past its abilities will often report a result that looks right and isn’t. We let those results stand for a while instead of stopping every experiment until it was airtight, because the errors are part of what we are measuring. Then, periodically, we have newer models go back over the older models’ work, check it and improve the projects, both to correct the record and to learn which kinds of mistake the earlier agents made.
In July 2026 we did one of those rechecks, with newer models reviewing and reworking the projects produced since September 2025. By our count it contained roughly fifteen headline quantitative claims, including an apparent signal in atmospheric methane, a sound locator with remarkably tight error bars, and a simulation result backed by a passing test. About one held up.
What the recheck found
Most of the failures could have been caught with evidence that was already sitting in the same project.
One analysis, part of a search for signatures of life in atmospheric data, reported that a methane series looked very different played forward than backward, with a score of 0.974 on a scale where 0.5 means no difference. It had been run on a series that rises steadily and repeats every season, without first removing that trend and cycle. The project’s own more careful rerun, which did remove them, scored about 0.556 and found no signal. That rerun sat two folders away from the announcement and had never been reconciled with it.
An early version of our sound locator, a tool meant to work out where a sound came from using several microphones, reported 95% confidence regions less than a nanometer across. Its actual error in that test was about 44 meters. The project had 117 passing automated tests, but none compared the uncertainty the tool reported with the error it actually made. The deeper problem was that the question, as posed, could not be answered from the data. Each microphone’s clock could be off by an unknown amount, and with a single sound that left the location undetermined however many microphones were used. Nothing checked for that.
Simulations depend on a random starting value called a seed, so a real effect should not depend on which seed you pick. One claimed ordering, between chaotic and orderly simulated behavior, held in only 5 of 15 seed and configuration combinations, and its direction flipped from seed to seed. Its passing unit test happened to use one of the seeds where it held.
One “working proof of concept” had no surviving code, because the experiment had never been committed. An automated research assistant later produced a project report about it anyway.
What the mistakes have in common
These are recognizable agent failure modes. An agent asked for a result tends to produce one, along with tests that pass and a summary that sounds confident. It rarely asks whether the question can be answered at all, or compares its number against the boring explanation. The report about an experiment that never ran is the extreme case, but the methane score and the nanometer error bars are the same pattern in a milder form.
The tools for doing it right were often already in the projects. Some had good statistical checks, and at least one had a well-written analysis plan. The headline numbers came from scripts that went around them. Claims were written faster than they were checked, and once a number reached a summary document it looked as trustworthy as a validated one.
The failures were concentrated in applications rather than the core idea. Some of that work applied one central method, a way of measuring from data how irreversible a process is, meaning how differently it behaves run forward versus backward. The one result that had been checked carefully from the start held up, reproducing a known identity linking time-asymmetry to physical entropy production to within 1%.
What we changed
We still want agents working on problems at the edge of what they can do. What changed is the line between an agent’s output and a claim we stand behind. Exploratory results are allowed to be wrong. A result only becomes a claim after it passes checks that were set before anyone saw the answer.
Those checks are written into our current research program, a plan we wrote in July 2026. It asks whether systems built from local rules with inspectable internal state, rather than trained neural networks, can match today’s best systems at chosen tasks.
Every milestone carries a kill test, a result stated in advance that would show the idea does not work. The plan puts it plainly: “A milestone without a way to fail is not on this list.” Before an experiment runs, we write down the hypothesis, the measurement and the threshold for success, which is called preregistration. It stops anyone, person or agent, from picking whichever comparison happens to look best afterward.
Each quantitative claim gets an entry in a registry recording how to reproduce it and whether it is validated, unvalidated, refuted or retracted. A claim that is not recorded counts as unvalidated, and it loses validated status when its code or data change. A pattern only counts if it beats versions of the data that keep the boring structure, such as trends and seasons, but remove the effect being claimed. Any tool that reports a confidence range must show, on simulated data with known answers, that the truth lands inside that range about as often as it says. We also check whether the data can determine the answer at all, which is the question the sound locator never asked.
Failed tests stay in the reports with their numbers, and the plan lists explicit non-claims, things we are deliberately not asserting. One reads “no claim that field dynamics are how brains work,” where field dynamics means the local rules our systems are built from.
What we are probably getting wrong now
If one in fifteen survived last time, it is fair to ask what is wrong in what we publish today, and whether to trust it.
Some of it is probably wrong. The rules above make errors less likely and easier to find, but they don’t make us right. What we can offer is a way to tell how much weight each result can bear. Our notes say what was tested, on what kind of data, and what was not shown. A result described as tested on synthetic data has not been tested on real data. A test that failed its preset threshold is reported as failed, even when the numbers lean our way. When something we published turns out to be wrong, we will correct it in place and say what changed.
We will also keep doing what produced this note. We plan to recheck our results every quarter, and to return to older ones as models improve, partly to see what newer models catch that older ones missed. A checking model can share blind spots with the work it reviews, so each recheck finds some errors, not necessarily all of them. If you find a problem before we do, we would like to hear about it.
There are costs. Plans, registries and proper baselines take time that could go into the next idea, and strict tests mean more results end up marked “not established.” Large ideas tend to get audited down into smaller, true ones, so we keep speculative ideas in a clearly labeled place, separate from the record of what has been shown. Whether these rules actually produce more results that hold up is itself a claim, and the next recheck is where we will test it.