The research in plain words

The question

Can you nudge a language model into a genuinely better answer, or only into a different one?

A large language model has read an enormous amount of what people have written. It is tempting to think that the answer to some unsolved problem is already in there, and that what is missing is the right way to ask. This research asks whether that is true, in a form that can be checked.

The phrase “genuinely better” is doing real work in that question. If a model writes something more surprising, that is a change in the text. If it writes a program that packs boxes more tightly than the best method known, that is a change in the world, and a machine can check it without anyone’s opinion. The experiments keep those two apart on purpose.

Why it is not obvious

Models are easy to make more interesting and hard to make more correct. Nudge one and it will write differently: stranger images, wider vocabulary, unexpected turns. Readers notice, and so do other models used as judges. The open question is whether any of that converts into a result a verifier would accept, and answering it honestly means resisting the strong impression that it did.

How you test this without fooling yourself

Three habits, borrowed from fields that learned them the hard way.

• Write the experiment down first. Before any data exists, the hypotheses, the conditions, the measurements and the analysis program are fixed and filed with a date. Then the data is read once. This removes the possibility of noticing a pattern afterwards and calling it a prediction.
• Do not let the judge see the trick. When a language model grades the text, it sees only what the model under test wrote, never the intervention. Otherwise the judge grades the experimenter’s own sentence and credits it to the model. That happened in an early version of this work, and finding it changed the result.
• Use a verifier wherever one exists. Some problems have exact answers: pack these items, place these circles, build this set of numbers. A program scores the candidate against the best construction known. There is nothing to interpret, which is the point.

And keep controls that are able to undo your own result. One kind copies the surface of an intervention while removing its content. Another gives the model nothing at all, to reveal what it already knew. Twice in this program the second kind dismantled a finding that had come out positive, and both times the negative version is the one that was published.

What has shown up so far

A pattern, across interventions at five different depths of a model, from the way it picks each word to the internal state it carries while writing.

• Interventions buy variation. Interrupting a model with a new subject every few hundred words makes it produce genuinely more varied material, three to four times as many distinct candidates when it is searching for a solution.
• They do not buy progression. That extra material never contained a better answer than the model was going to find anyway. Training the model on its own verified successes moved its average toward the best known method, and its best answer reached that method’s level and stopped exactly there.
• Form matters more than expected. When an idea is handed to the model instead of coaxed out of it, the same construction given as working code is picked up and improved, while given as a correct description in words it is obeyed and makes the result worse. Adding the description next to the code does damage.
• The models were also examined from the inside. When a text changes subject, the old subject keeps pulling the new one through a code the two share inside the model. In models with a recurrent memory, resetting that memory and hiding the old subject’s words from the model removes most of the pull and keeps what the model knows about the old subject; in one of them, this cut the continuations that drifted back to the old subject from about a quarter to 3%.

What this does not show

It does not show that models cannot do these things. It shows that the interventions tried here, at the scale of one laptop and a few processors rented by the hour, did not do them. One of the boundaries claimed earlier in the program was tested directly and gave way: a model that appeared unable to produce a certain kind of idea on its own produced it once it was given four times as much room to think. The claim was narrowed in public, and the experiment that narrowed it was published with the rest. In October an audit of every paper against its own records found errors in all of them; the corrected versions, each with what changed, are the ones on the home page. A report on those errors, which kinds got through and which checks would have caught them, is there too.

That is the part of this work I would most like to be judged on. The results that survive are the ones that survived an attempt to break them.

Where to go next

The home page has the papers, each with its data and its code, a table of what every intervention bought and did not buy, the working hypotheses, and the list of results that would change the conclusion. Questions are welcome at roberto@robertoono.com.