Roberto I. Ono Filho
Independent researcher, São Paulo. Machine creativity: what stimulating a language model buys, measured in what it writes and inside it.
The question
Can an intervention at inference time make a language model more creative, reaching a genuinely better solution, or does it only change how the model explores what it already knows?
I have been trying to find out at office scale, one laptop and rented GPUs by the hour, under one discipline: pre-registered batteries read once, blind judges that score only generated text, exact verifiers with classical baselines, sham controls, no-source controls, every candidate logged. The models are read from the outside, in what they write, and from the inside, in what their internal states carry and in what a text loses when part of that is cut, erased or reset. The answer so far is a pattern rather than a verdict. Across the interventions tested, each bought variation, integration or obedience, and none bought progression. Negative results are published with the same care as positive ones, because the object here is not another method that moves a benchmark. It is a map of what each kind of intervention actually buys.
Three things an intervention can buy
The distinction the program is built on. The three come apart, and almost everything that raises the first two leaves the third where it was.
- Variation
- Does the model produce something different? Measured as judged surprise on windows of generated-only text, as distinct valid candidates in a search, as novel word sequences against the training corpus.
- Integration
- Does it use what it is given, and hold together what it has? Measured as judged connection, as integration at the level of a whole document, as whether a handed construction propagates through a search.
- Progression
- Does it reach something objectively better? Measured by an exact verifier against classical constructions and published records, with no judge in the loop and no room for interpretation.
What each intervention bought
| Intervention | What changed | What did not |
|---|---|---|
| Anti-probable sampling 01 | Surface novelty: about twice the novel four-word sequences against the training corpus. | Judged surprise, with no detectable effect; nothing in verified search. |
| Interruption: a new subject every 150 to 300 tokens 01 | Judged surprise by 1.2 to 1.4 points and connection by 0.7 to 0.9 over habituation alone; three to four times the valid candidates in a search. | The best verified heuristic, which does not move. |
| Schematic memory: the model’s own recap, re-injected 02 | Integration at the level of the document, the highest of the series. | Development: the model remembers and does not advance. |
| A verifier written into the flow 02 | Nothing measurable, on a carrier where the classical heuristic is already near optimal. | Progression. The model wrote 16.4 fabricated verdict lines per notebook, imitating the format of the reward it could not earn. |
| Consolidation: training the model on its own verified findings 02 | The mean, on variants it never saw, replicated in three independent lineages. | The best candidate, which equals the classical heuristic to six decimals; and better-than-classic candidates on distant families become rarer per candidate. |
| Operator packages in verified search 03 | 0.196 more of the seed-to-record gap (a nominal result), construction diversity in eight of nine problems, and with memory and repulsion together, no collapses. | The plateau: local, served and frontier proposers stall within 0.008 of one another. |
| A stronger proposer 03 | Sample efficiency: tens of samples for what the local model does not reach in hundreds. | That same plateau. |
| Pulses of a concept direction in the model’s internal state 04 | Judged surprise, by an amount that depends on the content of the direction: sham directions displace the state more and produce less. | The written word. The pulse trails text interruption at every matched offset, and costs coherence. |
| The same construction handed as code instead of words 05 | The result of a run: code above the no-hint baseline in 14 of 16 matched pairs, words below it, and the correct description placed next to the code does damage. | Anything about invention: in every one of these experiments the content was handed to the model. |
| A worked example of the same kind, in a formal world 05 | The exit from reasoning: a verified bridge in minutes, where the target alone spent its whole budget thinking and submitted nothing. | An operator the model lacked. At four times the budget, the target alone produces one too. |
| Training the model on its own verified proofs, three iterations 06 | Transfer within the family, and once past what any template showed; on new families, more solves than the base in three of four, at a fraction of the tokens. | The reach: every family an adapter solved, the base also solved with eight attempts and a large thinking budget. The loop stops at one rung. |
How the question narrowed
Read in order the papers look like a straight line. They were not written that way, and the line below is a reconstruction, useful to a reader and not a record of how it happened.
- The stream. What produces novelty in a model writing with no task? Of the operations usually credited for it, one survives measurement: interrupting a habituated stream with a new subject.
- The verifier. Does that carry to a problem where value can be checked? It multiplies candidates three to four fold and leaves the best one exactly where it was. Variation is cheap, and it does not convert.
- The weights. If inference cannot do it, can training the model on its own verified findings? The mean moves toward the known good, the frontier does not move at all, and better-than-classic candidates become rarer per candidate.
- The search. With operators instead of stimulus, how far does a loop get? Further, and then every proposer stops at the same plateau, one record away; a later analysis reads that wall as the resolution of the encoding rather than the family.
- The state. Is the model’s internal stream a channel that a prompt cannot reach? It carries content, and it still trails the written subject.
- The idea. If stimulation cannot produce it, what does the model do with an idea it is handed? It operates on the idea when it arrives as code, and the same idea in words makes the run worse.
- The loop. Can a model climb by training on the proofs it produced and a verifier accepted? Within a family, yes, and once past what any template showed; then it stops at one rung, and what moved was the prior toward forms it had seen and its repair after a rejection, not its reach.
- The inside. When a new subject interrupts a text, what does the old one leave in the new one, and where does it live? Its facts and its pull take different paths; the pull is read from its content words, in a code the two subjects share, and on natural text, in the hybrid models, resetting the recurrent memory removes more of it than stopping that reading. Read in October; not yet in a paper.
Open now
The questions the program is working on, each pre-registered before its data, as the others were. What is stated here has been read but not yet written up; when it is, it appears above, with its data and the script that produced it.
- What a model keeps when the subject changes. Eight pre-registered studies, in two hybrid models and a matched dense twin, read between 23 and 25 October; the line is closed and not yet written up. The previous subject’s facts and its pull on the new text take different paths, and the pull lives in a code the two subjects share. On natural text, stopping attention from reading the old subject’s words removed less than half of the pull, while resetting a hybrid model’s memory and cutting those words removed most of it and kept the facts. In the model whose continuations drifted back to the old subject, that brought them from 23% to 3%.
The two questions listed here earlier in October (where a handed construction does its work inside a 27B model, and a catalogue of forms aimed at the walls) were set aside on 22 October, before collecting any data, in favour of the question above. The census of the series’ errata, listed here in October, became report 07. The three questions listed here in September have since been read or set aside. The loop that trains a model on its own verified proofs became paper 06. Operators that change what the model computes (which experts it routes to, what the attention reads) were read and reached a limit; they are not yet in a paper. Learning on blind targets with no literature was not run.
Papers
- 07
Errata in AI-Executed Research: A Census of Six Papers, a Blind Replay of the Checks, and What the Record Says About Why
Every paper of the series was executed with an AI assistant, and an audit against the records found errata in all six documents it read. Replayed blind on the versions frozen before that audit, one fresh review by a strong model catches about a third of the errata, twelve reviews from six model families 60%, and all checks together 71%; what they miss is mostly what the text alone cannot show, claims about the work itself among them. Given the whole record, current models reconcile a stale draft without error, which fits errors that came from the process rather than from the writing.
- 06
Fine-Tuning a Language Model on Its Own Verified Lean Proofs: Transfer, Generalisation and Saturation Across Three Iterations
A served 27B model, fine-tuned for three iterations on the Lean proofs it produced and a verifier accepted, with no human selecting them. Its own proofs transfer within their family and, once, solve rungs no template showed; on four new families its adapter beats the base in three at a seventh to two fifths of the tokens. Then the loop stops at one rung, and a third iteration changes what it tries first, not what it solves. What moved was the prior toward forms in the training data and the repair after a rejection, not the reach: every family an adapter solved, the base also solved with eight attempts and a large thinking budget.
- 05
Operable Form, Not Stimulus: What Seven Pre-Registered Programs Say a Language Model Does with the Ideas It Is Handed, and Where Stimulating It Had a Ceiling
The series read as one experiment. Every inference-time stimulus bought variation, integration or obedience, none bought progression; every proposer stalled in the same construction family. Handed an idea, the model uses it if it arrives in operable form: code beats words in a verified search (14 of 16 pairs) and the correct description of a family hurts next to its code; in formal worlds a same-kind example buys the exit from reasoning within a small budget, not an operator the prior lacked.
- 04
What State-Space Injection Buys in Open-Ended Generation, and What It Costs
Intervening in the residual stream during generation. A guarded pulse of a concept direction buys a content-specific gain in judged surprise (shams in random or shuffled directions fail) and beats pressing the direction continuously, which collapses coherence; but against the written subject the pulse trails text interruption at every matched offset, and it costs coherence.
- 03
Operator Packages, Proposer Strength, and Construction-Family Plateaus in Office-Scale Verified Search
A FunSearch-style loop instrumented at office scale, on nine construction problems from a public repository. Three operator packages compose additively and close more of the seed-to-record gap; a frontier proposer reaches in tens of samples what the local model does not in hundreds; and every proposer stalls at the same plateau, a record away; a later analysis reads the flagship wall as the grid's resolution limit rather than the family. Named in words, the record's family is adopted and loses; handed as code, it is optimized.
- 02
Moving the Mean Toward the Known Good, Not Beyond It: What Inference-Time Interventions and Weight Consolidation Buy in Open-Ended Generation
Every inference-time intervention bought variation or integration and none bought progression; a verifier placed inside the flow was imitated rather than obeyed. Consolidating the model on its own verified findings moves what it writes toward the known good, replicated in three lineages, and not beyond it: the best candidate reaches the classical heuristic and stops there, and better-than-classic candidates on distant families become rarer per candidate.
- 01
Interrupting the Loop: Periodic Subject Changes Raise Judged Surprise and Connection in Base Language Models
A base model writing with no task drifts into repetition. Of the operations usually credited for novelty, one survives measurement: a new subject injected every few hundred tokens into a habituated stream raises judged surprise and connection, replicated across models and on held-out premises. The anti-probable sampler and the improbable prompt do not.
Working hypotheses
These are not findings. They are what the results suggest, stated so that the next experiment can take them apart.
- Interventions move the distribution more readily than the frontier. Reasonably supported across four substrates, with one caveat that matters: two of the nulls sit on carriers where the classical baseline is already near optimal, so there was little room to move.
- Conceptual access to an idea and the ability to operate on it may be different things. A search loop edits objects present in its context. Code is such an object and a description is not, and next to the code the description competes with the search. Held on one problem and one proposer family so far.
- What a worked example buys may be time rather than capability. In the formal worlds an example of the same kind made the model submit in minutes, and four times the budget let it get there with no example at all. Whether a model can produce an operator that is absent from what it learned is not settled by any of this, and needs a target with no literature behind it.
What would change our mind
Stated in advance, and one of them was met by an experiment run to test it. It is listed as met, and it is why the claim above is narrower than it was before that experiment.
- Not observed. An inference-time intervention that raises verified value beyond the proposer’s construction family, on a problem whose verifier can express the exit. A consolidation that raises the rate of better-than-classic candidates in a distant family. An intervention in the model’s internal state that beats the written word at matched offsets. A description of an operator that a search loop tunes to the level of that operator’s code. A proposer that produces a record’s construction family, without being handed code, on a target with no literature.
- Met. A formal bridge produced from the target alone, at four times the output budget. This is why the program no longer says that the model does not fabricate an operator, and says instead that the operable form buys the exit from reasoning within a small budget.
Method
- Pre-registration, single read. Hypotheses, arms, outcomes and the analysis script are fixed and committed before any data; each battery is read once.
- Judges that cannot be gamed by the intervention. Language-model judges from another model family score windows of generated-only text and never see the injected material; a human blind round checked the instrument.
- Exact verifiers and classical baselines. In verified search, every candidate is scored by an exact verifier against published records and classical constructions; claims are stated as “below the published value”, never “better than”.
- Controls that can undo a result. Shams that copy the surface of an intervention without its content, and no-source controls that reveal what the model already knew. Twice in this program a no-source control dismantled a result that had come out positive, and both times the negative version is the one that was published.
- Inside the model, with the same controls. In open models run in full precision, linear probes read what the internal states carry; interventions then remove part of it, by stopping attention from reading given words, erasing a direction or resetting a hybrid model’s recurrent memory, and measure what the text loses and what it keeps. Every intervention runs through a path checked against the untouched model and is paired with a sham of the same size.
- Everything released. Code, every logged candidate and model judgment, and the dated lab notebook through 28 August; individual human ratings only in aggregate.
Code and data
github.com/RobertoOno/interrupting-the-loop: the harness (samplers, the reverie loop, state-space operators, the verified-search loop and its domains), the analysis scripts, the papers’ LaTeX, and the lab notebook through 28 August. Run data of the first four papers as GitHub releases, run-data-v1 and run-data-v2, archived at doi:10.5281/zenodo.22714138. The later programs, with every session trace, verifier report and dated pre-registration, at doi:10.5281/zenodo.22713807. The corrected versions of October, with their run data, judge verdicts and evidence, at doi:10.5281/zenodo.23163298. The census of those errata and the blind replay of the checks, with the raw responses of every check, at doi:10.5281/zenodo.23562923.
Scale
The whole program runs at private-laboratory scale: one laptop, GPUs rented by the hour, and frontier models through APIs as judges and, late in the series, as proposers. A battery costs tens of dollars and a night. The constraint shapes what can be asked, and it is also part of the point: every experiment here is one that a single person can run, and that anyone can re-run from the released code and data.
Contact
- roberto@robertoono.com
- ORCID
- 0009-0006-8650-629X
- GitHub
- RobertoOno