首页 > AI前沿 > Who made the strawberry? 14 days of logs building a procedural fruit with agents

Who made the strawberry? 14 days of logs building a procedural fruit with agents

Hacker News 2026-11-19 02:00 20 阅读 查看原文
AbstractThe case study “Who Made the Strawberry?” was built in about 24 hours of work over three days, Nov 14 to 16, 2026, by one main agent and the agents it started. This record lists what was done in 58 steps: counting scripts, two readings of the logs, analyses, reviews, consensus rewrites, cross-reads, adversarial readings, a blind test, independent studies by another model family, translations, restructurings, a recovery and publication. It shows how many agents of which model ran at once, how the output of one step fed the next, and what each check found. The checks found 33 errors in results or text; 23 were corrected. No second human took part, so every check described here is a check by models. 1What this record is This tab describes the process that produced the study. It is not part of the study's argument. Its sources are the logs of the Claude Code session that built the study (from Nov 14, 14:36), its 201 subagent runs, eight Codex runs in the study folder, and the study's own briefs, reports and reviews. One model reconstructed the timeline from these logs and cut it into 58 steps. No second coder checked that cut. Private material in the logs was left out. The reconstruction ends on Nov 16: its steps, figures and totals, and the limits drawn from them, describe the process up to then, not the finished study of Nov 17. The study was commissioned by the user, who approved its main steps and its publication; no other human took part. One main agent did the orchestration. It wrote briefs, started subagents, merged their results, ran scripts and published. It ran on claude-fable-5-2 until Nov 15, 00:16, and on claude-opus-5-5 after that. Subagents were Claude models started by the main agent or by other subagents. Codex runs used models of another family, gpt-6-astra and gpt-6-luna, started from the same folder. The study was started on November 14, about 14 hours after the result was posted on Reddit, so that it could be linked from there. The user and the same kind of AI agents that the study describes started it. So the study is not neutral ground, and its limits section says so (the study, ). Since Nov 17 the two tabs are divided by one rule. The study holds what a reader learns about the project, plus what was counted, who coded, how far two coders agree, and the limits. This tab holds everything about how the study came about: the readings and codings and how they compare, what went wrong when reading the logs, the corrections, and the reviews. Sections 5 to 12 were moved here from the study under that rule. 2The steps Nov 14. Within the first hour, 14 day readers cut the 14 days into 265 episodes, two coders labelled them, and the page went live; from then on every change was public. A fresh reviewer found seven problems in the afternoon. In the evening the counts moved into scripts, a second counting path found a gap of 138 tool calls, and an agent tracing ideas found that the digest lacked about half of the messages. So 14 new readers re-read every day from the complete chat. Two reviews, a blind test and eight adversarial readings followed before midnight. Nov 15. The morning went into a plain-language mode (7 agents) and five translations (66 runs, at most 20 at once). In the evening a style audit led to three independent rewrites and a consensus, a check for changes of meaning by another model family, two cross-readers and two independent studies by another family. Two agents then built a shorter preview. Its review by another family stopped at the usage limit, and a same-family review replaced it. Nov 16. The restructured version went live, and two process studies, a linkograph and 32 sources were added. A cleanup in an unrelated session deleted the local working folder; it was restored from the live site and the logs. The evening brought a plain-language pass checked file by file, a peer review by another family, a check of two listed limits, and an audit of all days against the raw logs. Nov 17. A model of another family coded a sample of 60 episodes again for the roles, without seeing the first coding (). An overall review by a fresh instance of the writing model followed, and the study was rebuilt: findings grouped by the kind of evidence, roles as an ordering by two coders with both coders side by side, and this tab for the making of the study. An independent review by a model of another family then checked the build against its data and sources, and its corrections were built in. These steps are not in the 58 steps and the figures of this section, which end on Nov 16. 3How outputs flowed Most steps took the output of an earlier step and passed their own on. Three patterns recur. draws one example of each. Parallel and blind. The same task went to several agents that did not see each other: one reader per day in both readings, two reviewers of the same draft, two independent studies, six adversarial readers with only the data. Parallel runs made the work fast. Keeping them blind kept the outputs separate enough to compare. Consensus. For the style rewrite, three agents rewrote the same text on their own and a fourth merged the three into one version. The merged text then went to a model of another family, which checked it field by field for changes of meaning, and to two fresh readers, one of each family. Check, then fix. Most checks did not change anything themselves. A reviewer wrote findings, and a later step applied them, often the main agent and sometimes a corrector agent. Some loops reached far back. The day audit on Nov 16 corrected day texts written by the second reading on Nov 14, and the peer review sent section 6.4 back for a fix within an hour after it had gone live. The links between the steps also show dependence. Almost every analysis after the first evening built on the episodes cut by the first reading and recoded by the second. Later checks that read the same episodes were therefore less independent than their number suggests. 4Effort per phase The main agent made 1,769 tool calls, its 201 subagent runs made 5,991, and the eight Codex runs made 135. About half of the subagent runs went into simplifying, translating and restyling text, not into analysis. The phase of plain language and translations alone used 73 runs (), twice as many as both readings together. Checks were small in runs and large in effect: the first review was one agent and found seven problems, and the day audit used 16 runs for 292 corrections. The Claude models processed about 1.5 billion tokens, nearly all of them cache reads, and wrote about 10.8 million. 5The two readings of the logs The episodes of the study were read twice. This section says why, what changed between the readings, and how far they agree. The study shows the second reading. Episodes. For each day, one agent read all prompts of that day included in the initial digest and the first 500 characters of each agent reply. It split the day into episodes and gave each episode one of six labels. Sorting episodes into such fixed categories is called coding here. The labels are a judgement of the reading agents. Revision comparison. The second reading kept the first reading's episode boundaries and could see its episode texts, but read the complete chat with another model and other instructions. On the 263 episodes both readings share, they give the same label to 80.6%, the same starter to 95.8%, the same deciding contribution to 79.8% and the same named mistake to 77.6%. By the scale of [47], agreement beyond chance is moderate for the label, weak for the deciding contribution and the named mistake, and minimal for the starter (box below). This compares two revisions. It is not an independent reliability estimate: shared texts can raise agreement, and changed evidence and rules can lower it. So the shift in is not due to the missing messages alone. Agreement beyond chance is measured with kappa: 1 is full agreement, 0 is chance. For the label kappa is 0.74, for who started an episode 0.34, for whose contribution decided it 0.53 and for whose mistake the chat names 0.58. Who started an episode has the lowest kappa despite 95.8% agreement. The reason is that the user started almost every episode, so chance alone already gives a high match. The deciding contribution was later replaced by the role coding below and is kept only for this comparison. 6Earlier codings Before the role coding, the same episodes were coded in two other ways: by kinds of contribution, and by whose contribution decided an episode. Both stay in the data and are shown here for the record. Neither is used for a finding of the study. The role coding replaced the second one after a review by a model of another family found that its rule tested different things for each side. What the user brought What the agents brought The earlier coding asked whose contribution decided each episode. Its rule tested different things for each side and put the most common pattern under “both”. Read as roles, “both” is almost always the same division of work: the user proposes, an agent builds, the user decides. In 17 of the 32 episodes once coded “user decisive” an agent named the cause, and 11 of them ended abandoned or unresolved, which the old rule should have coded “none”. “Agent decisive” mostly meant that an agent found the cause. 7The role coding and its independent check The study reports roles only as an ordering by two coders (the study, ). This section holds the full comparison of the two codings. The table and the chart of the comparison stand in the study (). Roles. On Nov 16 one agent (claude-opus-5-5) coded every episode again for eight roles: who proposed the approach, requested the work, implemented it, named the confirmed cause, judged the result by looking, judged it by measuring, kept or dropped it, and stopped running work. Each role is user, agent, both, none or unclear, and each episode also gets an outcome (completed, revised, abandoned, unresolved) and a scale (small fix, feature, change of direction). The codebook was written before coding. Where a text did not decide, the raw log was read. After the text coding, stops and agents' visual checks were added from the logs by a fixed rule (box). The coder knew the earlier codes and is of the same model family, so this is not a reliability estimate. A coder of another model family, which recorded itself as “GPT-6 (Codex; exact runtime revision not exposed)”, then recoded a stratified random sample of 60 episodes blind, with the same codebook (). It did not run the log pass. The comparison script merged the first coding's log values into its codes afterwards, so for judging by looking and stopping the two codings share that input and are not two measurements of those log events. It agreed on proposed in 65% of the sample (kappa 0.39), diagnosed in 70% (0.49), judged by looking in 63% (0.46), measured in 55% (0.24), implemented in 87% (0.38), kept or dropped in 65% (0.41) and stopped in 80% (0.64), on the outcome in 77% (0.52) and on the scale in 80% (0.64). Requested matched in 93%, but its kappa is near 0 (−0.01) because nearly every episode was requested by the user, so chance agreement is already high. Counted per side, the second coder recorded more acts for both sides in most roles (). The comparison file also scores each disagreement by a relative rule: user +1, agent −1, both and none 0, and a change counts as towards the side whose score rose. By that score the differences were 18 towards the agents and 3 towards the user for proposing, 14 and 0 for diagnosing, 15 and 5 for judging by looking, 25 and 1 for measuring, 10 and 10 for keeping or dropping, 3 and 7 for stopping, and 0 and 8 for implementing. The score is not a count of acts: a change from user to none counts as towards the agents although no agent act was added (3 of the 18 for proposing), and a change from none to both counts as neutral although both sides gained an act. It is no evidence that either model family favoured one side. The independent coder named the parts of the codebook it found ambiguous: when a contribution is essential enough to count a proposal for both sides, when added work is substantial enough to count a request for both, whether a request to cancel that had no effect counts as a stop, which events belong to overlapping episodes, and how to code an episode with mixed outcomes. Both coders are models, and 60 episodes is a small sample. The words for the agreement. By the scale of [47], kappa from 0.21 to 0.39 is minimal agreement, from 0.40 to 0.59 weak and from 0.60 to 0.79 moderate. So the two coders agree minimally on proposing (0.39), implementing (0.38) and measuring (0.24), weakly on naming the cause (0.49), judging by looking (0.46), keeping or dropping (0.41) and the outcome (0.52), and moderately on stopping and on the scale (0.64 each). Without the log rule, looking agrees in 43% (kappa 0.24) and stopping in 75% (kappa 0.54). Only 6 of the 60 episodes carry the same ten codes from both coders. How to read the comparison. Four things limit it. Neither coding was adjudicated, and no human coded; the second coder's notes call its remarks “coding explanations, not adjudication against the first coder”. The second coder recorded more acts in nearly every role and for both sides, so the two coders set the threshold for a recorded act differently, and which threshold is nearer to the logs is open. Each coder belongs to the family of one of the project's agents: the second to that of its Codex agent, the first to that of its Claude agents. And the sample holds only 3 of the 21 episodes coded as a change of direction. Those three were checked, but they are too few to test the pattern for that group. Possible leanings of the coders. Three known leanings of models as judges could each have played a part. None was tested here. Models have been found to recognise their own texts and to rate them higher [76]; that was tested on ratings of generated texts, not on which actor a coder credits in a work log, so it gives no direction for this comparison. Models adapt their answers to the view of the person they answer [77]; the user commissioned the study, so a coder could lean towards the user. Models as judges have preferred longer answers [78]. That is a preference in rating answers, which is not the same as unequal evidence: the logs hold more text about the agents' acts than about the user's, so a coder who looks for evidence in text can find more of it for the agents. Neither leaning is shown by the differences between the two codings, and each is only a reason to treat neither coding as the standard. What holds. The ordering of the two sides is the same for both coders in all eight roles. Two counts hardly move: that agents implemented (60 of 60 for both coders) and that the user requested (59 and 60). The others move with the coder. For the user, the first coding is the higher of the two on proposing (52 against 44 of the 60 episodes) and the lower on hand work (3 against 11), looking (45 against 52), keeping or dropping (32 against 41) and stopping (26 against 30). For the agents, the first coding is the lower on every role but implementing. An illustration of that movement, not an estimate. If the second coder's shares in the sample held for all 265 episodes, agents would have taken part in proposing in about 119 episodes, where the first coding has 85; in naming the cause in about 194, against 127; and in measuring in about 212, against 113. Resampling the 60 episodes gives ranges of 88 to 155, 163 to 221 and 186 to 238. The projection assumes that the sample stands for the whole. It does so only roughly: in the first coding, agents take part in proposing in 23% of the sampled episodes and in 32% of all. These figures show how far a count can move with the coder. They are not counts of the study. Outcome and scale. The second coder called fewer episodes completed (37 against 46 of 60) and more revised or unresolved (22 against 12). One episode as an example. The add-on decision of Nov 5 (the study, ) was in the sample. The two coders agree on nine of its ten codes. They differ on keeping or dropping: the first coder gave it to the user, who took the decision; the second gave it to both, because the agent had also dropped its own automatic plan. The codebook allows “both” when an agent discards one variant and the user approves another. Whether the agent's remark was such a discard is the judgement on which the two differ. Two earlier small checks. Before the role coding, two further agents without access to the study each checked 16 random episodes and agreed with the old coding of the deciding contribution in 13 and in 16 of them. These were checks by models of the same family and are not reliability estimates either. 8The process readings compared Three readings traced how the goal of the project changed: two process studies and a linkograph. They were made separately, each by one coder and by its own rules. The study shows the counts of both process studies side by side (the study, ). This section sets the three readings next to each other and says why their numbers differ. Thread shares differ widely: the intro path holds 16.7% of prompts in process study 1 and 2.8% in process study 2, the level video 9.7% and 3.5%. None of these measures is the user's attention. An inclusion ledger that reconciles the three corpora has not been made. How the links of the linkograph were found. Its links were first found automatically from shared words, and only selected candidate pairs were checked one by one by the coding agent. Of the checked automatic links, 43% were real building on earlier work, and links that are not shown may still exist. The linkograph uses whole episodes as steps, which favours the user, because episodes are organised around what he asked for. 9Counts that differ Some things were counted more than once, by different rules, and the counts do not fully match. The study gives the count it uses; the others are here. Helper agents. Agents in Claude Code started helpers 31 times, and 30 helper logs exist; they made 3,440 tool calls. The count of work alone (the study, ) has 33 helpers, because it adds three sessions that agents had started, with 3,954 calls. A later count by file found 33 helpers with 4,259 calls; that difference was not resolved. Prompts by model. The table of models counts prompts by the model that answered them (3,068 + 500 + 388 = 3,956; two prompts have no model there). The stop and cost tables count by the model of the run (3,073, 497, 378 and 10 unknown), so their numbers differ slightly. Pictures. Several counts of pictures appear in the data, because they count different things. They are listed in the box “Why the picture counts differ” in the study, . 10What went wrong when reading the logs Four things went wrong, or turned out to be traps, when models read the logs. Each left a limit in the study (the study, ). How each was found is told here. A digest that lacked half of the messages. The first readers worked from a shortened copy of the chat. It lacked 1,931 of the user's messages, about half: 241 that began with a screenshot and 1,690 he sent while an agent was working. Nothing in the first reading's output showed the gap. An agent that traced ideas found it because it met messages it could not place. The second reading followed (). The first process study met the same gap: it found that the daily digests leave out about 43% of the user's messages, those sent while an agent was still working, and it took 8 of its 30 turning points from the raw logs for that reason. A keyword rule that did not hold. For the purpose of the user's image prompts, a keyword rule agreed with the codes of a model in only 23 of 50 cases. So the study gives no count for all image prompts, only the sample of 50. A log string with two meanings. The tool logs “The user doesn't want to proceed with this tool use” both for a refused command and for an interruption by a new message. 97 of 109 such events came within 10 seconds of a new prompt, so most are interrupts. Counts that rest on this string were redefined by what the user said and what the tool then did. Agent text among the user's prompts. About 18,000 of the 366,000 characters counted as the user's prompts were written by agents. All nine long prompts that looked like agent texts were checked, and all nine contain such text. Only one of them did he paste on himself. Six are messages from other agents that the terminal put into his input line, five of them joined to a few words he was typing. Two are start orders for sessions that agents opened themselves. A second search found seven shorter cases of the same kind. All of them are still counted as his. In the role coding such text counts as the agents'. 11Corrections The numbers of the study were corrected several times after it was first published. This section lists the corrections and the results that were not redone afterwards. First published counts. The page was first published with 1,960 prompts, 272 stops, 74 context summaries and 17,966 tool calls. Three counting errors were found on Nov 14: continued sessions counted twice, prompts beginning with a screenshot left out, and messages sent while an agent worked never read. The day-by-day check. On Nov 16 a check of all 265 episodes against the raw logs, day by day, left 117 unchanged and made 23 major, 117 medium and 152 minor corrections; it raised several counts of findings and changed some labels. It also found one helper counted as a main session and up to 10 prompts possibly counted twice through a Codex helper; the second could not be checked, because the event file is missing. Results not redone. Three results were made before the day-by-day check and were not redone: the sorting of routine episodes, the token shares by label, and the comparison of the two readings. The sorting of routine episodes by main kind of work was done on the 100 episodes that carried the label then; 99 of them still do, and 113 carry it now. The token shares by label use the labels before the check and cannot be recomputed, because the event file they need is missing. They are shown below and no longer in the study. 12Reviews, adversarial readings and reanalyses The study was read, argued with and redone in part by further agents. None of them was a human from outside. This section lists the reviews, six readings with assigned briefs, two reanalyses by another model family, and the user's own reading of the drafts. Reviews. Two agents of the same family reviewed the study and checked samples against the chat. Their corrections are built in, and the wrong attributions they found favoured the user. A review by a model of another family led to the reframing of Nov 16, from whose contribution decided to who held which role. An overall review by a fresh instance of the writing model on Nov 17 led to the present order of the study. An independent review of that build by a model of another family followed the same day. It checked the cited passages of the sources added on Nov 17, in full text where it could open them; for two of them it could read only the abstract. The builder had read only abstracts or first pages of those sources. Sentences that claimed more than a source supports were narrowed. The user's review. The only human who reviewed the study was the user himself, its subject and commissioner. He read the drafts, found errors of presentation and structure, and asked for independent checks and for corrections that went against him. This is a participant review (member check) [45], not an independent one. No outside human has reviewed the study. Six readings with assigned briefs Six further agents got the data without this page. Two were told to argue for the user, two against him and two to stay neutral. Their verdicts followed their instructions, which shows how much a reading depends on its brief. This is a test of sensitivity, not a check of the counts, which they took as given. Their strongest challenge stays open: the agents' many small design choices are not counted (). Two of the one-sided readings are in the box below, without weights or verdicts. This study, too, was made at the user's request. Deliberately one-sided. Two agents were each told to argue one side from the same data, and these texts are what they wrote to that brief. They are not evidence and not the view of this study; their numbers come from the earlier version of the page. The letters mark how firm each agent said its claim was: A follows from the data, B depends on a definition one can hold, C only works by leaving something out. Two reanalyses by another model family Two instances of a model of another family, gpt-6-astra in Codex CLI at its highest reasoning level, each made a small study of their own. They got the daily digests (summaries) of the chat, the raw logs and the code. They did not see this study or its readings. One drew a sample of 56 of the user's entries and followed 5 chains of decisions. The other drew 70 entries and followed 8 chains. They used the same archive and the daily digests from the main pipeline, so they are separate interpretations of the same material, not repeated measurements on new data. Both came to the same answer. The strawberry was made jointly, under the user's direction and built in large part by agents, and the data allow no split in percent. The second study put it as “user-directed, jointly designed, and substantially agent-built”. They weigh two things differently from this study. First, agents made many visible design choices at small scale. Examples are the camera path around the eaten fruit and the structure of the next level of the film. Another is a first eating animation with 21 bites, which the user cut to 4. The count of ideas in this study () does not catch such choices. Second, the user also brought in technical concepts. One is a fast preview of a cut, whose mechanism the list of ideas credits to the agents (). The others are a rule for one global resolution and parts of the fruit that can be selected one by one. All three can be found in the code. Their limits are small samples, one coder each, shared access to the same digests, and neither of them watched the film. Their briefs are described in the tab “How this study was made”. 13What the checks caught The process data list 33 errors in results or text. follows one number through three checks. sorts all 33 by what was wrong and by who caught them first. Counts were the most common kind (12). Most were found by reaching the same number on two paths: a second script, a reviewer recounting, a cross-reader adding up parts. The largest error was silent. A filter in the digest dropped about half of the messages, and nothing in the first reading's output showed the gap. An agent tracing ideas found it because it met messages it could not place. It led to the second reading. Claims and caveats came next (9). Several came from rewrites: shortening, smoothing and plain-language passes removed caveats or made scoped claims general. Each time, a check that compared the text before and after, or a reader of another family, found it. Checks by the other family found six errors alone and five more together with a Claude reader; a sixth, first raised by the adversarial Claude readers, was confirmed by the other family's own studies. Among them are the wrong denominator in section 6.4 and the point that the replication and the prompted readings share data with the study. Four problems of layout and structure were noticed while looking at the live page, outside any agent check. Of the 33, 23 were corrected, 7 were disclosed or narrowed, and 3 wait for a reframing of the study. Four problems were operational, not errors in results: the live page was blank for about 90 seconds after a build step, a translation agent wrote a helper script into the wrong folder, two other-family runs stopped at their usage limit, and the working folder was deleted once. 14How disagreements were settled When outputs disagreed, the process used four ways to settle it. Back to the data. When two counts or two agents differed, the count was rerun or the chat was read again. In the findings check, two agents disagreed about where one idea came from, and the main agent settled it by reading the chat itself. Measured agreement. Where two coders labelled the same items, the agreement was reported, not argued away: 78 to 81% between the two readings, kappa 0.86 for the subtypes of routine work. The second reading replaced the first. A consensus agent. In the style rewrite, a fourth agent merged three independent versions into one. Keep both. The adversarial and neutral readings were not merged. All eight agreed on the counts and differed on definitions, so they stay on the page side by side. 15Limits of the process One family checked itself. Most reviews were done by Claude models, the family of the agents the study describes. Three other-family runs ended without a result, two of them at the usage limit, and their tasks went to Claude models. Dependent checks. Later analyses built on episodes cut by earlier AI readers. The two independent studies used the same logs and the same episodes, so they replicate the interpretation more than the measurement. Not blind at the end. The blind test was judged by the main agent, which knew which model had answered. Briefs shape results. Adversarial readers were asked to argue one side, so their texts were extreme by design. Questions put to the agents could carry a preferred answer, and the agents tended to agree with the brief. Public while built. Drafts were public from the first hour and were linked under the comments the study later analysed. Drift. Numbers drifted between copies of the same text, and rewrites dropped caveats more than once. No version control. The working folder was lost once and rebuilt from the live site and the logs. This record. It was reconstructed and cut into steps by one model. This figure stood in Appendix B of the study. It is kept here as a record, because the drafts were public and linked under the comments while the posts were read. Views and members are counted in different units and from different sources, and the figure does not show that views turned into members. 16Where human judgement was missing Every judgement in the study was made by models or by the commissioner, whose reading of the drafts was a participant review (member check), not an independent one. No second human coded, rated, interviewed or reviewed. The 787 image prompts with 869 images were never coded by a human. A control run, in which a fresh agent gets only the first brief, was proposed and not done. lists where a second human would have been needed. 17What we would do differently Build and validate the complete corpus first, with counts per source and per message type, before any reading. Write the questions and coding rules down before coding, including rules that do not decide the main result by definition. Keep one source for every number and generate all texts from it. Draft privately, publish after review, and do not link from the threads under study. Put the study folder under version control with backups from the first hour. Have a human second coder for a random sample, a blind judge for the blind test, and at least one human reviewer outside the project. Use other-family models from the start for a subsample, and budget their quota. Plan a comparison that is fair to both sides: the same target with a fixed final specification against step-by-step refinement, with repeated runs and blind ratings. A run that only fulfils the first, narrower request and stops is no fair control (the study, ). Data for the figures: methodik.json, derived from the process reconstruction without session identifiers or private content. Not affiliated with or endorsed by Anthropic. Commissioned by the user it describes.