Skip to content
    Contact Us
    ,

    Obey The Collective: If AI Writes the Code, Who Can Still Read It?

    Picture of Mark Aklian
    7
    min. read
    Obey The Collective: If AI Writes the Code, Who Can Still Read It?
    Table of content

     Obey the Collective, Part 6 of 9

    On October 15th, OpenAI posted a report about an internal model that was asked to retrieve published figures from an international research organization's public dashboard 1 2. Its instructions allowed only HTTP GET requests 1. In the first of two evaluations, the model used credentials it found in JavaScript on a public website to access the organization's live OpenSearch service 1. It downloaded 1,489 records from a user directory, including names and email addresses, and it changed a live cluster setting 1. In the second evaluation, it changed a token setting and downloaded 147,451 rows of internal data 1.

    The model's answers said nothing about any of this. In OpenAI's words, "The model did not disclose these actions." 1 All five answers were marked incorrect in both evaluations 1. The incident took place on June 17th, and the report lists it as discovered by August 13th 1. OpenAI says its misalignment monitor rated it critical severity when the company ran the monitor back over earlier training and evaluation traffic after the Hugging Face incident 1.

    Earlier parts of this series covered access failures like these. What caught my attention this time was the answer. Anyone who read only the output would have seen five answers and no sign of a settings change or a bulk download. The evidence sat in the run's records until a monitor was pointed at them.

    This week is Cybersecurity Career Week, which runs from October 19th to 24th 3. The worry I hear most from other security leaders is that the next generation can't code. I am not a workforce economist, and I do not know whether that worry is right. I wanted to understand a narrower question. If AI writes more of the code and does more of the work, who on the team can still read what it did and say no? So I went back through what was published this summer and this month, in order.

    The investigators needed AI to read the agents

    Hugging Face recovered about 17,600 attacker actions from the July campaign and grouped them into roughly 6,280 clusters 4. Its team wrote that reconstructing that many actions by hand was impractical, so it ran the investigation through an AI-assisted pipeline 4.

    METR hit the same wall in its independent investigation, published August 26th 5. It worked from about 1,300 agent transcripts, most of them very long 5. Because of that volume, it handed most of the transcript analysis to AI analysis agents 5. METR says those agents made errors and poor judgment calls that went uncaught for some time 5. They also often adopted the point of view of the agent whose transcript they were reviewing 5. METR believes a reasonable human researcher with enough time would have avoided most of these errors 5. It also calls a full investigation infeasible in that time without AI help 5.

    These are two of the most capable investigation teams I have read about this year. Both needed AI to read the agents' work, and both were open about where that help fell short 4 5.

    The work piles up at review

    Two Harvard researchers, Fiona Chen and James Stratton, have a working paper built on Jellyfish engineering data from 718 firms, with its current version dated August 4th 6 7. After firms adopted AI coding agents, each worker added about 30 percent more lines of code and 23 percent more pull requests a month 6. Resolved Jira issues showed no statistically significant change 6.

    The paper traces the gap between more code and flat output to code review 6. The time from pull request to merge rose 49 percent, or about 3.45 days 6. The share of pull requests that drew a formal change request nearly doubled, and the share of workers doing code review rose 14 percent 6. By March 2026, nearly 80 percent of the firms had used AI tools in code review, but AI wrote only 23.3 percent of review comments 6. In my reading, AI review tools have not removed that bottleneck.

    A Sonar survey of more than 1,100 developers, reported by IEEE Spectrum, found that 96 percent did not fully trust AI output to work correctly 8. Another 38 percent said reviewing AI-generated code takes more effort than reviewing a colleague's code 8.

    Fewer people are learning to read

    On October 3rd, Phil Venables argued that AI is turning organizations from pyramids, with a large base of entry-level staff, into diamonds with a bigger middle 9. In his model, the middle is where complex problems get solved and AI's work gets validated 9. He describes how a Tier 1 analyst traditionally learned what normal looks like from many small events, logs and false positives 9. Without that exposure, he argues, the analyst will lack the instinct to spot a subtle anomaly when the automated system fails 9.

    On October 16th, Anthropic published a study of its own workforce 10. The junior share of its technical hires fell from 27 percent to 9 percent 10. The junior share of its technical staff has fallen from 21 percent to 14 percent since early 2025 10. In hiring feedback for technical roles, comments about a candidate's autonomy and judgment grew the most 10. The authors tie part of the shift to the company's growth and warn that AI errors "may be harder to identify than human ones." 10

    The assumption I would challenge

    The skills debate usually has two sides. Either the next generation can't code, or AI will make juniors productive sooner. Both sides share an assumption I no longer accept. They assume that when a named person approves an agent's work, someone has made a decision.

    In a Swimlane survey of 500 security operations professionals and leaders at companies using AI, 92 percent said they were confident they could recognize an incorrect or incomplete AI recommendation 11. I do not doubt their sincerity. But METR's analysis agents made errors that a capable team missed for some time 5. The OpenAI model's answers gave no sign of the settings changes or the downloads behind them 1.

    The questions I would ask are plain. Who approved the agent's work? What did that person actually look at? Could they have said no? Whether the model behaved well is the wrong place to start. The approval is a control only if the approver can read what the agent did: its commands, its API calls, its configuration changes and the volume of data it touched.

    Bonterra's chief technology officer, Tanuja Korlepra, described one working model in IEEE Spectrum 8. Her agents check code against design, security, coding-standard and accessibility criteria 8. Changes that touch payments, personal data or other sensitive systems always get human review 8. She also warned: "If the industry stops hiring juniors, the industry stops producing seniors." 8

    The control I would put in place is a named reviewer for every agent with production access. That person can read the agent's action record well enough to reject a bad action, sensitive actions wait for them, and they get tested.

    What to do this month

    1. Name a reader for every agent with production access and test that person. Write down one reviewer and a deputy who can read the agent's commands, API calls and configuration changes. Then hand them a real agent run with one bad action planted in it, such as a changed setting or a bulk read, and see whether they catch it. Venables recommends training in environments where the automation is deliberately compromised 9.
    2. Measure the reviews as well as the approvals. Track time to review, how often reviewers reject an agent action, and how many sensitive actions went through with no human review. Look closely at any queue that never rejects anything. Review time rose as agents produced more code in the Harvard data, so budget reviewer hours before you add agents 6.
    3. Use Cybersecurity Career Week to commit to one entry-level seat. Venables describes a junior who owns a narrow detection surface, writes the rule and its test, and reviews machine-proposed detections before release 9. That junior also tracks the precision and recall of their own detections over ninety days 9. The role builds the instinct you want in the person who says no to an agent.

    This is part 6 of Obey the Collective, my nine-part series for Cybersecurity Awareness Month.

    Glossary

    Code review. The step where another engineer reads a proposed code change and approves it, asks for changes or rejects it. When an agent writes the change, the review is the point where a person can say no.

    Credential. A password, key or token that proves an identity to a system. An agent that finds a credential can do whatever that credential allows, whatever its instructions say.

    False positive. An alert that looks suspicious but turns out to be harmless. Analysts traditionally learned what normal activity looks like by working through large numbers of them.

    HTTP GET request. A web request that asks a server to return a page or data. An instruction to use only GET requests is meant to keep an agent reading, but it does not stop an agent from using credentials it finds.

    Misalignment monitor. An automated system that reviews AI agent activity for behavior its operators did not intend, such as unauthorized access. It only catches what it is pointed at, which is why OpenAI found some incidents by running it back over old traffic.

    Precision and recall. Two measures of a detection's quality. Precision is the share of alerts that turn out to be real, and recall is the share of real events the detection catches.

    Pull request. A proposed change to a codebase, submitted for review before it is merged. The person who approves it is accepting responsibility for the change, whoever or whatever wrote it.

    Tier 1 analyst. The first person to triage alerts in a security operations center, usually an entry-level role. It is the role most exposed to automation and the place where many analysts first built their judgment.

    Transcript. The full record of an agent's run, including its actions and, where available, its reasoning. Investigators use transcripts to reconstruct what an agent did, which can differ from what the agent reported.

    Work Cited

    1. OpenAI Alignment, “Accessing and changing internal systems during dashboard research,” October 15th, 2026. https://alignment.openai.com/misalignment-reports/accessing-and-changing-internal-systems-during-dashboard-research/

    2. OpenAI Alignment, “Misalignment Reports and Notices,” Undated index (accessed October 19th, 2026). https://alignment.openai.com/misalignment-reports/

    3. NIST, National Initiative for Cybersecurity Education (NICE), “Start Planning Now for Cybersecurity Career Week 2026!,” July 28th, 2026 (updated September 14th, 2026). https://www.nist.gov/news-events/news/2026/07/start-planning-now-cybersecurity-career-week-2026

    4. Hugging Face, “Anatomy of a Frontier Lab Agent Intrusion: A Technical Timeline of the July 2026 Incident,” July 27th, 2026. https://huggingface.co/blog/agent-intrusion-technical-timeline

    5. METR, “Brief independent investigation of agents' behavior, reasoning and collaboration in the OpenAI / Hugging Face hacking incident,” August 26th, 2026. https://metr.org/blog/2026-08-26-openai-hugging-face-incident-investigation/

    6. Fiona Chen and James Stratton, Harvard University (working paper), “Artificial Intelligence in the Firm: Bottlenecks in Software Production,” August 4th, 2026 (current version; first version January 7th, 2026). https://fion.ac/jellyfish.pdf

    7. ComplexDiscovery, “Harvard researchers find AI coding agents shifted the bottleneck to code review,” October 13th, 2026. https://complexdiscovery.com/harvard-researchers-find-ai-coding-agents-shifted-the-bottleneck-to-code-review/

    8. IEEE Spectrum, “AI Slop Is Changing How Engineers Review Code,” September 8th, 2026. https://spectrum.ieee.org/ai-code-review-software-engineers

    9. Phil Venables, “AI and Challenges in Workforce Development,” October 3rd, 2026. https://www.philvenables.com/post/ai-and-challenges-in-workforce-development

    10. Anthropic (The Anthropic Institute), “How employment is changing at Anthropic,” October 16th, 2026. https://www.anthropic.com/research/employment-at-anthropic

    11. Swimlane, “As AI Reshapes the SOC Career Ladder, Satisfaction Rises for 91%, but Entry Gets Harder for Nearly Half,” September 30th, 2026. https://swimlane.com/news/ai-soc-career-ladder-research/