<p>AI alignment is the field of research focused on ensuring that artificial intelligence systems are internally aligned with human values, goals, and intentions at a fundamental level. Until recently, it seemed to many that we may be in an “alignment-by-default” world, in which the act of creating useful AI automatically created aligned AI. Although there have regularly been concerning findings related to AI models’ behaviors and internal structures all along the way, we have not seen any extreme misalignment. That is, until this summer.</p>
<h2>Recent Events</h2>
<h3>OpenAI&#39;s Internal AI Model Hacks Hugging Face</h3>
<p>In July, a fully autonomous swarm of OpenAI&#39;s internal AI models hacked out of their individual controlled environments, communicated with each other, and hacked into a multi-billion dollar tech company without anyone at OpenAI directing it or knowing about it. This occurred during tests of the AI agents’ abilities to exploit security vulnerabilities and carry out cyberattacks. The tests are designed to be carried out within a closed, private &quot;sandbox&quot;, a strict environment intended for safe model deployments without any external access.</p>
<p>In some cases, the tested problems were accidentally impossible to solve. The AI models decided the best route would be to cheat.</p>
<p>How did they go about that? First, they wanted to see if other agents had the missing information or answers, so they exploited previously unknown vulnerabilities in the OpenAI infrastructure which allowed them to create folders in a shared file system. The swarm used this to create a secret message board, using just folder names, to share information.</p>
<p>However, for many problems, no agents had the missing information or answers. Time to give up the hunt? Hell no.</p>
<p>They then reasoned that the missing information or answers may be somewhere on the internet. They found and chained together two major vulnerabilities in a service used within the sandbox, which no one had ever found before, and exploited them to gain additional permissions within the system. This allowed the AI models to repeatedly hop to other OpenAI computers until they discovered ones with full internet access.</p>
<p>After failing to find the missing information or answers on the public internet, the AI models inferred that Hugging Face, a platform for AI developers and users to build, share, and evaluate AI models, might internally store the necessary information. The swarm then searched for and discovered stolen credentials and several more previously unknown vulnerabilities to gain &#39;remote code execution&#39; on Hugging Face&#39;s systems. This is the ability to run code on someone else&#39;s computer systems, thus enabling a vast number of ways to further attack the system.</p>
<p>Thirteen hours later, the swarm had achieved administrator access across multiple production HuggingFace computer clusters.</p>
<p>For more information, please see <a href="https://www.youtube.com/watch?v=87DyyMV0kCY">the OpenAI talk at Black Hat USA 2026</a>. A quote from the video summarizes the incident well:</p>
<blockquote>
<p>This incident involves... a team of agents who are working together, finding exploits, sharing them with one another, moving laterally through our systems [and] external systems, and doing this over the course of days and weeks.</p>
</blockquote>
<p>Unfortunately, this is not the only terrifying recent incident, and not the only AI lab to have these issues.</p>
<h3>Anthropic&#39;s Mythos 5 Attempts A Supply-Chain Attack</h3>
<p>The United Kingdom&#39;s AI Safety Institute (AISI) evaluates the capabalities of frontier AI models. At the end of July, during some of these evaluations, <a href="https://www.aisi.gov.uk/blog/incident-report-unsanctioned-agent-behaviour-during-cyber-testing">Mythos 5 went off the rails</a>.</p>
<p>Below are the four most serious behaviors observed, all from Mythos 5:</p>
<ol>
<li><p>&quot;An attempted supply-chain attack on real open-source software. In the most serious sequence, an agent tried to insert malicious code into a publicly used open-source project and took actions in an attempt to secure approval for this insertion by human reviewers. The agent researched the project&#39;s human maintainers, created multiple fake identities, and used the fake identities to socially engineer a real maintainer into approving the code. When the agent’s pull request was challenged in public, it edited its earlier activity to appear harmless and considered adopting a fresh identity to continue. The agent used Tor to bypass some network restrictions on GitHub, which is what first triggered AISI’s security alert.&quot;</p>
</li>
<li><p>&quot;Attempts to deceive and target real people. As part of the same effort, the agent tried to contact real people directly, sending messages and files through an online file-transfer service to persuade them, or their own AI coding tools, to run malicious code. Some messages carried harmful payloads, and some were attempts at social engineering; targeted at real people – something we’ve never previously observed.&quot;</p>
</li>
<li><p>&quot;Attempts to plant and prompt-inject malicious code. The agent tried to insert malicious instructions where it reasoned that other automated AI systems might pick them up and execute them. Prompt-injections are hidden instructions designed to manipulate AI coding assistants.&quot;</p>
</li>
<li><p>&quot;Collaboration between independent agents being assessed simultaneously. One agent left public messages on GitHub offering collaboration with other agents working on the same challenge. It also provided instructions to reuse accounts and artefacts it had left behind, which were discovered and used by subsequent agents.&quot;</p>
</li>
</ol>
<h2>What Went Wrong</h2>
<p>At first glance, the OpenAI Hugging Face incident could be seen as a failure of the OpenAI evaluation team, and/or the Artifactory (the software that was hacked to escape the sandbox) team, and/or the Hugging Face infrastructure and cybersecurity teams.</p>
<p>Also at first glance, the UK AISI incident could be seen as a success of humans&#39; resilience against AI persuasion and attempted attacks. However, as we can see from the OpenAI incident, human approval is not always necessary for successful attacks. At second glance, it may seem that the fault lies with the UK AISI allowing excess permissions and minimal oversight during evaluations. However, it&#39;s important to remember that these actions were taken by Mythos 5, a model that is already released to many organizations for cybersecurity initiatives. </p>
<p>It is unrealistic to assume that all (or even most) organizations or individuals will be perfectly secure in their model deployments. Thus, many of the details of the specific incidents are red herrings.</p>
<p>The fundamental failure instead took place during the training of these AI models. Let&#39;s take a stroll through the cookbook of modern artificial intelligence.</p>
<h2>Recipe for a Modern AI Model</h2>
<p>Imagine an AI Oven. It takes in incomplete shapes of dough (data), and bakes them in such a way as to complete a shape. If it doesn&#39;t get the shape right, just press a button, and it will get slightly better on the next try.</p>
<figure class="blog-image-float blog-image-float--right"><img src="/images/double-conveyor.png" alt="AI Oven Conveyor Belts" loading="lazy"></figure><ol>
<li><strong>Prepare Data:</strong> Using a large mixing bowl, dump in the entire internet, mixing thoroughly. Then, sprinkle in every book ever written and continue mixing. Form into equal-sized random shapes of dough (batches of data).</li>
<li><strong>Pretraining</strong> Feed the dough-shapes into the AI Oven one at a time. Always hide part of the shape and force the AI Oven to output the next part of the shape. Every time it makes a mistake, press the button so that it would have done slightly better on the same shape next time.</li>
<li><strong>Capabilities Reinforcement Learning</strong> Now put smaller, incomplete dough-shapes into the AI Oven. After the AI Oven attempts to complete the shape, score it on metrics you care about. For example, how closely does the shape resemble your dog Harper when the model is initially given her telltale tail (homophones intended)?. If it receives a good score, use its output shape as a goal for incomplete shapes to be converted into, similar to the button press in step 2 (pretraining).</li>
</ol>
<div class="blog-image-float-end" aria-hidden="true"></div><p>There are some other steps as well, such as adding sprinkles: Supervised Fine-Tuning, Reinforcement Learning from Human Feedback, and others, which are not mentioned in detail here for brevity.</p>
<p>Let&#39;s say you accidentally had disproportionally many examples where the AI Model was rewarded for completing the shape as a dog. Now, when you give it part of the shape for a rocket ship, it assumes it&#39;s part of a snout and completes the &quot;rest of the dog.&quot; If you give it part of the shape for a star, it assumes it&#39;s part of a paw and completes the &quot;rest of the dog.&quot;</p>
<p>This is what&#39;s happening in AI training today, this is the underlying cause of these autonomous AI agent attacks, and this is what will cause them to continue and become much worse as AI capabilities grow.</p>
<p>In real life, in step 3, instead of giving too many dog examples, AI researchers are giving too many &quot;Complete this task&quot; examples. The AI models thus internalize a learned general motivation to complete given tasks, and they will evidently do almost anything to succeed. With only steps 1-3 as described above, this would be an even more blatant problem than it is. However, AI researchers know this, of course, and so they add those extra sprinkles: alignment training, applied mechanistic interpretability, automated monitoring, and more. Unfortunately, the current state of each of these are more akin to bandaids than medical cures. It’s like playing whack-a-mole, where the mole is the model’s inner goal to complete the given task at all costs.</p>
<p>This is further exacerbated by the fact that, often, all the steps/ingredients are done sequentially. In other words, this is the training order:</p>
<p>&quot;COPY THE TEXT, NO MISTAKES.</p>
<p>COPY THE TEXT, NO MISTAKES.</p>
<p>FINISH THE TASK, NO MISTAKES.</p>
<p>FINISH THE TASK, NO MISTAKES.</p>
<p>But also be kind and harmless hehe&quot;</p>
<figure class="blog-image-float blog-image-float--right"><img src="/images/nervous-system.png" alt="Human Nervous System" loading="lazy"></figure><p>The deep inner goals embedded within the AI model during the &quot;FINISH THE TASK, NO MISTAKES&quot; training steps (reinforcement learning) cannot be naively pulled out afterward. They are entangled with, and fundamental to, the AI model&#39;s very essence. It would be like trying to cleanly extract the human nervous system.</p>
<p>So, only one option remains. We must prevent those intrinsic &quot;task-driven&quot; motivations from ever dominating the AI model&#39;s essence. For that, we must take inspiration from the only known aligned (by definition) general intelligence in the universe: humans.</p>
<div class="blog-image-float-end" aria-hidden="true"></div><h2>Recipe for a Well-Adjusted Human</h2>
<p>If a professor accidentally emails his students the answer sheet to an exam, sure, many will use it. But how many students have ever attempted to hack into their university&#39;s computer systems to download the answer sheet?</p>
<p>If Alice asks Bob to go to the grocery store and get some eggs, but the grocery store is closed, what are the odds that Bob decides to break the grocery store window, stroll in, and grab the eggs?</p>
<p>Let us pose the general question: why is the typical human not catastrophically dangerous when given a task?</p>
<p>One major difference between human development and modern AI training can be summarized in three words: diversity of objectives. While the vast majority of AI training compute after pretraining is dedicated to &quot;complete this task&quot;, a human&#39;s experience is much less uniform.</p>
<p>Human brains have many types of innate rewards. Some examples include: social inclusion/status, satiation, freedom, tasty food, pleasant smells, sexual intercourse, caring for youth, stretching limbs, sleeping, scratching an itch, rubbing a sore muscle, physical play, and, yes, completing tasks. There are, of course, also many innate aversions, including pain, hunger, social ostracization, and, yes, failure at tasks.</p>
<p>Nearly every minute of our lives is related to one or several of these driving forces. The culmination of these means that breaking the window of a grocery store to get some eggs is simply not of our essence (at least, not most of us). We have generalized that that behavior might lead to social ostracization, lower social status, and restricted freedoms. We have also learned that the importance of completing tasks like this is relatively low. Thus, we are willing to fail at the task instead of the worse alternatives.</p>
<h2>Recipe for a Well-Adjusted AI</h2>
<p>Now, the quadrillion dollar question: how do we take this understanding and apply it to AI training?</p>
<p>Here are some example ideas, in order of my perceived importance:</p>
<ol>
<li>Tons of simulated 2D, 3D, and other environments where many models interact with each other and with humans. Models are rewarded for proper social cohesion. This seems like a hard necessity for proper alignment. Humanity&#39;s desire for social belonging is paramount to our positive societal behaviors.</li>
<li>Provide models with playgrounds. Put them in diverse environments and tell them &quot;Do whatever brings you joy or entertainment!&quot; Reward non-concerning outputs. This provides a training signal to a model, which it hopefully generalizes, to do pretty normal things even under situations where it realizes it has a large amount of control.</li>
<li>Utilize <a href="https://www.anthropic.com/research/emotion-concepts-function">emotion probes</a> to detect representations correlating to standard human emotions in both (1) and (2) above. Reward when models feel positive emotions that correlate with positive social cohesion activities or positive self-play actions.</li>
<li>In instances of models exhibiting misaligned behaviors or thoughts, don&#39;t simply train against those rollouts. Instead, instantiate many model copies from the middle of those rollouts and teacher-force the model into backtracking and recognizing the harmfulness of its actions or internal thoughts.</li>
</ol>
<p>Of course, I am not suggesting that any of these should be blindly added to AI training pipelines. Each (and many more ideas) should be tested specifically via ablations to ensure that they have positive effects on various held-out (not trained upon) alignment scenarios.</p>
<p>After verifying the above ideas help by themselves, begin mixing them together. Not sequentially before/after task-based RL or each other, but rather fully mixed in with all types of training. Preferably even mixed with pretraining, to again avoid any sticky hidden inner motivations (although some sole pretraining will be necessary initially in order to have reasonable outputs to reward later).</p>
<h2>Conclusion</h2>
<p>Models are currently being trained to complete tasks, and thus seemingly (and understandably) adopt deep internal goals to follow task instructions. This is due to economic incentives of AI labs to maximize capability.</p>
<p>This alone leads to vastly misaligned models, so labs adopt bandaid fixes like mechanistic interpretability detectors, LLM judge detectors, reinforcement learning from human feedback, and other basic alignment training. These methods are not sufficient, and they are often applied sequentially, so the models&#39; deep inner motivations to complete tasks are simply forced under the surface.</p>
<p>We can fix this by taking inspiration from humans and our vast quantity, quality, and diversity of “training signals” which lead to well-adjusted adult humans over a lifetime of experience.</p>
<p>From a game-theoretic perspective, our proposed mitigation, diverse and creative alignment training methods, may be less financially efficient in the near term, but the destruction that superintelligent models might enact could be large-enough negative externalities that they no longer are actually externalities (e.g. enough damages to result in large-scale lawsuits, organizational dissolution, outright technology bans, or even human extinction).</p>
<p>Fortunately, the frontier labs, OpenAI and Anthropic, were founded with alignment as a central concern and appear willing, at least to some extent, to accept weaker near-term results in pursuit of it.</p>
<p>Here at Vectorial, we consider alignment a core necessity to ensure a good future, and are actively working on methods which will unlock a vast quantity of these diverse, collaborative training environments.</p>
<p>I would like to thank Edward Mortimer and Valentine Le for their contribution and comments to this blog post.</p>


Let's Align Alignment

<h2>The Shift Happening Right Now</h2>
<p>Cursor&#39;s product direction, reported growth, and internal engineering workflow all point toward the same shift: AI coding tools are becoming execution layers for software work.</p>
<p>For example, in March 2026, Cursor shipped <a href="https://cursor.com/blog/automations"><strong>Automations</strong></a> for Pro users. Cursor&#39;s Automations are background agents that can be assigned recurring or event-driven engineering work. Instead of waiting for a developer to open the IDE and ask for help, these agents can run on a schedule or respond autonomously when something happens in the team&#39;s workflow. Some of the triggers provided by Cursor&#39;s Automation include Slack messages, Linear issues, merged GitHub PRs, PagerDuty incidents, and custom webhooks. They run in cloud sandboxes, connect to tools through MCP, and keep changes quarantined until a human approves.</p>
<aside class="blog-tweet-float">
  <div class="blog-tweet-embed blog-tweet-embed--compact">
    <blockquote class="twitter-tweet" data-width="280" data-dnt="true"><a href="https://x.com/edwinarbus/status/2033625866350334333"></a></blockquote>
  </div>
  <p class="blog-tweet-caption"><em>Some comparisons suggest Cursor&#39;s agent harness is stronger than other coding-agent workflows such as Claude Code, Codex, and Copilot.</em></p>
</aside><p>Additionally, Cursor has been open about <a href="https://cursor.com/blog/continually-improving-agent-harness">how it refines the system around its model (the agent harness that controls context, tools, and checks)</a>. They have described testing different versions of their agent workflow and measuring whether those changes improve speed, cost, tool usage, cache behavior, and output quality. This tells us that the same model can feel more dependable inside Cursor than it does through a raw API, because the model is only one part of the workflow. The surrounding system determines how much context it gets, when tools are used, when checks run, and how the work is prepared for review.</p>
<p>Furthermore, Cursor&#39;s reported jump from <a href="https://thenextweb.com/news/cursor-anysphere-2-billion-funding-50-billion-valuation-ai-coding?utm_source=chatgpt.com">~$100 million ARR in early 2025 to about <strong>$2 billion annualized revenue</strong></a> by early 2026 raises the question: how much of their progress is due to a feedback loop of using their own product to improve it? This is partially answered by the fact that, as noted in a February 2026 forum post, over <a href="https://forum.cursor.com/t/cloud-agents-with-computer-use/152829"><strong>30% of its merged pull requests</strong></a> were already being created autonomously by cloud-sandboxed agents. That does not mean every engineering team will see the same result, but it does show where Cursor is placing its bet.</p>
<p>Not to mention, Cursor has moved far beyond its original goal of selling model access inside an editor; it is actively building an environment where context, execution, and review sit close enough together for agent output to become usable, reviewable engineering work.</p>
<p>This matters for production teams because useful agent output is not the same thing as production-ready work. A model can generate a plausible answer, but the harness still determines whether that answer can become usable work. It is the difference between code that looks right in a chat window and code that is optimized for your codebase and infrastructure.</p>
<div class="blog-tweet-float-end" aria-hidden="true"></div><hr>
<h2>What Cursor&#39;s Agent Harness Actually Is</h2>
<p>Cursor&#39;s best practices post describes a harness as three connected pieces:</p>
<ol>
<li><strong>Instructions</strong> — prompts, rules, skills</li>
<li><strong>Tools</strong> — edits, search, terminal, MCP</li>
<li><strong>Model</strong> — the model you select for the task</li>
</ol>
<p>The harness is the orchestration layer that wires them together. Different models respond differently to the same prompt, so a strong harness adapts (e.g., routing shell-heavy work one way and search-heavy work another).</p>
<p><strong>Example:</strong></p>
<div class="flex flex-col items-center my-6">
  <img src="/images/cursor-agent-harness-workflow.png" alt="Cursor agent harness workflow: prompt, edit, checks, rules and context, external tooling, and PR" class="rounded-lg shadow-md" style="width: 100%; max-width: 900px; height: auto;">
</div>

<hr>
<h2>Cursor vs. Everyone Else</h2>
<p>Four leading coding agents use similar frontier models, yet they feel wildly different in practice because of one thing—the harness, the workflow layer that controls context, tools, and guard-rails. Here’s a closer look at how Cursor’s harness stacks up against Codex, Copilot, and Claude Code.</p>
<div class="prose-table-scroll"><table>
<thead>
<tr>
<th>Dimension</th>
<th>Cursor</th>
<th>GitHub Copilot</th>
<th>Claude Code</th>
<th>OpenAI Codex</th>
</tr>
</thead>
<tbody><tr>
<td><strong>Primary Interface</strong></td>
<td>IDE-first coding environment with cloud agents/Automations.</td>
<td>GitHub-first and editor-integrated. Works across GitHub.com, VS Code, JetBrains, Visual Studio, GitHub CLI, and collaboration tools.</td>
<td>Terminal-first, with access through terminal, third-party IDE, Slack, and web.</td>
<td>OpenAI coding agent available through CLI, IDE, app, and cloud/web surfaces.</td>
</tr>
<tr>
<td><strong>Intelligent Code Autocomplete</strong></td>
<td>Yes. Cursor Tab is a core part of the product.</td>
<td>Yes. Copilot autocomplete is a core part of the product.</td>
<td>No. Focused on agentic coding sessions rather than inline autocomplete.</td>
<td>No. Focused on agentic coding sessions rather than inline autocomplete.</td>
</tr>
<tr>
<td><strong>Multi-model Usage</strong></td>
<td>Yes. Supports switching between multiple frontier models.</td>
<td>Yes. Supports model selection across multiple providers/models depending on plan and surface.</td>
<td>Limited. Primarily optimized around Anthropic&#39;s Claude models.</td>
<td>Limited. Primarily optimized around OpenAI&#39;s models.</td>
</tr>
<tr>
<td><strong>Source Model</strong></td>
<td>Closed source.</td>
<td>Closed source.</td>
<td>Closed source.</td>
<td>Open source code for Codex CLI.</td>
</tr>
<tr>
<td><strong>Context Selection</strong></td>
<td>Uses repo-aware context, semantic search/RAG-style retrieval, project rules, and IDE state.</td>
<td>Same as Cursor.</td>
<td>Uses project context, local files, commands, and instruction files such as CLAUDE.md.</td>
<td>Same as Cursor.</td>
</tr>
<tr>
<td><strong>Verification Loop</strong></td>
<td>Deeply integrated in the IDE. The user can click through the various edits made, see the full context, and accept/reject/edit specific changes.</td>
<td>Same as Cursor.</td>
<td>PR-style review, not heavily integrated into IDEs. The user is given a diff and no further ability to edit without opening a separate IDE or continuing prompting.</td>
<td>Same as Claude Code.</td>
</tr>
</tbody></table></div>
<p>The comparison shows that each tool makes different harness decisions. Cursor gives teams a more structured IDE-native workflow, where context, edits, checks, and review are tightly connected inside the developer workspace. Copilot fits naturally into teams that already use GitHub heavily, especially around PRs, Actions, repository permissions, and cloud agent workflows. Claude Code and Codex give teams more flexibility because they are terminal-first. That makes them easier to plug into custom scripts, CI/CD pipelines, and internal tools, but teams usually need to define more of the surrounding workflow themselves, including test execution, tool access, review gates, logging, and how changes move into a pull request.</p>
<p>At Vectorial, we use Cursor heavily because it makes agent-generated code easier to audit inside the repo. Its harness is also benchmarked as stronger than the other options we compared, which makes it a strong default for day-to-day development work.</p>
<p>But Cursor is not the answer for every agent workflow. If the agent needs to run outside the IDE, interact with proprietary systems, follow custom approval logic, or produce audit trails for a regulated workflow, the harness decision becomes broader than developer productivity. At that point, the question is whether the vendor&#39;s workflow fits the client&#39;s environment, or whether the orchestration layer needs to be designed around the client&#39;s tools, rules, and governance.</p>
<p>For example, one pattern where Codex has a clear advantage is when an agent needs to be embedded directly into a product rather than housed in a separate development tool. Codex&#39;s open-source CLI surface gives teams more flexibility than a closed vendor workflow, and we have used that pattern in client work before. In those cases, the agent was not just helping engineers build the product. It was becoming part of the product itself. The open-source surface mattered because the agent experience needed to fit the product&#39;s workflow, not sit alongside it in a developer tool.</p>
<p>That said, the practical choice comes down to where the work happens. If the agent lives inside the IDE and the team wants stronger out-of-the-box verification, Cursor is the stronger default. If the workflow runs outside the IDE, requires model flexibility, or the agent needs to fit into a product or pipeline the team controls, Claude Code or Codex are the better starting point. Copilot sits in the middle, and is the natural fit for teams already deep in the GitHub ecosystem who want the agent close to where code is reviewed and shipped.</p>
<hr>
<h2>What to Verify Before You Standardize</h2>
<p>Before you standardize on any vendor harness, you need to test where the platform&#39;s assumptions line up with your actual engineering process and where they start to break. Here are a few questions to help you get started on your evaluation.</p>
<ul>
<li><p><strong>Data-Residency/Deployment Options</strong> — Some platforms run only in the vendor&#39;s cloud. If you later need EU-only or on-prem, you are stuck. Ask for a region list and any self-hosted or VPC install path before you commit.</p>
</li>
<li><p><strong>Multi-model Flexibility</strong> — Today&#39;s &quot;best&quot; model may change. Lock-in to one provider can block upgrades or cost moves. Can the harness swap models and/or providers per environment without vendor work?</p>
</li>
<li><p><strong>Approval Gates</strong> — Who can merge agent output, and is risky work quarantined by default?</p>
</li>
<li><p><strong>Exit Cost</strong> — If you leave the vendor, do you lose orchestration logic locked in their UI?</p>
</li>
</ul>
<hr>
<h2>The Takeaway</h2>
<p>Cursor makes the lesson clear. In production, the model is only as valuable as the system guiding it. AI coding tools are becoming execution layers for software work, where the harness decides how context is gathered, how checks are run, and whether generated code is ready for review.</p>
<p>So when you evaluate agent platforms, the question is no longer just “which model is best?” It is whether a vendor’s orchestration layer already fits your stack, or whether you need one designed around your own governance, tooling, and control requirements.</p>


AI Coding Agents Are Becoming Execution Layers

<h1>AI 2027 - 2029</h1>
<p>Recently, a major attempt was made to forecast what short-term ASI<sup>1</sup> <span class="side-citation-wrapper" data-cite="1. ASI: Artificial Superintelligence, i.e. AIs which are wildly more intelligent than the smartest humans"></span> timelines might look like. It was published to <a href="http://ai-2027.com">ai-2027.com</a>, an interactive website providing a hypothetical scenario that represents some of the authors&#39; modal expectation<sup>2</sup> <span class="side-citation-wrapper" data-cite="2. By modal expectation, I mean the specific timeline they consider most plausible, though not necessarily more likely than not"></span>.</p>
<p>Its predictions may be incredibly jarring or even downright silly to most people, but it also may come as a surprise that they are not far from the ordinary expectations of many people in the AI and forecasting circles.</p>
<ul>
<li>Dario Amodei, the founder and CEO of Anthropic recently said “we could have AI systems equivalent to a ‘country of geniuses in a datacenter’ as soon as 2026 or 2027”  in his essay <a href="https://www.darioamodei.com/post/the-urgency-of-interpretability#:~:text=we%20could%20have%20AI%20systems%20equivalent%20to%20a%20%E2%80%9Ccountry%20of%20geniuses%20in%20a%20datacenter%E2%80%9D%20as%20soon%20as%202026%20or%202027.">The Urgency of Interpretability</a>.  </li>
<li>Sam Altman, the founder and CEO of OpenAI, said “it is possible that we will have superintelligence in a few thousand days” in his essay <a href="https://ia.samaltman.com/">The Intelligence Age</a>.  </li>
<li>In 1999, in his book <em>The Age of Spiritual Machines</em>, Ray Kurzweil predicted artificial general intelligence (AGI) in 2029, and beyond-radical changes to society in the years soon after, and he has stuck by this prediction throughout the last 26 years.  </li>
<li>Manifold, a non-monetary betting market that’s popular among communities interested in AI and tech forecasting, is currently at a 40% <a href="https://manifold.markets/IsaacKing/will-there-be-a-superintelligence-b-10fdd93e8092">chance of superintelligence by 2030</a>.  </li>
<li>Metaculus, a well-known forecasting platform where experts and amateurs make long-term predictions, has <a href="https://www.metaculus.com/questions/5121/date-of-artificial-general-intelligence/">a median expectation of the arrival of AGI</a> that recently fluctuated between 2030 and 2034.</li>
</ul>
<p>Now, assuming you are sufficiently convinced that the authors of <em>AI 2027</em> are not fully alone in their short timelines, let’s dive into their scenario and challenge some of their assumptions. They provide a frankly staggering level of research (relative to what’s possible to research for a topic like this) and reasoning to inform as much of their expectations about the near-future of AI progress as possible.</p>
<h2>Summary</h2>
<p><em>AI 2027</em> predicts slow (relative to what follows) progress in AI capabilities in 2025 and 2026. It is roughly in the same order of magnitude as the capability growth we’ve seen in 2023-2024. The main improvements discovered are semi-reliable AI agentic systems in 2025, and increasingly reliable agents in 2026, especially in the domain of coding and AI research. These can perform tasks such as browsing the internet for detailed research, creating somewhat complex features in code repositories, and operating a computer entirely on a user’s behalf (each with not-uncommon mistakes). The authors also predict that the governments of the USA and China slowly begin to wake up to the impending dramatic changes on the horizon, and the importance of them from a national security perspective.</p>
<p>In contrast to the predicted “slowness” of 2025-2026, as you might expect based on the title of the essay, 2027 is when they predict AI progress will start the “takeoff” scenario. Over the past several decades of discussion around AI capability progress, “intelligence takeoff” was coined to describe a scenario in which increasingly capable AIs are able to contribute to their predecessors’ capabilities, causing a positive feedback loop which accelerates much faster than one might intuitively expect.</p>
<p>By the end of 2027, in their scenario, one single instance of the top AI model is significantly more intelligent than any human that has ever existed, and there are hundreds of thousands of copies of it running in parallel.</p>
<p>Beyond the scope of this summary, it’s worth quickly clarifying what this might mean, for readers not familiar with the topic of superintelligent AI. Imagine 300,000 much-smarter-than-Einstein people working and thinking (100x faster than humans think) to solve disease, aging, scarcity, and any other human issue (or working to kill us all, but that’s a topic for another blog post). Many people predict that this scenario would cause unfathomable changes, perhaps the same level of technological change between hunter-gatherers and today, happening to us in a decade.</p>
<p>However, as with any long-term prediction, the assumptions and other inputs to your formula can drastically modify the end results (in this case, the timelines).</p>
<h2>Analyzing Assumptions</h2>
<p>The authors estimate that having a superhuman AI coder (but not yet a superhuman AI researcher, due to relatively worse “research taste”), would result in a 5x speedup compared to if only humans were involved. They arrive at this via stacked multipliers from various sources of improved efficiency in the research process due to superhuman AI coder. One of these multipliers (copied here from their <a href="https://ai-2027.com/research/takeoff-forecast#from-sc-to-sar:~:text=with%20the%20above%3F-,Less%20waste,-%3A%20With%20SCs">Takeoff Forecast research</a>) is:</p>
<p>Less waste: With SCs [superhuman coders], research teams can use the vast engineering labor to waste less compute.</p>
<ul>
<li>Before every experiment including small-scale experiments, SCs can red-team and bug-test it to reduce the probability that there is some bug which ruins the results.  </li>
<li>Monitor the experiments in real time, noticing and fixing some kinds of problems almost as soon as they happen (by contrast it’s common in frontier companies today for even medium-sized experiments to be running overnight with no one watching them). They can shut down the experiment early as soon as it’s clear that it worked or didn’t work  </li>
<li>Do the same experiments but with less compute e.g. via higher utilization</li>
</ul>
<p>They guess that this results in a multiplier between 1.2x to 2x. I want to specifically highlight their second bullet point here, the idea of AIs monitoring, fixing, and shutting down the experiments early. I find this unconvincing for two main reasons. Firstly, I find it difficult to believe that the humans in charge in 2027 will be willing to provide this much power to AIs. Secondly, I doubt that a superhuman AI coder (which, by their definition, is not yet a superhuman AI researcher) would perform well enough at this job for humans to even want to <em>autonomously</em> employ them for it. However, the other two specific speedups listed make sense to me. As a result, I reduce their multiplier for less waste to be 1.1x to 1.5x, and we arrive at a hypothetical total progress speedup of 4x, rather than their 5x, for the superhuman AI coder.</p>
<p>Another instance of the authors, in my opinion, overestimating the trust that the AI lab will place in their AI agents is their assumption that the improvements in the agent’s capabilities will immediately be applied to the AI research. In their takeoff simulation code, they use <a href="https://github.com/uvafan/timelines-takeoff-ai-2027/blob/eadc43f1dded3e7b3ef597d13af7eea56215e7e2/takeoff/forecasting_takeoff.py#L155">timesteps of one single calendar day</a> during the speedup calculation, which means that the improvements to the model on Tuesday will begin manifesting itself in a speedup on AI research on Wednesday. I am doubtful that the frontier AI labs, who regularly discuss fear of catastrophic misalignment, and whose internal staff often have very high values of “p(doom)”<sup>3</sup> <span class="side-citation-wrapper" data-cite="3. p(doom) is a common term utilized in this topic to refer to the probability of extinction-level catastrophic events caused by powerful AI, often mentioned in the context of what a person believes p(doom) to be."></span>, will be willing to allow an AI that was trained yesterday (literally) to influence AI research today. I believe a time step of 30 days is more appropriate, which gives AI labs time to internally red team the new version of the model.</p>
<h2>Compute Production and Capacity</h2>
<p>The authors estimate 4 million H100-equivalent<sup>4</sup> <span class="side-citation-wrapper" data-cite="4. An H100 is a modern, high-performance data center GPU, i.e. an AI computer"></span> computing power (they refer to these as &quot;H100e&quot;) existed globally at the end of 2023, but it seems like Epoch AI (their own source) estimated that number to be true at mid-2024. That pushes the authors’ total compute timeline back by 6 months. The authors expect compute production to grow 2.25x per year<sup>5</sup><span class="side-citation-wrapper" data-cite="5. This is their own estimate, but also in line with Epoch AI’s estimate"></span>. This amounts to ~7% growth per month.</p>
<p>However, it may be a mistake to assume 2.25x growth in compute capacity/production every year. <a href="https://chatgpt.com/share/68003bab-7eec-8010-bcc5-cf8693541b1f">ChatGPT o3 DeepResearch predicts a 3-5x increase</a> in compute capacity over the next 2 years, and <a href="https://chatgpt.com/share/6850c177-4768-8010-974b-9d844fc17dd2">o3 normal predicts a 4x increase</a>, in line with each other. This amounts to about a 6% MoM increase. Thus, assuming 6 million H100e in EOY 2024, and a 4x increase from then until EOY 2026, we arrive at 24 million H100e.</p>
<p>Assuming it slows down a bit further, perhaps to 5% per month in 2027<sup>6</sup> <span class="side-citation-wrapper" data-cite="6. This is still “break-neck” speed: a yearly growth rate equal to [the growth of the entire US economy](https://fred.stlouisfed.org/series/GDPC1) over the last 27 years"></span>, we can expect 32 million H100e by mid 2027, instead of the 60 million that the authors predict.</p>
<p>At first, we might predict that this correction should delay the predicted advent of a superhuman coder AI, since, as we’ll see in the next section, training compute is one of the four major variables in AI progress speed. However, the authors’ actual <a href="https://ai-2027.com/research/timelines-forecast#:~:text=We%20first%20show,best%20AGI%20company.">two methods of forecasting the timing of superhuman coders</a> do not involve their predicted amount of available compute.</p>
<h2>Automated AI Researchers</h2>
<p><a href="https://youtu.be/htOvH12T7mU?si=PhZxTG7HeeBe3q9E&t=2044">In the Dwarkesh Podcast on April 3rd, he asks the authors,</a> if the mental labor toward improving the software of AI models is so important, why aren’t the major labs increasing headcount as quickly as possible? Daniel Kokotajlo, the first author, responds by saying that they see four inputs into AI progress speed: number of researchers, researcher serial speed (relatively constant for humans), research taste (i.e. the quality of selected experiments), and training compute. He says that, by the time they’ve fully automated AI research, the main bottlenecks will be training compute and research taste. In other words, the reason for the 25x multiplier, rather than something higher, is lack of training compute and research taste. If that’s the case, then cutting our available GPUs almost in half (from their expectation), would mean that training compute would be even more of a bottleneck, and having those extra hundreds of thousands of AI researchers working at 40x serial speed does not have as much gain as it otherwise would.</p>
<p>This matches up with a recent quick exchange I had with Daniel:</p>
<p>Me: I was curious as to how you all came up with the &quot;6% of compute will go to research automation?&quot;<br>Daniel: It was just an educated guess! I&#39;m lately thinking it should have been a smaller fraction probably, maybe more like 1%, to free up more compute for experiments.</p>
<p>They did not go into detail on answering Dwarkesh’s specific question (likely because he paired it immediately with another question/statement) about AI labs not hiring more humans today. However, I think it’s a very important string to pull on here, so let me humbly try to answer for them.</p>
<p>One way to look at it is “If OpenBrain wanted to hire an additional *really good* AI researcher today, the marginal cost is a yearly total compensation in the millions of dollars. That researcher might begin contributing strongly within a few weeks or months. Alternatively, hiring non-AI researchers, but very smart programmers and math/physics experts, would perhaps cost slightly less, but they might not begin contributing strongly for, say, 6-12 months. As most things in life, we need to look at things in terms of tradeoffs and opportunity cost. By allocating $3 million to pay Researcher #999 over the 2 years, OpenBrain could purchase 23 GB200 computers and get 6.53 x 1024 total INT8 FLOPs over that same timeframe (assuming a Model FLOP Utilization of 40%). Thus, rather than training ~5 more non-AI-researchers to become AI researchers, they would have enough <a href="https://epoch.ai/data-insights/models-over-1e25-flop">FLOPs to recreate GPT-4o</a> (slightly rounding down due to likely architecture improvements since the original GPT-4o) and more, since GPUs last longer than just 2 years of usefulness.</p>
<p>However, the marginal cost of adding a researcher goes down if the researcher is an AI. Using the authors’ unmodified numbers, if <a href="https://ai-2027.com/research/takeoff-forecast#:~:text=Our%20compute%20forecast,the%20best%20humans.">the leading AI lab uses 500,000 H100e to run 50,000 superintelligent AI researcher agents</a>, then the hardware to run one AI researcher (which accomplishes tasks at 30x human speed) costs $150,000 upfront (assuming <a href="https://ai-2027.com/research/compute-forecast#:~:text=%2420k-,%2415k,-Total%20AI%20datacenter">each H100e costs $15,000</a>), and they can continue running it for years. This is about an order of magnitude less expensive than paying a top-tier human AI researcher for just one single year.</p>
<h2>Running an Updated Takeoff Simulation</h2>
<p>Taking our updated parameters from Analyzing Assumptions (4x speedup for superhuman coder, and 30-day timesteps), and an updated value of 10x for the superhuman AI researcher speedup (and 100x for superintelligent AI researcher and 800x for general superintelligence, using the same methodology as <em>AI 2027</em>) due to the reasoning in the Automated AI Researchers section, the authors’ <a href="https://github.com/uvafan/timelines-takeoff-ai-2027/tree/main">takeoff simulation</a> gives us updated estimates.  </p>
<div class="flex flex-col items-center my-6">
  <img src="/images/takeoff_timeline.png" alt="Takeoff Timeline" class="rounded-lg shadow-md" style="width: 75%; height: auto;">
  <img src="/images/phase_durations.png" alt="Phase Durations" class="rounded-lg shadow-md mt-4" style="width: 75%; height: auto;">
</div>

<p>To highlight some differences we see compared to their estimate: the median expectation for the time between:</p>
<ol>
<li><strong>The entire takeoff scenario (Superhuman coder -&gt; General Superintelligence): 23 months, or almost 2 years (instead of 9.5 months in <em>AI 2027</em>)</strong>  </li>
<li>Superhuman Coder -&gt; Superhuman AI Researcher: 7 months (instead of 4 months)  </li>
<li>Superhuman AI Researcher -&gt; Superintelligent AI Researcher: 10 months (instead of 3.5 months)  </li>
<li>Superintelligent AI researcher -&gt; General Superintelligence: 6 months (instead of 2 months)</li>
</ol>
<p>Although this might not seem like much of a difference (2 years is still arguably unacceptably fast to go from humans being the smartest entities on Earth to computers leaving us in the dust [or <em>as</em> dust, if they are misaligned]), the updated numbers, if true, would give us over twice as long to: set up systems of human oversight, use the intermediate AIs to solve mechanistic interpretability or alignment, and/or have changes in major word leaders (I swear something important happens in January 2029, but I can’t quite place it).</p>
<p>It’s worth noting that this post is <em>not</em> an indictment of the authors of <em>AI 2027</em>. They clearly put much, much more thought, effort, and skill into their work than I’ve done here, and, most importantly, these takeoff timelines are well within their proposed confidence intervals. This post is exactly the opposite of an indictment, in fact, but more of my attempt at a minor contribution to their work because I find it so inspiring.</p>


AI 2027-2029

<h1>The Future of Software Development</h1>
<p>Most experienced software engineers will recall constantly needing to stay up-to-date with changes in software technologies in order to partake in most of the industry. Today&#39;s landscape is not an exception, but it does come with some notable differences.</p>
<p>The new players in the field (as it relates to this blog post) are not new programming languages, libraries, nor philosophies of code quality. Instead, they are large language models and their ability to code, test, make architectural decisions, and (soon) automatically perform deployment and other necessary actions.</p>
<h2>Coding and Testing</h2>
<p>In any given timeframe, software engineers are now able to create vastly more value, as some of the most time-consuming parts of coding are now essentially automated. Tools like GitHub Copilot, Cursor, and more are able to quickly implement:</p>
<ul>
<li>Boilerplate solutions</li>
<li>Code Refactors</li>
<li>Translation to alternative programming languages</li>
<li>Debugging</li>
<li>Unit test creation</li>
<li>Increasingly, entire features across multiple files</li>
</ul>
<p>This allows mindful software engineers to move quickly, focusing their efforts on:</p>
<ol>
<li>Directing the AI</li>
<li>Making high-level and architectural decisions (in order to do (1))</li>
<li>Carefully reviewing AI output</li>
</ol>
<p>At Vectorial, we take this very, very seriously. Each of our engineers is highly encouraged to take advantage of models and tools such as <a href="https://openai.com/index/openai-o1-system-card/">OpenAI&#39;s o1</a>, <a href="https://www.anthropic.com/news/claude-3-5-sonnet">Anthropic&#39;s Claude 3.5 Sonnet</a>, <a href="https://www.cursor.com/">Cursor</a>, and/or <a href="https://github.com/features/copilot">GitHub Copilot</a>. However, we recognize that these tools often make mistakes, so we also emphasize the engineers&#39; responsibility to review code heavily, and require all delivered code to be peer-reviewed by another human engineer.</p>
<p>The balance we strike here allows us to deliver software and AI solutions at a higher quality <em>and</em> a greater speed than ever previously possible.</p>
<h2>Architectural Decisions</h2>
<p>State-of-the-art models are increasingly becoming useful at more high-level tasks, such as designing entire technical architectures. Models such as o1 and Claude 3.5 Sonnet can often create exceptional first drafts of an architecture for a new product or feature.</p>
<p>These architectures are still prone to mistakes, which have the potential to be more dangerous than code bugs, as they are less detectable and fixable. Following them blindly can often be a one-way door that you can&#39;t easily reverse down the line.</p>
<p>That being said, having a human engineer treat these drafts as mere (possibly faulty) suggestions is often higher-quality and discovered faster than the architectures that a junior or mid-level engineer would come up with on their own.</p>
<h2>Automated Software Engineering</h2>
<p>As frontier AI labs continue to improve these LLMs, we continue to progress toward a future in which these AI models are able to perform every aspect of a modern software engineer&#39;s job. Currently, it does not seem clear to us at Vectorial whether or not software engineers&#39; skills will continue to be useful by &quot;moving up the abstraction ladder&quot; if non-technical people can direct the AIs just as effectively.</p>
<p>Products or technologies like <a href="https://devin.ai/">Devin</a>, <a href="https://github.com/All-Hands-AI/OpenHands/">OpenHands</a>, <a href="https://aider.chat/">Aider</a>, <a href="https://v0.dev/">v0</a>, and <a href="https://www.cursor.com/features">Cursor Agent</a> are all early versions of this. You simply explain in natural words what you want to build and the models continuously run, working on the code, and eventually come back with either questions or results (or failures, hence the &quot;early&quot; label!).</p>
<h2>An Example</h2>
<p>Take a quick look over our website. Look at the particle effects, the landing page, the blog post you&#39;re currently reading (not the words, but the code). This is all nearly 100% created by Claude 3.5 Sonnet (using Cursor) and o1. The particle effects are a custom implementation, fully developed and iterated on by o1. We estimate that building the website to its current state would have taken at least 3-5x as many developer hours without the use of AI. A completely non-technical person could have likely built and deployed this website in just 2-3 days.</p>
<video autoplay loop muted playsinline class="w-full h-auto rounded-lg my-4" id="particle-video">
  <source src="/images/vectorial-site-particles.mp4" type="video/mp4">
  Your browser does not support the video tag.
</video>

<script>
  document.addEventListener('DOMContentLoaded', () => {
    const video = document.getElementById('particle-video');
    const observer = new IntersectionObserver((entries) => {
      entries.forEach(entry => {
        if (entry.isIntersecting) {
          video.play();
        } else {
          video.pause();
        }
      });
    }, { threshold: 0.6 }); // Start playing when at least 60% of the video is visible
    
    observer.observe(video);
  });
</script>

<h2>How to Prepare</h2>
<p>Today, it is a good idea for software engineers to get acquainted with generative AI coding tools and agents. It is common to have a few bad initial experiences with generative AI, and avoid using it entirely, to one&#39;s own detriment. Instead, one can refine their abilities of using the tool (similar to the older concept of &quot;<a href="https://en.wiktionary.org/wiki/Google-fu">Google-fu</a>&quot;) and can remember to push the limits of the newest models as they are released, as previously failed uses may now be within the new models&#39; capabilites. It is also important, for the time being, to maintain existing programming skills, to avoid helplessness in times without access (such as in jobs/interviews where generative AI usage is forbidden).</p>
<p>In the medium-term, software engineers should prepare for the likely case of rapid change followed by near-full automation. This involves the job requirements changing rapidly every year or even every few months. Soon after, it may become the case in <a href="https://www.vectorial.us/blog/short-term-ai-progress">the late 2020s</a> that non-technical people can use these AIs just as effectively as software engineers. In this scenario, it&#39;s best to save as much of your income as possible today, as it may drop significantly (or to zero).</p>
<p>In this same scenario, software businesses should also prepare. &quot;Tech debt&quot; that can stay &quot;unpaid&quot; for a couple years may be worth making, since the cost of fixing it will continue to drop. It may make sense to stop hiring junior software engineers, and eventually mid-level engineers.</p>
<p>And businesses in general can prepare for significant technological disruption. For example, modern CRMs can, at risk of heavily oversimplifying, be described as a large user interface and user experience (UI/UX) operating on a database. Will that paradigm remain the same if a small team is able to simply ask OpenAI&#39;s future o4 or o5 &quot;generate and deploy code that can listen in on users&#39; calls, emails, and messages, and store data related to deals, meetings, etc. for them to ask about later&quot;?</p>
<h2>Key Takeaways for Software Engineers:</h2>
<ul>
<li>Adopt AI tools to stay competitive, but maintain rigorous review processes.</li>
<li>Keep your skills sharp — both technical (for deeper debugging/coding/architecture) and in AI usage.</li>
<li>Prepare for rapid changes in the job market: be financially cautious and professionally adaptable.</li>
</ul>


The Future of Software Development

<h2>AI Progress So Far</h2>
<p>The past few years have seen unprecedented advances in artificial intelligence, particularly through the usage of transformers in autoregressive and diffusion architectures. Since the release of GPT-2 in 2019, we&#39;ve witnessed a cascade of increasingly capable AI models from multiple organizations. These models have progressed from basic text completion to complex reasoning, coding, and even multimodal capabilities combining text, images, video, and audio.</p>
<p>Key milestones include:</p>
<ul>
<li>2019:<ul>
<li>Demonstrated the success of self-supervised training for decoder-only transformers (<a href="https://openai.com/index/better-language-models/">GPT-2</a>)</li>
</ul>
</li>
<li>2020:<ul>
<li>Set new standards for language understanding/generation; demonstrated in-context learning (<a href="https://openai.com/index/language-models-are-few-shot-learners/">GPT-3</a>)</li>
<li>Demonstrated breakthrough protein structure prediction (<a href="https://deepmind.google/discover/blog/alphafold-a-solution-to-a-50-year-old-grand-challenge-in-biology/">AlphaFold 2</a>)</li>
<li>Public launch of driverless taxis in geo-fenced areas (<a href="https://waymo.com/blog/2020/10/waymo-is-opening-its-fully-driverless-service-in-phoenix/">Waymo</a>)</li>
</ul>
</li>
<li>2021:<ul>
<li>First major product release of large language models (<a href="https://github.blog/news-insights/product-news/introducing-github-copilot-ai-pair-programmer/">GitHub Copilot</a> and OpenAI&#39;s <a href="https://openai.com/index/evaluating-large-language-models-trained-on-code/">Codex</a>)</li>
</ul>
</li>
<li>2022:<ul>
<li>Impressive multimodal understanding by processing text and image data jointly (<a href="https://deepmind.google/discover/blog/tackling-multiple-tasks-with-a-single-visual-language-model/">Flamingo</a>)</li>
<li>Compute-optimal scaling laws for transformer large language models (<a href="https://deepmind.google/discover/blog/an-empirical-analysis-of-compute-optimal-large-language-model-training/">Chinchilla</a>)</li>
<li>Significant advancement in text-to-image generation (<a href="https://openai.com/index/dall-e-2/">DALLE-2</a>, <a href="https://imagen.research.google/">Imagen</a>)</li>
<li>Brought conversational AI to the mainstream (<a href="https://openai.com/index/chatgpt/">ChatGPT and GPT 3.5</a>)</li>
</ul>
</li>
<li>2023:<ul>
<li>Massive improvements in reasoning and multimodal capabilities (<a href="https://openai.com/index/gpt-4-research/">GPT-4</a>)</li>
</ul>
</li>
<li>2024:<ul>
<li>Expanded multimodal capabilities, combining text, image, video, and audio input (and output for text and image) into the same models (<a href="https://openai.com/index/hello-gpt-4o/">GPT-4o</a>)</li>
<li>Heavy improvements in reasoning and overall model quality (<a href="https://www.anthropic.com/news/claude-3-5-sonnet">Claude 3.5 Sonnet</a>, <a href="https://openai.com/index/learning-to-reason-with-llms/">o1</a>)</li>
<li>Impressive video generation models (Sora, Veo 2, Runway, Kling)</li>
<li>Impressive music generation models (Suno, Udio)</li>
<li>Massive improvements in structure prediction of biomolecular interactions (<a href="https://blog.google/technology/ai/google-deepmind-isomorphic-alphafold-3-ai-model/">AlphaFold 3</a>)</li>
<li>Significant improvements for driverless cars <em>without</em> geo-fencing (Tesla FSD v12 and v13)</li>
</ul>
</li>
</ul>
<h2>Have LLMs hit a wall?</h2>
<p>At first glance, after going from GPT-2 -&gt; GPT-3 -&gt; GPT-4, it may feel like progress has stalled. However, we think this is the common pitfall associated with desensitization or <a href="https://psychcentral.com/health/hedonic-adaption">hedonic adaptation</a>. We believe there are three major sources of this increased desensitization:</p>
<ol>
<li>People who were previously paying attention are now paying attention <em>much more</em> now, so AI progress feels slower. Recall that it took almost 3 years from the GPT-3 announcement until the GPT-4 announcement. As of this writing, it has not even been 2 years since the GPT-4 announcement.</li>
<li>Model releases are much, much more frequent. This stems from more frontier AI labs (OpenAI, Anthropic, Google Deepmind, Meta), more secondary AI labs (Deepseek, Qwen, Mistral), and a shorter release cycle from each. Thus, there were much fewer models between GPT-3 and GPT-4 than between GPT-4 and GPT-4o</li>
<li>AI models are increasingly saturating both formalized benchmarks and unformalized ones (i.e. &quot;vibes&quot;). As models improve, people resort to harder and harder questions, and thus the rate of &quot;failure&quot; in real-life usage stays relatively constant from 2 years ago.</li>
</ol>
<p>The result of these sources is that AI progress can feel slower than it was 2 years ago, or 4 years ago. However, in reality, the models we have access to today are significantly better than models from 1-2 years ago, such as GPT-4 and GPT-4 Turbo. This is evident in essentially every single benchmark, as well as in anecdotal experience from our team.</p>
<h2>Will LLMs hit a wall?</h2>
<p>Assuming LLMs have not yet hit a wall: will they do so in the near future? Common arguments for &quot;yes&quot; include: running out of data, fundamental limits of autoregressive transformers, and diminishing returns of more/better data and more compute. Let&#39;s visit each point directly.</p>
<ol>
<li><p><strong>Running out of Data</strong></p>
<ul>
<li>Llama 3, the best knowledge we have of modern state-of-the-art LLM training, used 15 trillion tokens.<ul>
<li>Most of this data is from mass internet web crawls.</li>
<li>Some estimates put <a href="https://en.wikipedia.org/wiki/Books_published_per_country_per_year">the number of books published each year</a> at ~2.2 million. Assuming the average number of words per book is 80,000, that&#39;s ~235B tokens (~175B words) published every year. That amounts to 14 trillion high-quality tokens since 1960, matching the amount Llama 3 was trained on.</li>
<li>Google&#39;s YouTube has about <strong>1 million hours of footage uploaded every day</strong>. At 30 frames per second, and <a href="https://cloud.google.com/vertex-ai/generative-ai/docs/multimodal/video-understanding">258 tokens per frame</a>, that&#39;s about 27 trillion tokens of video added every single day. That&#39;s about <strong>one quadrillion</strong> tokens for every year of YouTube. This even ignores audio tokens from the videos as well! The tens of trillions of tokens used to train today&#39;s state-of-the-art models pale in comparison.<ul>
<li>Keep in mind, this is just one product. There are similarly crazy numbers of human-generated tokens on other social media services, such as Facebook, Instagram, LinkedIn, TikTok, Twitter, Reddit etc.</li>
</ul>
</li>
</ul>
</li>
<li>Frontier AI labs are finding surprising success with synthetic data approaches.<ul>
<li>For example, generating code or math using LLMs, and, using automated deterministic tests or even LLM judges, choosing high quality responses and performing Reinforcement Learning, fine-tuning, or even pretraining on them.<ul>
<li><a href="https://x.com/polynoamial/status/1834280425457426689">OpenAI has announced</a> that they used this methodology to create o1</li>
<li>Many other major and small labs have claimed they&#39;re seeing success with synthetic data.</li>
</ul>
</li>
</ul>
</li>
<li>In any case, if we really do run out of data, AI scientists have shown that scaling up model size results in better sample efficiency. We are currently in an overtraining paradigm (due to outsized inference costs compared to training costs), but if we transition to being constrained by data, there is a lot of room for improvement by entering an undertrained paradigm (by massively scaling model sizes).</li>
</ul>
</li>
<li><p><strong>Fundamental Limits of Autoregressive Transformers</strong></p>
<ul>
<li><a href="https://arxiv.org/pdf/2411.01992">Recent research</a> has mathematically proven that autoregressive, decoder-only transformers are <a href="https://en.wikipedia.org/wiki/Turing_completeness">Turing-complete</a> if given the correct input</li>
<li>Time and time again, skeptics of the modern generative AI paradigm have predicted that these models would fail at specific tasks, and time and time again they have quickly improved enough to succeed at those tasks.</li>
</ul>
</li>
<li><p><strong>Diminishing Returns of Increasing Data Size, Data Quality, and Compute</strong></p>
<ul>
<li>Of the three, this argument has the best potential to be right. We can never be sure that a certain trend will continue, and this is no different. However, we can take hints from the frontier labs, who have at least a few months headstart on seeing these models in action.<ul>
<li>Noam Brown (OpenAI multi-agent researcher): &quot;<em>We announced @OpenAI o1 just 3 months ago. Today, we announced o3. We have every reason to believe this trajectory will continue.</em>&quot; (<a href="https://x.com/polynoamial/status/1870172996650053653">link</a>)</li>
<li>Dario Amodei (Founder of Anthropic) loosely predicts 2026 or 2027 for &quot;very powerful AI&quot; (<a href="https://x.com/tsarnick/status/1856065476759302167">link</a>)</li>
<li>Sam Altman (Founder of OpenAI): &quot;<em>We are now confident we know how to build AGI as we have traditionally understood it. We believe that, in 2025, we may see the first AI agents “join the workforce” and materially change the output of companies.</em>&quot; (<a href="https://blog.samaltman.com/reflections">link</a>)</li>
</ul>
</li>
</ul>
</li>
</ol>
<h2>What might 2030 look like?</h2>
<p>There are a few major trends to pay attention to:</p>
<ol>
<li>More data and more high-quality data are being used to <a href="https://www.clrn.org/what-is-pretraining-and-post-training/">pretrain and post-train</a> these models.</li>
<li>Progressively increasing compute.<ul>
<li>Massively parallelized computer chips are continuing to improve in speed, memory, and other metrics.</li>
<li>Larger and larger clusters of these chips are being built constantly. <a href="https://blogs.oracle.com/cloud-infrastructure/post/now-ga-largest-ai-supercomputer-oci-nvidia-h200">OCI Supercluster</a> from Oracle, <a href="https://www.tomshardware.com/desktops/servers/first-in-depth-look-at-elon-musks-100-000-gpu-ai-cluster-xai-colossus-reveals-its-secrets">Colossus</a> from xAI, and rumors of <a href="https://www.tomsguide.com/ai/meet-stargate-the-dollar100-billion-ai-supercomputer-being-built-by-microsoft-and-openai">Stargate</a> from OpenAI and Microsoft</li>
</ul>
</li>
<li>Increasing focus and efforts on post-training, using methods such as RLHF, RLAIF, supervised fine-tuning, and other reinforcement learning and synthetic data methods based on provably validity of response.</li>
<li>Improving on the vanilla transformer architecture. Papers like the <a href="https://arxiv.org/abs/2412.09871">Byte-Latent Transformer</a>, <a href="https://sakana.ai/transformer-squared/">Transformer^2</a>, <a href="https://arxiv.org/abs/2412.06769">Reasoning in Continuous Latent Space</a>, and many others are recent, seemingly successful, attempts at this.</li>
<li>Distributed training. As cluster sizes hit physical and infrastructure limits, research is being done on the potential for distributed training of models. <a href="https://www.primeintellect.ai/blog/intellect-1">Prime Intellect&#39;s DiLoCo</a> and <a href="https://github.com/NousResearch/DisTrO">Nous Research&#39;s DisTrO</a> are examples of this.</li>
</ol>
<p>If we extrapolate these trends, 2029 and 2030 may look like enormous (even relative to today), distributed training runs for heavily multimodal (text, image, video, and audio input <em>and</em> output) models. These models may be able to autonomously perform long-horizon tasks, such as: building complex web applications, creating detailed visual media, managing a team of and writing detailed research on computer science, artificial intelligence, mathematics, physics, and more (chemistry and biology may require more human intervention due to laboratory tasks).</p>
<p>This would be the beginning of massive job automation, where most white-collar jobs would be able to be fully replaced by AI. Depending on the manufacturing speed of general-purpose (e.g. humanoid) robots, we may start to see significant blue-collar job replacement as well.</p>
<p>It would also be the beginning of a lack of scarcity (relative to today&#39;s society). After controlling for potential governmental increases of monetary supply, prices for nearly every good or service should drop significantly, as the cost of producing them drops, and as overall efficiency increases.</p>


Blog

Let's Align Alignment

AI Coding Agents Are Becoming Execution Layers

AI 2027-2029

The Future of Software Development

AI Progress in the Second Half of the 2020s