Where the loop came from

When ChatGPT launched it had no loop at all. Request, response, done. You asked, it answered, and if the answer was wrong you asked again. You were the loop.

The first real loop arrived with reasoning models. Instead of answering in one shot, the model was allowed to think, call a tool, look at the result, think again and running until it decided it had a response. Give that loop the ability to act, edit files, run commands, open PRs, and you get agents like Claude Code. That’s still the first loop. The halt condition is the model’s own sense that its reasoning is done.

The moment agents got useful we started handing them goals instead of steps. Not “write this function,” but “make this test pass,” “ship this feature.” That’s the second loop, and it’s the one we’re in now: the thing runs until the goal is met, not until the reasoning effort is spent. The halt condition moved up a level, from “I’m done thinking” to “the objective is achieved.”

It’s not hard to see where this goes. Each loop swallows the one below it and takes a more abstract goal. It looks a lot like climbing a company hierarchy: at the bottom someone is told exactly what to do, a level up they’re given a task and left to work out the steps, a level up from that they own an outcome and decide the tasks themselves. This is what the conversation about loops and factories is all about. Now let’s see how we can put it in practice.

Operate one level up

Here’s the whole method in one line: build the level of abstraction one step above where you’re working today, and do your work from there.

Make it concrete. Right now most engineering with AI happens at the keyboard. You, or your engineers, instruct a model to write code and correct it interactively through a CLI, prompt by prompt. That’s operating inside the first loop. The move up is to stop being the thing that prompts the agent.

You can call this way of working loopcraft, loop engineering but regardless of the name what it means is: replacing yourself as the person who prompts the agent. So instead of correcting output live, you build a coding agent that takes a brief and produces a PR, ideally correct in one shot or with minor feedback, and you stop iterating on the code. You iterate on the design of the agent. When a PR comes back wrong, you don’t fix the PR. You fix the thing that produced it, so the next hundred come back better.

This is different from a coding agent inside a CLI. You’re not sitting in front of it. It picks its work up from where the work already lives, a ticket in your project tracker, a message in your team channel and delivers a PR back. The best demonstration of this is the recently released Claude Tag: you @-mention it on an issue and it goes away, does the work, and opens the pull request. No terminal, no live prompting. The task comes in through the tools your team already uses and the result comes back the same way.

Once you have that, you do the same move again, one level up. Now the interesting question isn’t how the code gets written, that’s handled. It’s where the tickets come from. And a ticket is usually a by-product: something said in a meeting, a request buried in an email, a bug reported in a channel, an issue in the repo. So you build the layer above the coding agent, a system that reads those sources and drafts the tickets, which you review and feed to the agent that solves them. Now you’re not writing tickets either. You’re steering a pipeline that turns raw signal into shipped code, and can interact with the sustem in different levels of abstractions: the ticket created, the PR produced etc

As far up as it goes

You can keep climbing. Every function of a business is, in principle, some loop of signal in and work out, and each one can get an agent that does the work a level below where you sit while you move up to steer it. Taken to its end, you run the whole organisation this way: agents wired into every function, doing the work the business theoretically needs done, and you intervening to correct the system rather than to do the work yourself. This is the loop-and-factory idea taken to its extreme, not one agent in a loop, but the org itself as a stack of loops you supervise.

I’m not claiming that end state is reachable for a whole business, or that it should be. Plenty of the work won’t fit the transformation, e.g. the judgment is the job and can’t be handed down a level. The exercise is still valuable. Find the bottleneck, push the agent one level of abstraction up from where it sits today, and watch how many loops you can close in your organisation

• Pay attention: GLM-5.2 — The new open model leader and once that many claim is good enough to replace your preferred closed ai model.

• Pay attention: Loop engineering emerges — The shift from one-shot runs to standing loops that keep agents working just got a name, used publicly by Boris Cherny (Claude Code) and Peter Steinberger (OpenClaw).

• Pay attention: Claude Tag: persistent agents in Slack — The async-agent form factor is crossing out of the terminal and into the surface teams already work in. Worth watching as the first mainstream place non-engineers meet a standing agent.

• Pay attention: Claude Fable 5 / Mythos 5 — The first generally-available Mythos-class model, SOTA on nearly every benchmark (AA Index #1 at 64.9, SWE-Bench Pro 80.3%) and a leap ahead of competition.

• Skip: OpenAI GPT-5.6 (Sol/Terra/Luna) — An expected frontier bump that matches rather than moves the story, hence skip.

Quick takes

• Anthropic takes the lead with Fable - Fable 5 broke a year-long pattern. Instead of trading a couple of benchmark points, it jumped ~10 points above the trend line on SWE-Bench Pro, and led on revenue, distribution and for the first time raw capability at once.

• Two monarchies, open vs closed - Closed AI is a US monopoly (Anthropic); open AI is effectively a Chinese one (DeepSeek, Kimi, GLM, MiMo), with the open crown rotating every few weeks. GLM-5.2 now leads the open pack, ~5 index points behind the frontier and for most real coding work, one generation behind already buys most of the value at a fraction of the cost, on weights you host.

• We might be measuring intelligence wrong - Benchmarks reward pass@k, right at least once in k tries. But agents act once, so what matters is pass∧k, right every time. Capability has raced ahead; reliability is climbing behind it which is why the impressive monthly numbers don’t always translate to impressive agentic performance.

It is not a multi-agent system

When people picture the factory, they picture fleets of specialised agents, a planner, a few doers, a reviewer, a tester, passing work to each other like stations on an assembly line. In fact the opposite is happening for the precise reason that Cognition, the company behind Devin, articulated in Don’t Build Multi-Agents. In short, every action an agent takes carries implicit decisions, and parallel agents that cannot see each other’s full traces make conflicting decisions that compound into incoherent work.

Instead the factory consists of many independent but powerful agents. Stripe’s minions are one agent loop wrapped in “blueprints” state machines that interleave deterministic steps (lint, push, run the right slice of a 3M-test battery) with agentic ones (implement, fix CI). Spotify’s Honk is one deliberately minimal Claude Code agent whose git cannot even push. The only genuine second agent anyone runs in production is OpenAI’s Auto-review, and it is a safety reviewer gating sandbox escalations, not a quality reviewer. The factory is one agent and a lot of plumbing.

The agent gets the environment you would give an engineer

If the factory is one agent, the first practical question is where it works. The convergent answer: a sandboxed environment that looks remarkably like what you would hand a new engineer on day one, the repo, a shell, the tests, and access to the tools your engineers get. Stripe spins up pre-warmed devboxes in about ten seconds, identical to the ones human engineers use, cut off from production and the internet. Ramp restores Modal sandboxes from repo snapshots rebuilt every thirty minutes, so the agent starts near-instantly on code that is at most half an hour stale, with the same telemetry, Sentry, Datadog, feature flags its engineers check.

This environment is not a new idea. Google’s engineers have worked in the cloud for over a decade. By 2023 around 80% of development on its main codebase happened in Cider, its web IDE, on top of a file system that streams the monorepo on demand. GitHub Codespaces made the dev environment a URL back in 2020, and plenty of startups run on that setup today. As it turns, this way of working is an even better fit for agents than engineers since a repo, a shell and a test runner is all an agent needs, it spins up a fresh one per task, and it does not keep your hours.

A hybrid, not a handover

The trending setup, on the other hand, is not “agents slowly replacing developers”, it is a hybrid. A background fleet independently completes the delegable work, while every developer keeps an agentic coding tool like Claude Code or Cursor in hand for the rest. This has two properties I like. First, you are permanently testing the frontier: when an autonomous run fails, the task degrades gracefully into an assisted one rather than a rewrite, and when the models improve, tasks graduate the other way. Second, it is assistive to the developer’s flow rather than disruptive, the developer and the agent work the same way, in the same environment, against the same verification, so nothing about the codebase has to be split into “human code” and “agent code”.

Which tasks go to the factory

Not everything belongs on the conveyor belt yet so it’s natural to ask which tasks do you send towards the autonomous agents vs your developer team. The tasks organisations actually delegate are well-scoped and mechanically verifiable: migrations, dependency upgrades, bug fixes, cleanup. Spotify’s Honk is openly a migration factory, its dataset migration shipped 240 automated PRs that the company estimates saved ten engineering weeks. Stripe’s minion PRs skew towards config changes, upgrades and small refactors. OpenAI runs recurring cleanup agents whose refactoring PRs are small enough to review in under a minute.

These task classes serve as an excellent testing ground as you keep exploring a boundary that keeps moving. Anthropic’s When AI Builds Itself charts exactly this: success on well-defined tasks saturated first, while success on open-ended coding tasks climbed to 76% by May 2026, which is how 80% of the code merged at Anthropic ends up written by Claude.

The practical rule: delegate where a pass/fail check exists, keep judgment work hybrid, and re-draw the line every quarter, because the open-ended curve is the one moving.

Keeping the agent on task

The main steering mechanism remains unglamorous: a CLAUDE.md or AGENTS.md file plus skills. It is a compact, inspectable way to inject specific instructions, org policies and practices, and a working memory into every run. The pattern that works is now well documented, OpenAI tried the one big AGENTS.md and reports it failed in predictable ways; what survives is a ~100-line table of contents pointing into real docs, kept fresh mechanically. GitHub’s analysis of 2,500+ repos lands on the same size, and finds “never commit secrets” the most common instruction in the wild.

All of this is scaffolding the labs are actively trying to absorb into the models, better memory, better context management, agents choosing what they need. Until they do, these files and loops are the interface to your factory.

So what do you do?

If you are an engineering leader, the factory reframes your questions. Not “should we use AI coding”, that ship has sailed, but three concrete ones. What environment do agents get? Give them what you would give an engineer, sandboxed and scoped. How do you keep them on task? An AGENTS.md worth maintaining, a verification loop the agent cannot talk its way around, and hard caps on iteration. What goes to the agent versus the human? Start with migrations, upgrades and bug fixes, run everything else hybrid, and move the line as the frontier moves.

What To Pay Attention To

• Pay attention: When AI builds itself — Pay attention as a long-term trend. In Anthropic’s own terms, they are unsure whether progress will stagnate and humans’ high-level direction will still be needed, or whether progress continues and we end up with recursive self-improvement. What’s worth keeping in mind is that at the frontier 80% of the code is already being written by AI, and there is a real possibility of the tech department in an org being completely overtaken by AI.

• Pay attention: The age of async agents — In the meantime, agentic coding is moving to the cloud, which has many benefits. This is not a new trend — see GitHub Codespaces, and of course orgs like Google where the coding environment always lived in the cloud — but this time it feels clearer and more pervasive because of how little setup a coding agent needs to get going. It means businesses need to start thinking about how they enable async agentic coding for their workforce, to get the maximum benefit from agents running 24/7.

• Pay attention: Pragmatic Engineer’s 2026 AI survey — Three pieces in one, with super useful insights: senior engineers now delegate to AI instead of junior devs; AI coding quality reflects the seniority and knowledge of the user, which disproportionately hurts the inexperienced; and hiring has never been stronger even as more and more code is written by AI — Jevons paradox in practice.

• Skip: Opus 4.8 and Dynamic Workflows — Opus 4.8 is an iterative improvement, hence skip. Dynamic Workflows is worth keeping in mind, but it sits on the same linear trajectory of more and more work being pushed towards agents through skills, workflows and other constructs — constructs which may soon be less relevant as AI gets better at choosing what it needs by itself.

• Skip: Cursor Composer 2.5 — Cursor is having a quiet comeback, slowly climbing the agentic coding rankings with its own model (fine-tuned on Moonshot’s open-weight Kimi K2.5), but it’s not something you should pay too much attention to — not important enough to even think about switching away from Claude Code.

A couple of years ago, prompt engineering was going to be a career. Then the models got good enough that the tricks stopped mattering, and the job evaporated before it really arrived. Nowadays, it’s all about harness engineering, the scaffolding around the model: how it reasons, calls tools, manages context, remembers, checks its work. So it’s worth asking: will harness engineering have the same fate as prompt engineering?

This is not to say that the harness is not important because it is. It is what turns your AI model into an agent. However, it currently serves two distinct roles, one of which gets more publicity and is more important in the short term, with the other being the core of what stays. Let’s start with the first.

Patching limitations of current models

One of the main reasons, harness engineering is getting a lot of attention is because it makes an existing model perform better on a task. That on its surface seems valuable and can be mistaken for long term business value. However, I don’t think most of those fixes are here to stay. Here is a list of such fixes, some of which are already obsolete:

🤔Reasoning like chain-of-thought, self-reflection loops (Reflexion, Self-Refine). Before reasoning models, there were plenty of techniques that were making models better at thinking which had a downstream effect in real world tasks, coding included but nowadays models are trained to reason so these techniques got removed from the harness level.

🛠️ Tools. Early harness code was also fixing issues with tool use, format the call, parse output, retry bad JSON. Models choose and call tools cleanly now, the skill is being internalised. Tools choice still matters but it plays an ever smaller role as models become better at making the right choice by themselves.

💿 Context. Pruning, compacting, pulling key facts back is where a lot of today’s lift sits. Manipulating the context is the latest art form. Bringing the right tools closer to the context, keeping the right information etc and suddenly your agent starts performing much better. However models are already getting better at managing their context so I am not so sure this will be an important part of future harnesses.

🏭 Domain knowledge is the single biggest lever right now. The reason being that current models, even though excellent at most domains, they lack on the job experience in your, or any, company. This means that there is plenty of room for adjusting how the agent performs in the tasks that are important to you. I am under the impression though that this will also go away when companies devise agent onboarding materials that the agent can refer back to.

In any case, it is important to be sceptical about additions to the “intelligence“ layer that will very likely go away at the next model iteration.

The agent runtime

The other role of the harness is more boring but more important in my opinion. It is the part that runs the AI model. This is the orchestration in some way, the software engineering engine around AI. It’s more the product than the intelligence itself. It contains things like

⚙️ Runtime. The engine that actually executes the model and its tools, reliably and at a sane cost. A smarter model still has to run somewhere, and keeping it up, retrying what fails and not blowing the budget on tokens is plain infrastructure work that no model iteration takes away.

💾 Storage. Sessions and memory have to live somewhere, which gives you an auditable trail of what the agent did but also the substrate for it to improve over time. However much a model can hold in its head, it still needs a notebook outside of it.

🔌 Integration surface. The access to your systems and data, and the ability to take real actions in the world. A brain with no hands does nothing, and wiring up those hands is ordinary software engineering that only grows as the agent is trusted with more.

🔒 Permissions and policies. The guardrails around what the agent can and cannot do, what it can touch, spend or send. The more capable the model gets the more this matters, not less, because a smarter agent left unchecked can do more damage and do it faster.

✅ Independent verification. One model doing the work and another checking it, because a model grading its own output goes soft the same way you shouldn’t review your own PR. This is a product decision about trust, not a trick to squeeze more intelligence out of the model.

So what do you do?

I would start by separating the above two aspects of the harness. The intelligence uplift , context juggling, the domain rulebook which is a tune, not a moat: the same kind that prompt engineering and fine-tuning gave. Don’t over invest in it, as it is a depreciating asset.

The durable element of the harness is the other half, the body and the product around the model brain. . That’s just good software engineering, and good software engineering is here to stay.

Pay attention: LLMs are finding real, unknown vulnerabilities now - Google reported the first known in-the-wild case of attackers using an LLM to discover a previously unknown vulnerability, and the UK AI Security Institute now clocks top models executing attacks that would take a human ~3 hours (up from a 1-hour forecast, and 30 minutes at Opus 4.6’s debut). The patch-vs-exploit math has changed, and LLM-found logical flaws are an active threat rather than a forecast.

Skip: All model labs are now agent labs - The whole industry capitulated to the harness in one news cycle (OpenAI, AI21, even DeepSeek), and the scary version of the story is that labs co-train models to only work inside their own agent, but I’m not convinced the lock-in bites, because even if the raw model stops being freely swappable, the agent layer on top of it probably still is.

Skip: The labs are moving into the consulting lane - OpenAI bought a 150-person FDE shop and Google joined the hiring spree, but labs chasing services revenue is just gravity, and unless you’re a consultancy or a thin wrapper you’re fine. The more useful takeaway is that the AI part itself won’t be your differentiator for much longer, because everyone, the labs included, is racing to embed it everywhere.

Skip: “Flash” is no longer the cheap tier - Gemini 3.5 Flash landed at frontier-class scores but 5.5× the price of its predecessor, so the “Flash = cheap” heuristic is not relevant anymore. At the same time, if you specifically need the speed, Flash is the fastest model on its intelligence level

Skip: An OpenAI model disproved an 80-year-old Erdős conjecture - A general-purpose model produced a genuinely novel result in under $1,000 of compute, a great story of AI scratching the surface of what’s possible, but it doesn’t change any practical use case you will build.

The mental model is the product.

When you program, a big part of the work isn’t typing code. It’s fitting what already exists, and what needs to exist, into your head, and coming up with a solution that delivers the result in a way that is clean, extensible, and consistent with what the owner of the ticket actually wants. The output you’re really producing is the mental model of the system. The code on the screen is a side effect of having that model in place. Asking an AI agent to do the work, while under specifying how or what needs to be done, breaks that. The mental model isn’t getting built, the AI is filling in the gaps with its own assumptions about your intent. And that doesn’t speed you up, because you’ll eventually need to sync mental models with the system anyway. You’ll still have to decipher what got built and confirm it matches what you wanted.

The difference is that now you do this iteratively. The AI produces something, you look at it, you realize you actually wanted X not Y, you tell it to adjust, it produces a new version, you refine again until you converge. Every individual step feels fast. But are you faster, or are you just moving fast down a longer road?

This isn’t speculation. Researchers and engineers building with AI are converging on the same observation, and proposing different responses, spec-driven development, agent scaffolding, treating the LLM as a junior engineer who needs a tight brief. What matters in practice is being able to inspect whether you’re in this situation and mitigate when you are. There’s no clean, well-proven solution yet. So here are some thoughts and observations from my own coding journey with AI.

How can we measure this?

The question is whether we are under specifying and paying the price by rewriting too much of our code too often. At the same time we want to know if some metric of business value or product velocity has meaningfully ticked upwards. Here is one way to quantify them (by no means the only or the right one)

1. Code half-life. Of the code you wrote N weeks ago, what percentage is still alive in the codebase today? A high half-life means the code you ship sticks. A collapsing half-life means you’re rewriting too often.

2. Adjustment rate in your AI sessions. What fraction of your AI coding sessions are refinements of prior AI output i.e. “remove what you added,” “different approach,” “also do X”, versus brand-new scope or status checks? A pre-AI analog exists too: PR descriptions and commit messages with fix: / refactor: / revert: prefixes.

3. Story points (externally validated). How many units of real value are landing per unit time? If you have a PM, a customer, or any external system that quantifies what was actually delivered, that’s one metric that speaks to increased product velocity or business value. This one is also harder to fake than the other two.

The three together are one way of trying to measure if you are moving faster towards a straight line or doing circles to reach the same destination. Writing less code raises half-life but tanks tasks shipped. Accepting whatever the AI produces lowers adjustment rate but tanks half-life. Only “specify up front, ship deliberately” pulls all three in the right direction at once. Here is how those metrics look for two of my projects. Project A is a pre-AI codebase while Project B is post-AI.

Throughput is way up. No ambiguity 3–4× more shipped per month, with similar effort per task. The rabbit is faster. But the durability collapsed. Code that should be settled by month four has been rewritten down to 11% of itself. And about a quarter of my prompts to Claude are refinements after the fact.

So how do you make the rabbit win?

What my data and others’ suggest is that you should treat speed-of-generation as a tax base, not a finish line. The tax you pay is in churn and rewrites. The way to lower the tax is upstream, by spending more time building a mental model first, so that what the AI does isn’t something that will require ten passes of refinement.

The turtle isn’t slow because it types slowly. It’s slow because it thinks before it moves. The rabbit can move much faster but it only wins the race if it spends enough of its speed advantage on knowing where it’s going.

Three signals to start with: code half-life, adjustment rate, externally-validated points shipped. Measure them, watch the trend, and see whether your road is getting straighter or just longer.

• Pay attention: OpenAI deprecates finetuning APIs. Fine tuning was an important lever for improving models till very recently. This marks the end of an “era“ in some sense.

• Pay attention: Anthropic meters programmatic Claude usage. Anthropic, and other labs have been subsidising subscription usage but it looks like the end of this might be coming closer so definitely worth planning for a future where token pricing and subscription are more equal.

• Pay attention: METR - speed up, value up… less. Yet another study that speaks to the rabbit and turtle phenomenon of speeding up code development does not necessarily translating to real productivity gains

• Skip: DeepMind AI Co-Mathematician - I would not pay too much attention on harness and manual gains on top of latest models that might become irrelevant as soon as the next version launches.

• Skip: Interaction Models (Thinking Machines Lab) - A super interesting work with questionable real world use cases, at least not as many.

Think of an Amazon-like catalogue. A product is not just a paragraph. It has a title, brand, category, price, colour, size, availability, reviews, tags, seller information, compatibility fields, and a free-text description. If you flatten all of that into one text blob and embed it, you may erase the structure that search actually depends on.

The Issue

This is why embeddings and “semantic search” may underdeliver. The embedding may understand the general meaning of the product description, but fail to preserve distinctions that matter: category, compatibility, size, colour, availability, jurisdiction, date, or whether a field is a hard requirement rather than background context.

There is an easy way to diagnose the issue. Take items you know are similar and confirm their embeddings are close. Take items you know are different and confirm they are far apart. Look at whether similarity scores are meaningfully spread, or whether everything clusters in a narrow band.

When choosing an embedding model, the question is not only “is this model trained on my domain?”, it is also “was this model trained or benchmarked on data shaped like mine?” A retail embedding model may still be a poor fit for structured product records if it mostly learned document retrieval. A more general model tested on tables or semi-structured retrieval may be more relevant.

Benchmarks such as STaRK, which evaluates retrieval over semi-structured knowledge bases, and TARGET, which looks at table retrieval, are useful because they focus on format, not just domain. The lesson from this literature is clear: pure embeddings are often weak when the record structure matters. Hybrid approaches tend to do better.

Before we look in hybrid approaches into more detail though let’s take a step back and start from the issue: “what kinds of queries are users asking, and why do they fail today?”

Query Types

In many systems, queries typically fall into the following buckets.

• keyword lookups. A user knows the exact product name, ID, model number, company, SKU, document, or phrase. These usually need strong lexical search, synonyms, analyzers, spelling tolerance, and sensible boosting.

• text-and-filter queries. The user writes natural language, but the query contains structured constraints: “wireless headphones under £100 with noise cancelling,” “contracts from Germany after 2022,” “suppliers with ISO certification in Spain,” or “grants about youth mental health in Wales.” These should not be treated as one semantic blob. They should be decomposed into free text plus filters.

• semantic queries. The user describes an intent in language that may not appear in the indexed record: “something comfortable for long flights,” “a laptop for light video editing,” or “insurance policies that cover climate-related disruption.” These benefit from semantic expansion, dense retrieval, or both.

• assistant questions. “Which one should I buy?”, “Can I return this?”, “What is the policy?”, “Which supplier is safest?” Those may not belong in search at all. Route them to a chat surface that has access to the same data, but treat it as a separate product.

Each type will require a slightly different approach to make it work.

A Better Search

Query decomposition

Take text-and-filter queries for example, the most useful move is query decomposition. A frontier model can turn a natural-language query into a text query, structured filters, and exclusions. For example, “wireless headphones under £100, not refurbished, for running” contains a product category, a price constraint, an exclusion, and an intended use. A less capable model may extract the obvious terms and miss the negation. A frontier model can preserve the structure. This is powerful because it improves the existing search engine rather than replacing it. Most mature systems already have useful filtering, field boosting, and keyword matching. The model’s job is to translate messy user language into the shape the search engine already understands.

This pattern is supported by work on LLM-driven structured query extraction. Fine-tuned models can convert messy user input into a combination of semantic terms, numerical constraints, and categorical filters that an existing search engine already understands [see]. The domain changes, e-commerce, enterprise search, internal tools but the pattern is the same: do not throw away structure when the user is asking for structure.

Query expansion

For semantic queries, there is an easier step before even looking at embeddings: query expansion. Use a frontier model to generate semantically similar reformulations or a hypothetical result description, then retrieve against those expanded queries. Take the query “running headphones that don’t fall out”. A short query like that has very few terms in common with a typical product description. A frontier model can expand it into “wireless sports earbuds with secure ear-hook fit, sweat-resistant, IPX4 or higher”, or even synthesise a hypothetical product description and embed that.

Work such as Query2doc and HyDE were among the first to explore this strategy but it’s one that lasted the test of time. The way it helps is by improving recall, since it changes what enters the candidate set. A reranker cannot recover a result that was never retrieved in the first place.

Reranking

After decomposition and expansion, reranking becomes the layer that fuses and corrects. Retrieve a wider candidate set from lexical search, filters, expansion, and possibly embeddings. Then use a reranker to reorder the candidates based on the full query intent. In production, dedicated cross-encoder rerankers are often a practical starting point because they are cheaper and faster than frontier models. Frontier models are still useful for prototyping, calibration, and generating training data.

There is plenty of research showing that cross-encoders are “very competitive” with the latest models as rerankers and “way more efficient” across standard IR benchmarks, making them the right first production choice, while frontier models earn their keep generating training data and listwise judgments that can be distilled into a custom reranker later.

AI judge

But none of this matters if you cannot measure whether search improved. This is where AI judges are useful, with one important caveat: do not ask a model for a vague relevance score. Decompose relevance into criteria. For a structured catalogue, criteria might be: does the product category match, do the key attributes match, does it satisfy the constraint, and how specific is the result? A result that fails a hard constraint should not score highly just because it is semantically related.

The judge should see structured fields, not a dumped text blob. It should explain each criterion before scoring. And it should be calibrated against human judgments. The LLM-as-judge literature increasingly supports this direction: criteria-decomposed rubrics, few-shot examples, compact grading scales, and periodic human spot checks.

Then compare systems with proper search metrics: nDCG@10 for ranking quality, relevant@10 for how many useful results appear, and a gate-failure rate for hard mismatches. Keep a frozen baseline so you know whether a new version improved search or only changed it.

Conclusion

Adding AI into your search sounds simple but there are a couple of moving components you need to get right, starting from which embeddings or whether you use embeddings at all, to understanding the queries of your users and choosing the right method for the problem. This is without even mentioning fusing results and judging improvements at scale.

What To Pay Attention To

• Pay attention: Mythos + Opus 4.7 — Mythos looks real after Mozilla used it to harden Firefox at scale, while Opus 4.7 is the deployable jump in Claude capabilities that matters because many teams already use Claude and Claude Code for serious work.

• Skip: GPT-5.5 — This feels like the expected OpenAI move: matching the frontier on agentic coding and knowledge work, but not changing the story enough to dwell on.

• Skip: Kimi K2.6 + DeepSeek V4 — These releases keep narrowing the open-model gap, especially on cost, but the evidence still points to them trailing the newest frontier models on the hardest real-world agentic work.

• Skip: Tokenmaxxing — Token consumption is a bad proxy for productivity, though it is worth watching because it may distort software engineering costs, team metrics, and hiring expectations.

• Pay attention: RIP PR / new coding practices — The important shift is that software work is moving upstream from writing and reviewing diffs to describing intent, setting architecture, defining checks, and supervising AI write-review loops.

This newsletter is for tech leaders under pressure to ship AI features while the ground keeps shifting beneath them. For over 12 years, and most recently through my own AI consultancy, I’ve helped organisations of all sizes make better use of AI. The goal here is simple: help you make sharper, more grounded decisions about building around AI.

Each week, one piece on a decision you’re facing. It might be architectural. How do you put an agentic layer on top of your product? It might be strategic. Train around today’s model limits, or wait for the next iteration? Plus a short read on the week’s AI news. What’s worth your attention. What’s noise.

Subscribe now