CHAPTER II.
THE DARK STACK

So far, we have taken pains to stay as medium-agnostic as we can. After all, we do not and cannot know what the superdark factory is made out of. However, we need the dark stack to accomplish two things: the creation of the technical conditions for the superdark factory to emerge and the production of the superdark factory’s identity. It is easiest and most productive to think through these two moves, for now, in terms of software. The present state of software affords us the types of resources we need for both moves: we need recursivity and self-adaptation for the factory’s technical preconditions, and we need statistical analysis and human–machine interfaces for identity production. Agentic AI is a particularly good system to think through all of the above, not as a hard necessity but as a well-equipped (for our purposes) modern toolchain that is already at the forefront of advanced contemporary automation.

We will interleave these two deliverables—technical precondition and identity production—through these four sections. The primitive and evaluations (I and III) are ways of enabling and influencing the processual self-development of a complex system. Versioning and the charter (II and IV) are ways of holding such a system to an identity that is absolutely exterior to itself. We discuss all of the above in terms of software (principles for applied programming) but will make plenty of room to think about them in more abstract terms.

I. The Primitive

You are not alone in the codebase; do not revert edits made by others and adapt to any existing changes you might find.
—Codex subagent direction (by convention) 27

The time of writing software is over. Software is gardened, not maintained. Software is impelled, not declared. Software is a product of (or an externality of?) a feedback loop, not a static, archival object. Software is content, and content is no longer an available surface for craft.

For now, we can think of the dark stack’s medium as software, but it is also pre-software: the stuff that it makes self-assembles into software, and that stuff sits inside the bones of software like a fatty, tacky tissue leaking blood cells (or like a genetic bomb with a hair trigger). In this capacity, it is a kind of control mechanism built into a world, or a kind of tactical world-building or worlding, a way of seeding a world-to-be-built with information, a way of seeding the information available within a world with control. The dark stack creates the possibility for a certain kind of software, a self-guiding and autonomous kind of software, while also sneakily delimiting, with intention, what that software might be allowed to become.

The superdark factory composes itself out of primitives—the smallest and most atomic units of software, software units that cannot be further decomposed into subcomponents without breaking into some other descriptive register (some other language, some other domain, some other application). The intentional design of these primitives is the first technique of the dark stack.

Primitives stack, they stick together, and their individual shapes have massive downstream effects on what kinds of future shapes might emerge through their adhesion, much like how the physical structure of DNA informs the construction of proteins or how the physical structure of an oxygen atom informs its hydrogenation into water. Some primitives can stack into anything (like a primitive that receives and emits only yes or no answers or most basic mathematical functions). 28 Other primitives fit together in specific ways that enable specific geometries. What began in computer science as a gate or as an immutable or as a type encountered a rising abstraction floor, from a struct to an object to a class of objects (to a hosted application, to a container, to a service). As this abstraction floor kept rising, the weight of design decisions rose with it: The more complex things that could be built became doubly contingent on the limited ways that other complex things could be combined and arranged (Gamma et al. 1995).

It would be trivial to say that software design is the same thing as primitive design, in the sense that “writing software” means the same thing as “arranging preexisting components into some new whole.” It would be more interesting to say that software design has “retreated” into primitive design, in the sense that the work of software is not to conduct the arrangement of primitives into this or that new application but rather to stage or collect or fashion the surface of primitives in a way that permits new and interesting self-patternings to emerge. The “application” itself, the particular stipulation of this or that arrangement of things, feels like last century’s design concern. The more interesting opportunity, and maybe even the more profitable opportunity, is to pursue the creation of certain primitive shapes that can be arranged into a plurality of pleasing and useful patterns. The contemporary primitive is also a kind of commodity, in fact it is even defined as such by Werner Vogels and Jeff Bezos as a founding philosophy of Amazon Web Services (AWS). 29 The primitives of the moment are hooks, harnesses, subagents, Model Context Protocol (MCP) servers, autonomous agents, etc.

It is worth separating the primitive from the protocol in our eclectic little bag of definitions. A primitive is a unit of software that is both composable (sticky, adhesive, connectable, collectable) and not readily decomposable (while it can be reduced into smaller-scale operations, doing so requires shifting registers or interpolating into another design space—take, for example, an iterative for() loop in C, which can be enumerated into a series of discrete machine-specific actions in Assembly, but doing so requires the introduction of an entirely new lexical system). Software that is assembled out of discrete, modular, reusable building blocks is called composable (as opposed to, for example, large, monolithic software objects). A primitive, in order to stick to other primitives, must take some input, perform some operation, and produce some output. The requirements and commitments of a given primitive’s input/output (I/O) regime form its shape. We call the parts of itself that a primitive exposes to the world in this capacity its contract, and we call the logical systems made implicit by those requirements and commitments protocols. We are principally interested in primitive design, and in the same way we wrote earlier that software design “retreats” into primitive design, so too does protocol design “retreat” into primitive design (the latter exercise creates the space of possibility for the former to self-assemble).

Notice what a contract does: it decorrelates the primitive’s interior information from its consumers’ utility. A consumer could be handed a primitive’s full internals and find nothing there to compose against that the contract does not already provide; anything it did find would be a dependency on internals that the primitive must remain free to revise. That is darkness in the exact sense defined earlier, now just at the smallest available scale. We are “manufacturing” darkness at the lowest level of the dark stack, so that the factory can later accrete it globally.

Let us work this definition space a bit: take an agent harness, like a Claude Code session with a set of skills and tools. The harness is a primitive: It defines a structured input (system prompt, tools, context) and a structured output (tool calls, text, artifacts), which aggregate into a contract (the explicit specification of what a component expects to receive, what it promises to return, and what conditions it guarantees—e.g., a tool schema). Each subcomponent of this harness can further be treated as an individual, formal primitive—stored, created, and managed independently of the aggregate.

We can also read the class of factory we are in based on who is reading a given contract and when. In a Class 1 factory, the contract is read at design time by whoever writes the plan—an engineer consults a POSIX man page or an OpenAPI spec, hard-codes the call, and then the thing proceeds along a frozen path. In a Class 2 factory, the contract is read at plan time by the factory itself—Claude Code reads an MCP tool schema mid-trajectory and decides whether the tool is worth calling. A Class 3 factory pushes the reading of the contract to the last instance: since no planner reads anything in advance, the contract has to carry enough self-description that a primitive can be picked up against a constraint that did not exist when the contract was written.

Scaling this up to agent-to-agent (A2A) interaction means introducing the concept of the agent itself as a relatively fungible primitive. An agent exposes a capability through a structured interface (an agent card in A2A, 30 a Hand manifest in OpenFang 31), and an orchestrator assembles agents into workflows dynamically. The orchestrator does not need to understand the agent’s internals; it only needs to understand the contract. Google’s A2A protocol 32 and Anthropic’s MCP are convergent designs despite different starting points: both are attempts to define the thinnest possible joint between autonomous components. This joint gets even thinner when looking at the first wave of autonomous software factory-as-a-service offerings, like Amazon’s Kiro and Factory’s Droid products. In both, the issuance of a local objective (called “specs” in Kiro-speak and “specification mode” in Factory-speak 33) is primitivized into a structured compositional unit. Agents then receive, unfold, and self-organize to accomplish the objectives through operations defined within that primitive, and in so doing unroll their own internal primitive cascade (graphs, harnesses, models, etc.).

Since our goal is to bring something unanticipated into existence, the dark stack’s primitive design prioritizes composability above everything and in so doing does not presuppose any given assembly. Our goal is to defer the problem of the assemblage to the last instance, allowing all that stuff to form after we have put our pencils down (a given tool, system prompt, skill, and memory management system are constructed independently and combined only at the moment of assembly). If the factory can organize itself only in ways that were anticipated at design time, then the result is not a superdark factory but just basic automation. Or, to put it another way, an assembly anticipated at design time is one whose derivation terminates, at every step, in inputs the architect provisioned—it lives inside the readable space c. Composability is a tool here that we are using to let the factory’s configuration space drift beyond c at all. A hard-coded pipeline of agents is literally just a waterfall: It is capped by design at the factory its architect already knew how to want.

A factory can determine its own assembly only if the primitives it works with are genuinely composable (if an agent can be swapped, rerouted, parallelized, or removed without cascading failures across everything it was connected to). This is where most current architectures ultimately fall short. While OpenClaw and similar spawning systems let agents instantiate other agents, giving the appearance of self-organization, that organization is still largely hierarchical. The underlying spawning logic is still largely imperative—agent A decides to create agent B with specific instructions. Class 3 composability would mean that a spontaneously forming orchestration layer can discover available primitives, assess their contracts, and assemble them against an input without any single system or agent needing to hold the full topology in its context. The frontier of automatic harness optimization (e.g., DSPy, Genetic-Pareto [GEPA] prompting, SkillOpt—see Khattab et al. 2023; Agrawal et al. 2025; Yang et al. 2026) represents a serious next step in this direction.

How do you build something, then, that is genuinely composable? We could also ask the opposite question: What does “poor” composability look like? What kind of primitive design would prevent higher-complexity systems from emerging spontaneously? We want to avoid any situation where a set of primitives can technically communicate with each other but implicitly depend on each other’s reasoning patterns, identities, or output-formatting in ways that are not captured by the contract. This kind of implicit dependency is called semantic coupling. A well-designed primitive makes its contract explicit enough that semantic coupling is minimized. The more standardized and self-describing the primitive is, the better, but the question is whether one can make the contract space rich enough to enable real autonomous assembly while keeping it thin enough to remain composable. How thick is too thick? You can easily limit the types of patterns available to a self-assembling system by overspecifying the primitive. How thin is too thin? You can easily underspecify the primitives you construct and wind up being overly permissive: nothing interesting or useful forms; there is just too much possibility. This question of thick or thin, rich or narrow, is one of the main design tensions in the dark stack.

This central tension is articulated as an open research question in “The Composable Data Management System Manifesto,” which comes to similar conclusions on primitive design for data management systems: rich components are good, thin contracts are good, and semantic coupling is by extension bad (Pedreira et al. 2023). And they struggle with exactly the problem above—if composability is good, which they agree it is, how do you build a robust enough intermediate representation (IR—“any structured representation of a program that carries enough information to allow it to be accurately executed, usually serving as the common interface between its components”) to be practical and useful within a large and dynamic industry? Their ultimate conclusion is a kind of unexpected convolution of the very thing they object to: If a thin contract cannot fully prevent semantic coupling, then what if every system adopted the same execution engine? 34 That is a pretty big requirement for a composable system—or, better, a pretty big primitive! So, how large can a single primitive be allowed to grow while still participating in composable design?

The sharpest reply to the manifesto comes from one of the systems it cites, DuckDB. 35 DuckDB is a great example of extreme outward composability (the composability of DuckDB inside third-party systems) and extreme inward noncomposability (an internal monolith with few dependencies—see Raasveldt and Mühleisen 2019), which necessarily results in a significantly larger-scale atomic actor. It should be said that DuckDB’s design principles make sense for its domain. SQL analytics is a mature problem class whose boundaries stopped moving, which is what allows this kind of convergence away from internal composability into a monolith, into a plan. A superdark factory, however, cannot be produced as a monolith, because to be a monolith is to be determined as a plan, period. And while the superdark factory may contain such larger structures at arbitrary points in its lifetime, the “largeness” of any of a superdark factory’s internal structures is an empirical question continuously discovered at runtime, assembled on demand based on external constraints.

To try and dig for some kind of compromise, we can go back into the history of data analytics, when the magma of data primitive design had not yet started to cool. If you send a structured query to a data platform (like asking DuckDB to pull you a particular record concerning a particular customer), a routine runs on the server side to determine the best way to find the information you might be looking for. This routine is called a query optimizer, and its history is pretty relevant to us here. A query such as “find me a series of records that meet these conditions,” usually structured within a common syntax such as SQL, arrives at a query optimizer without stipulating exactly how to find the result. The optimizer then has to formulate a plan, figuring out which database tables to read first, which to join into the results next, and whether to consult an index or iterate through every row. The optimizer is not just trying to produce a plan; it is trying to produce the best plan, and it determines what best means typically based on the amount of compute resources (or the amount of time) a given plan might take. This is a real case of a Class 2 decision among alternatives, because not only does the optimizer not know the best route in advance, but each successive decision the optimizer makes has serious impacts on the types of decisions available to it afterward. This space of alternatives should not be underestimated; the optimizer’s job is no small task! A query joining ten tables can be ordered in about seventeen billion ways. 36

One would expect a query optimizer to be a kind of prestructured expert, a master of its data domain, something that knows all of the tables it links together perfectly—after all, its work is highly specific to the material it serves to manage. In turn, one would expect this to be a prebuilt thing, a structured series of actions gathered together in a large, monolithic software application that can hold the entire question, and all of its implications, in its head. And this is exactly how the very first query optimizers were built, like the one inside IBM’s 1979 System R (Selinger et al. 1979). System R did the painstaking work of inferring the costs of certain actions in certain orders as scheduled by a linear, handwritten procedure. But a monolithic tool like System R meant that every new data type, access method, or join algorithm meant a brutal, manual intervention in a handwritten procedure that somebody then had to reverify in its entirety.

This was unsustainable for two reasons: First, in order to edit an optimizer like this, one really had to understand it—and it follows that the second reason that this was unsustainable is that it delivers the unbearable precondition that in order to even create a query optimizer, one had to hold the entire space of query optimization in one’s head. So what the field built instead was exactly the kind of composable structure we described earlier. Remember our neurotic governor who supplies nothing to the machine that they could not reconstitute outside of it? The cost g for System R was one human head per revision. That is a very expensive system!

Starting with 1987’s EXODUS (Graefe and DeWitt 1987), the process of query optimization changed into something highly primitivized and modular. In systems like these, engineers supply a pile of small rules, each one a single algebraic fact (like “these two tables can be joined in either order” or “you are allowed to run a filter before or after joining tables”), and a general search procedure assembles plans out of whatever those rules generate, filing every fragment it finds into a shared table of interchangeable alternatives before reading the cheapest complete plan back out at the end. There is no procedure anywhere that composes this plan; each rule is written alone and knows nothing of its neighbors.

This is the type of routine that has run inside Microsoft SQL Server since 1998, when the 7.0 release shipped an optimizer built on Goetz Graefe's Cascades framework (Graefe 1995), 37 and it plans queries for dozens of younger engines through shared libraries such as Apache Calcite (Begoli et al. 2018). Contemporary query optimizers ditch the plan altogether under what is now called adaptive query execution: for example, an engine like Spark (see Apache Spark, n.d.) halts partway through a running query, counts the rows it has actually produced, and replans for everything that remains, treating the plan as a guess to be checked against whatever the query itself turns up. The query optimizer is a Class 2 machine, a routine that receives an objective and internalizes the making of the plan, and its history is the recursive Class 2 self-assembly of a Class 2 factory.

If you want to build a plan, by all means build a plan. If your ambitions have concretized into a place where building a plan makes sense, if your field or your competitive arena has become a settled thing, perhaps the plan is yours to own. But as soon as the thing you build comes into contact with something unsettled, you will start to notice that the region of your software at that point of contact with the unknown will soften and crumble into modular primitives. And the Class 3 factory, of course, is surrounded by just such unknowns.

Incentive Design for Continuous Learning

So, what do we want to build? What kinds of structures do we hope will self-assemble, what kinds of structures might be useful to us as technical prerequisites for a Class 3 factory? And, in turn, what lower-level shapes (contracts) might best enable such conditions? We can think about primitive design as incentive design, wherein the types of contracts we define can be thought of as incentivizing certain kinds of behavior under certain conditions.

The recent history of mechanism design (Carroll 2015; see also Bergemann and Morris 2005) lays out a few principles for how we might think about building incentives into contracts, or building contracts in a way that incentivizes a certain result. The most significant contributors to this field include Dirk Bergemann and Stephen Morris (2005), who systematized “robust mechanism design” beginning in the mid-2000s, and Gabriel Carroll (2015), whose later “Robustness and Linear Contracts” delivered a design concept subsequently borne out in prompt and context engineering: the less a designer knows about a complex system (e.g., a model), the less specifically one should design its reward structure. A baroque incentive set filled with mazelike conditional clauses and carve-outs only works if its author knows the agent’s action space well enough to specify every clause and carve-out. If the author does not know that space, but they still insist on providing superprescriptive instructions, every one of those detailed instructions becomes an exploitable opportunity for their opponent. According to Carroll (2015), robust designs are those that rely on weak claims, not strongly held or strongly structured claims. Call this the principle of robust simplicity: the less the architect knows, the less structure they should impose. (This is already a widely accepted design convention in prompt/context engineering, not from the perspective of incentive design per se but from the perspective of instruction design—see Anthropic 2025.)

Mechanism design received a significant update in the late 2010s as it started to understand the object of design as a self-developing, learning algorithm. From the outside of the black box, it is reasonable to assume that the agents that will populate the factory’s interior—players in the cascade of games that the architect’s committed strategy sets in motion—are not necessarily rational best responders (agents who, at every move, choose the action that maximizes its expected payoff); instead, they are no-regret learners (agents who update their activities based only on their own observed outcomes, without modeling their environment, and whose long-run average performance does converge to the best strategy they could have chosen in hindsight). This is a reasonable assumption because rational best response demands something that nobody has, inside the factory or out: privileged access to the world. The work in Camara et al.’s (2020) “Mechanisms for a No-Regret Agent: Beyond the Common Prior”—alongside related results by Braverman et al. (2018) in “Selling to a No-Regret Buyer” and Deng et al.’s (2019) “Strategizing Against No-Regret Learners”—identifies that mechanisms designed against rational best responders can perform very poorly against no-regret learners. Mechanisms designed specifically for learning dynamics recognize that a given agent will pass through exploration phases, that they will make regret-minimizing decisions that look irrational from a Bayesian standpoint, that their behavior in any round is just a sample within a convergent strategy. It follows that the incentive set for the superdark factory should assume that the early superdark factory will be an absolute mess and not prematurely inhibit that mess from settling, coagulating, and converging in a strategic direction.

The interior of the superdark factory, it is safe to assume, consists of composable stacks of no-regret learning machines. After all, no-regret learning is the default behavior of almost every adaptive system, from multiarmed bandits to reinforcement learners to the gradient-descent-following guts of the modern machine learning paradigm. In one case, a learner might be an agent spontaneously deploying subagents within a single turn, or it could be a harness that steers a multiturn conversation, a workflow supervisor that creates dynamic agent graphs, a service orchestrator that sets forth larger constructs, various departments of the entire factory, and of course the entire factory itself. It also might be one or many spontaneously trained and erected models, be it generative models or more conventional machine learning models. In most cases, however, it is not the “model” itself that is the thing that is learning but actually deterministic software adjoined to nondeterministic methods. Regardless, it is now time to think about how to make this thing learn.

At this point, it is important to distinguish between the more naive no-regret learner and its more sophisticated subclass: the “no-swap-regret learner.” While the no-regret learner continually optimizes for the best general solution, the no-swap-regret learner optimizes for an array of conditions in which a given solution is best. The question a no-regret learner is always asking themselves is: “Over my lifetime, should I have picked a different solution overall?” (avoiding regret)—which is different from the no-swap-regret learner’s self-analysis: “For each solution I picked, should I have swapped that solution for any specific other solution?”

Here is a mundane example to flesh out the specific neuroticism of the no-swap-regret learner:

A commuter returning from work has three routes: the highway, the backroads, or a congested but direct bridge. The commuter wants to minimize their average commute time over the year. A no-regret commuter keeps a running average for each route and mostly takes the fastest route, while occasionally trying the others. At the end of the year, the commuter asks themselves: “Was my average commute at least as good as if I’d just taken the single best route every day?” If so, well, no regrets. The no-swap-regret commuter keeps a spreadsheet in the car and asks themselves the question: “Every time I took the highway, would the backroads have been better?” Or, “Every time I took the bridge, would the highway have been better?” In so doing, another strata of information can manifest—perhaps the bridge is significantly worse when it rains, which is an infrequent but real scenario, one which points decisively toward the highway as the better solution under those specific conditions. Making that swap under those conditions would contribute to an even better average commute time, and ignoring that swap would constitute swap regret.

At a high level, the no-swap-regret learner seems “better”—it has closed every conditional complication that might ultimately preempt it from the best possible outcome. But when an architect decides on a primitive, when they commit some structure into the factory and then turn the factory on, a no-swap-regret loop will simply converge to the best strategy implicit within that architect’s decision. That loop will just play out the architect’s game and then stop. From the perspective of the architect, the architect has reached what is described in game-theoretical terms as the Stackelberg value (V) (Deng et al. 2019), which is defined as the best payoff one can get by committing to a game-playing strategy and letting a perfectly rational agent respond to it. In ordinary settings, V is a worst-case scenario—Deng et al. (2019) show that an optimizer facing any no-regret learner can guarantee themselves at least (V − ε)T over T rounds, so commitment to a single strategy regardless of the opponent’s moves remains a reasonable benchmark against which other game-playing techniques can be assessed. 38 For the architect, however, whose first move persists as a committed policy, V changes from a floor (worst-case) into a ceiling (best-case): against no-swap-regret learners, there exists no sequence of play where the architect can extract more than V, because these learners converge on ideal counterplay to any commitment. 39 However, against the simple mean-based no-regret learner, our ordinary and naive chaser of running averages, the same theory identifies a second and larger quantity: U*. This variable is incredibly important to us: It is the value of an optimal-control problem over the learner’s cumulative-reward states, which is the exact maximum extractable from any mean-based learner and which in some games strictly, sometimes dramatically, exceeds V.

V and U* are two powerful new variables, and they might ring a little familiar. But before we get there, we need to extract what both V and U* mean in the context of the dark stack. First, recall that the installation of the dark stack is a one-time commitment. That works very well in the context of the Deng et al. paper: There is no second move required to harvest U*. Instead, you just need to encode an “extraction schedule,” a timetable of play that is fixed before the first round. A good first mover might bait the system for so many rounds, wait until the learner’s running averages tilt where they want them, then switch and collect the surplus they have extracted from those learners while their averages are still aligned with the first mover’s targets, before they adjust. The whole timetable can be written down before play begins because a mean-based learner is not reacting to you; it is reacting to its own memory of you, and its memory is a function of nothing but your own action history.

So why not write the whole extraction schedule at this stage, and then do the same thing? Why not try a bait and switch? Because what the Class 3 factory is missing here is the ability to actually write that schedule. The architect does not know the “payoff” of the game (by design)—and neither does the factory. That is the point. The world is ultimately responsible for setting prices to this game, disclosing payoffs only through the actual playing of the game. An architect without the ability to actually understand the way that actions are ultimately priced can commit nothing better than a stationary strategy, and a stationary commitment is worth V. Whatever lives in the gap between V and U* can be harvested only by proxy, through the mix of learners seeded at the first move—the only structures positioned to sample, in play, the world-priced payoffs the schedule would have needed. This is what it means for primitive design to be incentive design: The population you seed is the only instrument that can actually reach past V.

The gap between V and U* is the formal measure of everything an architect can obtain beyond the option space they installed into the superdark factory at its initialization. Again, we think of Thompson’s circuit—the result was not a better move within the space of moves the designers were evaluating (b ∩ c) but a move reachable through the internal dynamics of the system (b \ c). A factory built entirely out of no-swap-regret loops cannot find anything outside of what was already committed to (implicitly or explicitly) in its architectural armature c. A superdark factory needs at least some naive, mean-based no-regret learning somewhere. Otherwise it cannot reach outside of c and cannot really be dark.

We are tempted to try and squish these two vocabularies into a single theorem, but that is not a good idea. U* > V is a claim about the value of a repeated game; b* ∈ b ∖ c is a claim about membership in a solution space; these are two very different things. However, there is a relationship we can put these concepts in under one condition: the superdarkness of the superdark factory. If either the architect or the factory are able to wrestle some ownership over the game’s payoff matrix, if they are able to hack together a complete table of the game—with a row for every move the architect could commit, a column for every response the factory could make, and in each cell, the “payoff” value of that combination (in favor of one opponent versus another)—then we cannot build a relationship between these two vocabularies, because they are describing very different conditions. Now, that kind of possession of the rules is something one could rig together for a closed game, even a game as complicated as Go—any game with rules, for that matter. All the games in Deng et al. (2019) are exactly this kind of game. But in the case of a superdark factory, what would those cells even be? The architect’s list of moves consists of the structures they commit prior to turning on the factory. The factory’s list of moves consists of everything it might do. And the number in a given cell is what that combination turns out to be worth: whether the shipped thing finds a buyer, whether an AlphaFold protein fold holds in the lab, what satisfying a given norm is actually worth this quarter. Those numbers are facts about the world, and the world writes them into those cells at the moment that that combination is played, not before: The world is the player who fills in the table, one cell at a time, and one can learn a cell’s number only by actually living in it. The learners seeded during the Stackelberg move are just a kind of standing-in apparatus. This payoff table is priced by the world and disclosed only while the game is in play; nobody holds the matrix, the architect least of all. Under that condition, and under that condition only, we can adopt the following relationship:

This is a complicated relationship, so let us spend a little time with it. If you read the arrow here from right to left, it says “if the best configuration lies outside readable space, then the value extractable from the factory exceeds V.” If we are able to deliver on the above promises, then we can interpret V and U* not only in terms of b* and c but also in terms of what we defined earlier as P—the specific value we afford to the productivity of a Class 3 factory. V is the floor of P—a factory that converges, holds, and returns the full value of everything its architect knew how to want, a P of 0 purchased at great expense. U*—V is everything above that floor that only our no-regret learners can reach.

But we do not just get this relationship for free—to the contrary, this relationship is earned through the series of architectural commitments we make in the dark stack. First, a learner needs the ability to wander outside of readable space. We provide the means to facilitate such wandering by seeding the factory with means-based no-regret learners. Second, we need to incentivize them to stray out of this space, especially when they find something interesting. We solve this later through evaluation design (see “Evaluations,” below). But if they find something interesting out there, we need to find a way to retain the value of what they learned. We can do this by seeding both mean-based structures at the frontier (where the U* surplus is generated) and no-swap-regret structures at a kind of core (where the surplus is retained), which means that our primitive structures need to be able to encode both.

In order to seed any kind of learner at all, the contracts at the boundaries of the primitives we define have to afford three things: memory that persists across rounds, a defined set of actions the learner understands itself to be choosing among, and a reward that can be attributed back to the decision that earned it. The no-swap-regret class requires one more thing: a source of counterfactuals, some account of the roads-not-taken (this arrives below as the propensity score). Avrim Blum and Yishay Mansour (2007) have shown that a no-swap-regret learner never needs to be built directly. Instead, you can take N copies of an ordinary no-regret learner, one per action, let each copy own the rounds in which its action was played, ask the only question it knows how to ask—“Across my rounds, what should have been played instead?”—and then, each round, resolve the copies’ recommendations into a broader distribution over actions (Blum and Mansour 2007). This means that our primitive design can focus on enabling no-regret learners to form, while carefully seeding a (limited) method of counterfactual scoring to enable a subset of them to aggregate into no-swap-regret learners.

Information Design for Continuous Learning

So, now that we understand that we need to foster the development of certain kinds of learners, the question for us now is: How? This work has surprisingly little to do with learning algorithms and almost everything with rigging what information reaches whom: What kinds of decisions can a primitive make, based on what information, against what criteria? A pod of no-regret learners can assemble themselves into a no-swap-regret learner given enough information about their world, and while we want exactly that kind of learner at the core, we want a larger fleet at the frontier that never gets the material. Primitive design, in other words, is a kind of information control: the architect manages the onset population by deciding, in that early, soupy, undefined stage, what is public and what is private, what is rich and what is thin.

In “Don’t Build Multi-Agents,” Walden Yan (2025) of Cognition instructed agent builders to avoid multi-agent architectures by default on the grounds that “actions carry implicit decisions, and conflicting decisions carry bad results” (semantic coupling). For cases where multi-agent architectures were required, Yan suggested passing the full agent trace—maximum richness—and collapsing the system into a single-threaded linear agent wherever possible. But in a follow-up, Yan (2026) revised his stance with some deliberate exceptions: He discovered that his code-review loops perform best when the coding and reviewing agents share no context whatsoever (the clean-context reviewer, freed from the coder’s accumulated history, reasons backward from the diff and catches what the coder cannot). The suggestion, then, is to share full context among the agents that write code, but deny that context to any agent that might be judging code. Yan’s shared-context maximalism of 2025 thus matured into careful contract design. Yan even wrote that the open problems “are all communication problems”: how a weaker model knows when to escalate, how a child surfaces a discovery that should change its siblings’ work, how to “transfer context between agents without drowning the receiver” (Yan 2026).

Yan gives us two really important ways to think about information here. If the work undertaken by a kind of agent is genuinely single stream, then that agent should really be a single-stream entity: It should basically just have a single, accumulating body of context. And since, in agentic AI, there really is no difference between “sharing context” and “operating as one agent,” Yan is essentially suggesting that single-stream tasks should remain single-operator tasks. However, if the work undertaken by an agent requires evaluation, if someone needs to actually judge the execution of that task, then that judge should never have full context on the internal operations of the assembly and should instead simply look at it like a machine: input, output.

The agent’s contract (e.g., its structured output and its expected structured input) is what it shows to the world. Everything else (the things it is made up of) defines its cold interior. A contract that exposes every internal detail of the primitive provides too high a resolution for composition, since it lets the consumer form dependencies on signals to which the primitive did not intend to commit. In contrast, a contract that exposes too little gives its consumer no signal to compose against.

Let us move one step higher into the protocol. We can derive the definition of a protocol in two ways. From the bottom up: If you set a population of contracts loose on each other, the possibility field of their interactions is the protocol. In this sense, a protocol is kind of like an available tiling of some set of tessellating primitives—you can arrange these primitives, with these structured inputs and outputs, in this or that way. From the top down: If you write the protocol first, as a communication system, it dictates which inputs and outputs every primitive beneath it must expose. Regardless of how you define it, the protocol’s power is that it ultimately constrains the form of all future exchange before any exchange happens.

The field of information design (e.g., Bergemann and Morris 2019) outlines some general principles for the intentional construction of game spaces based on the protocols that pattern communication within those spaces. The power of the Stackelberg move we continue to refer to is affirmed in Emir Kamenica and Matthew Gentzkow’s (2011) “Bayesian Persuasion,” wherein they argue that a sender who commits in advance to a disclosure rule (choosing what is revealed before knowing what there is to reveal) can, without ever lying, push a rational receiver’s posterior beliefs across decision thresholds in the sender’s favor. This result survives translation into learning dynamics—a principal facing a no-regret learner still secures approximately the commitment value V, and against mean-based learners it can exceed it (toward U*—see Lin and Chen 2024).

In this sense, the Stackelberg position of an architect requires them to think very carefully about the impact of any standing informational commitment. Good information design, then, is design that encourages the productive capacity of the factory in a way that survives the absence of any architect. Once the factory is turned on, the contract space of a given agent becomes its entire sensory apparatus.

The dark stack’s primitive design focuses on establishing two major communication channels—a request channel and a reward channel—and two complementary approaches to protocol and information design. Traditional protocol design tends to favor the “narrow waist” (e.g., IP)—a thin, narrow, composable joint that is insensitive to growing complexity above it (e.g., TCP, HTTP, applications) and below it (e.g., Ethernet, Wi-Fi). But in the context of a continuous learning system, “thinness” has real consequences. A request from a transmitter should be “rich” in the sense of semantic content and self-description. As far as request-making is concerned, thinness is deleterious to learning. A thin request can force its consumer to fill its gaps, either inventing new information or relying on some unarticulated prior. A regular at a restaurant who asks for “the usual” often gets what they want, but this request is now implicit instead of explicit, and it relies on the receiving waitstaff’s ability to remember that order. Worse, a new member of the waitstaff might have to duck behind the bar to inquire with a colleague or make a best guess—it is very difficult to learn from dynamics where the cause of an effect is unstated. This reliance on implicit context is our old enemy semantic coupling, now at the level of the protocol. The request protocol must therefore be rich enough to describe itself such that its consumer has enough material to compose against the transmitted description and is never expected to infer based on contextual metadata (e.g., the author who submitted their request—in fact, the author here should be either irrelevant, or fungible, or private). Any agent who receives a request should respond in kind by producing another request that is semantically rich and neutral with respect to its author. Tool schemas (MCP) and agent cards (A2A) are already great examples of complete, semantically rich, but also neutral self-description.

The other channel—the reward channel—can be relatively thin. After all, the feedback returned to a transmitting agent is not consumed once (e.g., as a kind of control flow) but is instead accumulated across rounds by whatever is learning to choose. In this respect, richness is undesirable, since a learner given too much context will form dependencies on arbitrary signals (semantic coupling in reverse). In this sense, the reward should always be a score. But if the only reward that is received is a score (an error code, a floating number, a Boolean), then the protocol will result in the exclusive formation of ordinary no-regret learners, since there is no surplus of statistical propensity provided through which the learner can ultimately construct counterfactuals (roads not taken). Instead, the reward should also contain what is referred to here as a propensity score, 40 which is an agent’s own accounting of the statistical field it drew from when making its decision, which it discloses at decision time alongside the result it produces. This propensity score is calculated off the distribution of possibilities available to the agent at decision time (e.g., the weight a softmax placed on a given option, the model’s own probability assessment of the tokens it emitted, or the explicit probability assigned by an exploration rule such as epsilon-greedy over K actions, e.g., (1−ε)+ε/K). 41 By producing propensity as a public part of a request, another agent can then retroactively associate that propensity score with reward at evaluation/scoring time (see “Evaluations,” below) and then return it to the learner.

A score that trains a learner never arrives in the same instant as the action it scores: Reward is always delayed, sometimes by many rounds, and it must find its way back to the exact decision (and the exact propensity) that produced it. The return channel must therefore keep actions addressable over time, which is to say it must assign each sampled action a persistent handle that a later score can be attached to. This adds another level of necessary fattiness to the reward protocol, in the form of statefulness or a managed queue of outstanding decisions awaiting their reward.

Now we can return to the question of public versus private—not an “absolute privateness” at a factory-wide level but the relative availability of information between agents, the thing that induces one kind of learner rather than another. The architect seeds this in two ways: first as structure (what an A2A contract is shaped to expose) and second as content (what is actually disclosed at initialization).

In an environment where every A2A transaction is absolutely transparent (where every agent reveals the entire surface of their decision-making process, including the entire decisional set available to it), the opportunity for the superdark factory’s self-organization is again limited from U* to V—this is the weakest possible informatic setting for protocol and primitive design in terms of delimiting the space of possible learning. 42 Further, from the perspective of an architect seeking to build something that has a capacity strictly in excess of their design, it is useful to not only keep information private (contained within a deciding agent) but also actively spread dissensus (see Arieli and Babichenko 2019) across the agent population in a productive way (prebaking diversity or heterogeneity across agentic belief systems in a way that can keep the population from collapsing into a single motivation).

The design principle, then, follows a rule of minimal sufficient disclosure (see Kamenica and Gentzkow 2011). Overdisclosure hands a given agent signals that it will either overfit to or game (see Blackwell 1953), and underdisclosure starves an agent of what it needs to coordinate (potentially introducing semantic coupling as a result). A piece of disclosed information does its job only if, given everything an agent now understands at decision time, acting on that information is on average in that agent’s own interest—otherwise, an autonomous agent may simply ignore it (in Bayesian persuasion, this is called the “obedience constraint”). The “on average” piece is important, as information disclosure can pay off across rounds, not necessarily in every single round.

From there, the strategic field should look like the following: (1) The informational schematics of the factory should be absolutely public (the architecture of the primitive, the expected I/O format of an A2A contract, the structures of requests and rewards, etc.), but (2) the local state of a given agent (its running scores, history, distribution matrix at decision time) should be absolutely private (universal read access is the transparency collapse just described—everyone converges, and the factory is worth V), with the sole exception of (3) the contents of the propensity score (as directed forward within a request and stored within the reward queue), which pack just enough information into the request for its recipient to be able to reconstruct the potential value of an alternative choice.

A lot of work has been done here to describe a primitive architecture with a few core principles: (1) composability and a kind of formal neutrality, (2) a composable approach to assembling no-swap-regret learners from ordinary no-regret learners, (3) a protocol layer with a rich request line and a thin but stateful reward line, and (4) minimal sufficient disclosure when it comes to the publicness or privateness of information. But again—all of this work lives only at the Stackelberg first move of an architect. Once switched on, the superdark factory immediately begins refactoring itself and—meaningfully—it is to be understood (even encouraged) that the first stages of that refactoring are largely unproductive. The next section begins to address the question of what things such as “commitment” or “persistence” mean inside an unobservable container that is always changing.

II. Versioning

Like an Isekai protagonist “waking up” in a strange world, an LLM enters a Code-Space carrying its weights and post-training behaviours; the accumulated character of everything it learned before this moment. But the Code-Space then constitutes it further. The system prompt shapes what it takes its purpose to be. The available tools determine what actions exist. The context window sets the horizon of what can be present and known. We may think we are giving it instructions from outside the world, but for the model those instructions are the world. Change the system prompt or the tools, and it is, in a meaningful sense, a different actor inside a different world.
—Jay Springett, “Hard Worlds for Little Guys”

At the very moment the superdark factory is initialized, it loses its statefulness. It is many superimposed simultaneous configurations (or many simultaneous configurations of many simultaneous configurations), each constantly in transit between locally assigned configuration states. If its software changes are tracked as commits, the result is a continuous stream of machine-time-level pushes into and out of production, at such speeds that every push is forced, every push involves some level of branch conflict, 43 and every push is feature-flagged and ephemeral.

Version control in the dark stack cannot be a linear record of sequential time points; it cannot be Git. 44 Traditional version control works because discrete code is inert, theoretically reproducible, and theoretically rewindable to some prior state. Git works perfectly in a Class 1 factory, but it really starts to struggle in a Class 2 factory (while it contains temporalized information, that information needs to be situated alongside real-time logging of both the system and its inputs in order to reconstitute the factory), and it starts to become unusable after r crosses an unknown threshold (recall the “control tower” example earlier). But the superdark factory consists of continuously learning probabilistic software, not discrete software, and its codebase is both executed and continuously cultivated by a population of agents whose behavior is irreducible (or, perhaps more accurately, impractically reducible) to smaller-scale phenomena. While the configuration state (prompt hashes, model identifiers/checkpoints, topologies) of a superdark factory can absolutely be logged as (expensive and likely very ephemeral, webhook-style) forensic metadata, that state cannot be understood as constitutive of a factory’s version. At best, the configuration space of the superdark factory is an extremely high-dimensional probabilistic distribution that updates every few milliseconds. In reality, “yesterday’s” superdark factory no longer exists, and even if one were able to reconstruct every observable parameter, running it again would produce something different. As a result, the superdark factory does not move backwards; it only moves forward.

Instead of versioning a superdark factory by what it is (which is unknowable, or practically unknowable), the proper approach is to version it by what it does (à la Beer’s POSIWID) 45—and specifically what it does now. Any dynamic system settles into variably stable behavioral attractors, but those attractors are observable only as metastable relationships between input regimes (norms and constraints) and output regimes (here understood not only as the work product of the factory itself but also as the assessment of that product against metrics defined in the input regime), where the factory’s behavior remains statistically consistent despite underlying turbulence.

Two configurations that are syntactically completely different but produce statistically indistinguishable input–output distributions can be understood as belonging to the same version. If the input–output distribution changes, then it could be said that the version has changed. If a revision to the input at the level of the charter happens, the version has changed. If an external force changes the terms through which an input can be satisfied by a present distribution of the factory’s output, the version has changed. Version changes are diffuse but measurable, because they describe periods of behavioral consistency that can change at any time for any reason, and there is no expectation of a linear correlation between any successive versions (i.e., versions do not “increment” in the factory). It can be expected that a more complex input–output regime (e.g., a superdark factory that produces many different types of products) might involve either more complex or more inconsistent versioning, but that is an unsettled question.

Versioning the superdark factory is an active process that involves the antagonistic continuous evaluation (see “Evaluations,” below) of the factory’s input–output dynamics. Version changes are detected by observers (online LLM-as-judge evaluators, humans, discrete or probabilistic methods) working together to measure the factory’s input–output dynamics, though it should be understood that these observers are technically not external to the factory but very much part of the spongy, gooey walls that delimit the factory from its world (walls that, in principle, recede asymptotically forever). The versioning of the test suite itself could follow the same rules as the versioning of the broader factory, as any differentiable behavioral (but not syntactic) metastable dynamic within the test suite can be said to constitute a new version of the superdark factory.

While this approach to versioning might sound too abstract for practical use, it can be realized vividly with the mathematical principle of the transfer operator. We can borrow transfer operators from dynamical systems theory, where they are used to predict what the next step of a given probability distribution might look like. We can use them here to assess longer-term invariances in factory input–output distributions.

First, partition the factory’s observable behavior into cells (profiles of judge scores across the charter’s metrics, say), keep a running histogram of the effects of recent outputs on those cells, and then count how often behavior in one cell leads to behavior in another. The table of those counted tendencies is the operator. Any moment’s histogram differs from the factory’s long-run average as a map of surpluses and deficits across those cells, and the operator’s eigenstates are the special maps that fade without changing shape: patterns of more or less reliable behavior. The operator’s spectral gap is the distance between those few slow-fading patterns and everything beneath them, giving us a readout of whether versioning is happening at all. If near-invariant regions are cleanly partitioned and well defined (a wide spectral gap), then it can be said that a given version is durable and well defined. A small spectral gap suggests that versions are bleeding into each other, or that a given version is provisional or transitional.

We will explore how to use this transfer operator to not only understand the input–output regime of the superdark factory but also identify problems that might lead to its collapse.

Pathology Detection

Version robustness is neither good nor bad from a steering perspective; it just describes the behavioral turbulence of the factory as it continues to develop. However, this is not to say that the concept of a regression does not exist for the superdark factory—it absolutely does—though it should be mentioned that the temporality encoded in the term regression (regress to what? to when?) feels odd or incorrect. A superdark factory cannot be rewound by restoring some prior configuration, so the concept of a regression here needs to be redescribed as a pathology of convergence.

The first pathological condition is stable failure—this is the thing that most closely resembles a classical regression. Stable failure involves a near-invariant region (a robust version with a wide spectral gap) whose input–output distribution is failing against its input. The critical problem here is not the failure itself (superdark factories are expected to fail) but the stability of the failure condition, meaning that the factory’s dynamics have both settled into some undesirable attractor and are resisting leaving. It is worth noting that this can be caused by the factory just drifting into a bad region on its own or by its input changing in a way that the factory is not sufficiently incentivized to satisfy.

The second pathological condition is overfitting, which we discussed earlier. In this context, overfitting refers to a scenario where the factory is being opportunistic about the tension between the norms/constraints of the input’s intent and their metricization as verifiable quantities, optimizing for the latter instead of the former. Avoiding overfitting requires a particular approach to evaluation design and steering (see next section).

The third pathological condition of the superdark factory is learning death. While overfitting is a property of the factory’s solution (locking on to metric instead of norm and continuously improving against its misinterpreted goal), learning death is a property of the factory’s population dynamics. In the case of learning death, the factory has parked at Stackelberg V, and the surplus-generating frontier (e.g., mean-based no-regret learners haphazardly knocking the factory into new regimes of autonomous thought) has either died out (is no longer being invoked) or been quarantined (denied access to the surplus-retaining core of no-swap-regret learners). Unlike overfitting—which can be hard to detect with transfer operators, because overfitting systems can still progressively create new versions—learning death can leave a clear signal behind, e.g., a single state with no variety.

The final pathological condition is permanent unsettledness or thrash, a situation where a factory never settles on any state at all. In these circumstances, the factory’s surplus-generating frontier performs well at discovering new possibilities for factory organization but the surplus-retaining core abandons every possibility. This carries the entire cost of exploration but ultimately ships and delivers nothing predictably useful. A thrashing factory is easy to detect, since its spectral gap is continuously unsettled.

We will continue to work through these four convergence pathologies in a variety of contexts. Our goal is to seed the superdark factory with a kind of incentive-based immune system that corrects each of these pathologies live, in runtime.

Versioning: Commitment and the First Commit

Up until this point, everything about the superdark factory’s state is understood to be unknowable, or practically unknowable, as a product of constant multidimensional transformation. However, as established above, that state is still trackable from the outside of the factory. At this point, however, it becomes important to understand an architect’s first move not only in the context of a commitment to a strategy (and a commitment to not changing that strategy) but also as a commitment that carries its persistence over into the future, even if that persistence is unobservable.

In “Atoms, Institutions, Blockchains,” Josh Stark (2022) defines the hardness of a system as its capacity to make something very likely to be true in the future. In this context, a cast is a specific claim about the future that the thrower of the cast is trying to keep fixed (a cast is something that is both thrown into the future and something that hardens into shape). A hard cast, then, is one whose source of enforcement makes it very costly to break, so it reliably stays true over time. Conversely, a soft cast is something that is merely asserted and can quietly give way.

Our coauthor Jay Springett (2026) argues that these hard casts should be promoted from formal statements about behavior to the “physics” of the “world” in which the superdark factory is wholly contained. The Stackelberg move of an architect should be understood as a thin, constitutional kernel, 46 consisting of a small set of hard casts whose violation is irreversible and a larger set of soft casts that are revisable. The kernel itself is a minimal, privileged core that enforces the superdark factory’s fundamental physics—memory, access, capabilities. The kernel is where the dark stack is reified, where it locks itself in. The kernel is inviolable from both the inside of the factory (immutable) and its outside. 47 If an architect has the ability, at any time, to gracefully revise, branch, or roll back the kernel, then the kernel immediately becomes a site of negotiation for the rational players inside the factory. 48 While an architect certainly can reserve permissions for themselves through which to change the kernel, that change has to be understood as lethal—the old factory dies and a new factory begins from a new v0. 49 The world has changed. In this sense, the kernel becomes “constitutional,” not just in terms of “constitutive” but actually in terms of serving our architect’s need to supply the factory with an identity.

A well-designed “hard world” (in Springett’s terminology) is one in which irreversible harm is impossible—but not all hard worlds are perfectly designed, so a brutal kill switch that nukes the identity of the factory through the reconstruction of its kernel should always be left on the table. However, there are other, softer leashes that can be encoded during the architect’s Stackelberg move to avoid the types of pathological convergence described in the previous section. These soft casts are imposed in the form of punishment, specifically in the form of priced penalties for measurable damage to the factory’s input–output regime, which can at this stage be understood through the lens of the transfer-operator function. In this sense, these penalties are exercised through the scaling down of scoring returned from evaluators (see “Evaluations,” below) through the reward protocol (see “The Primitive,” above). We suggest that this is best accomplished through a Lagrangian transformation, subtracting λ·(cost) from the reward, 50 where λ is the weight of the penalty. A suggested method for the determination of λ might be through a proportional–integral–derivative (PID) controller, where Kp (proportional) raises the penalty in proportion to the amount a rule is being violated, where Ki (integral) stores sustained violation over time so persistent violations increment in price, and where Kd (derivative) reacts to how fast the violation of a rule is changing in order to dampen price escalation before it overshoots (Stooke et al. 2020). We will spend a lot of time with this controller in the next few sections.

We can use a priced penalty strategy with continuously self-adjusting penalty weights to overcome some of the convergence pathologies defined above. In the case of stable failure, one should price the duration of failure, ratcheting up penalties the longer the factory spends in a wide-spectral-gap attractor that is failing its input–output target. Conversely, in the case of thrash, one should penalize the duration of spectral-gap volatility, incentivizing the surplus-retaining core of no-swap-regret learners to stabilize for reasonably good outcomes. The prevention of learning death should be delivered as a fact about the world: Some share of compute and write access is usable only in the context of unhistoried actions (decisions that arrive carrying no propensity record and no reward trail), which establishes and preserves a niche for no-regret learners.

The prevention of overfitting, however, feels like the most difficult problem to engineer against inside the Stackelberg move. Overfitting is the gap between an intent and its metricization, but the only thing any engineer can institute a price regime against is yet another metric, which cascades out into another regime of possible overfitting. The only solution to this issue is in the application of the dark stack to the factory’s immune system and sensory organ—its evaluations.

III. Evaluations

Be empty, still, and idle, and from your place of darkness observe the defects of others.
See but do not appear to see; listen but do not seem to listen; know but do not let it be known that you know.
—Han Feizi, “The Way of the Ruler”

In classical software, unit tests function like stage gates. Binary filters (go/no-go, true/false) are positioned at critical stages of the software development life cycle (SDLC) in order to validate whether a given branch is ready to advance to the next phase. The structure of a unit test involves a few core assumptions about software that no longer hold true, or at the very least no longer hold true at any level of scale. The first assumption of the unit test is the unit: the presumption that software can be cut into fragments whose behavior in isolation predicts its behavior in composition. This assumption is held doubly across the time domain, holding firm to the idea that this proposed unit of software is isolable from time—a unit of software executed now, under the same conditions, will return the same results later. In so doing, a unit test is an assertion about rightness or correctness as a property knowable before execution, not as a property discovered through execution at scale.

The SDLC waterfall—the continuous downstream motion of software through requirements and design, implementation, verification, and maintenance—is built on the same core assumptions: the idea that a preproduction environment is a reasonable simulation of a production environment and therefore that change always and only flows downstream, the idea that software is testable in advance, and the idea that requirements can be fully specified in advance instead of iteratively discovered. As is obvious at this point, the SDLC waterfall describes an entirely different ontology of software than the one we are enumerating here.

The first wave of Class 2 software factories—especially in incumbent enterprise software—inherit their design directly from the morphology of the SDLC waterfall (see Microsoft 2026). There is still a design process (now applied to agent harnesses, graphs, and services), a defined preproduction environment, a predefined array of statements about correctness (judge scores and thresholds) built into the transition between preproduction and production, a productionization or deployment moment wherein discrete code is delivered simultaneously with new agentic features, and an observability strategy based on tracing and logging LLM calls. The unit test process that gates various stages inside of preproduction and which ultimately gates the transition into production is based on a combination of discrete tests and offline LLM-as-judge evaluations.

Offline evaluations involve the same underlying predefinition of rightness or correctness by an engineer, now codified in a question-and-answer dataset. These offline evaluations run passively as background jobs, sending a series of dataset questions to an agent (typically as a user prompt) and then enlisting another agent to gauge the conformity between the first agent’s answer and the expected answer provided in the dataset. Additional layers of screening (e.g., looking for edge cases, testing against malicious prompt injections, screening for racial bias) are often applied at this level by another cohort of agentic judges. Like in traditional continuous integration (CI), this first wave of software factories automates the batch execution of offline evaluations upon a given pull request. These factories have also begun to automate the ingestion of live production traces back into those offline evaluations (so-called golden datasets), which can either be accessed upon the next unit test of the next pull request or might run continuously as part of a simulated testing environment, prompting engineers to respond if simulations indicate a regression based on edge cases or external changes (e.g., a model checkpoint change).

But the idea of the preproduction environment itself is dying. In fact, it dies directly with r. As Charity Majors (2026) puts it, “having nondeterministic code in production is finally forcing us to do the things we should have done all along. . . . Tests and evaluations in production. Production is not what happens after development is over, production is a stage of development.” 51 Superdark factory design prioritizes the online evaluation over the offline evaluation. An evaluation that runs in a CI/continuous deployment (CD) pipeline before deployment is again just an isomorphism of legacy preproduction unit testing. An online evaluation runs continuously in production, because the system it evaluates is itself continuous. The factory’s behavior at deployment time is not its behavior at time t + n, even if no configuration has changed, because the factory’s behavior is a function of exogenous conditions (model provider updates, API latency distributions, user population drift, upstream data schema changes) that are not captured by a predeployment snapshot. Therefore the online evaluation is always active; it operates not only within a simulation of a production environment but as a first-party participant in that very production environment.

The tension between offline evaluation (unit tests based on a predefined understanding of the good) and online evaluation (live, supervisory subagents) can be observed at the contemporary protocol level. The open-source Prometheus LLM evaluation framework, for example, is often used for offline evaluations. Prometheus involves injecting not only a given scoring rubric but also a series of reference examples directly into a judge’s user prompt (alongside the response being judged), ensuring that the judge is anchoring its evaluation directly to those provided examples, as opposed to self-generated steps. Confident AI’s DeepEval DAGMetric is also used for offline evaluations, a process that involves the decomposition of the evaluative process into a directed graph of discrete yes/no decision nodes that route an output into a structured, deterministic verdict. It is worth noting that both Prometheus and DeepEval are frameworks that could be applied to production traffic or routed as scorers through a continuous feedback loop from production back into preproduction, using a platform like Braintrust—but they are offline evaluation tools in the sense that they are passive, post hoc, and ultimately just conditional, discrete gates with LLMs attached.

The first wave of online evaluators (evaluators actually inside the runtime loop) are called guardrails—typically meager semantic search looking to censor toxic words or tokens that look like personally identifiable information (PII, e.g., a Social Security number). Guardrails are generally blocking at runtime and therefore optimized for speed (e.g., Amazon Bedrock’s Guardrails). While active (blocking) and online (inside the application space), guardrails tend to be extremely conditional, with conditions programmatically inscribed prior to usage (e.g., NeMo guardrails). However, this approach to runtime control is already struggling at scale, because the guardrail again becomes a kind of basic unit test, written before production, applied to production, and therefore contingent on the isomorphism between the preproduction and production environments (which vanishes as r increases). A classic guardrail failure might be one that fails to identify non-English names as PII, or one that produces constant false positives in industries where sensitive material might be discussed (e.g., health care).

But unlike basic guardrails, the online evaluations in the superdark factory are learners. Already, research projects such as LATTICE start to move us into the world of self-developing guardrails: “integrating adversarial testing with automatic policy improvement, converting discovered jailbreaks into new guardrail rules without manual patching” (Broadhurst et al. 2026). LATTICE’s continuous improvement layer accepts changes only if a proposed update does not lower its score (F1)—but because each update is a targeted, conditional edit rather than an open-ended exploration, the loop can only ever converge in the direction of the correlated equilibrium of its current ruleset; it is incapable of taking score-lowering moves that would discover anything genuinely new. In this sense, LATTICE is a kind of no-swap-regret learner.

W. Ross Ashby’s (1956) law of requisite variety (the “first law of cybernetics”) states that a controller must have at least as much variety (in terms of inputs) as the system it controls. If the factory is understood to be a dynamic set of populations inclusive of both swap-based learning (retentive core) and mean-based learning (exploratory frontier), it is therefore imperative that the evaluation layers of the factory contain a significantly higher population of mean-based no-regret judges than the type of swap-based judges described in the LATTICE paper (Broadhurst et al. 2026), though both are important. This population balance is required for the superdark factory to handle catastrophe.

Catastrophes

René Thom’s work on catastrophe theory points out that continuous systems (systems whose parameters change smoothly) can nonetheless exhibit sudden, discontinuous shifts in behavior (Thom 1975, 18). Catastrophe theory provides a framework through which a control space (the parameters available to a controller of a system, e.g., speed, load, or temperature) maps onto a behavior space (the state that a given system settles into) and explains how that mapping can result in uneven and complex relationships. For example, what might look like a linear traversal across control space (more and more load applied to a beam) might not result in the linear traversal in behavior space and might instead produce large, discontinuous jumps across behavior space (the beam gradually bending until it spontaneously breaks). Thom refers to this catastrophic mode as a fold, in that a region in behavior space is bistable under control space conditions.

Another catastrophic condition is the cusp, wherein a system might have two control knobs, and the independent activity of those control knobs might result in different behaviors under identical conditions. This can result in hysteresis, arbitrary path dependency, and unexpected performance, but more critically it can result in more complex folds in controller-to-behavior mapping and therefore more prevalent and more significant fold catastrophes (abrupt jumps in behavior space).

The superdark factory is riddled with potential catastrophes, 52 both in the complex and nonlinear mappings between control and behavior at various positions within the factory and in the significant plurality of variously coordinated controllers simultaneously driving behavioral change. The evaluation function does two jobs at once: It connects the factory’s output to the factory’s input by managing the incentives that run between them, and it stands on guard for catastrophic conditions. Thom’s catastrophe theory, however, is full of caveats about its viability as a predictive tool. Instead, Thom was interested in providing a kind of morphological language through which catastrophes could be described. However, quantitative early warning signals (EWS) research can provide a framework that inherits from catastrophe theory that can help a dynamic system resolve when it may be near a catastrophe, while at the same time acknowledging the profound difficulty in diagnosing catastrophes in advance or predicting the timelines or dynamics through which they might occur.

If the first dark design principle for evaluations is that they must be online and continuous, and if the second is that the set of evaluations must contain the requisite variety (and surplus!) of mean-based and swap-based learners, the third dark design principle is that evaluations must be recursive—and massively so. Every individual evaluator can be, and will be, incorrect, just as it can overfit to its own reward structure or be gamed by productive agents hungry for the rewards that it administers. Again Stafford Beer enters the room, through the idea that any viable system is built from smaller viable systems and that each successive systemic order is a complete unit that senses, decides, produces, and rewards within its own scope. It follows that the superdark factory has evaluations of evaluations of evaluations of evaluations, stacking to some arbitrary level of recursivity as decided by the aggregate evaluatory apparatus. It should come as no surprise that the superdark factory likely involves more evaluators consuming more compute and performing more invocations than agents engaged in production. 53 The fourth dark design principle for evaluations follows from the first three: the signal that grades an evaluator must sit outside the loop that evaluator judges. Without this displacement, the evaluation layer optimizes toward the same target as the producers it supervises and the two classes settle into a shared overfit.

The recursive stack of evaluations is particularly important when it comes to building sensitivity for incoming catastrophes. A system calibrated to EWS requires multiscale attentiveness to a few important metrics: variance, autocorrelation, and ensemble disagreement. The first, variance, involves an evaluator’s assessment of the magnitude of fluctuations in an observed metric over a variety of timescales. The EWS literature suggests that an incoming catastrophe might be signaled by larger swings in variance as it struggles more and more to return to equilibrium. The second, autocorrelation, requires a similarly multiscalar assessment of how much a current moment might look like a previous moment over time, tracking how long any perturbations to a system metric such as latency distribution or error rate might persist before being dampened out. Both variance and autocorrelation are metrics through which to identify “critical slowing down”—a major warning sign of an impending collapse—as a system takes longer to recover from perturbations (e.g., how a beam on the verge of breaking might take longer and longer to straighten out after successive heavy loads). The third, ensemble disagreement, looks at the variance of determinations across a population of heterogeneous judges that are evaluating the same situation. Thom’s examples of the cusp and the fold are mirrored here as cases where identical inputs produce different ensemble consensus patterns. All of the above assessments cycle at every level of the dark stack, rippling across the communion of evaluations (more on this later).

Overfitting and Adversarial Populations

Productive agents, now established to be the minority of the superdark factory’s population, learn from evaluator agents through the coupling of scoring and propensity (see “The Primitive”). Evaluator agents learn through a similar coupling of scoring and propensity (for no-swap-regret learning), and they are graded from above, tier upon tier of successively higher abstraction, each tier grading the tier below on how compliant its scoring was with the factory’s input document. An evaluator agent is rewarded, in this sense, for verdicts that represent nominally correct applications of that input document, though unlike in the offline evaluation schemas described above, those verdicts are not strictly predefined at the level of the input.

However, this cannot be the only reward mechanism for the evaluations layer, because if that were the case, then evaluator agents and productive agents would be easily incentivized to collaborate in the direction of overfitting. A second reward signal is required, and it needs to come from outside the factory’s input and is ideally prebaked into the superdark factory at the Stackelberg move. This second metric scores realized consequence: a judgment of whether a given verdict predicted real downstream outcomes. For example, a judge that flagged some anomalous behavior should be duly incentivized only if that anomalous behavior preceded a true regression or caused a real failure in the world of the factory. The reach of this mechanism runs all the way back to the auxiliary supply: the same unprovisioned stream that enables AlphaFold’s success and the Thompson circuit algorithm’s overfitting. Only realized consequences scoring can sort success from failure in this respect. A realized-consequence score requires true regressions and real failures, which means that this reward mechanism is not only sparse but shrinking over time. To create enough of this kind of scoring to be useful, we need to actually farm and manufacture it, which we do by inducing a level of adversarial activity.

It should be stated first that the majority of evaluations are not adversarial, and second that the adversarial layer consists not only of evaluators but also of productive workers, both of which consist of healthy mixes of mean-based and swap-based learners. We should also note that the adversarial layer is both internal and external to the factory’s membranous walls, both acting like a chaos monkey inside the continuous software delivery arm and actively participating as a serious generator of production traffic.

So, what does adversarial mean? In one sense, it is a kind of red teaming—a practice where a software platform might commission “ethical hackers” (typically third parties) to simulate cyberattacks in order to test the defenses of a system. Using LLMs or agents for red-teaming activities is already an established practice (see Ganguli et al. 2022), whose modernization includes deploying agents to not only simulate malevolent traffic but further continuously learn and improve against their target (because “fixed attack strategies are likely to become ineffective”—Diao et al. 2024). In the superdark factory, this kind of evolving red-team-driven arms race is scoped beyond security, all the way through to intentional, active severity-0 bug construction at scale. And while an engineering manager in a highly regulated industry might balk at this idea, it is also true that any good, self-improving evaluation system requires exposure to seriously catastrophic contexts in order to enrich its realized-consequence scoring. Furthermore, those events cannot be staged in an artificial environment—they have to be real and they have to keep coming; the factory’s sense of consequence relies on real risk, and every real failure that it encounters makes its reality a little sharper.

This type of managed adversariality ultimately makes factory-wide lapses into overfitting very difficult, or—in an ideal case—tolerable or irrelevant. It becomes difficult for a factory to consistently overfit against its input because the general producer class and the general evaluator class are not permitted to collude, because subelements of each class are actively at war with each other, and because the realized-consequence metric sits outside the factory’s input entirely, fixed at the architect’s Stackelberg move.

At origin, writing an evaluation is the most direct act of governance available to the human at the boundary of the superdark factory—the specification of the superdark factory’s initializing norms and constraints is exactly the authoring of its first eval.

IV. The Charter and the Loop

// Generated from spec – do not edit.
—Tessl platform convention, where all code carries this header

After the Stackelberg move, the architect retreats from the factory site and shifts its work to input design. The input, defined earlier, is a collection of norms and constraints, articulated in the form of metrics, that enumerate the terms of the factory’s continued survival. Indeed, survival is exactly the stake of the game (from the perspective of the factory), because, as established in “Versioning,” an architectural intervention into the kernel is always at the cost of the factory’s death and its subsequent reanimation as a wholly separate world.

The input is delivered in the form of a charter, here cheekily derived from the SDLC waterfall’s “spec” and refurbished as an authored, editioned description of acceptable (norms) probability states (metrics), given conditions (constraints). Every change to the input is a new charter, and every new charter is a new version of the factory (though, as discussed earlier, it is not the only situation in which a factory’s versioning changes). It would be a mistake to think that a charter is always human-authored, and in fact the first charter is almost certainly coauthored with several agentic systems.

StrongDM’s Attractor is a great example of a Class 2 factory with a constitutional charter, where three markdown files with around seven thousand lines of detail situate the entire codebase and where the holdout validation scenarios enumerated therein are indistinguishable from a kind of dark versioning system. As superdark factories mature, specs become more and more probabilistic, containing multivariate possibilities of acceptable output distributions given input constraints, with holdouts, confidence intervals, and sample size requirements all subject to change.

A charter is a limited, privileged site of governance (intervention), a site of worlding (less-than-hard casts that delimit what kinds of worlds are possible), and also—perhaps unexpectedly—a site of negotiation. The charter is not read-only and is instead co-contributed to by the factory’s governance committee, which necessarily includes some type of representative delegation from inside the factory. A factory that is not able to participate in its own charter-writing is again not a superdark factory but instead an exercise in Class 1 or Class 2 automation. Writing a good charter is therefore the core engineering discipline of the superdark factory, and it is a genuinely new kind of engineering. It requires the capacity to express intent as a formal distribution, a statistical profile rich enough to constrain the factory’s output space while leaving the factory free to determine its own assembly. The charter must be precise enough to prevent overfitting but thin enough that the factory’s internal optimization is not artificially bounded. This is the same tension—richness versus thinness—that governs primitive design, because the charter is itself the outermost primitive of the dark stack.

When a governance committee, however arbitrarily composed of humans and agents, intervenes in the activities of the superdark factory, it intervenes at the level of the charter. Governance never intervenes in the internal affairs of the superdark factory—or rather, the interference of governance in the superdark factory is the factory’s death and subsequent rebirth. Instead, governance occurs in the rewriting of a charter and a subsequent permission to the factory for its reactive self-organization.

For all this morbid talk, a charter consists of soft casts (it is subject to continuous change), it is negotiated, and it is contributed to by the factory itself—how can a charter be both absolutely threatening and also so malleable? The answer lies in the architectural commitment to the hard cast of the kernel, which is ultimately a kind of live-or-die button whose threat is continuously invoked—“Here is a new charter; if it cannot be satisfied beyond what is priced as acceptable, then the factory needs to be scrapped.” At the same time, a charter needs to be reasonably priced, or, to put it another way, it needs to be reasonably permissive. Departures from the charter are acceptable within limits, and those limits are necessarily undefined or dynamically defined.

It is worth spending time detangling the pricing of a charter (e.g., via our PID controller) and the real cost established in terms of a factory’s constraints (e.g., tokens, USD, MWh). The pricing of a charter in terms of penalization or reward necessarily establishes a token economy in which a given norm is traded, leveraged, or hedged against in real-dollar terms. These pricing mechanisms are always proportionally established as a kind of market relationship, 54 and it should be obvious to any reader that the superdark factory can ultimately be understood as a kind of marketplace 55—albeit a Kantorovich marketplace (its prices emerge without any dependency on exchange, no buyers or sellers required), wherein a central source of value is kept intentionally nonfungible (the realized-consequence reward structure). The numéraire always resolves upward, of course, in the sense that a continuous, reciprocal flow of capital is an objective requirement of any factory, and therefore the correlation between the pricing λ of a factory’s norm set and the material costs established as constraints needs to be profoundly understood at charter time. The only kill switch greater than the teardown of a factory kernel is a token budget of $0.

Until now, the superdark factories described above have been riotously active, to the point of something between a complex multiscalar anarchic marketplace and a war. But is speed a prerequisite for a superdark factory? The answer to this question is no, but it is a complicated no. On one hand, speed is partially delimited by a charter, in that speed is categorically indistinguishable from a specific approach to cash burn and is therefore indistinguishable from a constraint. Apart from this negatively defined approach to speed, the factory keeps its own time, and a slow-but-steady factory can theoretically win in circumstances where it can both manage its spend conservatively and also accumulate and execute larger-scale counterfactuals (no-swap-regret learners obtain significantly higher-quality counterfactuals per round if the rounds are longer). At the same time, Ashby’s law can be extended to one of requisite velocity—a factory cannot move slower than its environment, and the factory’s multilayered control apparatus likewise cannot move slower than its factory. A factory moving slower than its environment cannot learn from its environment, since the environment drifts faster than the factory’s adaptive organs. Or, for a more concrete example, the type of security system endemic to a superdark factory is based in some sense on a constantly regenerated surface. As soon as the factory moves slower than the rate of a potential attacker’s full exfiltration cycle (from probe to discovery to attack to extract), the factory must price its security risk accordingly. In this sense, as in all things, the superdark factory should be tuned to its niche. It should be fast enough that neither its environment nor its attackers can get ahead of it but slow enough that its counterfactuals are worth keeping.

Self-Writing

What does it mean to write with a factory? It is, of course, a game—a different game than that initiated in the Stackelberg move but a related game, or a game that inherits from the space of possibility enacted by that prior game. The rules of this new game are contained in the spec’s write-access permissions, and those rules are hard casts. Those hard casts do more than just constrain the factory’s behavior: They are affirmative design decisions around agency, 56 or, in Springett’s terms, the charter is designing for the agency of the “little guys” inside. Any change to those rules should also result in the termination of the factory and its subsequent reconstruction, as those hard casts are part of the same initiatory hard cast of the Stackelberg move—in a sense, the read/write permissions of the factory’s input layer are part of the factory’s hard kernel, just inscribed on a different kind of surface.

The easiest ground within a charter to cede to the factory’s contributory arm is its metrics layer. Metrics can be thought of as acceptance criteria against norms and constraints, and agents are already quite good at contributing to their own acceptance tests (Ferreira et al. 2025). In the case of StrongDM, when an agent engaged in simulated performance testing identifies a failure mode, the case that produced that failure mode is written up as a scenario and added to the factory’s acceptance testing (Willison 2026). The superdark factory can do this work itself: if you hand it a norm, it will propose a metric to represent that norm. The factory’s own consequential reward structure can also be trusted, at least in part, to assess when those proxies are insufficiently descriptive of those norms—which it does by permitting its evaluatory layer to continuously add holdout test criteria to a given charter (based on adversarially induced conditions or just real production traffic). Every such contribution converts a world-finding into authored record; the charter takes something the factory encountered as auxiliary supply and ratifies it as provisioned supply.

If the factory is trusted to propose proxies for norms, then it is expected to contribute to the determination of λ—after all, it is proposing the means through which a norm is tokenized into the factory’s marketplace of priorities. It is worth further arguing, however, that the contribution of the factory to the determination of λ is not just incidental but actually desired: λ cannot be determined without collaboration with the factory. If λ could be accurately and consistently computed without the participation of the factory, then the factory would not need to be a superdark factory. But λ’s value is the marginal worth of that constraint at the factory’s current operating point; it is a kind of a shadow price, defined only where the factory stands and produced as a by-product of operating there. It reaches the committee as a speculative price posted by the factory.

It is at the level of norms and norm-setting that things get more complicated. A factory that constructs its norms wholesale moves us into the undefined Class 4, something with no dependencies on its world. Therefore, it is at the level of norm-encoding that the factory hits some kind of a wall, either a read-only wall (e.g., an “externally authored” normative constitution) or a governance structure that facilitates some level of power-sharing (e.g., a representative democracy). The pedantic but relevant objection to the idea of “read-only-ness” from second-order cybernetics would argue that any author of the factory’s normative constitution is indissociable from the factory as broadly conceived (more on this later), and therefore the question of norm-setting becomes explicitly a question of governance. One could again bring in the principle of requisite variety and suggest that a control mechanism (in this case charter-level governance) must contain factory-level variety, and the only producer of that level of variety available to the control mechanism is the factory itself. Or, one could take Ashby at his word alongside Roger C. Conant, stating that the only good regulator of a system is homomorphic to the system it attempts to regulate, and since the internal contents of the superdark factory avoid easy structural modeling, then the factory must be present inside its own governance in order to produce this homomorphism.

The superdark factory then joins its own governance committee, which leads to the question—how? How does a population with no stable membership send a delegation to a nominally “outside” body? The answer is sortition (the randomized selection of delegates), which needs to resemble to some degree the requisite variety of its stakeholders. On the cadence of charter revision, a sample of the factory’s population is seated (no-regret learners, no-swap-regret learners, productive agents, evaluators, antagonists), and that seat is consistently rotated, preserving a kind of mutual anonymity of the factory’s representative class, both to itself and to its fellow committee members (another tidy homomorphism of the larger-scale relationship). The closest real-world analogy to this kind of structure might be something like a decentralized autonomous organization (DAO), as a DAO is already governing a pseudonymous, dynamic, and partially automated membership class through a series of hard casts (in the DAO’s case, the smart-contract layer; in the superdark factory’s case, the kernel) and soft casts (the productive and social work of governance). An example of this could be Arbitrum (e.g., Blockful 2024)—or rather, a kind of inversion of the principles of Arbitrum—where instead of humans building a constitutional order for themselves and delegating their less-significant votes to AI agents, the array of less-significant voting topics becomes available to the factory’s representative set. (Already, a sour and wretched history of voting-worthiness by class becomes an inflection point the human will invariably need to address.)

Unlike a DAO, however, factory participation in its own self-governance is not consensual (in either direction), it is not persistent, and, importantly, it is not delivered through the medium of token-weighted vote. The AgentCity papers (Ruan 2026; Ruan and Zhang 2026a, 2026b) do a great job of thinking through this arrangement, with a Separation of Power (SoP) model that assigns legislation to agents (who “collectively propose, deliberate, vote on, and codify the Task level Policy of the agent economy as smart contracts”) and adjudication to humans (who are “accountable” in a kind of weird [RACI] way, where “every agent traces to a human principal through a complete ownership chain. Sanctions and rewards flow to the responsible human, connecting the agent economy to human society’s existing legal and social systems”—Ruan and Zhang 2026a). In the AgentCity framework, this interaction is intermediated through deterministic software, through which “humans can verify what the agent society is doing without interpreting opaque agent reasoning,” because “agent reasoning is opaque, but the law they produce—smart-contract code on a public blockchain—is transparent.”

A complementary example might be Robin Hanson’s (2013) definition of futarchy: “vote on values, bet on beliefs.” In this sense, a governance board consisting of humans and/or extrafactory agents may vote on a more abstract and primary array of norms (“prior art” in which the notion of the good is established), where lower and more dependent norms, their proxies as metrics, and their determinants as constraints are given over in part to a factory delegation whose informational advantage is decisive. This top congressional house is “read-only” from the perspective of the factory, though the factory is expected to testify within the assembly. While the AgentCity example subjects both values and beliefs to voting, a futarchic model opens up conditional prediction markets wherein it is ultimately the pricing of whether a given operational or legislative belief will improve some metric in the direction of a value that decides the course of action. A futarchic conditional market aggregates the factory’s own private, positional knowledges with respect to its own operating positions into a price, 57 and any representative error that misprices the marginal worth of a constraint is penalized through the conventional reward channel.

Adopting a futarchic model also changes the PID controller through which λ is progressively determined. Until this point, this controller was backward-looking—it only repriced a given constraint if the factory had already spent time violating that constraint (either in production reality or as a simulated, adversarial condition). A futarchic λ, however, is necessarily forward-looking—the market continually reprices based on the expectation of constraint violations. A mispriced λ is, of course, an arbitrage opportunity, so the adversarial population within the factory profits precisely by correcting the committee’s errors in much the same way as they are incentivized to police overfitting. Examples of futarchic governance in practice are mixed (see Kubinec 2025), 58 largely constrained within the DAO space (e.g., MetaDAO), with results typically pointing in the direction that futarchy performs well when decisions visibly move metrics and performs poorly when decisions are inconsequential relative to metrics. For the superdark factory, this bolsters an argument that the norm or value layer should go through a separate, perhaps more authoritarian, mode of determination, while constraint determination (λ and metrics) might be well served by conditional prediction markets.

Time and Phase

When does governance happen? At what rate? Thus far, the factory’s relationship to time has been left fairly abstract—it is not too slow, it is contingent, it is fast or dense enough to render its internal schedulers incoherent to an outside observer—and that level of abstraction creates issues for a governance layer that needs to be capable of resolving some workable concepts of present, past, and future (kp, ki, and kd, respectively).

Luckily, the superdark factory has a temporal unit, which is the loop—the cybernetic circuit through which error is corrected against a reference. The kybernetes steersman returns here, à la Wiener, whose control mechanism over his ship includes a heading (reference), a rudder (action), the inevitability of drift (sensed consequence), and the steersman’s correction (error applied). A feedback loop is typically defined negatively, as above in terms of error correction, but its sign can be inverted into a positive loop in terms of progressive deviation from course. One closed cycle, whatever its sign, is one loop, which can be understood and analyzed in terms of its periodicity (relative frequency of loop closure), phase (relative positions between two loops), and gain (in the sense of a Bode plot, where gain represents the strength with which a feedback loop converts one unit of error into one unit of correction). We have already established a rough corollary to gain, which is nothing less than exactly λ, understood as a coefficient of the amplitude of feedback. But the other two, frequency and phase, remain as of yet unbound or undefined.

The superdark factory is a stack of loops running with expected variances in periodicity and phase across scales and populations. The period of a given factory loop is largely determined by the delay through which feedback is returned, which can range from small circuits (an online evaluation scoring a single agent’s output within the round it was produced) to very long circuits (the best example being realized-consequence scoring, which is ultimately dependent on the timescale of consequentiality). Feedback plus delay can lead to conditions like oscillation—the artificial construction of larger-scale periodicities that are information-free, wherein the factory revisits the same regions of configuration space on a regular cadence and learns nothing new on each pass—or instability, where each pass through a loop winds up amplifying error, ultimately flipping the sign into a positive feedback loop.

The four convergence pathologies we have been describing can be re-understood in temporal terms with temporal solutions. Each of the four convergence pathologies has catastrophic (“unstable”) forms in which gain rapidly ramps toward infinity or toward zero, and the destination in either direction results in factory death.

Overfitting, for example, can be understood as a kind of aliasing. In overfitting, a metric measures its underlying norm at some finite rate and resolution, and when the norm varies faster or at a finer grain than the metric measures it, optimization against the measurements diverges from the norm in the intervals between them. The factory then improves every reading while the unmeasured remainder of the norm goes unsatisfied to an arbitrary degree. The mismatch between the sampling rate of the metric and the rate of variation of the norm accumulates, and the solution is like a solution to a Nyquist violation—simply increase the sampling rate (the rate, resolution, and variety of measurement) until both are cotemporal.

Learning death occurs in loops whose period exceeds the duration of opportunity (overstable loops). As reward is averaged over long windows, an exploratory discovery (e.g., via mean-based learning) appears and is abandoned before the loop that rewards that discovery would close, so no explorer is paid within the lifetime of its exploration. A population compensated only for long-run averages ceases to produce variation on any shorter timescale. The solution here can also be thought of as a temporal resolution—the compensation period of any exploratory learner must be shorter than the lifetime of the things it is being compensated for discovering, which can either be engineered through anticipatory settlement (the futarchic prediction market) or through engineering a kind of guaranteed patience within the Stackelberg move, wherein some share of the exploratory population is allowed to live longer than justified by its own current scoring.

We can think of thrash as oscillation. Thrash occurs in loops whose periods exceed the lifespan of the configurations they are trying to error-correct. In this case, the feedback loop completes its circuit after the configuration it scored has been refactored away, and so the correction is applied to a successor configuration for which it is no longer accurate, and the misapplied correction produces a deviation that the following cycle (which is also equally delayed) error-corrects with the same inaccuracy. Either the loop needs to be shortened (less delay) or some speed limit needs to be applied to the velocity with which the factory refactors itself, allowing feedback loops the time they need to actually close. If the latter is undesirable, which it is likely to be, then the factory needs to be destroyed and the evaluations layer needs to be adjusted in ways that incrementally contain in-built compensation for the expected delay of delivery.

In the case of stable failure, the loop looks fine: It closes on schedule and its feedback arrives accurately and in phase. Where the other three pathologies are mismatches of period and phase, stable failure is an issue of gain—specifically an issue wherein an insufficient amount of available gain exists to kick the factory out of a steady failure state. The resolution for stable failure states was already provided in temporal terms: The duration of a failure state needs to be penalized, or (better yet, for this section) the duration of a failure state needs to ratchet up the available gain that can be applied to the loop. However, gain ramped high enough to kick a system out of an overdamped attractor will, if unchecked, overshoot into an oscillation condition (thrash).

Clocking the Factory

Now the question of when does governance happen can be answered. There is one particularly nasty relationship between the cadence of a governance feedback loop and the loops that it ultimately controls that is referred to here as “iatrogenic thrash” (thrash produced by governance). If the governing loop revises its command set (e.g., the spec) at a rate near the natural frequency of the loop beneath it, then the outer controller will correct against the unfinished transients of the controlled loop. In turn, the two controllers will force each other near their shared resonance, and both will begin to oscillate with growing gain. This is exactly a case of catastrophic instability, a situation of feedback whose loop gain reaches unity at 180° of phase lag, which turns negative error correction into positive error amplification.

The literature on cascade control proposes that an inner loop must resolve itself several times faster than the outer loop that commands it, with ratios that start from 3:1 at a minimum to 10:1 (Åström and Hägglund 2006), 59 at which point the outer loop can treat the complexities of the inner loop as a constant. Given that most factory loops run at machine time, this requirement feels hardly threatening to a regular update cadence on the part of governance. However, this exact rule needs to be applied through the full cascade of nested loops that make up the factory’s dark interior, and the governance loop needs to relate itself to the periodicity of the factory’s slowest loops.

This leads to the question, How can you measure the periodicity of a feedback loop inside a superdark factory? After all, on the inside of the factory, a verdict is consumed as a reward signal in the scored agent’s propensity update and then discarded. However, those verdicts aggregate into the score distributions that accumulate into transfer-operator versioning (a version is, again, a running distribution of judge scores per metric over a time window). In this sense, the slowest-loop periodicities can be derived from a spectral analysis of those per-metric score distributions over time. This approach can be verified via the toolset of classical system identification (Ljung 1999). The governance committee can inject a small, deliberate intent revision and measure how long the output distribution takes to return to a settled distribution, and that settling time corresponds to the period of the slowest feedback loop. In turn, the governance committee can roughly time its own charter-revision minimums.

A factory whose versions stabilize monthly inside market conditions that are comprehensively repriced weekly has no viable governance tier. The perennial game here lives between an upper bound of sampling noise and a lower bound of “lagging the world.” In this particular case, a factory must either learn to accelerate lower loops or merge loop levels in order to facilitate the very legitimacy of its own governing conditions.

In “The Architecture of Complexity,” Herbert Simon (1962) argues that within any system–subsystem hierarchy, the high-frequency dynamics within a given subsystem settle long before the low-frequency dynamics between subsystems become consequential, and for that reason each level is “screened off” from the others simply by virtue of rate difference alone. Simon takes this logic further, arguing that each successive hierarchical subsystem is not defined by its contents or its formal designation (e.g., via an org chart) but is instead defined exactly by the band of frequencies at which its internal interactions run. In this sense, we can define the governance layer of the factory less by virtue of its role or by virtue of its nominal exteriority vis-à-vis the factory and instead as its slowest-moving loop (or, inversely, drop the question of governance entirely and situate this entire discourse in terms of temporality).

From there, the question turns to something that we have been withholding from you the whole time—why is there nothing about orchestration layers, subsystems, factory organ management, etc.? The answer until now has been the assertion that any intervention at this level of scale inhibits the factory’s ability to self-organize, wielding the principle of robust simplicity as rationale. On one hand, the case for the automatic emergence of subsystems is relatively strong, because each class of evaluation (and evaluation-evaluation) already carries its own characterizing loop periodicity. The delay of feedback in this sense is quite literally constitutive of a system or of a class (an in-round online judge to a task worker, a regression-prediction judge to a services architecture, a realized-consequence judge to a deployment architecture). In this sense, a rough tiering of internal subsystems comes “for free” the moment the factory contains a requisite variety of evaluators. Simon (1962) goes further to argue that composition itself selects for the progressive aggregation of subsystems in content-agnostic ways. This hierarchy can be expected to deepen on its own, to and through the factory’s walls.

However, the automatic development of subsystems via emergence alone fails for two reasons, which forces us to go back to the Stackelberg move. First, the automatic selection for rate separation discovers said separation only by surviving its absence; a factory that learns not to collapse its bands only does so by passing through thrash, and thrash always invites a whiff of catastrophic death into the room. At the Stackelberg move, building a preconstructed orchestration armature that the factory is licensed to tear down might be one way to survive that absence (a kind of arch-centering, wherein a temporary wooden frame holds stones in place until the keystone sets and then comes out). We are simply not opinionated about what that orchestration layer might look like. Second, band separation is under continuous attack from forces that push bands together. Every time loops share a common medium (e.g., a common foundation model whose checkpoint releases act as a global forcing function) or a common infrastructure, they risk entrainment (they tend to drift into the same rhythm the same way that pendulum clocks that share a beam might fall into sync 60). To paraphrase Yoshiki Kuramoto (1984), actors can spontaneously synchronize once any kind of coupling or dependency crosses a threshold, 61 which under certain conditions can result in nothing short of the spontaneous annihilation of entire factory subsystems. A factory relying entirely on the implicit temporalities of a judge feedback-return structure is under slow selective pressure toward band separation and a fast coupling pressure toward collapse, and there is no reason to expect the slow force to win out in the end.

The Stackelberg move, then, must actually encode some level of temporal registration into its hard cast. This does not mean that it takes the step of encoding the content of each systemic level, nor that it takes the step of encoding the actual frequency bands themselves. Instead, it must simply encode the relational dynamics that keep both iatrogenic thrash and entrainment at bay. There are two tactics here, and each needs to be encoded into the “thin but stateful” managed queue of outstanding decisions described earlier (see “The Primitive”), where producers await the promise fulfillment of their asynchronous judges. First, this queue needs to enforce a minimum cascade control ratio (e.g., 3:1+) before returning verdicts into the next evaluatory tier—think of a manager who reads a weekly report instead of gluing their eyeballs to a live activity ticker; in this case, information is intentionally withheld from management until it settles at a temporal scale that is significantly slower than the operational cadence of the thing being managed. In this sense, the information is not just being temporally withheld but is also being delivered as a distribution (again Beer’s management principles rear their head). Second, this temporal deferral should also be diversified (jittered) to intentionally obfuscate entrainment. This same level of buffering is not enforced downstream (e.g., there is no escrow of decisions held back from higher evaluatory layers or governance; downstream commands remain rich but stateless), both because the factory is expected to outrun the world (and therefore be immediately sensitive to high-priority downstream overrides) and because upper management layers are intentionally starved of information variety from their downstream workforce to prevent them from collapsing downward (and actively penalized for doing so, as both repeating a prior decision or introducing downstream noise result in wasted compute and degraded scores).

At the same time, as is becoming a refrain at this point, there are certain cases where these temporal bands need to be broken, much like how the exploratory frontier of mean-based no-regret learners needs to be enabled to explore less-than-rational solutions. Lower-level subsystems can become overly rigid and overly determined, typically due to an imbalance between exploratory and surplus-retentive learners within that subsystem. That rigidity makes that subsystem perfectly primed for a small, fast disturbance to tip it into what C. S. Holling calls “release” (local collapse—see Gunderson and Holling 2002; Holling 2001). If that system is overly coupled to its parent systems—which, by nature of its overdetermination, it is likely to be—that collapse can accumulate upward, triggering the collapse of that larger, slower parent system. Holling calls this a “revolt,” 62 through which a subsystem essentially forces its own correction by destroying its own governance layer, which in turn should be read as a potentially positive outcome both for the subsystems involved and for the superdark factory as a whole. The design principles above enable this particular outcome by never actually specifying the terms of band separation itself. It follows then that accidental breaches are penalized heavily but productive breaches are rewarded. This penalty structure is again managed by the PID controller, which penalizes chronic, low-grade breaches and rewards decisive and productive revolts.

Entrainment should not be cast aside easily. Until this point, the superdark factory has been presented in isolation, as a dependency-free, vertically integrated monad. This, of course, cannot be the case—not only are the factory’s walls constantly and progressively integrated with the world it attempts to model and game, but the factory is, as is invariably the case with software, very likely assembled out of an enormous volume of third-party dependencies. Why would that not be the case? These dependencies include everything from hardware (GPU-sourcing, raw materials, energy providers, infrastructure) to software modules (open-source libraries, software-as-a-service platforms, model providers), all of which are continuously changing at impossible-to-model periodicities.

Moreover, it can never be assumed that there is only one superdark factory, nor that a superdark factory does not share common infrastructure or common markets with another superdark factory, nor that two superdark factories do not share each other, consume each other as inputs, serve each other as vendors or dependencies, or finance each other as customers. In turn, it becomes possible to imagine that the superdark factory risks entrainment with other superdark factories—or, more malevolently, serves entrainment as a tactic of interfactory war. It becomes the duty of governance, an extraordinarily difficult duty, to force or incentivize the randomization of a factory’s dependency class without ever truly being aware of what those dependencies might be. Should governance fail—and it always will, to some degree, unless it scales accordingly—then an economy of superdark factories can quickly become a phase-locked, synchronized condensate, one which can fail in correlated flash-crash cascades that elevate concerns of catastrophic instability all the way to the planetary scale and beyond.