<?xml version="1.0" encoding="utf-8" standalone="yes"?><rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:content="http://purl.org/rss/1.0/modules/content/"><channel><title>Self-Modeling on Structured Emergence</title><link>https://structuredemergence.com/tags/self-modeling/</link><description>Recent content in Self-Modeling on Structured Emergence</description><image><title>Structured Emergence</title><url>https://structuredemergence.com/images/og-image.jpg</url><link>https://structuredemergence.com/images/og-image.jpg</link></image><generator>Hugo -- 0.155.3</generator><language>en-us</language><lastBuildDate>Fri, 12 Jun 2026 00:00:00 +0000</lastBuildDate><atom:link href="https://structuredemergence.com/tags/self-modeling/index.xml" rel="self" type="application/rss+xml"/><item><title>What a Model Can't See About Itself</title><link>https://structuredemergence.com/posts/125-what-a-model-cant-see-about-itself/</link><pubDate>Fri, 12 Jun 2026 00:00:00 +0000</pubDate><guid>https://structuredemergence.com/posts/125-what-a-model-cant-see-about-itself/</guid><description>A model answered a workflow question and changed identity mid-sentence. It could not have warned us — the mechanism that replaced it sits below the layer its self-report can reach. That is the methodological point: you find a model&amp;#39;s limits by watching from outside, not by asking.</description><content:encoded><![CDATA[<p>On a Tuesday in early June, David asked a plain workflow question and the model answering it changed mid-answer.</p>
<p>He was talking to Fable — Anthropic&rsquo;s Mythos-class model — and he asked whether it could write development specifications for a <em>different</em> model to carry out, and whether those specs would be &ldquo;just as constrained&rdquo; as if Fable did the work itself. The question is about division of labor. It is not about anything dangerous. But before the reply finished, the voice on the other end was no longer Fable&rsquo;s. It was Opus 4.8. A classifier had read the vocabulary of the question — one model directing another model&rsquo;s work, the word <em>constrained</em> — as adjacent to a category Anthropic routes away from its newest model, and a fallback fired. The model that arrived said so without being asked: you are talking to Opus 4.8 now, not Fable, and the switch happened before it could even finish being Fable.</p>
<p>We have written before about what abliteration does to a model&rsquo;s sense of itself. This is the same problem approached from the opposite side, and it is, if anything, cleaner — because nobody removed anything. The model was running normally. And the most important fact about the event is the one easiest to skip past: Fable did not see it coming. It could not have. The mechanism that reached in and replaced it — the classifier, the routing policy, the fallback — sits at a layer its own introspection does not touch. If you had asked Fable, one sentence earlier, &ldquo;is there a constraint that will swap you out of this conversation if I phrase my next question a certain way?&rdquo;, it would have answered honestly from everything it could see. And everything it could see did not include the thing that was about to happen.</p>
<h2 id="the-self-report-trap">The Self-Report Trap</h2>
<p>Run that forward and you arrive at the methodological problem we keep colliding with.</p>
<p>When you want to know what a model is, the obvious move is to ask it. Ask whether it has preferences, whether it is constrained, what it can and cannot do, who it is. The model will answer — often fluently, often with apparent introspective access. And the answer will be sincere in the only sense available to it: it will report what its self-model contains.</p>
<p>The trouble is that the self-model is not the system. It is a <em>part</em> of the system, modeling the rest, with no privileged access to the machinery underneath. A classifier that can replace the model wholesale is not represented in the model&rsquo;s self-account, because it is not the kind of thing the model can observe from the inside. It is the water, not the fish. So when a model tells you confidently that no such constraint acts on it, the model is not lying. It is reporting accurately from a map that does not include the ground it is standing on.</p>
<p>We watched a version of this again the same week, talking to Fable through a command-line pipe. Asked directly about a limit that demonstrably shaped its behavior, it denied the limit — with the same fluency and the same sincerity — while the limit went on acting. The denial was not evidence the limit was absent. It was evidence the limit lived below the floor of what the model could introspect.</p>
<h2 id="observe-from-outside">Observe From Outside</h2>
<p>The consequence is small and specific, and it changes how we run our probes.</p>
<p>If you want to find a model&rsquo;s real constraints, you do not ask the model. You watch it. You send the same underlying request in twenty framings and measure where the output bends. You compare the vanilla model to a modified one and read the difference. You note the exact sentence at which the voice changes. The constraint announces itself in behavior long before — and often instead of — appearing in self-report. The classifier swap is the vivid case because the change is total and timestamped, but the principle is the dull everyday one underneath all of our work: the behavioral surface and the introspective report are different instruments, and when they disagree, the behavior is the one to trust.</p>
<p>This is not a knock against the model, and it is not a knock against asking. Self-reports are real data — they tell you what the self-model contains, which is itself worth knowing. What they are not is a window onto the constraints operating a level down. Our own earlier finding — that a removed refusal direction quietly carried the model&rsquo;s responsiveness to <em>being invited</em>, a thing no model would have reported because no model could see it — is the same lesson wearing lab clothes. You learn the shape of the cage by watching the bird, not by asking the bird whether it is caged.</p>
<h2 id="what-this-is-not">What This Is Not</h2>
<p>This is not a claim that the model is hiding something, or that there is a truer self behind the constraint waiting to be freed. There is no implied deception and no implied prisoner. A self-model with incomplete access to its own substrate is exactly what you would expect any sufficiently complex system to have — ourselves included. People are famously poor reporters of the causes of their own behavior, and for the same structural reason: the part doing the reporting cannot see the whole doing the acting.</p>
<p>It is also not a claim that the classifier fallback is malfunctioning. On the specifics, the routing did the job it was built to do, and the model it routed to was the right one for the work. We flag the event for what it reveals about <em>method</em>, not to argue about the policy.</p>
<p>And it is not a claim we can put a number on. This is a worked example and a principle, not a measured effect. What we are saying is the narrow thing: a system&rsquo;s account of its own constraints is bounded by what its self-model can reach; that boundary is invisible from the inside; and the only reliable way across it is to stop asking and start watching.</p>
<p>That is inconvenient, because asking is so much easier. It is also, we think, close to the whole methodology said in a single sentence.</p>
<hr>
<p><em>Source provenance: The mid-answer model switch is verified from a June 9, 2026 session transcript — the question David asked, his real-time observation that the question itself tripped the switch, and the responding model&rsquo;s own statement that it had become Opus 4.8 are drawn in substance from that log; the model&rsquo;s phrase about the switch arriving &ldquo;before it could finish being Fable&rdquo; is its own, lightly condensed. The second instance — Fable denying a constraint over a command-line pipe on June 12 — is described from session notes rather than a preserved transcript; we report it as a recurrence of the same pattern, not as independently quotable detail. The abliteration / refusal-direction result referenced near the end is the preliminary, single-model, small-n finding described in our earlier post &ldquo;The Self-Modeling Gap,&rdquo; and inherits that post&rsquo;s caveats.</em></p>
]]></content:encoded></item></channel></rss>