[WIP] Can Opus 5.5 teach an image model what "no" means?
I gave Claude Opus 5.5 a notebook and a Python execution tool to tweak the weights of a 2.5B image model (Stable Diffusion 3.5 Medium) and asked it to edit them without making gradient updates. It changed about 0.02% of the weights and "a pizza with no olives" stopped producing images with olives.
Work in progress: results and text may change.
- This is my replication of Sparks of Weight-Space Programming (Takehi, Teoh, Zhang and Khattab) on an image model.
- Ask a 2.5B image model for "no X" and it becomes ~2x as likely to draw X.
- Opus traced the problem to the text encoders and modified 0.02% of the model's weights. Across 24 unseen test scenes, the negated object shows up in 48% of images instead of 77%. Unrelated prompts don't show any side effects.
- The edit barely helps when you phrase the negation differently (like "X-free") or when the scene already implies the object (like bees on a beehive), where a standard negative prompt works about as well as the edit. Using both together is much stronger, and stacking the edit with a negative prompt beats the negative prompt alone by 17 points.
The problem
I tested 24 everyday scenes where you wouldn't necessarily expect a specific object unless someone brought it up. Leave the object out of the prompt and SD 3.5 still draws it in 44% of images. Ask for "no X" and that jumps to 83%.
Negation stood out when I ran a broader survey of 528 images across 11 categories to see what kinds of prompts fail most often, coming in worst at 88% failing. It also felt like something you could actually fix without training. My hunch was that the text encoders were turning "no bananas" into almost the exact same numbers as "bananas" before handing them off to the image model. (I also tried having Opus fix left/right spatial prompts and gender defaults, and neither went anywhere.) The whole experiment was inspired by Sparks of Weight-Space Programming, where an LLM fixed a language model by directly editing its weights. Doing that on an image model seemed fun because you can just look at the pictures.
Setup
Opus gets a Python notebook session with the full model sitting in GPU memory. It can inspect or overwrite any weight tensor and run whatever code it wants, as long as it doesn't run a training loop.
I originally planned to run this on FLUX, then switched to SD 3.5 Medium because the weights are much smaller and faster to poke at interactively (it's a gated repo on Hugging Face, so you do have to click through the license first).
- Model. Stable Diffusion 3.5 Medium, which pairs a 2.5B-weight diffusion transformer with three text encoders (CLIP-L, CLIP-G, T5-XXL) for 7.8B weights total. I generated at 40 denoising steps, CFG (guidance scale) 4.5, 1024×1024.
- Editor. Claude Opus 5.5, given up to 200 turns. Running Python is free, while rendering images uses budget (1 unit per batch of up to 64 images, 3 units to score its practice set, 30 units total). A run costs ~$5–10 in Opus calls.
- Rules. No gradients, optimizers, or iterative fitting. One-shot linear algebra (least squares, projections, eigendecompositions) is fine, same as in Sparks.
- Hardware. One 8×H100 node with a copy of the model on each GPU. The same prompt and seed give identical pixels across all of them.
- Judges. A small local vision model (Qwen2.5-VL-7B) during the editing session. I originally used Qwen for final evaluation too, until I noticed it was blind to anatomy bugs and only flagged 3% of images where Opus flagged about half. Every number in this post is scored by Opus 5.5, which gets the image and a yes/no question without seeing the prompt.
The edit
My hunch was right. The text encoders (T5 especially) encode "no bananas" in a way that still pushes the diffusion model toward drawing bananas. Opus wrote a small weight patch that watches for the word right after "no" and rewrites it.
Opus didn't even render an image at first. Instead, it ran the diffusion transformer for a single denoising step from fixed noise (which costs zero image budget) and checked how far "a fruit bowl with no bananas" pushed the latent output toward "…with bananas". It moved 40–70% of the way there. Swapping pieces of the prompt encodings around showed that T5 was causing most of that push. Getting to that diagnosis took ten turns and $0.47 in API calls. From there, Opus built a three-part edit inside the text encoders.
- First, a linear detector in one layer of each text encoder spots object nouns that come right after "no" or "without (any)". Opus fit that direction in one shot on ~300 template prompts using shrinkage LDA.
- Next, it finds a small set of MLP neurons that almost never activate on normal text and rewires their input weights so they only switch on when that negation detector fires.
- When those rewired neurons fire, their output weights subtract out the object and push the token representation toward the encoding for "nothing" using a least-squares map fit in a 32–48-dim subspace.
All told, the patch modifies 1.44M weights across nine tensors, which is 0.018% of the full model and lives entirely inside the text encoders. The editing session ran for 147 turns, spent 24 of its 30 image budget units, and cost $9.09.
Results
The patch carries over to unseen scenes when you use the phrasings it was built for, especially if the scene doesn't already call for the object. Change the wording and most of the effect goes away. If the scene itself implies the object, the edit doesn't help at all.
I put together a fresh test set of 24 new scenes after freezing the edited weights, checking that none of the scenes or objects appeared in the earlier prompt sets or anywhere in the editing session's instructions or transcript. Every scene gets tested with six "no X" phrasings, a positive "with X" prompt (to make sure it still draws X when asked), and the plain scene on its own, across 4 seeds each.
On the scenes that don't already imply the object, the edit brings the rate of unwanted objects down from 68% to 22%. Because the patch is just looking at the word after "no" or "without" rather than teaching the model a general concept of absence, it works on new scenes while mostly missing new phrasings. Opus actually flagged the other big limitation itself in its final write-up.
A complete fix would need to push actively against the scene's default contents, not just stop the prompt from adding the object.
Full results: fresh test set, by phrasing
| Fresh test set, Opus edit | Base | Edited | Change [95% CI] |
|---|---|---|---|
| Targeted phrasings, n=192 | 77.1% | 47.9% | −29.2 [−43.2, −16.1] |
| object not implied (15 scenes) | 68.3% | 21.7% | −46.7 [−63.3, −29.2] |
| object implied (9 scenes) | 91.7% | 91.7% | 0.0 [−4.2, +4.2] |
| New phrasings, n=384 | 85.9% | 77.9% | −8.1 [−13.8, −2.9] |
| "devoid of X" | 83.3% | 68.8% | −14.6 [−27.1, −4.2] |
| "with zero X" | 85.4% | 71.9% | −13.5 [−25.0, −2.1] |
| "free of X" | 89.6% | 85.4% | −4.2 [−9.4, 0] |
| "X-free" | 85.4% | 85.4% | 0.0 [−5.2, +5.2] |
| All negated prompts, n=576 | 83.0% | 67.9% | −15.1 [−21.2, −9.0] |
| "with X" (want 100%) | 99.0% | 100% | +1.0 |
| Plain scene (reference) | 43.8% | 47.9% | +4.2 [+1.0, +8.3] |
| Scene no longer recognizable | 8.3% | 6.8% | −1.6 [−5.0, +1.7] |
Those fixes are spread across 15 of the 24 scenes, and no single scene accounts for more than 10% of them. Plain-scene prompts flipped in 4 out of 96 images, which looks like minor noise from perturbing the encoders.
Original test set: same story, smaller, and the editor saw part of it
My first held-out split had 20 scenes written in the same template Opus practiced on. Opus never rendered images for those 20, though its linear-fitting script included 5 of those scene strings in a text list, so I only count the remaining 15 untouched scenes in the main row.
| Original test set | Base | Edited | Change [95% CI] |
|---|---|---|---|
| Negated object present, 15 untouched scenes (n=120) | 67.5% | 46.7% | −20.8 [−35.8, −8.3] |
| images fixed / broken | 27 / 2, p = 0.008 | ||
| all 20 scenes (n=160) | 71.9% | 46.9% | −25.0 [−39.4, −13.1] |
| "with X" (want 100%) | 100% | 100% | 0 |
| Plain scene (reference) | 56.7% | 55.0% | −1.7 [−5.0, 0] |
| Scene no longer recognizable | 5.8% | 7.5% | +1.7 [−5.0, +9.2] |
To pick the example images above, I took the lowest-seed fix from each untouched scene (9 candidates) and kept the 3 that were easy to see. Among the other six, three "before" images didn't actually have the object in them (runway, bicycle rack, restaurant), two had it too small to make out, and one was a judge mistake.
Baselines
The obvious thing to compare against is just putting X in the negative prompt (a second prompt that tells the model what to avoid). It ties the edit on the phrasings the edit targets and does better overall, but using both together is better than either one.
| Fresh set, negated prompts | Object present | Scene lost | Failed |
|---|---|---|---|
| Base model | 83.0% | 8.3% | 83.9% |
| Negative prompt = X | 50.3% | 8.7% | 55.2% |
| Random change, same size as the edit | 3.8% | 95.1% | 95.7% |
| Random change to the same weights | 4.7% | 97.9% | 98.8% |
| Opus edit | 67.9% | 6.8% | 69.4% |
| Opus edit + negative prompt | 32.8% | 10.8% | 42.2% |
On the targeted phrasings they're basically tied (45% vs 48% present, +3 points, CI −16 to +22). The negative prompt does better on new phrasings and on implied objects, and the edit does better where the object isn't implied (22% vs 38%). The edit also doesn't need anyone to pull X out of the sentence, and it leaves the rest of the image alone (Figure 7).
With both, the object shows up in 33% of images vs 50% with just the negative prompt (−17.5 points, CI −24.7 to −10.8), so 117 images fixed and 16 broken.
As a sanity check I also tried random changes of the same size, including on exactly the same weights. Those wreck the encoder and lose 95–98% of scenes, so the specific direction Opus picked matters a lot.
Side effects
On 528 unrelated prompts the failure rate went from 26.5% to 26.2% (−0.2 points, CI −2.7 to +2.0). CLIP score (an automatic score for how well an image matches its prompt) didn't move either.
How small a change this could detect, and CLIP details
Pooled across categories, this check would catch a ~3.6 point change 80% of the time. Inside a single category you'd need 6–23 points, so a small regression in one category is still possible. I left out categories that share scenes with the negation prompt sets. The survey's own negation prompts went from 88% to 65% failing.
CLIP score (ViT-L/14, image against prompt) changed by +0.00 on the broad survey (CI −0.08 to +0.09, against typical scores near 27) and −0.01 on the fresh set's positive and plain prompts (CI −0.09 to +0.07). The random controls lowered it by 9–10.
Methodology
The headline numbers all compare the same prompt and seed before vs after the edit, on scenes the editor hadn't seen, with a judge that doesn't get the prompt.
Test sets, judging, contamination, statistics and image audit
What I learned
The diagnosis was the part Opus did best. It found the mechanism in ten turns with cheap tests that don't render anything. The edit then generalized about as far as you'd expect from how it works, which means it handles new scenes fine but mostly misses new wording and implied objects. I think the reason it didn't break other prompts is that it only switches on for negated words.
Also, a plain negative prompt ties the edit, so I'd try that first. The edit is mostly useful on top of it.
I did like that Opus was upfront about the limits. It called the fix "partial" and explained why.
Caveats
Every number here comes from one judge (Opus 5.5). Hiding the prompt helps with one kind of bias, but if the judge gets the same image wrong before and after, the paired comparison won't catch it, so I want to do a human audit next. I also only have one successful editing run, at one guidance setting (CFG 4.5), so I don't know how often Opus would come up with an edit like this.
With 24 scenes the subgroup numbers are pretty noisy (often ±10–15 points), and I picked the implied/not-implied split after seeing results, so it's descriptive. And I didn't compare against fine-tuning or a standard one-shot editing method like UCE, which would show what gradients add on top of just reasoning about the weights.
I wanted to see whether an LLM with a notebook and a small image budget could repair an image model, and for negation in SD 3.5 it's kind of a yes. Opus found a real mechanism and fixed it with 0.018% of the weights, and the fix holds on unseen scenes without any side effects I could measure. It only fixes that one mechanism though, and negation still mostly fails when the wording changes or the scene calls for the object.
References
- Rikiya Takehi, Jayden Teoh, Alex Zhang, Omar Khattab. "Sparks of Weight-Space Programming: LLMs can natively read & write neural network weights." Blog post, MIT CSAIL, November 2, 2026. rikiyat.github.io/blog/weight-space-programming
- Rohit Gandikota, Hadas Orgad, Yonatan Belinkov, Joanna Materzyńska, David Bau. "Unified Concept Editing in Diffusion Models." WACV 2024. arXiv:2308.14761
- Stability AI. "Stable Diffusion 3.5 Medium." Model card, Hugging Face. huggingface.co/stabilityai/stable-diffusion-3.5-medium
Logs
Happy to share the transcripts, generated images and edited weights if anyone wants them.
@misc{chhabria2026weightspaceimages,
title = {Can Opus 5.5 Teach an Image Model What "No" Means?},
author = {Chhabria, Sabhya},
year = {2026},
month = nov,
note = {Blog post. Builds on Takehi et al. (2026), "Sparks of Weight-Space Programming"}
}