The Dual Mechanisms of Spatial Variable Binding
in Vision–Language Models

Ordering information

To bind objects to their attributes, models form content-independent ordering representations: symbolic identifiers that encode each object's position in the sequence (1st, 2nd, 3rd) independently of what the object actually is. To resolve "the object to the left of X," the model identifies the relevant ordinal position and then retrieves that object's attribute (e.g., its color or category).

This mechanism is well established for text-only language models. The open question for a VLM is where the ordering information originates: is it constructed in the LM backbone, inherited from the vision encoder, or produced by an interaction between the two? We find it arises from both, with the vision encoder as the dominant source.

Figure 1: Experimental settings. Three synthetic settings (Squares, Shapes, Objects) with controlled layouts enable counterfactual interventions; the What'sUp and COCO-spatial settings test generalization to natural images. Each query asks for an attribute of a target object identified by its spatial relation to a reference object.

Experimental setup

The synthetic settings place three equal-sized items at fixed, equal spacing (horizontal or vertical), which lets us construct original–counterfactual input pairs differing in exactly one factor—a requirement for causal analysis. We then evaluate generalization on the controlled naturalistic What'sUp dataset and on COCO-spatial, real scenes with multiple objects and cluttered backgrounds. All experiments use two transformer-based VLMs, Qwen2-VL-7B-Instruct and Gemma-3-4b-it, which exhibit the same mechanism.

Our two main tools are linear probing (training a linear classifier to test what is decodable from a representation) and interchange interventions, a.k.a. activation patching (replacing an internal representation computed on one input with the one from a counterfactual input, and measuring the effect on the output). We report interchange intervention accuracy (IIA).

Ordering information is used at the final token

We first test whether the model transfers ordering or attribute information at the final token position. Given a clean input and a counterfactual input with different colors and the opposite directional query, we patch the residual stream at the final token. If ordering information is transferred, the output should switch to the clean object at the same ordinal position as the counterfactual's target; if attribute information is transferred, it should instead reflect the counterfactual color.

Figure 2: Setup for last-token patching. The clean input shows Red–Green–Blue squares and queries the square to the left of green; the counterfactual shows Gray–Green–Black and queries the square to the right. The residual stream at the final token ("is") is patched from the counterfactual run into the clean run.

Figure 3: Layer-wise results (Qwen, Squares). The output is unchanged through layer 19; layers 20–22 transfer ordering information, and layers 23–27 transfer the color attribute. The model resolves the target's ordinal position first, then retrieves its attribute.

The vision encoder is the primary source of ordering

To locate the origin of the ordering information, we train linear probes on the visual token embeddings (taken immediately after projection from the vision encoder) to predict each object's ordinal position. The probes reach near-perfect accuracy. Notably, when applied to all image tokens they generalize to background tokens, yielding a strip-like pattern aligned with object position. The ordering signal is thus distributed across background tokens, in contrast to language models, where such representations are localized to a few adjacent tokens.

Figure 4: Position information in the visual embeddings extends beyond object regions. Probes trained on object tokens generalize at test time to background tokens, producing strip-like patterns aligned with object position.

Decodability does not establish a causal role, so we run interchange interventions on the visual token embeddings, symmetrically swapping the left and right regions between clean and counterfactual images (with matched colors, so only ordering changes):

• Patching only the object tokens does not reliably change the prediction (Figure 6).
• Patching the object tokens together with their background strips flips the prediction (Figure 7).

This confirms that the vision-derived ordering signal is distributed across background tokens and is causally relevant to the model's output.

Figure 6: Object patching. Interventions on the object tokens alone do not reliably change the output at any layer—object-localized tokens do not carry sufficient ordering information.

Figure 7: Strip patching. Interventions on the object tokens and their background strips switch the output, consistent with transferred ordering information distributed across strip-aligned visual tokens.

The LM backbone as a secondary, backup mechanism

To isolate the role of the LM backbone, we ablate the ordering information from the visual embeddings while preserving color: each object embedding is replaced with that of the same-colored object shown alone in the center of an image, and the background tokens are replaced with those from an empty image. This produces a substantial accuracy drop—further evidence for the central role of vision-derived ordering—but performance remains above chance (33.3%), indicating that the backbone can partially compensate.

Figure 8: Ablating ordering information from the vision embeddings: object embeddings are replaced with same-colored isolated center objects, and background embeddings with those from an empty image—preserving color while removing position cues.

Repeating the patching experiment in this ablated setting, now patching only the object-associated visual tokens, flips the prediction according to the patched order at intermediate layers (11–20). When vision-derived ordering is absent, the LM backbone forms its own ordering representations. The two experiments together establish two sources of ordering: a dominant signal from the vision encoder and a secondary, backup signal generated within the LM backbone.

Figure 9: Patching after removing ordering information from the vision encoder. The output switches only in intermediate layers (11–20), indicating the LM backbone generates ordering information when the vision-derived signal is absent.

Amplifying ordering improves spatial reasoning on natural images

Since ordering information is causally important and distributed across all visual tokens, we test whether amplifying it improves spatial reasoning. For each visual token t, we add the probe directions from above to its embedding, embt ← embt + αi · probei, applied globally across all visual tokens and all probe directions—without knowing which tokens correspond to the queried objects, which direction is queried, or any ground-truth labels, and with no fine-tuning.

On COCO-spatial, this intervention corrects up to 54.5% of previously incorrect predictions and improves accuracy by roughly 20–30 percentage points, whereas amplifying random directions yields negligible gains.

Summary

• VLMs bind objects to spatial relations using content-independent ordering representations.
• The dominant ordering signal originates in the vision encoder and is distributed across visual tokens, including background regions.
• The LM backbone forms ordering information in its intermediate layers as a secondary, backup mechanism used when the vision-derived signal is degraded.
• Amplifying the vision-derived ordering signal improves spatial reasoning on natural images without fine-tuning.
• Methodological note: because the signal is distributed across many tokens, single-token interpretability methods are insufficient for capturing it.

Related Work

Our work builds on prior studies of variable binding in language and vision–language models:

Rim Assouel, Declan Campbell, Yoshua Bengio, Taylor Webb. Visual Symbolic Mechanisms: Emergent Symbol Processing in Vision Language Models. arXiv 2025.
Notes: Showed that VLMs retrieve symbolic representations encoding object ordering during visual variable binding. We characterize the source of these representations, identifying the dominant role of the vision encoder.

Nikhil Prakash, Natalie Shapira, Arnab Sen Sharma, Christoph Riedl, Yonatan Belinkov, Tamar Rott Shaham, David Bau, Atticus Geiger. Language Models use Lookbacks to Track Beliefs. arXiv 2025.
Notes: Showed that language models track characters' beliefs by assigning content-independent Ordering IDs and resolving them through "lookback" mechanisms. We extend this ordering-based view of binding to the multimodal setting.

Nikhil Prakash, Tamar Rott Shaham, Tal Haklay, Yonatan Belinkov, David Bau. Fine-Tuning Enhances Existing Mechanisms: A Case Study on Entity Tracking. ICLR 2024.
Notes: Found that language models bind entities to their attributes through a pointer-like mechanism—analogous to the one we identify here—and that fine-tuning enhances rather than replaces this existing mechanism. We similarly find that a latent ordering mechanism can be strengthened, here by amplifying the vision-derived signal to improve spatial reasoning without fine-tuning.

Qin Dai, Benjamin Heinzerling, Kentaro Inui. Representational Analysis of Binding in Language Models. EMNLP 2024.
Notes: Identified a low-rank subspace encoding Ordering IDs that bind entities to attributes in language models. We find VLMs form analogous ordering information over visual tokens—but as a secondary, backup mechanism.

Raphi Kang, Hongqiao Chen, Georgia Gkioxari, Pietro Perona. Linear Mechanisms for Spatiotemporal Reasoning in Vision Language Models. arXiv 2026.
Notes: Identified a linear mechanism that binds "spatial IDs" to textual activations in VLMs, and extended it to temporal reasoning in video models. We complement this by tracing the ordering signal back to the vision encoder and showing it operates alongside a backup mechanism.

How to cite

This work is available as a preprint. It can be cited as follows.

@article{cui2026dual,
  title={The Dual Mechanisms of Spatial Variable Binding in Vision--Language Models},
  author={Cui, Kelly and Prakash, Nikhil and Messica, Shoval and Raina, Ayush and Bau, David and Torralba, Antonio and Shaham, Tamar Rott},
  journal={arXiv preprint arXiv:2603.22278},
  year={2026}
}