Imagine, Don’t Narrate:
The Generative Bottleneck in
World Models of Interaction Dynamics

Summary

Modern AI agents can plan, reflect, reason, and act over long-horizon digital tasks. Even so, they cannot plan to steer a social interaction in real time. Doing so requires anticipating how latent factors like trust and resistance will evolve under the agent’s actions, fast enough to plan within a conversational turn. Generative world models approach this by narrating possible futures as text, but autoregressive generation is both too slow for real-time planning and fundamentally lossy.

Social-JEPA imagines instead. It is a joint-embedding predictive architecture for dialogue: a frozen 125M-parameter encoder reads the conversation, and just 500K trainable parameters roll the interaction forward in a learned latent space, conditioned on the agent’s candidate actions, without ever decoding text. To compare the two paradigms against known dynamics, we build LID-Bench, the first controlled testbed for interaction dynamics in which the true hidden states are known.

Three generative world models spanning 117M–350M parameters all learn interaction dynamics internally, yet by five turns ahead (k=5) they lose 89–100% of that signal when they render it as text. Social-JEPA bypasses text entirely, forecasting latent dynamics 2.8–3.9× more accurately while performing rollouts 1,059–2,314× faster. At ~11 ms per turn, real-time planning within conversational turn-taking latencies becomes feasible.

Narrate or imagine?

When AI agents interact over multiple turns—tutoring a struggling student, negotiating a contract, de-escalating a crisis—they must reason about the partially observable internal states of the people they talk to. Trust, resistance, and cognitive load do not evolve monotonically: an intervention that reduces confusion under low resistance may trigger a defensive “backfire effect” when resistance is high. Planning under such state-dependent dynamics requires a world model of the interlocutor.

How should that model represent the future? One approach is to narrate: generate the interlocutor’s next response in text, then re-encode it to estimate how their state has changed. The alternative is to imagine: predict future interaction states directly in a learned representation space, bypassing text entirely. In continuous visual control, latent world models already dominate observation-space approaches in both accuracy and compute [1][2]. Whether that advantage transfers to dialogue, where observations are natural language full of surface cues that tempt shortcut solutions, was an open question.

Social-JEPA

Social-JEPA learns latent interaction dynamics strictly in representation space, following the joint-embedding predictive architecture (JEPA) recipe [3]. It trains fully self-supervised on observed conversations, the agent’s actions, and the time gaps between turns. It never sees the true hidden states.

A frozen RoBERTa-base encoder reads the first and most recent tokens of the dialogue history, preserving both stable traits and the current state, and a learned projection maps it to a 64-dimensional state. A 3-layer MLP predictor then rolls that state forward one turn at a time, conditioned on the agent’s abstract strategy (for example VALIDATE or REFRAME) and on the time gap before the next reply. During training, each of the K=5 predicted states is matched to a target: the conversation’s real future, read by the same frozen encoder and an exponential-moving-average (EMA) copy of the projection. VICReg variance and covariance terms [4] keep the embedding from collapsing to a constant.

Every component matters. Training with single-step instead of multi-step rollout drops R² at k=5 from +0.053 to +0.009. Removing the EMA target as well nearly doubles the spread across seeds, turns the k=5 mean negative (−0.011), and halves the effective rank of the embedding (23.1 → 11.4). Without VICReg, training diverges within the first epoch. And a 2.8× larger generic encoder without dynamics-aware training (SimCSE-RoBERTa-Large, 355M) reaches only +0.106 at k=1, versus Social-JEPA’s +0.128.

LID-Bench

Just as MuJoCo [5] and DMControl [6] were prerequisites for progress in model-based RL, evaluating interaction world models requires a testbed with known dynamics. LID-Bench (Latent Interaction Dynamics) simulates an interlocutor with nine latent coordinates: three stable traits (openness, patience, expertise) and six dynamic ones (trust, resistance, arousal, cognitive load, confusion, stress). These are generator-defined control variables, not psychological claims. They evolve under the agent’s actions through a nonlinear, state-dependent transition with a backfire gate: in regions of high arousal and resistance, interventions reverse sign, so actions that normally build trust erode it.

GPT-4o writes each reply from descriptions of how the person responds (the scope, stance and focus of the reply), and each agent turn from a paraphrase of the agent’s intended move. It never sees the state values or the action names. Four leakage controls stop simple word-matching from reading the state off the text: a TF-IDF baseline reaches only R² = 0.085 out of distribution.

The benchmark contains 4,750 episodes of 25 turns (~119k transitions) across five domains: tutoring, medical triage, customer service, career mentoring, and persuasion. It includes five distribution shifts, each changing a single mechanism, and three tasks: multi-step state forecasting, counterfactual action effects, and planning in latent space. Models observe only the text, the agent’s actions and the time gaps; the true hidden states are used only for evaluation. LID-Bench will be released publicly.

Experimental results

We compare three world-modelling paradigms on LID-Bench. Generative world models (GenWM: GPT-2 117M, OPT-125M, OPT-350M) are fine-tuned to predict the interlocutor’s next reply given the history, action, and time gap. Reconstruction (ReconWM) uses the same frozen encoder and predictor as Social-JEPA but is trained to reconstruct the text. Finetuned-Rep, an ablation, keeps Social-JEPA’s architecture but drops the EMA target and multi-step rollout. Every method is scored the same way: a linear probe, trained on the training split, maps its predicted state to the true hidden state. For generative models, each generated reply is first re-encoded by a frozen SimCSE-RoBERTa-Large encoder.

Forecasting interaction dynamics

The benchmark behaves as intended: a reference model given the true hidden states and the full dynamics reaches R² > 0.94 at every horizon, while shortcut baselines fail at k=1 (TF-IDF +0.037; zero-shot Claude 3.5 Sonnet −0.132). Social-JEPA outperforms GenWM GPT-2 by 3.9× at k=1 (+0.128 vs +0.033, p < 10⁻⁷) and stays positive through k=5, while the generative models decay to near zero (Fig. 1a). OPT-125M and OPT-350M both beat GPT-2 at every horizon, yet Social-JEPA still leads by 2.8×: better pretraining and ~3× more parameters do not close the gap.

Where the signal goes

Probing each generative model’s hidden states directly, without decoding text, separates what it learns from what it can express (Fig. 1b). GPT-2 loses 56% of its learned dynamics signal through text rendering at k=1. OPT-125M loses 20% at k=1, growing to 89% at k=5. OPT-350M learns more internally at k=1, but its loss grows from 32% to 91% by k=5: scaling improves single-step encoding, not what survives text. Even read out internally, every generative model falls well below Social-JEPA.

Reconstruction fails differently. ReconWM achieves the best encoding R² of any method (0.196) yet catastrophic dynamics (−0.607 at k=1). Because it shares Social-JEPA’s encoder and predictor, this isolates the cause: the reconstruction objective forces the latent space to encode surface features needed for text, not dynamics. Generative models lose the signal at inference; reconstruction loses it at training. Observation-space coupling is the common failure mode.

Counterfactual reasoning shows the same pattern. Asked how each possible action would change the state, Social-JEPA and the generative models predict the direction of the effect about equally well (sign accuracy 0.607–0.611, against 0.524 for a baseline that ignores the state). But the generative models cannot rank actions by the size of their effect (Spearman ρ at most +0.027, versus +0.330 for Social-JEPA). Coarse directions survive text rendering; fine-grained magnitudes do not.

Robustness under distribution shift

Each of LID-Bench’s five shifts changes exactly one mechanism: delayed action effects, a lower backfire threshold, rare starting states with high arousal and resistance, a different family of transition dynamics, or noisy text (dropped words and inserted filler). At k=5, Social-JEPA remains positive on the in-distribution test set and four of five shifts, while the generative models and Finetuned-Rep are often near zero or negative, and scaling from 125M to 350M does not improve robustness. Noisy text is the exception for Social-JEPA (−0.001 at k=5); diagnostics point to the frozen encoder rather than the dynamics model.

Planning in real time

The practical payoff of latent dynamics is planning. We run the cross-entropy method (CEM: 256 candidate action sequences, 5 iterations, horizon 5) over each latent-space model and report normalised improvement (NI), where random actions score 0 and planning with the true hidden state scores 1. Social-JEPA plans well under a cooperative reward (NI = 0.640) and an adversarial one (0.553) with the same frozen dynamics model; only the reward changes. Under the adversarial reward, the policy that generated the training data scores −0.104, worse than random, so the planner is not copying behaviour it saw in training: it exploits the learned dynamics.

Speed is what makes this usable. A Social-JEPA rollout takes 0.43 ms and a full CEM step 5.7 ms; the same step with a generative world model takes ~628–972 s. Social-JEPA’s complete per-turn cost, encoder plus CEM step, is ~11 ms, under 1.1% of a 1 s conversational turn-taking window [7]. It is the only paradigm where sampling-based planning is compatible with real-time interaction.

Limitations and real dialogue

LID-Bench’s latent states are generator-defined control variables, not real-world ground truth. On KODIS [8], a corpus of real multicultural dispute-resolution dialogues, the gap between what generative models learn internally and what survives in their text replicates in 12/12 runs across all three model families. It holds for three of the four traits that map onto LID-Bench’s coordinates; Honor (mapped to resistance) is the exception. Effect sizes are small, and multi-step rollout and planning are evaluated on LID-Bench only. Absolute R² values are modest (0.128 at k=1, 0.053 at k=5), reflecting how hard it is to infer state from text alone. Planning, however, only needs predicted states to rank action sequences correctly, and R² = 0.053 at k=5 is enough for NI = 0.64.

The same reward-agnostic dynamics that let one model serve cooperative or adversarial objectives could also serve manipulation under a different reward. Because the probe translates latent predictions into the quantities a reward operates on, it is the natural point of governance: deployments should pair this work with consent from those whose states are inferred, audit logs of probe outputs, and reward functions vetted against externally validated objectives.