Sparse Autoencoders Reveal Interpretable and Steerable Features in VLA Models

An SAE feature is a learned latent direction that activates sparsely to help reconstruct a model's internal activations. Concretely, the SAE learns a dictionary where each feature corresponds to one basis direction in activation space, and a given activation vector is represented by a sparse linear combination of these features. In LLMs, many features are strikingly interpretable and can align with localized concepts, but this is not guaranteed. Because there is no theoretical guarantee that a learned direction corresponds to a single human concept, we treat interpretability as an empirical property. We evaluate it by (i) inspecting the highest-activating episodes/timesteps and checking whether the activating contexts share a coherent explanation, and (ii) performing interventions: steering along a feature direction and testing whether it produces predictable, causal changes in model outputs. We assess robustness by training multiple SAEs with different random seeds and verifying that common features reappear.

PCA is a global linear decomposition optimized for variance, not interpretability or sparsity. In our preliminary research, we found that PCA decompositions of the activations were not interpretable with respect to episode, task, sub-task, or episode completion. The same applies to inspecting MLP weights directly, as these often exhibit a high degree of superposition among concepts. While linear probes are an excellent choice when supervision data is available, the unsupervised nature of SAEs makes them superior for our exploratory purposes. Our approach is ultimately pragmatic; we aim to use SAEs to identify properties of VLAs that warrant further study.

SAEs were popularized as a tool for mechanistic interpretability of transformers, but the approach is not language-specific: it is a generic dictionary-learning method over activation vectors. Recent work has extended SAE analysis to multimodal settings, and emerging VLA interpretability work already uses SAE-like interventions to steer action-level behavior. Traditionally, SAEs are trained on a per-token level. However, we find that, for robotics data, the natural base unit is the timestep, which spans many tokens in a VLA model.

Mechanistic interpretability can be important even when it does not immediately increase policy performance. With VLAs, especially in their applications to home robotics, safety must go far beyond collision avoidance and stability to a more contextual safety. Understanding how VLAs function is paramount to building the trust required to place them into people's homes. Our feature quantification pipeline allows us to evaluate the generality of a model's features without ever running a single rollout. In this way, we hope interpretability can guide future improvements, even when the immediate goal is understanding rather than performance.