Going with the Flow: Koopman Behavioral Models as Pseudo Planners for Visuo-Motor Dexterity

We introduce Unified Behavioral Models (UBMs), a framework for representing dexterous visuo-motor skills as coupled dynamical systems between visual flow and robot action flow. Our first instantiation, Koopman-UBM, models these behavioral dynamics as a structured latent linear system, enabling smooth long-horizon execution, efficient inference, and online replanning when observed visual flow deviates from prediction. Across simulated and real-world dexterous manipulation tasks, the method matches or outperforms strong diffusion- and transformer-based baselines while offering faster and more coherent execution.