Publications

Mnemis: Dual-Route Retrieval on Hierarchical Graphs for Long-Term LLM Memory

Zihao Tang, Xin Yu, Ziyu Xiao, Zengxuan Wen, Zelin Li, Jiaxi Zhou, Hualei Wang, Haohua Wang, Haizhen Huang, Weiwei Deng, Feng Sun, Qi Zhang

Accepted by ACL 2026.

AI memory is increasingly important for long-term interactions with large language models, but existing RAG and Graph-RAG methods mainly rely on similarity-based retrieval. We propose Mnemis, a dual-route memory framework that combines fast System-1 similarity search with a System-2-style global selection mechanism over hierarchical memory graphs. By jointly retrieving semantically and structurally relevant memories, Mnemis improves long-term memory reasoning and achieves state-of-the-art performance on LoCoMo and LongMemEval-S benchmarks.

Going with the Flow: Koopman Behavioral Models as Pseudo Planners for Visuo-Motor Dexterity

Yunhai Han, Jiaqi Fu, Linhao Bai, Ziyu Xiao, Zhaodong Yang, Yogita Choudhary, Krishna Jha, Chuizheng Kong, Shreyas Kousik, Harish Ravichandar

We introduce Unified Behavioral Models (UBMs), a framework for representing dexterous visuo-motor skills as coupled dynamical systems between visual flow and robot action flow. Our first instantiation, Koopman-UBM, models these behavioral dynamics as a structured latent linear system, enabling smooth long-horizon execution, efficient inference, and online replanning when observed visual flow deviates from prediction. Across simulated and real-world dexterous manipulation tasks, the method matches or outperforms strong diffusion- and transformer-based baselines while offering faster and more coherent execution.

Video2Sim2Real: Full-Stack Autonomous Dexterous Skill Acquisition from a Single Human Video

Yunhai Han, Jianuo Qiu, Linhao Bai, Ziyu Xiao, Zihang Zeng, Yangcen Liu, Zhaodong Yang, Shalin Jain, Wenrui Ma, Jiaqi Fu, Yuqian Zheng, Manisha Natarajan, Muhammad Zubair Irshad, Kenneth Shaw, Matthew Gombolay, Zsolt Kira, Harish Ravichandar

We present Video2Sim2Real, a full-stack framework for learning dexterous robot skills from a single human manipulation video. The system reconstructs a simulator-ready digital twin, extracts object-centric keyframes and motion priors, optimizes robot configurations in simulation, and bridges the sim-to-real gap through imitation learning, residual reinforcement learning, and collision-aware motion planning. Across everyday manipulation tasks, Video2Sim2Real improves simulated success, safety, trajectory coherence, and real-world transfer over prior approaches.