Towards Self-Growing Worlds, Self-Taught Agents
COM2 Level 4
Executive Classroom, COM2-04-02
Abstract:
Today, virtual worlds are built by hand and embodied agents are taught by hand: one scene per artist, one skill per dataset. Skills learned in simulation also tend to stay there. In this talk, I will present the research vision of the IIIXR Lab at Korea University: seed virtual worlds from reality, let agents teach themselves inside them, and bring both back into the real world. The talk has three parts. In Create, I will cover worlds that are created from minimal input: generative 3D scenes and language-based scene editing, text-to-motion and camera trajectory generation, efficient Gaussian splatting and neural global illumination, and simulation-ready assets built from scans and video. In Run, I will discuss agents that learn skills with expert demonstrations or without demonstrations, from dexterous hand manipulation to vision-language-action agents, world models that let agents imagine outcomes before acting, dynamics learned as executable code, and uncertainty-aware cooperative planning. In Bridge, I will show how these results transfer to physical AI, including robot hands, humanoids, and vehicles, through counterfactual data generation and predict-before-you-act safety.
Bio:
HyeongYeop Kang is an Associate Professor in the Department of Computer Science and Engineering at Korea University, where he founded the IIIXR Lab in 2021 and has directed it since. His research aims to build virtual worlds that grow themselves and embodied agents that teach themselves, and to transfer both back into the real world. His work spans neural computer graphics and 3D generation, embodied agents with vision-language-action models, world models, extended reality and interaction, and physical AI. His group has recently published at SIGGRAPH, SIGGRAPH Asia, NeurIPS, CVPR, ICLR, AAAI, and IEEE VR. He received his Ph.D. from Korea University in 2017.