The project integrates advanced robotic hardware, real-time control algorithms, multimodal perception, and AI-driven human–robot interaction. Two representative systems have been developed: a flexible endoscope robot for intraoperative visualization and a robotic scrub nurse for instrument handling.
Intelligent Surgical Robotic Assistant with Multimodal-Al Perception and Interaction
Researchers
Introduction
The Main Impact
1
A key contribution of this project is a quadratic programming (QP)-based control framework solved by a finite-settling-time adaptive neural network (FST-ANN), enabling millisecond-level convergence with high robustness. This ensures stable visual servoing, accurate motion control, and effective constraint handling such as remote center-of-motion and joint limits. In parallel, a multimodal AI framework combining vision, speech recognition, and large language models (LLMs) enables intuitive, real-time surgeon–robot interaction.
2
To support perception, the project also introduces the Gastro28 dataset and an interactive annotation system (ISSAS), significantly reducing labeling effort while improving segmentation accuracy. Extensive simulations, preclinical experiments, and cadaveric studies demonstrate the effectiveness of the proposed systems. Overall, the CRF project advances the development of safe, efficient, and human-like surgical robotic assistants for future clinical deployment.