The National Natural Science Foundation of China

Human could perceive the environments around and perform effectively and efficiently with multi-sensory information, and we hope the machines with multiple sensors could have the same ability to understand the world, which is the important foundation for the general artificial intelligence. Thus, this project focus on the multi-modal perception and understanding with the motivation of cognitive neuroscience, and want to resolve these problem below:

• Audio-Visual perception in complex and heterogeneous scenarios
• The visual knowledge construction with the guidance of auditory information
• Unconstrained audio localization and separation with visual semantic knowledge