For instance, as shown in this Figure, when answering the audio-visual question “Which clarinet makes the sound first” for this instrumental ensemble, it requires to locate sounding objects “clarinet” in the audio-visual scenario and focus on the “first” sounding “clarinet” in the timeline. To answer the question correctly, both effective audio-visual scene understanding and spatio-temporal reasoning are essentially desired.

Audio-visual question answering requires auditory and visual modalities for multimodal scene understanding and spatiotemporal reasoning. For example, when we encounter a complex musical performance scene involving multiple sounding and nonsounding instruments above, it is difficult to analyze the sound first term in the question by VQA model that only considers visual modality. While if we only consider the AQA model with mono sound, the left or right position is also hard to be recognized. However, we can see that using both auditory and visual modalities can answer this question effortlessly.

Basic informations

We choose to manually collect amounts of musical performance videos from YouTube. Specifically, 22 kinds of instruments, such as guitar, cello, and xylophone, are selected and 9 audio-visual question types are accordingly designed, which cover three different scenarios, i.e., audio, visual and audio-visual. Annotations are collected using a novel by our GSAI-Labeled system.

Characteristics

• 3 typical multimodal scene
• 22 kinds of instruments
• 4 categories: String, Wind, Percussion and Keyboard.
• 9,290 videos for over 150 hours
• 7,423 real videos
• 1,867 synthetic videos
• 9 audio-visual question types
• 45,867 question-answer pairs
• Diversity, complexity and dynamic

Personal data/Human subjects

Videos in MUSIC-AVQA are public on YouTube, and annotated via crowdsourcing. We have explained how the data would be used to crowdworkers. Our dataset does not contain personally identifiable information or offensive content.

Data and Download

---

Raw videos:

• Google Drive
• Baidu Drive (password: cvpr)
- Real videos (36.67GB)
- Synthetic videos (11.59GB)
Note: Please move all downloaded videos to a folder, for example, create a new folder named MUSIC-AVQA-Videos, which contains 9,288 real videos and synthetic videos.

Raw videos frames (1fps): Available at Baidu Drive (14.84GB) (pwd: cvpr) or Google Drive (coming soon!). In fact, we thought it might be more convenient to execute the code above to extract video frames.

Features (VGGish, ResNet18 and R(2+1)D):

• VGGish feature shape: [T, 128]  , Download from  Baidu Drive (pwd: cvpr) or Google Drive, (~112.7M)
• ResNet18 feature shape: [T, 512]  , Download from  Baidu Drive (pwd: cvpr) or  Google Drive, (~972.6M)
• R(2+1)D feature shape: [T, 512]  , Download from  Baidu Drive (pwd: cvpr) or Google Drive, (~973.9M)

Annotations (QA pairs, etc.): Available for download at GitHub.

How to read the annotation files?

The annotation files are stored in JSON format. Each annotation file contains seven different keyword: "video_id", "question_id", "type", "question_content", "templ_values", "question_deleted" and "anser". Below, we present a detailed explanation of each keyword.

• "type": the question's modality information and type.
• "question_id": the unique identifier to QA pairs. .
• "video_id", "question_content", "templ_values" and "anser": The contents of these keywords together construct the Q-A pairs corresponding to the video with id "video_id". The form of <FL> is the template word in the question, and its specific content is the information contained in "templ_values". See the paper for a more specific question template description
• "question_deleted": The check code during data annotation process and it will not be used in the dataloader.

Below, we show an example entry from the JSON file:

        
        {
          "video_id": "00000272",
          "question_id": 50,
          "type": "[\"Audio-Visual\", \"Temporal\"]",
          "question_content": "Where is the <FL> sounding instrument?",
          "templ_values": "[\"first\"]",
          "question_deleted": 0,
          "anser": "right"
        }
        

The example shows the information and annotations related to the QA pairs corresponding to the video with id "video_id". As noted, we assign the unique identifier "50" to that QA pairs. From the entry, we can retrieve the video name, questions, answer and type where the QA pairs belongs. For example, the example above needs to combine audio and visual modality information to make a correct answer to this temporal question.

Publication(s)

If you find our work useful in your research, please cite our CVPR 2022 paper.

        
        @ARTICLE{Li2022Learning,
          title={Learning to Answer Questions in Dynamic Audio-Visual Scenarios},
          author={Guangyao li, Yake Wei, Yapeng Tian, Chenliang Xu, Ji-Rong Wen, Di Hu},
          journal   = {IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},
          year      = {2022},
          
        }
        

Disclaimer

The released MUSIC-AVQA dataset is curated, which perhaps owns potential correlation between instrument and geographical area. This issue warrants further research and consideration.

Copyright

All datasets and benchmarks on this page are copyright by us and published under the Creative Commons Attribution-NonCommercial 4.0 International License. This means that you must give appropriate credit, provide a link to the license, and indicate if changes were made. You may do so in any reasonable manner, but not in any way that suggests the licensor endorses you or your use. You may not use the material for commercial purposes.