Datasets

To further investigate the effectiveness of imbalance methods, we build a balanced audiovisual dataset containing samples with various modality discrepancies, and the allocation of such modality discrepancy remains uniformly distributed over the dataset.

To explore scene understanding and spatio-temporal reasoning over audio and visual modalities, we build a largescale audio-visual dataset, MUSIC-AVQA, which focuses on question-answering task. As noted above, high-quality datasets are of considerable value for AVQA research.

To our knowledge, the audiovisual aerial scene recognition task has not been explored before. For further facilitating the research in this field, we construct a new dataset, with high-quality images and scene labels, named as ADVANCE, which in summary contains 5075 pairs of aerial images and sounds, classified into 13 classes.

To jointly utilize ambient sounds and visual contexts for crowd counting, we construct the auDIoviSual CrOwd dataset (“DISCO” for short), which consists of 1,935 images and audios from various typical scenes, a total of 170, 270 instances annotated with the head locations.

We build category-balanced multi-source videos by artificially synthesizing solo videos from the MUSIC and VGGSound dataset to facilitate the learning and evaluation of multiple-soundings-sources localization in the cocktail-party scenario.

To further evaluate our model in real-world scenes, we collect real multi-source instrument videos and real multi-source daily life scenes on YouTube. Each instrument video includes 3 to 4 instruments and each daily life scene includes 2 events Because of the high noise of the real data, we perform manual labeling for bounding box annotation.

Shuttersong is a social sharing software, just like Instagram2. We collect almost the entire updated data from Shuttersong, which consists of 36,646 pairs of images and song clips.