Datasets

In this work, we introduce a novel task called Reference Audio-Visual Segmentation (Ref-AVS), which seeks to segment objects within the visual domain based on expressions containing multimodal cues.

To achieve a better understanding of long form audio-visual videos, we propose to focus on the multisensory temporal event localization task, which essentially requires the model to predict the start and end time of each audio and visual event in the video. Concretely, we divide the video into several non-overlapping snippets, then predict the event categories of all snippets. To study the proposed multisensory temporal event localization task, we elaborately build a large-scale Long Form Audio-visual Video (LFAV) dataset with an average video length of 210 seconds and a total video length of 302 hours.

To further investigate the effectiveness of imbalance methods, we build a balanced audiovisual dataset containing samples with various modality discrepancies, and the allocation of such modality discrepancy remains uniformly distributed over the dataset.

To explore scene understanding and spatio-temporal reasoning over audio and visual modalities, we build a largescale audio-visual dataset, MUSIC-AVQA, which focuses on question-answering task. As noted above, high-quality datasets are of considerable value for AVQA research.

To our knowledge, the audiovisual aerial scene recognition task has not been explored before. For further facilitating the research in this field, we construct a new dataset, with high-quality images and scene labels, named as ADVANCE, which in summary contains 5075 pairs of aerial images and sounds, classified into 13 classes.

To jointly utilize ambient sounds and visual contexts for crowd counting, we construct the auDIoviSual CrOwd dataset (“DISCO” for short), which consists of 1,935 images and audios from various typical scenes, a total of 170, 270 instances annotated with the head locations.

We build category-balanced multi-source videos by artificially synthesizing solo videos from the MUSIC and VGGSound dataset to facilitate the learning and evaluation of multiple-soundings-sources localization in the cocktail-party scenario.

To further evaluate our model in real-world scenes, we collect real multi-source instrument videos and real multi-source daily life scenes on YouTube. Each instrument video includes 3 to 4 instruments and each daily life scene includes 2 events Because of the high noise of the real data, we perform manual labeling for bounding box annotation.

Shuttersong is a social sharing software, just like Instagram2. We collect almost the entire updated data from Shuttersong, which consists of 36,646 pairs of images and song clips.