In different multimodal scenarios, it needs to integrate and utilize information across modalities in a specific way based on the demands of the task. Different integration ways between modalities are referred to as “multimodal interaction”. How well a model handles various multimodal interactions largely characterizes its multimodal ability.

In this paper, we introduce MIBench, a comprehensive benchmark designed to evaluate the multimodal interaction capabilities of Large Multimodal Models (LMMs), which formulates each instance as a ($con_v$, $con_t$, $task$) triplet with contexts from vision and text, necessitating that LMMs employ correct forms of multimodal interaction to effectively complete the task. MIBench assesses models from three key perspectives: the ability to source information from vision-centric or text-centric cues, and the ability to generate new information from their joint synergy. Each interaction capability is evaluated hierarchically across three cognitive levels: Recognition, Understanding, and Reasoning. The benchmark comprises over 10,000 vision-text context pairs spanning 32 distinct tasks.

Evaluation of state-of-the-art LMMs show that: (1) LMMs' ability on multimodal interaction remains constrained, despite the scaling of model parameters and training data; (2) they are easily distracted by textual modalities when processing vision information; (3) they mostly possess a basic capacity for multimodal synergy; and (4) natively trained multimodal models show noticeable deficits in fundamental interaction ability. We expect that these observations can serve as a reference for developing LMMs with more enhanced multimodal ability in the future.

MIBench introduces a comprehensive evaluation framework designed to deeply assess the multimodal interaction capabilities of LMMs. It evaluates LMMs on three interaction patterns: Vision-centric, Text-centric, and Synergy, which require models to source information from visual cues, textual cues, or a combination of both modalities, respectively. These patterns are evaluated across three progressive cognitive levels: Recognition, Understanding, and Reasoning.

Within this framework, each evaluation instance is formulated as a ($\textbf{con}_\textbf{v}$, $\textbf{con}_\textbf{t}$, $\textbf{task}$) triplet. For vision- and text-centric tasks, they can be resolved by leveraging cues from the centric modality. We introduce various contexts from another modality to evaluate the model's ability to selectively utilize cues from the target modality, which range from helpful contexts (e.g., correct guidance, concept visualization) to misleading guidance and unrelated contents. For the synergy part, the model is presented with one coupled visual-textual pair with complementary cues, necessitating effective cross-modal collaboration.

To meet these requirements, we first identify suitable tasks based on the proposed taxonomy, and subsequently collect, re-annotate, or synthesize test samples following the carefully designed annotation pipeline. Ultimately, this structured approach culminates in a large-scale benchmark comprising 32 distinct tasks and 2,980 base samples, which are expanded into 10,030 context pairs through diverse contextual variations, enabling a fine-grained analysis of model interaction capabilities.