首页 > AI前沿 > NYUAD at ImageEval Shared Tasks: A Multimodal LLM for Audio-Visual Question Answering and Hallucination Detection in English and Modern Standard Arabic

NYUAD at ImageEval Shared Tasks: A Multimodal LLM for Audio-Visual Question Answering and Hallucination Detection in English and Modern Standard Arabic

arXiv自然语言 2026-11-17 13:00 18 阅读 查看原文
用户需要我作为一个专业的新闻排版工程师,将提供的纯文本新闻正文排版成语义化的HTML片段。 需要遵循的规则: 1. 只输出HTML片段本身,不要Markdown围栏、不要解释。 2. 每个自然段落用`

`包裹,长文用`

`/`

`划分小节。 3. 列表内容用`
  • `,引用用`
    `,代码用`
    `,重要句子可用``。
    4. 保留原文所有信息,不得增删改内容。
    5. 不要输出``/``/``等文档级标签。
    
    分析提供的正文:
    正文是一段关于多模态大语言模型(MLLMs)在阿拉伯语视听问答(AVQA)和幻觉检测方面的论文摘要/新闻。
    内容包含:
    - 背景与机会:MLLMs使得视觉和语音信息交互更自然,为AVQA和幻觉检测创造机会。
    - AVQA的定义与挑战:AVQA扩展了传统VQA,需要联合理解图像和口语问题。在现代标准阿拉伯语(MSA)中,由于多模态数据集有限及跨模态推理需求,仍具挑战性。
    - 本文方法:提出基于MLLM的AVQA框架,输入图像和口语问题,生成答案;并利用该框架检测图文对的幻觉。
    - 评估与结果:在ImageEval 2026 Task (Ayn-VQA)基准上评估,该基准聚焦文化根基的阿拉伯语AVQA。实验结果表明框架有效理解口语查询,将回答基于视觉内容,并产生准确答案。结果强调了不同LLM架构在英语和MSA及不同条件下的能力与局限。
    
    这段文本可以划分为几个小节,或者直接作为几个段落。为了符合“长文用`

    `/`

    `划分小节”的规则,我可以将其合理划分为几个小节,比如: 1. 背景与挑战 (Background and Challenges) 2. 方法与框架 (Proposed Framework) 3. 评估与结论 (Evaluation and Results) 但是规则4要求“保留原文所有信息,不得增删改内容”。如果我添加小标题,算不算“增删改内容”? 通常,添加HTML标签(如`

    `)是排版的一部分,但如果小标题本身是新加的文字,就可能违反“不得增删改内容”。为了严格遵守“不得增删改内容”,我不应该添加任何原文没有的文字作为标题。 不过,规则2说“长文用`

    `/`

    `划分小节”,如果我不添加文字,怎么划分小节呢? 或者,我可以把原文的某句话作为小节标题?但原文没有明显的标题。 或者,我可以将原文的几个自然段落分别用`

    `包裹,因为原文本身就是由几个句子组成的一个长段落,或者我可以按逻辑拆分成几个`

    `,不使用`

    `/`

    `,因为原文没有提供可以作为标题的文本。 等一下,如果原文没有标题,我强行加标题就违反了“不得增删改内容”。因此,最好的做法是直接将原文按语义拆分成几个`

    `,对于其中的重要句子使用``。 或者,我可以把第一句话作为背景,第二句话作为AVQA定义,等等。 让我们仔细看看原文: "The increasing adoption of multimodal large language models (MLLMs) has enabled more natural interaction with visual and spoken information, creating new opportunities for intelligent audio-visual question answering (AVQA) and hallucination detection systems. AVQA extends traditional visual question answering by requiring models to jointly understand an input image and a spoken question to generate accurate, context-aware responses. Despite recent advances, AVQA and hallucination detection tasks remain challenging for modern standard Arabic (MSA) due to the limited availability of multimodal datasets and the need to reason across Arabic speech and text modalities alongside visual inputs. In this paper, we present an MLLM-based framework for AVQA that takes an image and a spoken question as input and generates answers. In addition, we utilized this framework to detect hallucination in text-image pairs. The proposed approach is evaluated on the ImageEval 2026 Task (Ayn-VQA) benchmark, which focuses on culturally grounded Arabic AVQA. Experimental results demonstrate that the proposed framework effectively understands spoken queries, grounds their responses in the visual content, and produces accurate answers. The results underscore the capacity and limitations of different LLM architectures for both English and MSA and under various conditions." 我可以将其拆分为以下几个`

    `: 段落1(背景与AVQA定义):

    The increasing adoption of multimodal large language models (MLLMs) has enabled more natural interaction with visual and spoken information, creating new opportunities for intelligent audio-visual question answering (AVQA) and hallucination detection systems. AVQA extends traditional visual question answering by requiring models to jointly understand an input image and a spoken question to generate accurate, context-aware responses.

    段落2(挑战):

    Despite recent advances, AVQA and hallucination detection tasks remain challenging for modern standard Arabic (MSA) due to the limited availability of multimodal datasets and the need to reason across Arabic speech and text modalities alongside visual inputs.

    段落3(方法):

    In this paper, we present an MLLM-based framework for AVQA that takes an image and a spoken question as input and generates answers. In addition, we utilized this framework to detect hallucination in text-image pairs.

    段落4(评估与结果):

    The proposed approach is evaluated on the ImageEval 2026 Task (Ayn-VQA) benchmark, which focuses on culturally grounded Arabic AVQA. Experimental results demonstrate that the proposed framework effectively understands spoken queries, grounds their responses in the visual content, and produces accurate answers. The results underscore the capacity and limitations of different LLM architectures for both English and MSA and under various conditions.

    这样拆分完全保留了原文的所有信息,没有增删改任何文字,只是用`

    `包裹了每个自然段落(将长段落拆分为几个逻辑上自然的段落),并用``强调了重要句子。 关于规则2“长文用`

    `/`

    `划分小节”,由于原文没有提供小节标题文本,添加标题文字会违反规则4“不得增删改内容”。因此,不使用`

    `/`

    `是更稳妥的选择,或者我可以把第一句话的某部分作为标题?不,那也改变了文本结构。直接使用`

    `和``即可。 再检查一下规则: 1. 只输出HTML片段本身,不要Markdown围栏、不要解释。 -> 好的。 2. 每个自然段落用`

    `包裹,长文用`

    `/`

    `划分小节。 -> 原文是一整段,我将其拆分为几个自然段落并用`

    `包裹。因为没有现成标题,不强行造标题。 3. 列表内容用`

    • `,引用用`
      `,代码用`
      `,重要句子可用``。 -> 没有列表、引用、代码,使用了``。
      4. 保留原文所有信息,不得增删改内容。 -> 严格遵守。
      5. 不要输出``/``/``等文档级标签。 -> 好的。
      
      微调段落拆分,确保自然:
      

      The increasing adoption of multimodal large language models (MLLMs) has enabled more natural interaction with visual and spoken information, creating new opportunities for intelligent audio-visual question answering (AVQA) and hallucination detection systems.

      AVQA extends traditional visual question answering by requiring models to jointly understand an input image and a spoken question to generate accurate, context-aware responses. Despite recent advances, AVQA and hallucination detection tasks remain challenging for modern standard Arabic (MSA) due to the limited availability of multimodal datasets and the need to reason across Arabic speech and text modalities alongside visual inputs.

      In this paper, we present an MLLM-based framework for AVQA that takes an image and a spoken question as input and generates answers. In addition, we utilized this framework to detect hallucination in text-image pairs.

      The proposed approach is evaluated on the ImageEval 2026 Task (Ayn-VQA) benchmark, which focuses on culturally grounded Arabic AVQA.