首页 > AI前沿 > Asymmetric Multimodal Learning with Decomposed JEPAs

Asymmetric Multimodal Learning with Decomposed JEPAs

arXiv机器学习 2026-11-12 19:13 23 阅读 查看原文
用户需要我作为一个专业的新闻排版工程师,将给定的纯文本新闻正文排版成语义化的HTML片段。 规则如下: 1. 只输出HTML片段本身,不要Markdown围栏,不要解释。 2. 每个自然段落用`

`包裹,长文用`

`/`

`划分小节。 3. 列表内容用`
  • `,引用用`
    `,代码用`
    `,重要句子可用``。
    4. 保留原文所有信息,不得增删改内容。
    5. 不要输出``/``/``等文档级标签。
    
    正文内容分析:
    这段文本是关于多模态表示学习的学术论文摘要或介绍,包含一些LaTeX标记(如`\emph{sole}`, `\textbf{Alignment Redundancy Bound}`, `$\Red(Z; X_1, X_2)$`等)。
    我需要将其转换为HTML,处理LaTeX标记,使其在HTML中合理呈现,同时不改变原文信息。
    原文:
    Cross-modal alignment is the standard objective for building multimodal representations. We show that alignment as a \emph{sole} objective is lossy wherever modalities are informationally asymmetric---where neither can express what the other carries. Visual question answering is an obvious case: a question is a selective relational operator probing a dense visual field whose fine-grained features no query recovers, so what alignment preserves is not what the task requires. We prove an \textbf{Alignment Redundancy Bound}: alignment-only Dirichlet energy minimization retains at most the redundant information atom $\Red(Z; X_1, X_2)$, annihilating private modality-specific structure $\Unq(Z; X_m)$ and cross-modal synergy $\Syn(Z; X_1, X_2)$. We introduce \textbf{Decomposed JEPA (D-JEPA)}, which factorizes latents into an aligned shared core and private modality-specific fibers. The shared core retains most of the structure alignment targets; the private fibers recover what alignment discards, via contextual masked prediction and asymmetric cross-subspace independence. Under information asymmetry, alignment-only training undergoes catastrophic semantic collapse on compositional reasoning benchmarks, surviving only on binary queries. D-JEPA averts this collapse, substantially improving downstream accuracy over both alignment-only and alignment-with-reconstruction baselines, and restoring a positive multimodal synergy gap in place of the baseline's inverted gap. Information-theoretic audits confirm that D-JEPA recovers most of the mutual information available from a frozen backbone, and causal fiber-swap interventions demonstrate that the recovered information is causally localized in the private fibers.
    
    处理LaTeX和排版:
    - `\emph{sole}` -> `sole` 或 `sole`,用``更语义化。
    - `\textbf{Alignment Redundancy Bound}` -> `Alignment Redundancy Bound`
    - `$\Red(Z; X_1, X_2)$` -> 可以保留为文本或者用``,或者直接原样输出(因为是数学公式)。为了HTML语义,可以直接输出文本 `Red(Z; X_1, X_2)`,或者用 `Red(Z; X_1, X_2)`。考虑到是数学符号,保留原样或者用``都可以。这里直接去掉`$`保留文本内容比较自然,或者用``。题目说代码用` 
               
    `,这里不是代码,所以直接去掉`$`作为普通文本,或者保留`$`?最好去掉`$`,写成 `Red(Z; X_1, X_2)`。
    - `\textbf{Decomposed JEPA (D-JEPA)}` -> `Decomposed JEPA (D-JEPA)`
    - `$\Unq(Z; X_m)$` -> `Unq(Z; X_m)`
    - `$\Syn(Z; X_1, X_2)$` -> `Syn(Z; X_1, X_2)`
    
    原文只有一段,是否需要划分小节?规则说“长文用

    /

    划分小节”,这段其实不算特别长,就是一段完整的摘要。我可以直接用一个`

    `包裹,或者稍微切分一下?不,规则说“每个自然段落用

    包裹”,原文是一个自然段,所以我就用一个`

    `。或者为了体现排版,可以加个`

    `作为标题?原文没有标题,只有正文。那就直接输出`

    ...

    `。 检查原文: Cross-modal alignment is the standard objective for building multimodal representations. We show that alignment as a \emph{sole} objective is lossy wherever modalities are informationally asymmetric---where neither can express what the other carries. Visual question answering is an obvious case: a question is a selective relational operator probing a dense visual field whose fine-grained features no query recovers, so what alignment preserves is not what the task requires. We prove an \textbf{Alignment Redundancy Bound}: alignment-only Dirichlet energy minimization retains at most the redundant information atom $\Red(Z; X_1, X_2)$, annihilating private modality-specific structure $\Unq(Z; X_m)$ and cross-modal synergy $\Syn(Z; X_1, X_2)$. We introduce \textbf{Decomposed JEPA (D-JEPA)}, which factorizes latents into an aligned shared core and private modality-specific fibers. The shared core retains most of the structure alignment targets; the private fibers recover what alignment discards, via contextual masked prediction and asymmetric cross-subspace independence. Under information asymmetry, alignment-only training undergoes catastrophic semantic collapse on compositional reasoning benchmarks, surviving only on binary queries. D-JEPA averts this collapse, substantially improving downstream accuracy over both alignment-only and alignment-with-reconstruction baselines, and restoring a positive multimodal synergy gap in place of the baseline's inverted gap. Information-theoretic audits confirm that D-JEPA recovers most of the mutual information available from a frozen backbone, and causal fiber-swap interventions demonstrate that the recovered information is causally localized in the private fibers. 转换后:

    Cross-modal alignment is the standard objective for building multimodal representations. We show that alignment as a sole objective is lossy wherever modalities are informationally asymmetric---where neither can express what the other carries. Visual question answering is an obvious case: a question is a selective relational operator probing a dense visual field whose fine-grained features no query recovers, so what alignment preserves is not what the task requires. We prove an Alignment Redundancy Bound: alignment-only Dirichlet energy minimization retains at most the redundant information atom Red(Z; X_1, X_2), annihilating private modality-specific structure Unq(Z; X_m) and cross-modal synergy Syn(Z; X_1, X_2). We introduce Decomposed JEPA (D-JEPA), which factorizes latents into an aligned shared core and private modality-specific fibers. The shared core retains most of the structure alignment targets; the private fibers recover what alignment discards, via contextual masked prediction and asymmetric cross-subspace independence. Under information asymmetry, alignment-only training undergoes catastrophic semantic collapse on compositional reasoning benchmarks, surviving only on binary queries. D-JEPA averts this collapse, substantially improving downstream accuracy over both alignment-only and alignment-with-reconstruction baselines, and restoring a positive multimodal synergy gap in place of the baseline's inverted gap. Information-theoretic audits confirm that D-JEPA recovers most of the mutual information available from a frozen backbone, and causal fiber-swap interventions demonstrate that the recovered information is causally localized in the private fibers.

    是否需要加标题?题目只给了“正文如下”,没有标题。所以不加标题,只输出段落。 检查是否修改了内容: - `\emph{sole}` -> `sole` (语义化,不改变文字) - `\textbf{Alignment Redundancy Bound}` -> `Al