Memes often derive their harmful, hateful, or sarcastic meaning from small but decisive visual, textual, or cross-modal cues.
Existing multimodal classifiers can miss such evidence when relying mainly on global image-text representations.
We introduce MemeCF
We introduce MemeCF, a cue-focused benchmark of 9,895 memes across harm, hate, and sarcasm, with annotations identifying the modality and rationale of the pivotal evidence.
We also propose MemePIVOT
We also propose MemePIVOT, a local-global architecture for meme classification.
MemePIVOT uses frozen CLIP features, unbalanced optimal transport to align words with image patches while allowing irrelevant evidence to remain unmatched, and an evidential fusion head to combine local grounding with global meme context under uncertainty.
Experiments
Experiments on HarMeme, PrideMM, and MemeCF show consistent gains over strong text-only, image-only, multimodal, and vision-language baselines.
Results
Cross-dataset and ablation results further show that explicit pivotal-evidence modeling improves robustness and contributes meaningfully beyond global multimodal representations.
Availability
Our code and dataset are publicly available at https://github.com/AkshitSharma1/MemePIVOT.