Interpreting Text-Enriched Dual-Head Multitask Learning for Indonesian Hateful Meme Detection Using Explainable AI

Abstract

Internet memes in Indonesia are frequently weaponized to disseminate implicit
hate speech through sarcasm and cultural nuances. Automatically detecting
such content is computationally challenging, and existing deep learning
frameworks predominantly operate as opaque black boxes, lacking decision
transparency. This study implements and optimizes a text-enriched dual-head
multitask learning architecture utilizing IndoBERTweet to concurrently classify
hatefulness and appropriateness within the INDOMEME dataset. Rather
than processing raw image pixels, we employ a text-enrichment strategy where
visual semantics are transcribed into textual descriptors via Optical Character
Recognition and vision-language captioning. To bridge the interpretability gap,
we deploy Local Interpretable Model-agnostic Explanations (LIME) to decode
the internal feature attributions of the architecture. Furthermore, advanced
training optimizations, encompassing cosine annealing, gradient accumulation,
class-weighted loss, and dynamic threshold calibration, were engineered to
enhance model generalization. Experimental evaluations demonstrate that
the optimized model achieves a Macro-F1 score of 0.812 for hatefulness and
0.820 for appropriateness, surpassing the established baseline. Crucially, the
LIME analysis unveils a pivotal finding: despite sharing an identical textual
backbone, the hate-specific head predominantly focuses on lexicons carrying
social agitation, whereas the appropriateness head prioritizes general norm
violations. These empirical findings substantiate that multitask learning enriches
semantic representation quality, offering a transparent framework for
trustworthy content moderation.