bibtext: @inproceedings{
  zhou2024enhancing,
  title={Enhancing Language Models with Idiomatic Reasoning},
  author={Jianing Zhou and Ziheng Zeng and Hongyu Gong and Suma Bhat},
  booktitle={First Conference on Language Modeling},
  year={2024},
  url={https://openreview.net/forum?id=2wtj0up8rv}
  }
abstract: Advancements in Large Language Models (LLMs) have significantly propelled the field of Natural Language Processing (NLP); however, nuanced reasoning in the presence of non-canonical language forms, such as figurative language, remains an intricate challenge. These language forms, integral to human communication, elude standard LLM comprehension due to their inherent non-compositionality, contextual ambiguity, and sparse representation in text corpora. Addressing these challenges, this paper introduces an innovative approach to seamlessly incorporate idiomatic knowledge into pre-trained language models (PTLMs). Our methodology first employs a multi-view data augmentation strategy that uses idiomatic instances representing one property to generate training data for various idiom-related tasks. When combined with a novel parameter-efficient tuning mechanism that accounts for the unique attributes of idiomatic language, we embed task-specific and idiomaticity-aware inductive biases within a PTLM.  Integrating a meta-pretraining protocol based on meta-learning principles, further equips the model with enhanced adaptability to diverse downstream idiom-aware tasks. Empirical validation on diverse benchmarks centered around idiom comprehension and reasoning, demonstrates the efficacy of our approach. Notably, our model surpasses various parameter-efficient fine-tuning baselines outperforming the conventional full fine-tuning paradigms, thereby creating more contextually aware and linguistically robust language models.
title: Enhancing Language Models with Idiomatic Reasoning
authors: Jianing Zhou Ziheng Zeng Hongyu Gong Suma Bhat