bibtext: @inproceedings{
  wu2024agentdocedit,
  title={Agent-DocEdit: Language-Instructed {LLM} Agent for Content-Rich Document Editing},
  author={Te-Lin Wu and Rajiv Jain and Yufan Zhou and Puneet Mathur and Vlad I Morariu},
  booktitle={First Conference on Language Modeling},
  year={2024},
  url={https://openreview.net/forum?id=1ba209BACA}
  }
abstract: Editing content-rich and multimodal documents, such as posters, flyers, and slides, can be tedious if the edits are complex, repetitive, or require subtle skills and deep knowledge of the editing software.
  Motivated by recent advancements in both Large Language Model (LLM) agents and multimodal modeling, we propose a framework that automates document editing which takes as input a linguistic edit request from the user and then performs sequential editing actions to the document the satisfy the request.
  Our proposed method, Agent-DocEdit, first grounds the edit request directly in the underlying document structure to identify the elements that need to be manipulated. Then, we rely on the agent capabilities of LLMs to generate an edit program which calls a set of pre-defined APIs to modify the underlying structure of the document.
  To improve the generated edit program, we leverage a feedback mechanism incorporating a deterministic code executor and a multimodal LLM.
  We demonstrate the effectiveness of our proposed modularized LLM editing agent on the DocEdit dataset, where Agent-DocEdit outperforms existing state-of-the-art methods by 70+% in document element grounding and 16+% on final rendition generation.
title: Agent-DocEdit: Language-Instructed LLM Agent for Content-Rich Document Editing
authors: Te-Lin Wu Rajiv Jain Yufan Zhou Puneet Mathur Vlad I Morariu