Parameter-Efficient Fine-Tuning in Large Models A Survey of Methodologies

23.04.25
Paper Link: "https://arxiv.org/pdf/2410.19878"

• Methodology used: This paper presents a comprehensive survey of Parameter-Efficient Fine-Tuning (PEFT) methodologies for large models. The authors reviewed and analysed over 100 research articles published between June 2019 and July 2024. They provide a structured taxonomy to classify different PEFT approaches, expanding beyond the traditional additive, reparameterized, and selective categories to include hybrid, quantization, and multi-task PEFT. The survey discusses the core principles and applications of various PEFT techniques across different model architectures and downstream tasks (NLP, vision, multimodal, diffusion). Finally, the paper proposes potential future research directions in the field.

• New things introduced/ Novelty: The novelty lies in providing a more up-to-date and broader review of PEFT methods, encompassing recent advancements up to July 2024. The paper offers an expanded and more detailed taxonomy of PEFT approaches, providing a more structured understanding of the field. It offers a comprehensive coverage of PEFT applications across various types of large models, including those beyond natural language. The survey also contributes by identifying and suggesting promising future research directions in PEFT.

• Key take aways and results: The survey highlights PEFT as an effective approach to adapt large pre-trained models to specific tasks while significantly reducing computational and storage costs compared to full fine-tuning. It provides a detailed overview of various PEFT techniques, including additive methods like adapters and LoRA, reparameterized methods such as prompt tuning, and selective methods like BitFit. The paper also discusses the application of PEFT in multimodal, visual, and diffusion models. Furthermore, it points out key future research directions, such as PEFT for multi-objective learning, continual learning, and privacy-preserving fine-tuning.

• Comparison with State of the Art (SOTA) and how better it is and under what circumstances: This survey builds upon existing surveys by offering a more comprehensive and recent overview of the rapidly evolving field of PEFT. It provides a more granular classification of PEFT techniques and covers a wider scope of applications, including domains beyond natural language processing, reflecting the increasing interest in parameter-efficient adaptation across different types of large models. By highlighting the latest advancements and future directions, this survey aims to provide a more complete and forward-looking perspective on PEFT.

• Drawbacks that are discussed in the paper: As a survey, the paper does not present new empirical results or directly discuss drawbacks of specific PEFT methods in detail. However, by outlining future research directions, it implicitly points to areas where current PEFT techniques might have limitations or require further development, such as handling complex multi-objective tasks or ensuring privacy. The paper also acknowledges the continuous advancements in the field, suggesting that any survey is a snapshot in time.

• Improvements that can be made: As a survey, its main contribution is in its comprehensiveness and structure. Future iterations could benefit from more quantitative comparisons of different PEFT methods across various tasks and model sizes, although the rapidly evolving nature of the field makes this challenging. Including more in-depth discussions of the trade-offs between parameter efficiency, performance, and computational cost for different PEFT techniques would also be valuable. Additionally, providing a more systematic analysis of the limitations of current PEFT methods could further guide future research.