【论文笔记】NEFTune: Noisy Embeddings Improve Instruction Finetuning

基本信息

标题: NEFTune: Noisy Embeddings Improve Instruction Finetuning
作者: Neel Jain, Ping-yeh Chiang, Yuxin Wen, John Kirchenbauer, Hong-Min Chu, Gowthami Somepalli, Brian R. Bartoldson, Bhavya Kailkhura, Avi Schwarzschild, Aniruddha Saha, Micah Goldblum, Jonas Geiping, Tom Goldstein
发表: ICLR 2023
arXiv: https://arxiv.org/abs/2310.05914

基本信息

摘要

我们表明，通过简单的增强，语言模型微调可以得到改进，有时甚至可以显著提高。

NEFTune在训练过程中向嵌入向量添加噪声。

使用Alpaca对LLaMA-2-7B进行标准微调，在AlpacaEval上的成绩为29.79%，而使用带噪声的嵌入时，成绩上升至64.69%。NEFTune在现代指令数据集上也优于强大的基线。

使用Evol-Instruct训练的模型提高了10%，ShareGPT提高了8%，OpenPlatypus提高了8%。

即使是经过RLHF进一步精炼的强大模型，如LLaMA-2-Chat，也能从NEFTune的额外训练中受益。

简介

AlpacaEval Win Rate percentage for LLaMA-2-7B models finetuned on various datasets with and without NEFTune

AlpacaEval中LLaMA-2-7B模型在各种数据集上经过微调（含和不含NEFTune）的胜率百分比。

NEFTune在这些数据集上均带来了巨大的性能提升，展示了生成答案的对话质量提升。

NEFTune: Noisy Embedding Instruction Fine Tuning

指令模型（Instruction models）是在包含指令和响应配对的数据集上训练的。每一步 NEFTune 的训练过程通过从数据集中抽取一个指令，并将其token转换为嵌入向量开始。NEFTune 的不同之处在于，它在标准训练基础上向嵌入向量中添加了一个随机噪声向量。该噪声通过抽取独立同分布（iid）的均匀分布值生成，范围为[−1,1][-1, 1]。然后将整个噪声向量按因子α/Ld\alpha / \sqrt{Ld} 进行缩放，其中LL 是序列长度，dd 是嵌入维度，α\alpha 是一个可调参数。

这种缩放规则来源于对抗性机器学习（adversarial ML）的相关文献，并产生了一个欧几里得期望幅度约为α/3\alpha / \sqrt{3} 的随机向量。算法 1 详细描述了我们的方法。

Algorithm 1 NEFTune: Noisy Embedding Instruction Finetuning

实验

AlpacaEval Win Rate versus Text-Davinci-003 for LLaMA-2 trained on different datasets, using GPT-4 as the evaluator, showing an average improvement of 15% across all datasets

AlpacaEval Win Rate with and without NEFTune on LLaMA-2, LLaMA-1, and OPT across Alpaca, Evol-Instruct, ShareGPT and OpenPlatypus datasets. Performance improves across different datasets and models with ChatGPT as the evaluator

OpenLLM Leaderboard tasks with and without NEFTune on LLaMA-2 across Alpaca, Evol-Instruct, and OpenPlatypus datasets and LLaMA-1 trained on Evol-Instruct. We observe that performance does not change across datasets and models

LLaMA-2-Chat (7B), LLaMA-2 (13B), and LLaMA-2 (70B) can be finetuned further to improve performance

AlpacaEval Win Rate (ChatGPT Eval.) reported across different datasets and model sizes

Left: training loss on the Alpaca dataset for models with and without NEFT, computed with no added noise. Training with NEFT yields a higher training loss. Right: loss of the same model, but evaluated on the “test” Evol-Instruct dataset. NEFT yields slightly lower loss.

Left shows the ROUGE-L of training with and without NEFT. Right shows BLEU score

(Row 1) Avg. Character lengths of AlpacaEval responses from LLaMA-2 models finetuned on different datasets. We also report average output length for each dataset (though we trained with max sequence length of 512). NEFT increases average length. (Row 2) Whitespace-tokenized lengths of generations. (Row 3) 2-Gram repetition rates. (Row 4) Log-Diversity measures.

We use the following meta-prompts to get longer responses: “Generate a long response”, “Generate a comprehensive response”, and “Generate a long and comprehensive response.” Longer responses score better, but do not close the gap with NEFT

Win Rate (and Avg. Character Length) on AlpacaEval as evaluated by ChatGPT for different levels and types of training noise. While length does increase with noise, it is not always indicative of AlpacaEval Win Rate.

总结与局限

NEFTune的成功指出了算法和正则化在LLM训练中常被忽视的重要性。与已经研究了多年正则化和过拟合的计算机视觉社区不同，LLM社区倾向于使用旨在优化器稳定性的标准化训练循环，而不是泛化。在这种环境中，LLM研究人员已经将数据集和模型扩展作为前进的主要途径。鉴于NEFTune的持续收益以及在小指令数据集上过度拟合的趋势，似乎在LLM环境中重新审视正则化是值得的。

我们的研究有几个局限性。我们将AlpacaEval作为LLM指令跟随能力的核心衡量标准，这受到单个裁判（GPT-4）的偏见。此外，由于计算资源有限，我们无法在多个数据集上验证NEFTune在70B变体上的成功，并且大多数NEFTune运行必须依赖于固定的超参数而不是扫描。最后，尽管我们进行了实证研究，但我们并没有对NEFTune为何有效有一个明确的了解。