Alpaca-13B is a 13-billion-parameter language model developed by the Stanford Center for Research on Foundation Models (CRFM). It is an instruction-tuned version of Meta's LLaMA-13B foundation model, designed to follow natural language instructions. The project was released in March 2023 to provide a replicable and accessible model for academic research into instruction-following behaviors.
The model was trained using supervised fine-tuning on a dataset of 52,000 instruction-following demonstrations. These demonstrations were generated using the Self-Instruct method, where OpenAI's text-davinci-003 was prompted to generate new instructions and corresponding outputs based on a small seed set of human-written examples. This approach allowed the researchers to create a high-quality instruction-following model with a significantly reduced training budget compared to traditional methods.
Stanford's evaluation indicated that Alpaca-13B performed qualitatively similarly to proprietary models like text-davinci-003 on a range of user-oriented tasks, including creative writing and summarization. Its release sparked significant interest in the open-source community, leading to the development of efficient fine-tuning techniques and numerous derivative models. Due to the terms of the underlying LLaMA model and the OpenAI data used for training, Alpaca-13B is restricted to non-commercial, research use only.