stablelm-tuned-alpha-7b

StableLM-Tuned-Alpha-7B is a 7-billion parameter decoder-only language model developed by Stability AI. Released in April 2023 as part of the initial StableLM suite, it is an instruction-tuned version of the StableLM-Base-Alpha-7B, specifically optimized for conversational interactions and chat-based applications. It was designed to demonstrate the potential of open-source models in handling complex natural language tasks with a relatively compact parameter count.\n\nThe model architecture is based on the GPT-NeoX framework, featuring enhancements such as Rotary Positional Embeddings (RoPE). It was pre-trained on a custom dataset containing 1.5 trillion tokens, building upon The Pile to provide a broader knowledge base. For its fine-tuning, the model utilized a combination of five prominent instruction datasets: Stanford Alpaca, GPT4All, Dolly, ShareGPT, and Vicuna.\n\nWith a context length of 4,096 tokens, StableLM-Tuned-Alpha-7B provides a larger window for processing and generating text compared to many contemporaneous open-source models. It was released under a non-commercial CC BY-NC-SA-4.0 license, aimed at researchers and the developer community to explore and refine conversational AI capabilities.