solar-10.7b-instruct-v1.0

SOLAR-10.7B-Instruct-v1.0 is a large language model developed by Upstage, specifically fine-tuned for instruction-following and conversational tasks. It is the instruction-tuned variant of the SOLAR-10.7B base model, which was built using a methodology called depth up-scaling (DUS). This technique involves scaling a 7-billion parameter model into a 10.7-billion parameter structure by duplicating layers and performing continued pre-training to refine the expanded architecture.\n\nThe model's foundation is based on the Llama 2 framework and integrates weights from Mistral 7B. This architectural choice results in a 48-layer transformer model that aims to deliver performance comparable to or exceeding larger models with up to 30 billion parameters. The training process for the instruct version includes supervised fine-tuning (SFT) and alignment using direct preference optimization (DPO) to enhance its response quality and adherence to user prompts.\n\nSOLAR-10.7B-Instruct-v1.0 is optimized for single-turn interactions and shows robust capabilities in reasoning and natural language understanding. It maintains a context window of 4,096 tokens and is designed to provide high-quality output while remaining more computationally efficient than traditional higher-parameter models.