phi-3-medium-4k-instruct

Phi-3-medium-4k-instruct is a 14 billion parameter language model developed by Microsoft, part of the Phi-3 family of small language models (SLMs). It is a dense decoder-only transformer designed to deliver high-performance reasoning, mathematics, and coding capabilities within a relatively compact footprint. This version is instruction-tuned and supports a context window of 4,096 tokens.

The model was trained on a 4.8 trillion token dataset comprising high-quality synthetic data and filtered public web content. Microsoft's training approach for the Phi-3 series focuses on "textbook-quality" data to instill strong logical foundations and general knowledge. After pre-training, the model underwent a post-training phase involving supervised fine-tuning (SFT) and direct preference optimization (DPO) to enhance its ability to follow instructions and adhere to safety guidelines.

Architecture and Performance

Phi-3-medium uses a similar architecture to the Llama series and utilizes the same 32,064-token vocabulary tokenizer as the Phi-3-mini variant. In various benchmarks, it has demonstrated reasoning capabilities that rival significantly larger models, particularly in tasks requiring logical inference and technical problem-solving. It is optimized for environments where memory and compute resources are constrained compared to those required by massive large language models (LLMs).