Phi-4 Multimodal Instruct is a 5.6 billion parameter foundation model developed by Microsoft as part of the Phi-4 family. It is a unified multimodal model designed to process and reason across text, image, and audio inputs simultaneously within a single neural network architecture. By aligning these modalities in a shared representation space, the model avoids the latency and information loss typical of multi-model pipelines, enabling more context-aware interactions.
The model utilizes the Phi-4-mini-instruct model as its language backbone, supplemented by specialized encoders and adapters for vision and speech. It was trained on a massive dataset comprising 5 trillion text tokens, 2.3 million speech hours, and 1.1 trillion image-text tokens. This data-centric approach prioritizes high-quality synthetic and filtered public domain data to achieve high performance in reasoning-heavy tasks despite its relatively small size.
Phi-4 Multimodal Instruct supports a 128K token context window and is optimized for tasks such as visual question answering (VQA), document reasoning, optical character recognition (OCR), and audio summarization. It underwent post-training using supervised fine-tuning (SFT), direct preference optimization (DPO), and reinforcement learning from human feedback (RLHF) to improve instruction adherence and safety across its supported languages.