K2 Horizon 375B A23B

K2 Horizon 375B A23B is the flagship language model of the K2 Horizon fleet, a series of open-source models released by the MBZUAI Institute of Foundation Models (IFM). It is a sparse Mixture-of-Experts (MoE) architecture engineered for high-performance enterprise applications, particularly in advanced reasoning, coding, and agentic task execution. As part of a "fully open" initiative, the model is published under the Apache 2.0 license with access provided to its weights, training code, and data construction methodology.

The model architecture utilizes a total of 375 billion parameters, with approximately 23 billion parameters active per inference pass. A core technical innovation is the Mixture of Value Attention (MoVA) mechanism, which extends expert routing directly into the multi-head attention layers. This design, combined with diffusion distillation techniques, is intended to accelerate generation speed and improve reasoning efficiency without significantly increasing computational costs during inference.

K2 Horizon 375B A23B supports a native context window of 524,288 tokens (512K), allowing for the processing of massive codebases and long-form document sets. The model was pre-trained on a 20-trillion-token corpus, of which nearly 17% consisted of explicit problem-solving trajectories and approximately 10 trillion tokens were synthetic. It defaults to Markdown-based tool-calling, which was found to be more token-efficient than standard JSON during training.

Performance evaluations indicate a strong emphasis on reliability and reduced hallucinations. The model is specifically tuned to prioritize abstention over guessing, meaning it frequently declines to respond to queries where its internal confidence is low rather than providing inaccurate information. This behavior has allowed it to maintain competitive accuracy in complex knowledge-work and software engineering benchmarks.