OLMo 2 7B is a 7-billion parameter language model developed by the Allen Institute for AI (Ai2). Released in November 2024, it is part of the second generation of the Open Language Model (OLMo) family. The model is designed to provide a high-performance foundation for AI research, adhering to a "truly open source" philosophy by providing not just the model weights, but also the full training data, training code, evaluation tools, and intermediate checkpoints.
The model utilizes a decoder-only transformer architecture and was trained on a massive corpus of approximately 5 trillion tokens. This training data includes the Dolma dataset as well as refined mixes such as OLMo-mix-1124 and Dolmino-mix-1124. The training process followed a staged approach, involving extensive initial pre-training followed by specialized data annealing and model souping techniques to optimize the final checkpoint.
Compared to the original OLMo-7B, OLMo 2 7B features significant performance improvements, including a 9-point increase in MMLU scores. These gains are attributed to improved architectural optimizations and a more sophisticated data curation strategy. The model and its associated assets are released under the Apache 2.0 license, facilitating transparency and reproducibility in the machine learning community.