OLMo 7B Instruct is a 7-billion parameter large language model developed by the Allen Institute for AI (AI2). It is a key component of the OLMo (Open Language Model) project, an initiative designed to advance the science of language models by providing full transparency. Unlike many contemporary models, OLMo is released alongside its entire training ecosystem, including the training data (Dolma), code, logs, and evaluation suites.
Architecture and Specifications
The model utilizes a decoder-only transformer architecture optimized for stability and performance. It incorporates modern architectural features such as SwiGLU activation functions, Rotary Positional Embeddings (RoPE), and the removal of bias terms in linear layers to improve training stability. It supports a context length of 2048 tokens and was trained on at least 2 trillion tokens of text data.
Training and Fine-Tuning
This instruct-tuned variant was developed from the OLMo 7B base model through a multi-stage post-training process. It underwent Supervised Fine-Tuning (SFT) using the Tulu 2 mixture and was further refined with Direct Preference Optimization (DPO) using a cleaned version of the UltraFeedback dataset. These stages were specifically designed to improve its ability to follow complex instructions and participate in natural conversational dialogue.
Open Science Commitment
As a research-focused artifact, the model is released under the Apache 2.0 license. AI2 provides over 500 intermediate checkpoints for the model, allowing researchers to study the evolution of capabilities during the training process. This level of access is intended to support reproducible research into model safety, bias, and performance dynamics.