Llama 2 Chat 70B is a large language model developed by Meta, representing the largest variant in the Llama 2 family. It is an auto-regressive transformer-based model with 70 billion parameters, specifically fine-tuned for conversational and dialogue-driven use cases. The model was trained on a dataset of 2 trillion tokens from publicly available sources and features a context window of 4096 tokens, which is double the capacity of its predecessor, Llama 1.
Architecture and Training
The 70B model variant is distinguished from the smaller 7B and 13B models by its use of Grouped-Query Attention (GQA), an architectural optimization that enhances inference scalability and performance. To optimize the model for dialogue, Meta employed a refinement process consisting of Supervised Fine-Tuning (SFT) and multiple iterations of Reinforcement Learning from Human Feedback (RLHF). This dual approach was designed to align the model's responses with human preferences for both helpfulness and safety.
Capabilities
Llama 2 Chat 70B is designed for complex reasoning, multi-turn dialogue, and sophisticated natural language tasks. Meta focused on responsible development by implementing safety-specific data augmentation, red-teaming, and safety-tuning to reduce the likelihood of generating harmful or inappropriate content. The model's training data has a knowledge cutoff of September 2022, though some fine-tuning data includes examples up to July 2023.