CodeLlama-34b-instruct is a large language model developed by Meta AI, specifically fine-tuned for code generation and instruction following. It is part of the Code Llama family, which is built upon the Llama 2 architecture. The 34B parameter version was designed to provide high-level performance for programming tasks while maintaining a more efficient footprint than the later 70B variant.
The model was trained on an additional 500 billion tokens of code and code-related data beyond the standard Llama 2 training corpus. The instruct variant is further refined with a specific dataset of human-generated instructions and synthetic code examples to better handle natural language prompts for software engineering. It supports a wide variety of programming languages, including Python, C++, Java, PHP, TypeScript, C#, and Bash.
CodeLlama-34b-instruct features an expanded context window, allowing it to process and reason over sequences of up to 100,000 tokens. This capability is particularly useful for tasks involving large codebases or complex project structures. To optimize inference performance, the model utilizes Grouped-Query Attention (GQA).
The model is intended for specialized applications such as code completion, bug fixing, and the generation of technical explanations. By utilizing specific instruction formatting, it can serve as a conversational assistant for developers, providing both code implementations and corresponding documentation.