llama2.c

[Edit section]

[Copy link]

This repository implements a Transformer-based language model for natural language generation. The core model architecture and training logic are defined in Python, while efficient C implementations handle inference for deploying the models.

The key functionality includes:

• Defining the Transformer model architecture in Python, including components like the multi-headed self attention blocks and position-wise feedforward layers (model.py).

• Implementing the training loop and optimization logic to pretrain language models (train.py), using techniques like mixed precision training, gradient accumulation, learning rate scheduling, and checkpointing.

• Efficient C implementations for inference (run.c) that can run on CPU and embedded devices, using core data structures to cache activations. These provide building blocks to run models.

• Tools for tokenization using SentencePiece (tokenizer.py), to encode text into integers for input/output from models.

• Sampling logic (run.c) to generate text conditioned on a prompt, using techniques like nucleus sampling and temperature.

The key design choice is to separate model definition and training in Python from inference implementations in C. This allows rapidly iterating on model architecture, while still having performant deployment. The C inference code handles the math-heavy Transformer operations efficiently.

Details on specific model architectures, training configurations, and inference benchmarks can be found in the documentation (doc). The tokenization and sampling tools provide the necessary components for building text generation systems.

Model Architecture and Training

[Edit section]

[Copy link]

References: model.py, train.py

The model.py file defines the core Transformer model architecture for language modeling. It contains functionality for preprocessing the input embeddings, passing the input through each layer, and generating predictions.

Inference

[Edit section]

[Copy link]

References: run.c

The C implementation in run.c focuses on efficiently running trained Transformer models for natural language processing tasks. It provides the core data structures needed to take a pre-trained model checkpoint and generate predictions.

Pre-trained Models

[Edit section]

[Copy link]

References: doc/stories260K.md

This section documents several pre-trained LLama models that are available for users to leverage in their applications. The main model documented is 'stories260K', which is a 260,000 parameter language model trained for the task of text generation.

Tokenization

[Edit section]

[Copy link]

References: doc/train_llama_tokenizer.md

This section details how to train sentencepiece tokenizers for LLama models. The file …/train_llama_tokenizer.md discusses Meta's approach to training their tokenizer. It shows the configuration used by printing a protobuf object.