TorchTitan

Полная документация ниже — автоматически из репозитория pytorch/torchtitan (ветка main). Оригинальные материалы проекта распространяются на условиях его лицензии.

A PyTorch-native platform for pretraining and post-training generative AI models at scale

Quick start · Models · Features · RL · Docs · Contributing

torchtitan is a clean-room, PyTorch-native implementation of large-scale
training. One readable codebase takes a model from pretraining through
supervised fine-tuning to reinforcement learning, composing FSDP, tensor,
pipeline, context, and expert parallelism without rewriting the model.

• Composable parallelism. Models declare how they shard; FSDP2, TP, PP, CP,
  and EP stack into N-D layouts through DeviceMesh and DTensor.
• Built for research. Minimal, hackable code; every run is a Python
  recipe you can read, copy, and modify.
• Pretraining to RL in one stack. TitanRL runs the
  same model definitions and kernels in the trainer and in vLLM generation, with
  a bitwise-reproducible batch-invariant mode.
• Frontier models and techniques. DeepSeek, Qwen, Kimi, gpt-oss, Llama, and
  FLUX, with MXFP8/NVFP4 low precision, Dist-MoE, CUDA graphs, and more.

[!NOTE]
torchtitan is under active development. To use the latest features, use a
recent PyTorch nightly.

Latest news

• [2026/08] TitanRL is a hackable RL stack for scaling and debugging. It reuses TorchTitan model definitions and kernels across training and vLLM generation and supports batch-invariant mode.
• [2025/11] AMD released an optimized fork of torchtitan for AMD GPUs.
• [2025/10] We released torchtitan v0.2.0.
• [2025/10] SkyPilot now supports torchtitan! See the tutorial.

Quick start

1. Install with a PyTorch nightly (Python 3.11+). Replace cu132 with your CUDA or ROCm build.

git clone https://github.com/pytorch/torchtitan && cd torchtitan
pip install --pre torch --index-url https://download.pytorch.org/whl/nightly/cu132
pip install -r requirements.txt

2. Train a debug model. It uses the tokenizer bundled with the tests, so no download is needed.

NGPU=1 ./run_train.sh   # Llama 3 debug model, 10 steps

3. Train a real model. Download the tokenizer (Llama weights require
access on Hugging Face), then
select a recipe by module and function name:

python scripts/download_hf_assets.py --repo_id meta-llama/Llama-3.1-8B --assets tokenizer --hf_token=...
MODULE=torchtitan_recipes.models.llama3 CONFIG=llama3_8b ./run_train.sh   # 8 GPUs

4. Make it yours. A recipe is a Python function returning a complete config,
so changing a run means writing a function, not passing flags:

# my_recipes.py
from torchtitan_recipes.models.llama3 import llama3_8b

def llama3_8b_tp2():
    config = llama3_8b()
    config.parallelism.tensor_parallel_degree = 2
    return config

MODULE=my_recipes CONFIG=llama3_8b_tp2 ./run_train.sh

See the configuration guide for the full model,
and torchtitan_recipes for the verified recipes.

Supported models

Every model has debug configurations exercised in CI. Verified recipes have
been run in training, convergence, or performance workflows. To add a model, see
the model guide.

Features

We report performance on up
to 512 GPUs and verify loss convergence
across parallelisms and techniques.

Test status

Hardware	Integration tests	Unit tests
CPU
NVIDIA GPU
AMD GPU (ROCm)
RL (NVIDIA GPU)

Installation

The quick start installs from source, which is the recommended
way to get the latest features. Other options:

pip install --pre torch --index-url https://download.pytorch.org/whl/nightly/cu132 --force-reinstall
pip install --pre torchtitan --index-url https://download.pytorch.org/whl/nightly/cu132

pip install torchtitan
# or
conda install conda-forge::torchtitan

#SBATCH --ntasks=2
#SBATCH --nodes=2

srun torchrun --nnodes 2 ...

Documentation

Contributing

We look forward to your contributions!

• New ideas should start in the experiments folder;
  follow the experiments guidelines.
• For fixes and contributions to core, follow the contributing guidelines.

Citation

Our paper gives a detailed look at the parallelisms and optimizations in
torchtitan, with advice on when to use each technique:
TorchTitan: One-stop PyTorch native solution for production ready LLM pre-training.

@inproceedings{
   liang2025torchtitan,
   title={TorchTitan: One-stop PyTorch native solution for production ready {LLM} pretraining},
   author={Wanchao Liang and Tianyu Liu and Less Wright and Will Constable and Andrew Gu and Chien-Chin Huang and Iris Zhang and Wei Feng and Howard Huang and Junjie Wang and Sanket Purandare and Gokul Nadathur and Stratos Idreos},
   booktitle={The Thirteenth International Conference on Learning Representations},
   year={2025},
   url={https://openreview.net/forum?id=SFN6Wm7YBI}
}

License

Source code is made available under a BSD 3 license. You may have
other legal obligations that govern your use of other content linked in this
repository, such as the license or terms of service for third-party data and
models.