Skip to content
FunCoding

distributed-training

Multi-GPU and distributed training patterns with PyTorch DDP. Use when scaling training across GPUs.

Star
14,635
Category
Science
License
MIT
Last commit
Aug 19, 2026

researchclaw/skills/builtin/tooling/distributed-training/SKILL.md

View on GitHub

Distributed Training Best Practice

  1. Use DistributedDataParallel (DDP) over DataParallel for multi-GPU
  2. Initialize process group: dist.init_process_group(backend='nccl')
  3. Use DistributedSampler for data sharding
  4. Synchronize batch norm: nn.SyncBatchNorm.convert_sync_batchnorm()
  5. Only save checkpoint on rank 0
  6. Scale learning rate linearly with world size
  7. Use gradient accumulation for effectively larger batch sizes

Similar Skills

wiki-query

PApaperclipai/paperclip101k2,948 new stars in the last 7 days

Official

clinical-case-report

NEnexu-io/open-design101k1,126 new stars in the last 7 days

systematic-literature-review

BYbytedance/deer-flow84k368 new stars in the last 7 days

academic-paper-review

BYbytedance/deer-flow84k368 new stars in the last 7 days

directory-submissions

COcoreyhaines31/marketingskills54k958 new stars in the last 7 days

academic-paper

IMImbad0202/academic-research-skills52k972 new stars in the last 7 days