跳到正文
FunCoding

distributed-training

Multi-GPU and distributed training patterns with PyTorch DDP. Use when scaling training across GPUs.

Star
14,635
分类
科研
许可证
MIT
最近提交
2026年8月19日

researchclaw/skills/builtin/tooling/distributed-training/SKILL.md

在 GitHub 上查看

Distributed Training Best Practice

  1. Use DistributedDataParallel (DDP) over DataParallel for multi-GPU
  2. Initialize process group: dist.init_process_group(backend='nccl')
  3. Use DistributedSampler for data sharding
  4. Synchronize batch norm: nn.SyncBatchNorm.convert_sync_batchnorm()
  5. Only save checkpoint on rank 0
  6. Scale learning rate linearly with world size
  7. Use gradient accumulation for effectively larger batch sizes

相似的 Skill

wiki-query

PApaperclipai/paperclip101k近 7 天新增 2,948 个 star

官方

clinical-case-report

NEnexu-io/open-design101k近 7 天新增 1,126 个 star

systematic-literature-review

BYbytedance/deer-flow84k近 7 天新增 368 个 star

academic-paper-review

BYbytedance/deer-flow84k近 7 天新增 368 个 star

directory-submissions

COcoreyhaines31/marketingskills54k近 7 天新增 958 个 star

academic-paper

IMImbad0202/academic-research-skills52k近 7 天新增 972 个 star