跳到正文
FunCoding

data-loading

Optimize data loading pipeline to prevent GPU starvation. Use when setting up DataLoader or data preprocessing.

Star
14,635
许可证
MIT
最近提交
2026年8月19日

researchclaw/skills/builtin/tooling/data-loading/SKILL.md

在 GitHub 上查看

Efficient Data Loading Best Practice

  1. Use num_workers = min(8, os.cpu_count()) for DataLoader
  2. Enable pin_memory=True when using GPU
  3. Use persistent_workers=True to avoid re-spawning
  4. Pre-compute and cache transformations when possible
  5. For image data: use torchvision.transforms.v2 (faster)
  6. For large datasets: consider memory-mapped files or WebDataset
  7. Profile with torch.utils.bottleneck to find I/O bottlenecks

相似的 Skill

receiving-code-review

OBobra/superpowers298k近 7 天新增 2,631 个 star

requesting-code-review

OBobra/superpowers298k近 7 天新增 2,631 个 star

systematic-debugging

OBobra/superpowers298k近 7 天新增 2,631 个 star

code-review

MAmattpocock/skills288k近 7 天新增 10,178 个 star

codebase-design

MAmattpocock/skills288k近 7 天新增 10,178 个 star

scaffold-exercises

MAmattpocock/skills288k近 7 天新增 10,178 个 star