wiki-query
PApaperclipai/paperclip101k近 7 天新增 2,948 个 star
attention-variants-from-papers
Implement paper-defined attention variants by extracting mechanism invariants, mapping tensor shapes, preserving module interfaces, validating numerical behavior, and integrating the custom attention into an existing transformer stack.
- Star
- 1,842
- 分类
- 科研
- 许可证
- Apache-2.0
- 最近提交
- 2026年7月24日
tasks-extra/diff-transformer_impl/environment/skills/attention-variants-from-papers/SKILL.md
在 GitHub 上查看Attention Variants from Papers
Use this skill when a paper changes how attention scores, branches, normalization, or head sharing work, but the module still needs to behave like a drop-in transformer attention block.
Workflow
- Read the paper for invariants, not names. Use paper-to-implementation.md to extract the external contract, the changed computation, and the training-time constraints.
- Build a shape ledger before coding. Use shape-ledger.md to track projections, head grouping, branch count, and output width.
- Choose the mechanism pattern. Use mechanism-patterns.md for subtractive attention, branch mixing, learned gates, and extra normalization.
- Preserve the module boundary. Keep the same input and output shape, mask semantics, positional encoding flow, and cache behavior unless the task explicitly changes them.
- Validate in layers. Start with random-tensor smoke tests, then compare against a baseline attention path. Use stability-and-validation.md.
- Integrate into the stack last. Swap the new module into one transformer block, verify the residual path, then roll it through the full model. Use transformer-integration.md.
Checklist
- extract the paper's invariants before writing code
- account for every reshape, branch, and repeat in a shape ledger
- preserve output width at concatenation or output projection
- apply masks and positional terms at the intended stage
- confirm random smoke tests stay finite
- compare unchanged behaviors against a baseline attention implementation
Reference Map
- paper-to-implementation.md: turn paper text into module invariants and coding decisions
- shape-ledger.md: keep dimensions consistent while branch structure changes
- mechanism-patterns.md: reusable patterns for nonstandard score composition and mixing
- stability-and-validation.md: numerical checks, smoke tests, and baseline comparisons
- transformer-integration.md: wire the custom module into an existing transformer block
相似的 Skill
官方
clinical-case-report
NEnexu-io/open-design101k近 7 天新增 1,126 个 star
systematic-literature-review
BYbytedance/deer-flow84k近 7 天新增 368 个 star
academic-paper-review
BYbytedance/deer-flow84k近 7 天新增 368 个 star
directory-submissions
COcoreyhaines31/marketingskills54k近 7 天新增 958 个 star
academic-paper
IMImbad0202/academic-research-skills52k近 7 天新增 972 个 star