wiki-query
PApaperclipai/paperclip101k2,948 new stars in the last 7 days
attention-variants-from-papers
Implement paper-defined attention variants by extracting mechanism invariants, mapping tensor shapes, preserving module interfaces, validating numerical behavior, and integrating the custom attention into an existing transformer stack.
- Star
- 1,842
- Category
- Science
- License
- Apache-2.0
- Last commit
- Jul 24, 2026
tasks-extra/diff-transformer_impl/environment/skills/attention-variants-from-papers/SKILL.md
View on GitHubAttention Variants from Papers
Use this skill when a paper changes how attention scores, branches, normalization, or head sharing work, but the module still needs to behave like a drop-in transformer attention block.
Workflow
- Read the paper for invariants, not names. Use paper-to-implementation.md to extract the external contract, the changed computation, and the training-time constraints.
- Build a shape ledger before coding. Use shape-ledger.md to track projections, head grouping, branch count, and output width.
- Choose the mechanism pattern. Use mechanism-patterns.md for subtractive attention, branch mixing, learned gates, and extra normalization.
- Preserve the module boundary. Keep the same input and output shape, mask semantics, positional encoding flow, and cache behavior unless the task explicitly changes them.
- Validate in layers. Start with random-tensor smoke tests, then compare against a baseline attention path. Use stability-and-validation.md.
- Integrate into the stack last. Swap the new module into one transformer block, verify the residual path, then roll it through the full model. Use transformer-integration.md.
Checklist
- extract the paper's invariants before writing code
- account for every reshape, branch, and repeat in a shape ledger
- preserve output width at concatenation or output projection
- apply masks and positional terms at the intended stage
- confirm random smoke tests stay finite
- compare unchanged behaviors against a baseline attention implementation
Reference Map
- paper-to-implementation.md: turn paper text into module invariants and coding decisions
- shape-ledger.md: keep dimensions consistent while branch structure changes
- mechanism-patterns.md: reusable patterns for nonstandard score composition and mixing
- stability-and-validation.md: numerical checks, smoke tests, and baseline comparisons
- transformer-integration.md: wire the custom module into an existing transformer block
Similar Skills
Official
clinical-case-report
NEnexu-io/open-design101k1,126 new stars in the last 7 days
systematic-literature-review
BYbytedance/deer-flow84k368 new stars in the last 7 days
academic-paper-review
BYbytedance/deer-flow84k368 new stars in the last 7 days
directory-submissions
COcoreyhaines31/marketingskills54k958 new stars in the last 7 days
academic-paper
IMImbad0202/academic-research-skills52k972 new stars in the last 7 days