Sinusoidal vs Learned vs Rotary
| Sinusoidal | Learned | RoPE | |
|---|---|---|---|
| Type | Fixed | Parameter | Fixed |
| Applied at | Input embedding | Input embedding | Q and K inside attention |
| Extrapolates past max_len | Yes | No | Yes |
| Encodes relative position | Weakly | No | Explicitly |
Why it matters: LLaMA, Qwen, Mistral all use RoPE because ⟨RoPE(q, m), RoPE(k, n)⟩ depends only on m−n — relative position becomes a first-class operation, not a learned approximation.