Using attention patterns to improve how language models route work between expert networks
This paper proposes Attention-Aware Routing, a method that improves how Mixture-of-Experts language models route tokens to different expert networks by feeding the router additional information about what the model is paying attention to, rather than relying only on the token's hidden state. The approach improves performance on math reasoning tasks and reveals that routing decisions and attention mechanisms are tightly coupled in how the model processes information.










