2 points | by gslaller 11 hours ago
1 comments
Did some work on visualising the firing pattern of MoE (Mixture of Experts) modules for Qwen models and other statistics. You'll see temporal consistency in the router, as one would expect.
Did some work on visualising the firing pattern of MoE (Mixture of Experts) modules for Qwen models and other statistics. You'll see temporal consistency in the router, as one would expect.