Testing a local LLM one question at a time hides the failure: what we measured on a 64 GB Mac
On a 64 GB Mac, two local models both scored 6/6 one question at a time. With the same questions bundled into one context, they scored 9/18 and 17/18.
9/18 vs 17/18
same questions bundled: MoE vs dense
0/3 vs 2/3
fully correct bundled runs
≈2.1×
decode speed, yet similar wall-clock time
