Here's the best benchmark I've found, one that isn't remotely saturated:
Take a recent paper or chapter(s) from a recent book that you are very familiar with and ask the LLM to create a 50-item multiple choice test on the provided information with an answer key at the bottom. Every LLM I've tried has failed. Notable:
- DeepSeek's sparse attention is genuine. It skips paragraphs and pages, then gets very confused when the skipped items are referenced.
- Training data almost always overrides contextual data, which often profoundly perverts the results when the two disagree.
The essential question for any benchmark is to ask whether you are modes of success (to justify investment) or failure (to identify architectural limitations). You be the judge: how do you want your tests biased?
Here's the best benchmark I've found, one that isn't remotely saturated:
Take a recent paper or chapter(s) from a recent book that you are very familiar with and ask the LLM to create a 50-item multiple choice test on the provided information with an answer key at the bottom. Every LLM I've tried has failed. Notable:
- DeepSeek's sparse attention is genuine. It skips paragraphs and pages, then gets very confused when the skipped items are referenced. - Training data almost always overrides contextual data, which often profoundly perverts the results when the two disagree.
The essential question for any benchmark is to ask whether you are modes of success (to justify investment) or failure (to identify architectural limitations). You be the judge: how do you want your tests biased?