I'm on the Hutter Prize committee (claim verification). Some context for HN:
The prize scores lossless compression of enwik9, a 1GB Wikipedia snapshot, and counts the size of the decompressor. Entries are limited to one CPU core, about 50 hours and under 10GB of RAM. These limits address 3 concerns:
* Specialized hardware biases research toward the industry’s current assumptions.
* Talented individuals, not only well-funded labs, can afford to compete.
* The public increasingly opposes the resource consumption of superintelligence efforts.
This year three entries improved the record by about 1%, another 1%, and then astoundingly 7.5%, compounded, the largest yearly gain since the prize started in 2006. The record now compresses enwik9 by more than 10x.
What I think matters most: there's no held-out validation set. The whole file is public so incentives to leak data in high stakes benchmarks are for naught. Most AI benchmarks can't say that, and it matters more as AI investment decisions lean on benchmark numbers.
Rules, records and the source code of past winners: http://prize.hutter1.net/. Happy to answer questions.
I'm on the Hutter Prize committee (claim verification). Some context for HN:
The prize scores lossless compression of enwik9, a 1GB Wikipedia snapshot, and counts the size of the decompressor. Entries are limited to one CPU core, about 50 hours and under 10GB of RAM. These limits address 3 concerns:
* Specialized hardware biases research toward the industry’s current assumptions. * Talented individuals, not only well-funded labs, can afford to compete. * The public increasingly opposes the resource consumption of superintelligence efforts.
This year three entries improved the record by about 1%, another 1%, and then astoundingly 7.5%, compounded, the largest yearly gain since the prize started in 2006. The record now compresses enwik9 by more than 10x.
What I think matters most: there's no held-out validation set. The whole file is public so incentives to leak data in high stakes benchmarks are for naught. Most AI benchmarks can't say that, and it matters more as AI investment decisions lean on benchmark numbers.
Rules, records and the source code of past winners: http://prize.hutter1.net/. Happy to answer questions.