I built DogLM (https://mikeushakov.github.io/doglm/), a benchmark that evaluates whether an LLM, when prompted to build a video game with a background dog in it, lets the player pet that dog.
I generated 804 playable games with 17 models and scored the generated games' code using Sonnet 4.6 as LLM-as-judge. First results:
1. When a background dog is mentioned in the prompt that generates a game, but the player-dog interaction is not mentioned, an LLM needs to be "forced" by some hint to generate such interactions. Without such a hint, LLMs almost never make background dogs interactive. The hint that allows models to generate interactive dogs doesn't have to explicitly mention dog petting. I used "Add 2-3 game mechanics that a player would enjoy" as such a hint.
2. Out of 804 games, only two games generated without such a hint had some weak form of player-dog interaction. None of those two games had pettable dogs.
3. In 56 of 804 generated games, the player was able to pet the dog. But all these 56 games needed such a hint.
4. In my first test, the most dog-friendly model was Gemini 3.7 Flash, generating 15 pettable dogs and 11 weaker player-dog interactions across 50 generated games.
The full data, including the types of generated player-dog interactions, are posted here on my GitHub (link above).
I also wrote a long blog post where I explained the idea of the benchmark, the first results, and the constraints of it that I see (it is very likely that I don't see all of them): https://mikeushakov.com/machines-of-spontaneous-warmth/
Important disclaimer for HN: I'm a PhD candidate in social sciences, not an AI developer. This is my first benchmark (a proof-of-concept of it). I do understand that it is narrow and not robust yet. Will appreciate your feedback to improve it.
I built DogLM (https://mikeushakov.github.io/doglm/), a benchmark that evaluates whether an LLM, when prompted to build a video game with a background dog in it, lets the player pet that dog.
I generated 804 playable games with 17 models and scored the generated games' code using Sonnet 4.6 as LLM-as-judge. First results:
1. When a background dog is mentioned in the prompt that generates a game, but the player-dog interaction is not mentioned, an LLM needs to be "forced" by some hint to generate such interactions. Without such a hint, LLMs almost never make background dogs interactive. The hint that allows models to generate interactive dogs doesn't have to explicitly mention dog petting. I used "Add 2-3 game mechanics that a player would enjoy" as such a hint.
2. Out of 804 games, only two games generated without such a hint had some weak form of player-dog interaction. None of those two games had pettable dogs.
3. In 56 of 804 generated games, the player was able to pet the dog. But all these 56 games needed such a hint.
4. In my first test, the most dog-friendly model was Gemini 3.7 Flash, generating 15 pettable dogs and 11 weaker player-dog interactions across 50 generated games.
The full data, including the types of generated player-dog interactions, are posted here on my GitHub (link above).
I also wrote a long blog post where I explained the idea of the benchmark, the first results, and the constraints of it that I see (it is very likely that I don't see all of them): https://mikeushakov.com/machines-of-spontaneous-warmth/
Important disclaimer for HN: I'm a PhD candidate in social sciences, not an AI developer. This is my first benchmark (a proof-of-concept of it). I do understand that it is narrow and not robust yet. Will appreciate your feedback to improve it.