I use to think small models needed more handholding but I think it’s more a question of age. Newer models do fine with less handholding regardless of size.
Yesterday for fun I tried Ornith 9G on my 16GB M1 with OpenCode. It took 3.5 hours do so what Kimi does in a minute (prompt: “analyze this codebase front end and back end and explain the auth mechanism”), but the subtlety and nuance it showed in its response was very surprising to me - I would have lost the bet if someone asked me if a 9G model could do this.
I don't think so. Having build a custom agent with Anthrpic's SDK (in Go) and even running the agent on smaller quantized (non-frontier) models using Ollama, I seriously doubt you'd get good results. I read the same blog post, and I get the impression that the larger model with greater capabilities is able to infer more accurately without being constrained by large(r) system prompts, but I don't think this works so well for smaller less capable models.
I use to think small models needed more handholding but I think it’s more a question of age. Newer models do fine with less handholding regardless of size.
Yesterday for fun I tried Ornith 9G on my 16GB M1 with OpenCode. It took 3.5 hours do so what Kimi does in a minute (prompt: “analyze this codebase front end and back end and explain the auth mechanism”), but the subtlety and nuance it showed in its response was very surprising to me - I would have lost the bet if someone asked me if a 9G model could do this.
I don't think so. Having build a custom agent with Anthrpic's SDK (in Go) and even running the agent on smaller quantized (non-frontier) models using Ollama, I seriously doubt you'd get good results. I read the same blog post, and I get the impression that the larger model with greater capabilities is able to infer more accurately without being constrained by large(r) system prompts, but I don't think this works so well for smaller less capable models.
[flagged]