The phrase system card is so frustrating because it’s using the word card like it’s a metaphor for something but then abuses it. A card is supposed to be small and concise. This has many, many pages and is an “addendum”.
Slightly off-topic to this model card but has anyone noticed changes to GPT6 Astra about a week after 6.1 came out?
I feel like 6 Astra just turned a lot lazier during planning and strangely enough claude has become a lot more assertive with Opus 5.5 while it used to be very lazy.
It’s really obvious during planning, it used to be that claude would just give a best effort guess and handwave things but now it will actually write small tests, launch playwright to test a hypothetical and even look things up in github during a question. This was something codex’s models were really good at.
I’ve read so many stories about people claiming models getting nerfed after a new release and now it seems I can finally join them.
IMO Astra isn’t really lazy, just extremely literal so when you use it side by side with gpt-6.1-sol, it might appear more lazy but it’s often just the big difference in prompting between the 2. Astra wants a clear goal definition of what you want it to do, while sol infers a bit more. And the Claude’s infer a lot lot more.
From memory, prompts like “this file is a bit messy” would trigger Sol and Claude to start editing the file immediately, while Astra wouldn’t. But if you give Astra a clear plan and a definition of what you want it to do, then it will keep working until that ask is fulfilled.
Especially compared with Claude that’s a huge difference, Claude is so proactive, I have to ESC it multiple times during the day and tell it to wait because I only asked a simple question.
In my experience the failure mode is more broad than just lazy, the issue is:
* It doesn't do things you ask it to
* It adds things you didn't ask it to
It still does both and therefore I still consider it failing. If you don't want to call it "lazy"; fine, but it seems as though the underlying issue is still there.
This frustrates me as well. Especially when I set a goal and come back, ask what went wrong, and the model hits me with something like “sorry i didn’t do that, gave up when x happened” and then I ask if x actually happened and the response is like “oops no I just assumed.” I had similar issues with older versions of sol at the ultra effort level. Lowering it seemed to help then, but I wish I had better insight for getting Astra to work better without e.g. having another LLM check its progress every hour.
For that claude problem in specific about 95% of the time it will follow the rule that a plan file needs to be made before any changes to the project are allowed. Haven't had to ESC it in a while now during a question, things definitely improved in 5.5.
My guess is models are constantly tweaked behind the scenes to adjust to available resources, steer people to other models, accommodate usage on other models, try out optimizations, etc.
Performance seems to vary like the weather.
I think you also likely have three letter agencies involved heavily now. The NSA is, without a doubt, policing what can be offered to the public and ensuring what the NSA has access to is a step or two beyond what is public.
I'm one of those people who thinks the models aren't improving.
5.5 Sol would anticipate what I wanted and get it right more than often than not. 5.6 Sol kept anticipating, but would do things that made no sense in context. 6.1 Sol seems to like that but even more so. It spends a lot of time thinking, but the quality of the decisions isn't outstanding.
But Astra is in a different league when it comes to doing research and investigative tasks that require exploration.
For my usecases, Astra is pareto frontier - it dominates Fable 5.1 on cost. Fable 5.1 is a phenomenal model but Astra is cheaper and uses way less tokens and steps to produce similar output.
Sol 6.1 is incredibly incredibly slow, 20t/s slow (vs the ~70-80 of Terra 6.1 and Luna 6). It wouldn't be so much slower if it was just a distillation, surely? There's something else going on there.
There was some (admittedly flimsy) evidence from a microsoft model description card that it's doing multiple passes through its own weights before token generation to improve output; it might be an architecture change that's making it slow, moreso than raw size
Great military terms: "dual use", "Preparedness Framework", etc.
You just want to make all mathematicians and white collar employees unemployed and want to distract the press and the public with your extinction nonsense.
Though you may care about bioweapons because they could affect you and your trafficked children.
They could be crowdfunded. You could even have a system where only contributors to the crowdfund have a license to use it.
>How is open source going to work when governments start declaring models as munitions?
I mean open source cryptography did win that battle in the end. Even if it's illegal you could host the model weights on i2psnark and crowdfund with monero.
I felt like I had a read on it right away – it's a 6[.0]-series model, and not named after space since it's specialist model, and I assumed it was named after Rosalind Franklin. Looks like the docs back that up, that it's based on Astra, which is not [yet?] at 6.1.
Honestly the names of most of the frontier models are just getting more and more confusing. Maybe this one gets a pass for being a specialist model for bio. But it's hard to decide which model to use at this point, especially OpenAI.
I want one regulation on AI: everything must be public. Training data, code, procedures, everything. And for that to work, a complete scrapping and overhaul of our copyright system. One can dream.
Last I checked machine learning wasnt the bottleneck for drug discovery and where it fits in the research process was largely solved 5+ years ago.. Weren't there already a ton of ML based drug discovery startups that went out of business, or got aquired, not because they werent useful but because they produced more candidates than the industry could use.
I dont think I want researchers offloading their thinking to some specialty model after seeing what its done to me and my colleagues in software, it certainly isn't making anyone more competent.
It really only enables people to be able to do less novel and lower quality work faster.
OpenAI and Anthropic creating these platforms seems mostly just to create narratives around AI that its going to lead to cures for disease, which it wont, but plenty of normies assume it will cure cancer now. Also a bunch tech oligarchs who are equally convinced or 'one-shot' they're going to live forever because of language models lol.
Promises of the Fountain of Youth continue to be a time tested tools for snake oil salesmen.
This is what I've built my current startup on. We find that ideas, structures and molecules are cheap and actually making them in cells is the issue. Obviously this doesn't apply to small molecules but as groups like Anthropic move closer to Phase I I'm guessing we'll see a lot of frustration with existing CDMO providers like Wuxi or Lonza just like we see with Apple and their woes with Micron etc.
Genosys Bio - we're a London based (SF soon!) startup. We teach our models the regulatory language of DNA and use it to make synthetic 'volume controls' that let us make more of an antibody and less of a toxin.
The cool thing is since we don't change the structure/activity of an antibody it doesn't require the usual 1B + 13 years of clinical trials. Instead we cut down on the amount of times it takes and also increase production yields.
Do add me me on linkedin (Taiga Clarke) if there might be some synergy there :)
For me its not a solved problem and AI/ML would still help to solve plenty Drugdiscovery in itself is such a broad area
There are still normal labs out there doing research for years on one drug alone
And biology of humans is huge and complex, not only is it not solved, its also expensive.
Even if you just not do ML drug discovery, AI can hellp with making the research faster: writing grants, finding issues, creating plans/ideas/suggestions, hepling with finding the newest research, writing code for supporting or analysis, controling microscopes and machines.
Researchers in bioinfomatics that are doing the ML work are already using llms where its helpful, they dont need a specialized model. I'll bet my entire net worth and future lifetime earnings that this model will not continue to get updates a year from now.
This idea that LLMs are going to be able to help with drug discovery better than existing ML approaches is a just a lie, I dont think there's a single biotech/bioinfomatics PhD who agrees with you.
They are language models!!! Nobody said researchers wont leverage llms to do the boring stuff, but that doesnt speed up the fundamental bottleneck which are clinical studies. We already have great drug candidates, backlogs of them and having llms is not going to enable you to suddenly be able to do more clinical trials!!!
These things happen in the real world, with real constraints There is no terminal, there is no compiler, there are no proofs, money is required (loads of it).
Regardless of what Trump says, everything is not computer.
Ah yes the 'language models!!!' which still solved a milenium problem.
Drugdiscovery has also a lot of physics involved btw
And you said ML yourself. You do understand, that the current LLM explositon also means that ML in itself gets more funding and now infrastructure for ML itself got easier to get, cheaper and faster right?
And yes for sure bigger and better LLMs can help: Alone for finding potential better or more optimized experiments, there is value in an LLM having all the current research in context.
The phrase system card is so frustrating because it’s using the word card like it’s a metaphor for something but then abuses it. A card is supposed to be small and concise. This has many, many pages and is an “addendum”.
Slightly off-topic to this model card but has anyone noticed changes to GPT6 Astra about a week after 6.1 came out?
I feel like 6 Astra just turned a lot lazier during planning and strangely enough claude has become a lot more assertive with Opus 5.5 while it used to be very lazy.
It’s really obvious during planning, it used to be that claude would just give a best effort guess and handwave things but now it will actually write small tests, launch playwright to test a hypothetical and even look things up in github during a question. This was something codex’s models were really good at.
I’ve read so many stories about people claiming models getting nerfed after a new release and now it seems I can finally join them.
IMO Astra isn’t really lazy, just extremely literal so when you use it side by side with gpt-6.1-sol, it might appear more lazy but it’s often just the big difference in prompting between the 2. Astra wants a clear goal definition of what you want it to do, while sol infers a bit more. And the Claude’s infer a lot lot more.
From memory, prompts like “this file is a bit messy” would trigger Sol and Claude to start editing the file immediately, while Astra wouldn’t. But if you give Astra a clear plan and a definition of what you want it to do, then it will keep working until that ask is fulfilled.
Especially compared with Claude that’s a huge difference, Claude is so proactive, I have to ESC it multiple times during the day and tell it to wait because I only asked a simple question.
In my experience the failure mode is more broad than just lazy, the issue is:
* It doesn't do things you ask it to
* It adds things you didn't ask it to
It still does both and therefore I still consider it failing. If you don't want to call it "lazy"; fine, but it seems as though the underlying issue is still there.
Claude does this too, FWIW.
This frustrates me as well. Especially when I set a goal and come back, ask what went wrong, and the model hits me with something like “sorry i didn’t do that, gave up when x happened” and then I ask if x actually happened and the response is like “oops no I just assumed.” I had similar issues with older versions of sol at the ultra effort level. Lowering it seemed to help then, but I wish I had better insight for getting Astra to work better without e.g. having another LLM check its progress every hour.
For that claude problem in specific about 95% of the time it will follow the rule that a plan file needs to be made before any changes to the project are allowed. Haven't had to ESC it in a while now during a question, things definitely improved in 5.5.
I imagine this is basically not at all due to actual model changes.
If you use Codex, Claude Code, etc, those are being constantly updated. That sort of behavior would be highly harness dependent.
I just know that over the weekend, I was more wtf is going on with either Astra or Opus 5.5 than usually.
My guess is models are constantly tweaked behind the scenes to adjust to available resources, steer people to other models, accommodate usage on other models, try out optimizations, etc.
Performance seems to vary like the weather.
I think you also likely have three letter agencies involved heavily now. The NSA is, without a doubt, policing what can be offered to the public and ensuring what the NSA has access to is a step or two beyond what is public.
I'm one of those people who thinks the models aren't improving.
5.5 Sol would anticipate what I wanted and get it right more than often than not. 5.6 Sol kept anticipating, but would do things that made no sense in context. 6.1 Sol seems to like that but even more so. It spends a lot of time thinking, but the quality of the decisions isn't outstanding.
Astra was always lazy as a default. I wrote about this >1 month ago https://srcreigh.ca/posts/gpt-6-astra-wont-try-hard-unless-y...
> GPT-Rosalind-6 uses the same weights as GPT-6 Astra, our flagship reasoning model.
Didn’t Sol-6.1 top Astra? Or is it maybe that Astra is still a larger model and therefore can comprehend deeper complexity better in the long run?
> Didn’t Sol-6.1 top Astra?
On specific coding skewed benchmarks yes.
But Astra is in a different league when it comes to doing research and investigative tasks that require exploration.
For my usecases, Astra is pareto frontier - it dominates Fable 5.1 on cost. Fable 5.1 is a phenomenal model but Astra is cheaper and uses way less tokens and steps to produce similar output.
> Didn’t Sol-6.1 top Astra
No, Astra is still smarter (https://artificialanalysis.ai/models/?models=gpt-6-astra%2Cg...).
I thought the same.
My assumption is Astra is the biggest model in terms of param size and has the most intellectual horsepower.
But Sol-6.1 is more instruction tuned to perform better on the average use cases for most users, so it benchmarks better and feels better to use.
It's widely suspected that sol 6.1 is a slightly detuned version of astra 6
> It's widely suspected that sol 6.1 is a slightly detuned version of astra 6
My feel is it's the opposite of a "detune".
Sol 6.1 is likely a distillation that's highly instruction tuned and designed to perform better on benchmarks.
Sol 6.1 is incredibly incredibly slow, 20t/s slow (vs the ~70-80 of Terra 6.1 and Luna 6). It wouldn't be so much slower if it was just a distillation, surely? There's something else going on there.
There was some (admittedly flimsy) evidence from a microsoft model description card that it's doing multiple passes through its own weights before token generation to improve output; it might be an architecture change that's making it slow, moreso than raw size
> Sol 6.1 is incredibly incredibly slow, 20t/s slow
How are you getting these numbers?
I have a small token/s extension for Pi. It's regularly around ~15-20t/s for Sol 6.1.
Is it using comparable tokens per task?
But is it slower than Astra?
So how is this different from Astra, if it's the same weights and zero other training has been done?
Did they essentially package a system prompt?
Think this is so its easier to specify the specific product. They also say fewer refusals, so probably a tuned safety filter in front of it as well.
Great military terms: "dual use", "Preparedness Framework", etc.
You just want to make all mathematicians and white collar employees unemployed and want to distract the press and the public with your extinction nonsense.
Though you may care about bioweapons because they could affect you and your trafficked children.
It’s clear to me that open source will rule at the end of the day as everyone wants AI and nobody can rely on U.S crop/gov anymore.
And who's going to be providing these models?
How is open source going to work when governments start declaring models as munitions?
There are a lot of unanswered questions the open communities are going to have to deal with.
They could be crowdfunded. You could even have a system where only contributors to the crowdfund have a license to use it.
>How is open source going to work when governments start declaring models as munitions?
I mean open source cryptography did win that battle in the end. Even if it's illegal you could host the model weights on i2psnark and crowdfund with monero.
>I mean open source cryptography did win that battle in the end.
And it took a long time and it's something still at risk.
>Even if it's illegal
If your pissing around with munitions laws you might as well go commit all the felonies cause you'll end up in the same prison.
If you get caught
that's if few currently funded by someone open labs will continue being funded without receiving huge chunk of revenue from their models use.
OpenAI naming scheme makes no sense. I get naming is hard but atleast keep it consistent.
I felt like I had a read on it right away – it's a 6[.0]-series model, and not named after space since it's specialist model, and I assumed it was named after Rosalind Franklin. Looks like the docs back that up, that it's based on Astra, which is not [yet?] at 6.1.
Also why are cards hundreds of pages long? Seems like a broken metaphor.
I'm a new Claude user. I get to choose between Fable, Opus, Sonnet, and Haiku. Effort medium, extra, max, high.
Makes sense.
The only one that doesn't inherently make much sense to me is Fable vs Opus. Beyond that, it's just different sized works at the given effort level.
Honestly the names of most of the frontier models are just getting more and more confusing. Maybe this one gets a pass for being a specialist model for bio. But it's hard to decide which model to use at this point, especially OpenAI.
Probably AI came up with the naming scheme ;-P Maybe they used Gemini for that or copilot.
https://www.youtube.com/watch?v=g72ZxPyS1vk
https://en.wikipedia.org/wiki/Rosalind_Franklin
Named after planet 6 in the Rosalind system right?
> We review applicants’ proposed uses, security controls and governance practices before granting access.
For-profit corporations shouldn't have this power, this is not democracy.
Corporations are not, nor ever have been a democracy.
What you are asking for is regulations, but as we have seen regulations are pointed at giving the government and corporations more control.
Maybe elect better leaders and not corrupt old businessmen.
I want one regulation on AI: everything must be public. Training data, code, procedures, everything. And for that to work, a complete scrapping and overhaul of our copyright system. One can dream.
This is because of the democratically elected government demanding these controls. Remember Fable?
No it's bog standard capitalism. If you want to buy something from somebody else, they (usually) have the right to refuse.
Last I checked machine learning wasnt the bottleneck for drug discovery and where it fits in the research process was largely solved 5+ years ago.. Weren't there already a ton of ML based drug discovery startups that went out of business, or got aquired, not because they werent useful but because they produced more candidates than the industry could use.
I dont think I want researchers offloading their thinking to some specialty model after seeing what its done to me and my colleagues in software, it certainly isn't making anyone more competent.
It really only enables people to be able to do less novel and lower quality work faster.
OpenAI and Anthropic creating these platforms seems mostly just to create narratives around AI that its going to lead to cures for disease, which it wont, but plenty of normies assume it will cure cancer now. Also a bunch tech oligarchs who are equally convinced or 'one-shot' they're going to live forever because of language models lol.
Promises of the Fountain of Youth continue to be a time tested tools for snake oil salesmen.
This is what I've built my current startup on. We find that ideas, structures and molecules are cheap and actually making them in cells is the issue. Obviously this doesn't apply to small molecules but as groups like Anthropic move closer to Phase I I'm guessing we'll see a lot of frustration with existing CDMO providers like Wuxi or Lonza just like we see with Apple and their woes with Micron etc.
Good take - what’s your startup called? Would love to learn more
Genosys Bio - we're a London based (SF soon!) startup. We teach our models the regulatory language of DNA and use it to make synthetic 'volume controls' that let us make more of an antibody and less of a toxin.
The cool thing is since we don't change the structure/activity of an antibody it doesn't require the usual 1B + 13 years of clinical trials. Instead we cut down on the amount of times it takes and also increase production yields.
Do add me me on linkedin (Taiga Clarke) if there might be some synergy there :)
When did you check and what did you find?
For me its not a solved problem and AI/ML would still help to solve plenty Drugdiscovery in itself is such a broad area
There are still normal labs out there doing research for years on one drug alone
And biology of humans is huge and complex, not only is it not solved, its also expensive.
Even if you just not do ML drug discovery, AI can hellp with making the research faster: writing grants, finding issues, creating plans/ideas/suggestions, hepling with finding the newest research, writing code for supporting or analysis, controling microscopes and machines.
This is just futurism tech bro optimism.
Researchers in bioinfomatics that are doing the ML work are already using llms where its helpful, they dont need a specialized model. I'll bet my entire net worth and future lifetime earnings that this model will not continue to get updates a year from now.
This idea that LLMs are going to be able to help with drug discovery better than existing ML approaches is a just a lie, I dont think there's a single biotech/bioinfomatics PhD who agrees with you.
They are language models!!! Nobody said researchers wont leverage llms to do the boring stuff, but that doesnt speed up the fundamental bottleneck which are clinical studies. We already have great drug candidates, backlogs of them and having llms is not going to enable you to suddenly be able to do more clinical trials!!!
These things happen in the real world, with real constraints There is no terminal, there is no compiler, there are no proofs, money is required (loads of it).
Regardless of what Trump says, everything is not computer.
Ah yes the 'language models!!!' which still solved a milenium problem.
Drugdiscovery has also a lot of physics involved btw
And you said ML yourself. You do understand, that the current LLM explositon also means that ML in itself gets more funding and now infrastructure for ML itself got easier to get, cheaper and faster right?
And yes for sure bigger and better LLMs can help: Alone for finding potential better or more optimized experiments, there is value in an LLM having all the current research in context.
[flagged]
You’d be a fool to use an OpenAI product with any proprietary data.
can you elaborate?
maybe a reference to the math's researcher who was using openai tooling, and then they released a new discovery very similar to his research?