It's needed in general-purpose models because there are billions of persons who can use these models to develop (personally or organizationally) valuable software, and all of these persons deserve the ability to test their software against the best exploit-development resources that exist.
The so-called "safeguards" preventing this, on balance, probably do more harm than good. The best argument in favor of these safeguards is perhaps the temporary one: today, there is simply too much deployed critical software that was developed in a more carefree age. Therefore, one might favor a tradeoff: try to introduce artificial friction against exploit development targeting old software (during the process of patching all of it), with the side effect that new software ends up less resilient than it could have been.
But in the steady state (which may be a very small number of years from now), every practical capability for exploit development should be a routine part of the software creation and maintenance lifecycle.
It's needed in general-purpose models because there are billions of persons who can use these models to develop (personally or organizationally) valuable software, and all of these persons deserve the ability to test their software against the best exploit-development resources that exist.
The so-called "safeguards" preventing this, on balance, probably do more harm than good. The best argument in favor of these safeguards is perhaps the temporary one: today, there is simply too much deployed critical software that was developed in a more carefree age. Therefore, one might favor a tradeoff: try to introduce artificial friction against exploit development targeting old software (during the process of patching all of it), with the side effect that new software ends up less resilient than it could have been.
But in the steady state (which may be a very small number of years from now), every practical capability for exploit development should be a routine part of the software creation and maintenance lifecycle.