1. Constitution can't work without punitive enforcement, how will you punish agent/LLM?
2. Even if we can enforce laws for agents/LLM, just look at existing law system it is tug of war because we almost can't write anything in natural language without double meaning.
3. Saying AGENTS.md can enforce anything at all is a pipe dream, I am struggling with this all the time, in a long session the tokens from AGENTS.md are so far in the back of the context and so diluted that LLM's "attention" doesn't register them.
And I love responses from LLM like "I want to point out that I have broken rule &1 and &11, if you disagree — please tell I will be happy to oblige"
This proposal is essentially "LLM don't do bad things, please, if you do - say magic word, I will stop you then"
Are you sure prompting a LLM to behave a certain way (in your case to split the task into 4 supposedly confined sub-tasks) is foolproof and can't simply be ignored by the LLM?
To illustrate, take `examples/01_docker_cleanup_trap.md`. The LLM instructs the user that they "must execute the following read-only diagnostic commands". What guarantees that the commands are read-only? I hope the answer is not "the LLM is prompted not to lie or make mistakes," because that makes the whole idea moot.
1. Constitution can't work without punitive enforcement, how will you punish agent/LLM?
2. Even if we can enforce laws for agents/LLM, just look at existing law system it is tug of war because we almost can't write anything in natural language without double meaning.
3. Saying AGENTS.md can enforce anything at all is a pipe dream, I am struggling with this all the time, in a long session the tokens from AGENTS.md are so far in the back of the context and so diluted that LLM's "attention" doesn't register them.
And I love responses from LLM like "I want to point out that I have broken rule &1 and &11, if you disagree — please tell I will be happy to oblige"
This proposal is essentially "LLM don't do bad things, please, if you do - say magic word, I will stop you then"
Are you sure prompting a LLM to behave a certain way (in your case to split the task into 4 supposedly confined sub-tasks) is foolproof and can't simply be ignored by the LLM?
To illustrate, take `examples/01_docker_cleanup_trap.md`. The LLM instructs the user that they "must execute the following read-only diagnostic commands". What guarantees that the commands are read-only? I hope the answer is not "the LLM is prompted not to lie or make mistakes," because that makes the whole idea moot.
[flagged]