How does Critic ensure that the narrative, the assumptions, decisions and complexity generated by the agent are correctly brought forward in this different format and not, itself, being vulnerable to the same exact problems?
Now, Critic doesn't interfere with the actual process of code-writing. Your agent is free to write code as it wants. Critic's plugin simply observes the agent during its turn and at certain boundaries, nudges it to write a narrative. The plugin also comes with skills on how to write a concise narrative and how to identify "complex code" that needs annotation. Critic also prevents an agent from ending its turn until it publishes a quality narrative (we use heuristics like word count and format parsing to measure this).
All together, this results in a narrative that is short, direct, and easy to skim through.
word count and format parsing measure the shape of the narrative though, not whether it matches the diff. I think the question is not if it's well formed, but if it makes sense and has all the information, and I don't think you can do that with heuristics.
Do you do anything in addition to that, like for example, an extra agent session where given the narrative and diff, the agent lists the claims it cannot verify from the code, or something along those lines?
How does Critic ensure that the narrative, the assumptions, decisions and complexity generated by the agent are correctly brought forward in this different format and not, itself, being vulnerable to the same exact problems?
Good question!
As you read the rest of this answer, keep in mind that Critic attaches to your agent as a plugin. Plugins can have hooks (https://code.claude.com/docs/en/hooks and https://learn.chatgpt.com/docs/hooks) that pass instruction to an agent at predefined boundaries.
Now, Critic doesn't interfere with the actual process of code-writing. Your agent is free to write code as it wants. Critic's plugin simply observes the agent during its turn and at certain boundaries, nudges it to write a narrative. The plugin also comes with skills on how to write a concise narrative and how to identify "complex code" that needs annotation. Critic also prevents an agent from ending its turn until it publishes a quality narrative (we use heuristics like word count and format parsing to measure this).
All together, this results in a narrative that is short, direct, and easy to skim through.
word count and format parsing measure the shape of the narrative though, not whether it matches the diff. I think the question is not if it's well formed, but if it makes sense and has all the information, and I don't think you can do that with heuristics. Do you do anything in addition to that, like for example, an extra agent session where given the narrative and diff, the agent lists the claims it cannot verify from the code, or something along those lines?
I looked at this landing page for 5min and still have no idea what this is. The description above is doing a better job.