How it is made should be irrelevant in the process. There is no way to stop people using LLM’s or other tools to create products like this report.
It should be globally a standard check to verify incoming documents and to determine their validity.
The interesting part on this case is that we see the real human aspect of it. Just asking for a report only finding the positive facts instead of doing real research.
That is an ethics question. When behaving like this (and it happened before LLM’s) it should be seen as an ethics violation. If not done there is no value in those legal procedures anymore.
The facts people have to accept is that the only place to check this is the deliverable. They are lucky now to have found the source, the prompts and the background but so many reports are entered without being so obvious.
> How it is made should be irrelevant in the process.
No? If you are employed (or requested in this case I guess) to offer your expert opinion, the expert they want to hear from is you, not someone else you farm your work out to - whether that’s an LLM or another person
How is this any different from any job? Or at least any knowledge worker job like an engineer or some sort? They hired you, so presumably they want you to do the work or solve the problem, not farm it out to another person or an LLM.
Does it matter how it was created? Whether you had a subordinate create it, or an employee, or a contractor, googled it, cited a study, or some other method. You as the expert look at the conclusion and either agree with it or disagree.
I think if people looked at LLM as just tools or interns, which you should always check or question their conclusions, then they'd 1) not believe everything LLMs tell you and 2) they would get better responses after critiquing the LLM response.
> How it is made should be irrelevant in the process.
Is that true? Almost all answers have value based on how they were reached, especially when (like this case) there's no trivial and objective verification step.
For example, suppose you live next to a volcano, and I sell you some software that predicts when it will erupt. One day you dig into it and realize I just hard-coded "not today" [0] which is 99.999% accurate.
I think you would be quite angry, and justifiably so... But why would that be if "not today" was correct, and "how it's made is irrelevant to the process?"
Tying it back to the court case, we expect professional witnesses to look at the details and then reach a conclusion, and to reach out based their own professional expert knowledge of cause and effect... Not to pick a conclusion and try to make the details fit, nor to offload executive function to an LLM. Even if they render the same boolean verdict, we care about what process was taken.
“How” as in “what tool” is a different “how” as in “with what intent”. LLM was used to produce a bunch of arguments supporting a pre-intended outcome. Doesn’t matter much whether it was a machine or a human (and whether it was “expert” himself or a ghostwriter), but it does matter - a lot - how the outcome was the pre-selected and the task was to cherry-pick the facts to support it.
> LLM was used to produce a bunch of arguments [...] Doesn’t matter much whether it was a machine or a human (and whether it was “expert” himself or a ghostwriter)
Even ignoring the intent-issue, the process used matters because we don't just care about whether each argument is individually true, we care about which arguments and claims appear at all. Do we trust that the LLM-brain is just as good at bringing up relevant issues as the brain of a human (real) expert?
Like the blind men and the elephant [0]: They're not wrong that there one part is like a rope, and one part is like a spear, and one part is like a fan, etc... But we'd vastly prefer "this is a large land mammal".
> One day you dig into it and realize I just hard-coded "not today" [0] which is 99.999% accurate.
It only becomes visible when you do the research.
The justice process in this example just has to check and verify the inputs. There is no way around it. Today it is this expert, tomorrow another one.
When someone is caught that's an ethical issue and there could be more structural consequences as well. Like blocking the person as an expert or other means. As the expert becomes untrustworthy.
The biggest problem is not really that chatgpt was used but more than the conclusion is decided first and chatgpt is used to find the arguments to support it.
Normal, an expert should look at the facts and elements provided without a predecided result and forge his conviction based on the elements.
Obviously experts might be biased, especially when paid by the company, but it should still be in the understanding and evaluation of documents. Otherwise they are not an expert. They are not "lawyers" with the task to find incriminating or exonerating arguments.
Their testimony should start with something like "in my honest opinion". That is obviously not compatible with taking chatgpt to generate your testimony by asking to prepare arguments that support your customer.
from the point of view of the judge and society, i guess, what matters, ultimately, is whether the arguments presented for a case hold, not where did they come from -- so it shouldn't matter if ChatGPT was the one that came up with the argument if the argument is good.
though one could point out that if one particular method of generating arguments tends to generate arguments that take time to analyse but are often enough pointless, we'd save time by not using them.
so assuming that the report will be analysed on its own merits, really, it was 3M who got scammed here, because they paid $475/hr for a guy that was just prompting ChatGPT. i mean, i don't know much about the subject, but that screenshot of the gas detector with the "what am i looking at here"... wow, it looks like i already know enough to be an expert witness. no wonder these people are so drawn to stuff like positive affirmations, when they do make it by faking it.
I find it very interesting that, from what I understand, the LLM use isn’t actually what’s at issue. It’s really just that he worked backwards, beginning with a conclusion and attempting to find evidence towards that end. The LLM chat logs are just a uniquely astounding piece of evidence towards his incompetence.
> During discovery in the case, Will Moye, one of the plaintiffs’ attorneys, found a five-page document called “Citation Overlay,” which appeared to have been generated by AI. Moye recognized the Citation Overlay document as being from ChatGPT, and demanded all of the prompts Autenrieth used from 3M’s lawyers. The deposition was paused for three hours while they were gathered, and Moye was given 350 pages of ChatGPT conversations that Autenrieth had when creating the report.
Fun fact: archive.is can even bypass a lot of hard paywalls and no one knows for sure how they do it. There's been at least on debate on HN with no clear conclusion: https://news.ycombinator.com/item?id=36060891
They probably use a combination of residential proxies, actually paying for a lot of subscriptions (including lots of small regional publications apparently) and cleverly removing the "My Account" link, referrer shenannigans, and who knows what else.
On the other hand archive.is is notorious for not working for many people. I am trapped in an endless reCaptcha loop, being repeatedly asked in Thai or in Japanese or any of these random Asian languages to mark "whatever" in the pictures, or being asked to scan QR codes for something, and so on.
This is also another obvious example of misalignment on the part of ChatGPT. If the thing had any semblance of an actual system of ethics, or emulation thereof, it would refuse requests to try to make facts fit a predetermined conclusion. In a legal case, no less!
No in the case of an expert witness. And ChatGPT is not, should not and can not be party to any legal proceeding. It isn't even remotely ethically sound to help people twist facts for their own purposes.
It doesn't feel intellectually honest to suggest that expert witnesses are there for any other reason than to promote the case of the side that hired them. They rarely outright lie, but "try to make facts fit a predetermined conclusion" is their job description. They would not be brought to the stand otherwise.
I agree that there are ethical problems around helping people twist facts for their own purposes. My point is that everyone in a courtroom except the judge and jury are there to do exactly that. They make the strongest possible case to get their desired outcome from the available facts.
My father was an expert witness in construction defect and he told me about some of the times he was hired to produce a report about the causes of some defect or failure and that sometimes the entity that hired him was at fault. He usually wasn't retained for trial in those cases, but it earned him a reputation for honesty.
Seeing these pay-for-opinion "experts" fills me with a deep disgust.
It’s a fun idea, I do wonder why the artifact is locked away at the end of the story - what harm could come from it?
Then again, I don’t think I’ve ever really needed help being happy - I’d be curious to hear its suggestions, but I’m not sure what it could tell me that I didn’t already know - and further, there would likely be plenty of times where it would make more sense to do the thing that doesnt bring me the most happiness. “Flex your arm by 35%” just doesn’t sound like a plausibly compelling suggestion, and “the ring is never wrong” is really only useful if my sole goal is to maximize my own happiness.
I hate that this is how AI is being used.. but tbh this is a nothing burger...
This is exactly what a good lawyer/team does.. they argue against what their client is accused of.
a good prosecutor should be able to dismantle the AI argument the same way they would a human argument. just because its AI generated doesn't mean its a corrupting of the process or cheating..
the process is the same either way, its corrupt or broken the same way either way, it works the same way either way.
This was an EXPERT WITNESS, not an advocate. Though unfortunately (and bizarrely) it appears that in the US, expert witnesses in civil cases can be hired to say whatever the hiring party wants them to say without it being perjury.
How it is made should be irrelevant in the process. There is no way to stop people using LLM’s or other tools to create products like this report.
It should be globally a standard check to verify incoming documents and to determine their validity.
The interesting part on this case is that we see the real human aspect of it. Just asking for a report only finding the positive facts instead of doing real research.
That is an ethics question. When behaving like this (and it happened before LLM’s) it should be seen as an ethics violation. If not done there is no value in those legal procedures anymore.
The facts people have to accept is that the only place to check this is the deliverable. They are lucky now to have found the source, the prompts and the background but so many reports are entered without being so obvious.
It’s a terribly huge and complicated task.
> How it is made should be irrelevant in the process.
No? If you are employed (or requested in this case I guess) to offer your expert opinion, the expert they want to hear from is you, not someone else you farm your work out to - whether that’s an LLM or another person
Cynically, the purpose of the expert witness is someone to provide something which is treated as fact but isn't subject to penalties of perjury.
How is this any different from any job? Or at least any knowledge worker job like an engineer or some sort? They hired you, so presumably they want you to do the work or solve the problem, not farm it out to another person or an LLM.
Does it matter how it was created? Whether you had a subordinate create it, or an employee, or a contractor, googled it, cited a study, or some other method. You as the expert look at the conclusion and either agree with it or disagree.
I think if people looked at LLM as just tools or interns, which you should always check or question their conclusions, then they'd 1) not believe everything LLMs tell you and 2) they would get better responses after critiquing the LLM response.
You're at work. You need to produce an artefact – a project proposal, say. 100-odd lines of Gantt, costs, that your BD team will send to the client.
You get Copilot to do this for you.
Later at the review meeting, errors are found. A bunch of stuff was missed, the costs were off.
Whose fault is it? Who looks stupid? If it's bad enough, who gets fired?
> How it is made should be irrelevant in the process.
Is that true? Almost all answers have value based on how they were reached, especially when (like this case) there's no trivial and objective verification step.
For example, suppose you live next to a volcano, and I sell you some software that predicts when it will erupt. One day you dig into it and realize I just hard-coded "not today" [0] which is 99.999% accurate.
I think you would be quite angry, and justifiably so... But why would that be if "not today" was correct, and "how it's made is irrelevant to the process?"
Tying it back to the court case, we expect professional witnesses to look at the details and then reach a conclusion, and to reach out based their own professional expert knowledge of cause and effect... Not to pick a conclusion and try to make the details fit, nor to offload executive function to an LLM. Even if they render the same boolean verdict, we care about what process was taken.
[0] https://xkcd.com/221/
“How” as in “what tool” is a different “how” as in “with what intent”. LLM was used to produce a bunch of arguments supporting a pre-intended outcome. Doesn’t matter much whether it was a machine or a human (and whether it was “expert” himself or a ghostwriter), but it does matter - a lot - how the outcome was the pre-selected and the task was to cherry-pick the facts to support it.
> LLM was used to produce a bunch of arguments [...] Doesn’t matter much whether it was a machine or a human (and whether it was “expert” himself or a ghostwriter)
Even ignoring the intent-issue, the process used matters because we don't just care about whether each argument is individually true, we care about which arguments and claims appear at all. Do we trust that the LLM-brain is just as good at bringing up relevant issues as the brain of a human (real) expert?
Like the blind men and the elephant [0]: They're not wrong that there one part is like a rope, and one part is like a spear, and one part is like a fan, etc... But we'd vastly prefer "this is a large land mammal".
[0] https://en.wikipedia.org/wiki/Blind_men_and_an_elephant
Totally agree on your example:
> One day you dig into it and realize I just hard-coded "not today" [0] which is 99.999% accurate.
It only becomes visible when you do the research.
The justice process in this example just has to check and verify the inputs. There is no way around it. Today it is this expert, tomorrow another one.
When someone is caught that's an ethical issue and there could be more structural consequences as well. Like blocking the person as an expert or other means. As the expert becomes untrustworthy.
>How it is made should be irrelevant in the process
Bullshit. I dare any lawyer to say that to the face of a client paying 100s per hour for "services rendered."
You say you want respect but honey for what, exactly?
The biggest problem is not really that chatgpt was used but more than the conclusion is decided first and chatgpt is used to find the arguments to support it.
Normal, an expert should look at the facts and elements provided without a predecided result and forge his conviction based on the elements.
Obviously experts might be biased, especially when paid by the company, but it should still be in the understanding and evaluation of documents. Otherwise they are not an expert. They are not "lawyers" with the task to find incriminating or exonerating arguments.
Their testimony should start with something like "in my honest opinion". That is obviously not compatible with taking chatgpt to generate your testimony by asking to prepare arguments that support your customer.
> It should be globally a standard check to verify incoming documents and to determine their validity.
Part of that process is bringing in expert witnesses and I don't think ChatGPT qualifies.
>It should be globally a standard check to verify incoming documents and to determine their validity.
And how do you do that? Do you ask... An expert?
Try to think about it.
from the point of view of the judge and society, i guess, what matters, ultimately, is whether the arguments presented for a case hold, not where did they come from -- so it shouldn't matter if ChatGPT was the one that came up with the argument if the argument is good. though one could point out that if one particular method of generating arguments tends to generate arguments that take time to analyse but are often enough pointless, we'd save time by not using them.
so assuming that the report will be analysed on its own merits, really, it was 3M who got scammed here, because they paid $475/hr for a guy that was just prompting ChatGPT. i mean, i don't know much about the subject, but that screenshot of the gas detector with the "what am i looking at here"... wow, it looks like i already know enough to be an expert witness. no wonder these people are so drawn to stuff like positive affirmations, when they do make it by faking it.
I find it very interesting that, from what I understand, the LLM use isn’t actually what’s at issue. It’s really just that he worked backwards, beginning with a conclusion and attempting to find evidence towards that end. The LLM chat logs are just a uniquely astounding piece of evidence towards his incompetence.
The article is pay walled.. Does it explain why the expert witness shared screenshots of these prompts? Voluntarily or required to?
https://archive.is/zKgVp
> During discovery in the case, Will Moye, one of the plaintiffs’ attorneys, found a five-page document called “Citation Overlay,” which appeared to have been generated by AI. Moye recognized the Citation Overlay document as being from ChatGPT, and demanded all of the prompts Autenrieth used from 3M’s lawyers. The deposition was paused for three hours while they were gathered, and Moye was given 350 pages of ChatGPT conversations that Autenrieth had when creating the report.
So he got caught by a keen eye, but could have likely got off scot free, and have his generated report accepted by the court.
https://archive.is/zKgVp
Many paywalls are fairly "soft", and can be got around by visiting an archived version of the page you're interested in, at https://archive.is.
Fun fact: archive.is can even bypass a lot of hard paywalls and no one knows for sure how they do it. There's been at least on debate on HN with no clear conclusion: https://news.ycombinator.com/item?id=36060891
They probably use a combination of residential proxies, actually paying for a lot of subscriptions (including lots of small regional publications apparently) and cleverly removing the "My Account" link, referrer shenannigans, and who knows what else.
On the other hand archive.is is notorious for not working for many people. I am trapped in an endless reCaptcha loop, being repeatedly asked in Thai or in Japanese or any of these random Asian languages to mark "whatever" in the pictures, or being asked to scan QR codes for something, and so on.
the guy who runs it has blocked it for Finland and will have infinite captchas, probably for other countries as well
Not to mention that they are actively DDoSing people
https://en.wikipedia.org/wiki/Archive.today
It's probably your browser setup. Try a clean profile.
This is also another obvious example of misalignment on the part of ChatGPT. If the thing had any semblance of an actual system of ethics, or emulation thereof, it would refuse requests to try to make facts fit a predetermined conclusion. In a legal case, no less!
Doesn't the entire court system hinge upon both sides attempting to make facts fit a predetermined conclusion?
No in the case of an expert witness. And ChatGPT is not, should not and can not be party to any legal proceeding. It isn't even remotely ethically sound to help people twist facts for their own purposes.
It doesn't feel intellectually honest to suggest that expert witnesses are there for any other reason than to promote the case of the side that hired them. They rarely outright lie, but "try to make facts fit a predetermined conclusion" is their job description. They would not be brought to the stand otherwise.
I agree that there are ethical problems around helping people twist facts for their own purposes. My point is that everyone in a courtroom except the judge and jury are there to do exactly that. They make the strongest possible case to get their desired outcome from the available facts.
I hate the expert for not cleaning up the url before sending it over
Interesting that he pointed it to the CSB video - those are great explanations for us laymen, but I assume the LLM would prefer a written report.
Luckily it's still up https://youtube.com/watch?v=CFVUSDzHL8A
I was worried the videos would already be disappearing given Trump's attempts to shut it down https://www.chemicalprocessing.com/safety-security/risk-asse...
We're in the meat proxy era, unplug your brain and feed the machines!
I personally know people who ask chatgpt what to eat, to which bar to go, to plan their vacations, it's insanity.
My father was an expert witness in construction defect and he told me about some of the times he was hired to produce a report about the causes of some defect or failure and that sometimes the entity that hired him was at fault. He usually wasn't retained for trial in those cases, but it earned him a reputation for honesty.
Seeing these pay-for-opinion "experts" fills me with a deep disgust.
Obligatory link: the whispering earring https://gwern.net/doc/fiction/science-fiction/2012-10-03-yva...
(I couldn’t remember the title or the author. So I asked claude for it of course)
It’s a fun idea, I do wonder why the artifact is locked away at the end of the story - what harm could come from it?
Then again, I don’t think I’ve ever really needed help being happy - I’d be curious to hear its suggestions, but I’m not sure what it could tell me that I didn’t already know - and further, there would likely be plenty of times where it would make more sense to do the thing that doesnt bring me the most happiness. “Flex your arm by 35%” just doesn’t sound like a plausibly compelling suggestion, and “the ring is never wrong” is really only useful if my sole goal is to maximize my own happiness.
If you let your legal process decide based on rhetoric, you'll have people use AI to generate rhetoric.
Why not just bypass the expert and just ask GPT to come to a determination? /s
I hate that this is how AI is being used.. but tbh this is a nothing burger...
This is exactly what a good lawyer/team does.. they argue against what their client is accused of.
a good prosecutor should be able to dismantle the AI argument the same way they would a human argument. just because its AI generated doesn't mean its a corrupting of the process or cheating..
the process is the same either way, its corrupt or broken the same way either way, it works the same way either way.
This was an EXPERT WITNESS, not an advocate. Though unfortunately (and bizarrely) it appears that in the US, expert witnesses in civil cases can be hired to say whatever the hiring party wants them to say without it being perjury.
the legal system is less about facts and more about the strength of your argument.