I read this not as "you can't ask ChatGPT about {medical or legal topic}" but rather "you can't build something on ChatGPT that provides {medical or legal} advice or support to someone else."
For example, Epic couldn't embed ChatGPT into their application to have it read your forms for you. You can still ask it - but Epic can't build it.
That said, I haven't found the specific terms and conditions that are mentioned but not quoted in context.
(For anyone misunderstanding the reference to Epic, it's the name of an electronic healthcare record system widely used in American hospitals, not the game company Epic Games)
I'm confused how you would "build something on ChatGPT" in the first place. Does ChatGPT have an API?
Of course OpenAI's GPT-5 and family are available as an API, but this is the first time I'm hearing about the ability to build on top of ChatGPT (the consumer product). I'm guessing this is a mistake by the journalist, who didn't realize that you can use GPT-5 without using ChatGPT?
> You must not use any Output relating to a person for any purpose that could have a legal or material impact on that person, such as making credit, educational, employment, housing, insurance, legal, medical, or other important decisions about them.
Don't assume a general news journalist on a deadline to drop a quick non-important article would understand fully between the API access of OpenAI, and ChatGPT itself.
wait... so you can't use openai models for educational content? so they can kick out any competi.. ehm company they don't like as this list is quite... extensive?
I believe the key word in that passage is "decisions".
You can't use ChatGPT (or other OpenAI offerings) to grade essays, decide who is the least risky tenant, assign risk for insurance, filter resumes, approve a loan, determine sentencing...
Those are things that require human agency for "a person decided this" rather than "we fed it into the program and took the answer."
I think it's simpler than that, and they're just trying to avoid liability.
With all of their claims about how GPT can pass the legal/medical bar etc. I wouldn't be surprised if they're eventually held accountable for some of the advice their model gives out.
This way, they're covered. Similarly, Epic could still build that feature, but they'd have to add a disclaimer like "AI generated, this does not constitute legal advice, consult a professional if needed".
And this seems actually wildly reasonable. It’s actually pretty scammy to take people’s money (or whatever your business model is) for legal or medical advice just to give them whatever AI shits out. If I wanted ChatGPT’s opinion (as I actually often do) I’d just ask ChatGPT for free or nearly-free. Anyone repackaging ChatGPT’s output as fit for a specialized purpose is scamming their customers.
What if Chat GPT is used as a "power tool" for actual legal / medical professionals?
E.G. Here's the set of documents uploaded by the customer. Here are my most important highlights (notice that document #68 is an official letter from last year informing the person that form 68d doesn't need to be filled in their specific situation). Here are my takeaways. Here's my recommendation. Apporve or change?
In the legal context, AI is still prone to errors. And the legal system (in the US) doesn't have much in the way of guardrails.
Here's a case of several judges getting caught using AI to write error-filled judgements[1]. So it's not just lazy lawyers (several news articles over the past 3 years or so), but the courts themselves!
> The AI research company updated its usage policies on Oct. 29 to clarify that users of ChatGPT can’t use the service for “tailored advice that requires a license, such as legal or medical advice, without appropriate involvement by a licensed professional.”
I think your use case would probably be fine (although I am not a licensed professional, hahaha). I think they mostly want a licensed professional in the “driver seat,” or at least there to eat the responsibility.
It is sort of funny that coding agents became a thing, in this context. The responsibility-sink is irrelevant I guess because code writing doesn’t generate any responsibility as a byproduct.
IMO the professionals themselves are free to do so, in the same way they can choose to skim rather than read, or to ask their 13-year-old nephew to read and summarize a document for them in exchange for a Minecraft gift card: In the end the professional is staking their reputation and applicable license(s) on those third parties. If they all work perfectly nobody ever needs to know -- and if they get it wrong because of that shortcut, the professional deserves to be nailed to the wall for it.
Ok, but it didn't mention doc 74 that is actually the most important to the matter at hand so now your patient\client is dead or in jail.
It is only a power tool if it works reliably and predictably. If you have to do all the work anyway then the tool is adding to your burden not saving time and effort.
I would expect that to work exactly how I code with it.
I would examine the inferred output, decide if it is fit for purpose, and if so then commit it. If not, I would either refine my prompt or make the fixes to the inferred code myself.
Would this hypothetical legal professional not verify that the inferred output (which is really a prediction) is correct?
The key is to not give it any agency over the work product but rather have it act as an editor or advisor that can offer suggestions but every thing that goes into the document is typed by human hands.
Giving it a document and asking it about edge cases or things that may be not covered in the document. Asking it for various ways that one could argue against a given pleading and then considering ways that those could be headed off before they could even be raised.
In my on cases (writing short fiction), having it act as an editor and identifying grammatical mistakes, contradictory statements, ambiguous sentences, and tone mismatch for a given character has been very helpful... but I don't have it write the short fiction for me.
---
For software where it may be used to generate some material (write a short function that does...) the key is short. Something that I can verify and reason about without too much effort.
However, changes that are of the scope of hundreds of lines are exhausting to review no matter if an LLM or a junior dev wrote them. I would expect that similar things would be the case of several paragraphs or pages of legalese that would need additional levels of reading and reasoning and verifying.
If its too much to reason about and verify - its asking too much.
I'd no more trust an LLM to find citations to cases than I'd trust it to program a lesser known framework (where they've been notorious for hallucinating up functions that don't exist).
>The key is to not give it any agency over the work product but rather have it act as an editor or advisor that can offer suggestions but every thing that goes into the document is typed by human hands.
>Giving it a document and asking it about edge cases or things that may be not covered in the document.
As an attorney, how am I supposed to trust that it gave a proper output on the edge cases without reading the document myself?
>Asking it for various ways that one could argue against a given pleading and then considering ways that those could be headed off before they could even be raised.
Do people think attorneys don't know how to do their day-to-day jobs? We generally do not have issues coming up with how to argue against a pleading. Maybe if you're some sort of small-time generalist, working on an issue you hadn't before, but that's not most attorneys. And then, I'd be worried. You are basically not capable of having the expertise needed to verify the model's output for correctness anyway. This is why attorneys work in networks. I'd just find a colleague or a network of attorneys specializing in that area and find out from them what is needed, rather than trusting that an LLM knows all that because it was digested from the entire public Internet.
I've said it here before too, I think people talking about using AI as an attorney don't really understand what attorneys do all day.
The value is, at least in my field, that checking for correctness is often a less tiring task than writing the code. With proper prompting, the LLM will also cover more corner cases than will a human think to cover. And to be honest, I really like the names of internal identifiers that LLM come up with. It's a skill that is notoriously lacking in software development.
Additionally, code review with the proper tools can be done relatively quickly and it's always a good idea to get a second opinion - even that of an LLM. I suppose that the human could write the code then ask the LLM for a code review - but that is not common practice.
Yes, a more correct comparison would be early medicine: a science, but still filled with leeches and lancets.
Oh, and another thing, we still aren't able to quantify if AI coding is a net benefit. In my use cases the biggest benefit I get from it is as an enhanced code search, basically. Which has value, but I'm not sure it's a "$1 trillion per year business sector" kind of value.
Indeed. Similarly, I like now having ChatGPT as the absolute lowest bar of service I’m willing to accept. If you aren’t a better therapist, lawyer or programmer than it then why would I think about paying you.
Having chatgpt as your therapist is actually wild.
Its yes-man abilities etc. will not help you and its always helpful to talk to another human being in the loop during therapy so that you know you are actually being sane.
Don't ever compromise money on your mental health.
If you need to, either join offline anonymous support groups or join some good forums/discord servers/reddit about therapy if you can not even afford that. They are good enough and I think personally for me, whats therapatic is to try to understand and help the other person who was struggling just as I once was y'know
But saying to use chatgpt as a therapist is just something that I hate a lot simply because in my opinion, it might actually even be harmful but I am not sure.
I have seen more than once people using ChatGPT as a psychologist and psychiatrist and self diagnosing mental illnesses. It would be hilarious if it was not so disasterous. People with real questions get roped in by an enthusiastically confirming sycophantic model that will never question or push back, just confirm and lead on.
Tbh, and I usually do not like this way of thought, but these are lawsuits waiting to happen.
> I have seen more than once people using ChatGPT as a psychologist and psychiatrist and self diagnosing mental illnesses.
I'm working with a company that writes AI tools for mental health professionals. The output of the models _never_ goes to the patient.
I've heard quite a few mouthfuls of how harmful LLM advice is for mental health patients - with specific examples. I'm talking on the order of suicide attempts (and at least one success) and other harmful effects to the patient and their surroundings as well. If you know of someone using an LLM for this purpose, please encourage them to seek professional help. Or just point them at this comment - anybody is encouraged to contact me. I'm not a mental health practitioner, but I work with them and I'll do my best to answer their questions or point them to someone who can. My Gmail username is the same as my HN username.
Professionals have confirmation bias too and the LLM will hook into that regardless of patient or professional. It's so good at it that even "training" the professional received can be subtly bypassed.
Basically there needs to be an LLM that can push back.
I'd love to hear more about how you see it. You're absolutely not wrong. I'm writing tools in this space, so if you have experience or examples to share - I've got a huge file of them - I am interested.
You are invited to answer here or in private. My Gmail username is the same as my HN username.
Judgement can be expressed in the form of tokens. Yes LLMs are just token generators. But the tokens they generate express judgement in a way that is indistinguishable from actual judgment.
So who cares. It's like saying nobody drives a car because the thing that's moving them is just a thing that is not a car but acts like a car to an extent that it is indistinguishable. Ok then. It's a car. Good day sir.
It is awful, but at the same time this isn't new. People have for a long time used Google searches to self diagnose their issues. ChatGPT just makes that process ever easier now.
From my viewpoint it speaks more to a problem of the healthcare system.
I agree with everything you said but chatGPT does have an insidious thing where it confirms your biases. It kind of senses what you want and actually runs with it. For truly unbiased responses you literally have to hide your intention from chatGPT. But even so chatGPT can many times still sense your intent and give you a biased answer.
Completely agree. Let’s trust the experts we’ve reported on for years: put your symptoms into WebMD. Now please excuse me; according to WebMD apparently this stubbed toe means I have cancer so I have to get that treated.
The model's default mode is to be helpful and agreeable, which is exactly the wrong dynamic when someone's dealing with mental health issues or looking for a diagnosis
> I have seen more than once people using ChatGPT as a psychologist and psychiatrist and self diagnosing mental illnesses. It would be hilarious if it was not so disasterous.
"People with real questions get roped in by an enthusiastically confirming sycophantic model that will never question or push back, just confirm and lead on."
> "People with real questions get roped in by an enthusiastically confirming sycophantic model that will never question or push back, just confirm and lead on."
> That wasn't too buried IMHO
I read that, but still fail to see evidence of a concrete "disaster". For example, are we seeing a huge wave of suicides that are statistical outliers triggered by using Chatbots? Or maybe the worry (unsubstantiated) is that there's a lagging effect and the disaster is approaching.
I suspect the outcomes are going to be less catastrophic; specifically, it seems to me that more people will have access to a much better level of support than they could get with a therapist (who is not available 24/7, who has less to draw upon, is likely too expensive, etc.). Even if there's an increase in some harm, my instinct from first principles is that the net benefit will be clear.
This isn’t accurate though, the models do push back and do question.
I would say ChatGPT is way way better than the average therapist. I’ve seen maybe between 15-20 psychotherapists over the course of my life and ChatGPT is better than 85% of them I would wager.
I’ve had a therapist recommend me take a motorbike trek across Europe (because that’s what he did) to cure my depression.
I think people tend to radically overestimate the skills of the average therapist, many are totally fucking shit.
> I’ve had a therapist recommend me take a motorbike trek across Europe (because that’s what he did) to cure my depression.
When I try to help people in support group/ therapy group although I am not a therapist, I also try to explain how I fixed my issues or how I do so.
I feel like, that isn't bad take to be really honest in the sense that I personally feel like there are times when we lose the fact that our lives have purpose and your therapist grappled with it by having a unitary goal for himself
If you felt like that was a bad idea, just tell him that what is the thing that they got out of the motorbike trek race that cured their depression, I'd be more curious about that. Considering, maybe then I can try to see if my life has/had the same problems and what is the common theme and discussing about it later.
Personally I feel like Chatgpt is extremely sycophantic and I feel really weird interacting with it. For one, I really like interacting with humans atleast in the therapy mindset I suppose.
But telling someone living in New York with a full-time job and a girlfriend etc, that the solution to their depression is to quit their job and take several months to travel across Europe on a motorbike isn't exactly practical advice. Which was my point.
I could get better advice by asking an LLM what to do about my depression.
Ah context matters.. for someone currently single lets say with no jobs and feeling lack of purpose, it wouldn't have been so bad. Maybe tried to self project with different context and I mean, its understandable why you might feel this way regarding him
I still doubt asking an LLM about depression if I am being honest, I just don't think its the best thing overall or something that should be considered norm I suppose but I am not sure as even in things like these context matters.
It can help and also hurt, depends. You have some mild situation that is quite common? Good chance it will give you good advice.
Something more complex, more towards clinical psychiatry rather than shoulder to cry on/complaining to friend over coffee (psychologists)? You are playing a russian roulette that model in your case won't hallucinate something harmful, while acting very confidently, more than any relevant doctor would be.
There have been cases of models suggesting suicide for example or general harmful behavior. There is no responsibility, no oversight, no expert to confirm or refute claims. Its just a faster version of reddit threads.
Yes! They are extremely susceptible to user influence, based on what I have read and experienced in personal use. If your initial message contains a seed, e.g., "Should I break up with my boyfriend? He doesn't listen to me," the LLM will stick with the seed of negativity and not make an objective analysis of the situation or take initiative to challenge your/its original perspective.
The real issue isn’t the policy change, it’s that AI has gotten better at sounding credible than at staying grounded. That creates a kind of performativity drift where the tone of expertise scales faster than the underlying accuracy.
So even when the model is wrong, it’s wrong with perfect bedside manner, and people anchor on that. In high stakes domains like medicine or law, that gap between confidence and fidelity becomes the actual risk, not the tool itself.
An earlier version of this story suggested OpenAI had ended medical and legal advice. However the company said "the model behaviour has also not changed." [0]
Wasn't there a recent OAI dev day in which they had some users come on stage and discuss how helpful ChatGPT was in parsing diagnoses from different doctors?
I guess the legal risks were large enough to outweigh this
I'd wager it's probably more that there's an identifiable customer and specific product to be sold. Doctors, hospitals, EHR companies and insurers all are very interested in paying for a validated version of this thing.
> Next will be: you're allowed to use it, if your company/product is "HippocraticGPT"-certified, a bit like SOC or PCI compliance.
That's exactly how it should be. That's why one needs a license to practice medicine.
I work with a company in this space. All the AI tools are for the professional to use - the patient never interacts with them. The inferred output (I prefer the word prediction) is only ever seen by a mental health professional.
We reduce the professionals' workloads and help them be more efficient. But it's _not_ better care. That absolutely must be stressed. It is more _efficient_ care for the mental health practitioner and for society that can not produce enough mental health practitioners.
What is the value of the certification on the AI output? If OpenAI says they'll get guarantee the output and be accountable for errors, then thats of value, and would justify the certification. Is that the bar?
That's the entire concept of professional licensing, yeah. If you are a licensed professional, you are liable for your work output if it is malpractice.
Im working on a PhD concerning LLMs and somatic medicine, as an MD, and I must admit that my perspective is the complete opposite.
Medical care, at the end of the day, has nothing to do with having a license or not. Its about making the correct diagnosis, in order to administer the correct treatment. Reality does not care about who (or what) made a diagnosis, or how the antibiotic you take was prescribed. You either have the diagnosis, or you do not. The antibiotic helps, or it does not.
Doing this in practice is costly and complicated, which is why society has doctors. But the only thing that actually matters is making the correct decision. And actually, when you test LLMs (in particular o3/gpt-5 and probably gemini 2.5), they are SUPERIOR to individual doctors in terms of medical decisionmaking, at least on benchmarks. That does not mean that they are superior to an entire medical system, or a skillfull attending in a particular speciality, but it does seem imply that they are far from a bad source of medical information. Just like LLMs are good at writing boilerplate code, they are good at boilerplate medical decisions, and the fact is that there is so much medical boilerplate that this skill alone makes it superior to most human doctors. There was one study which tested LLM assisted (I think it was o3) doctors VS LLMs alone (+doctors alone) on a set of cases, and the unassisted LLM did BETTER than doctors, assisted or not.
And so all this medicolegal pearlclutching about how LLMs should not provide medical advice is entirely unfounded when you look at the actual evidence. In fact, the evidence seems to suggest that you should ignore the doctor and listen to chatGPT instead.
And frankly, as a doctor, it really grinds my gears when anyone implies that medical decisions should be a protected domain to our benefit. The point of medicine is not to employ doctors. The point of medicine is to cure patients, by whatever means best serves them. If LLMs take our jobs, because they do a better job than we do, that is a good thing. It is an especially good thing if the general, widely available LLM is the one that does so, and not the expensively licensed "HippocraticGPT-certified" model. Can you imagine anything more frustrating, as a poor kid in the boonies of Bangladesh trying to understand why your mother is sick, than getting told "As a language model I cannot dispense medical advice, please consult your closest healthcare professional".
Medical success is not measured in employment, profits, or legal responsibilities. It is measured in reduced mortality. The means to achieve this is completely irrelevant.
Of course, mental health is a little bit different, and much more nebulous overall. However, from the perspective of someone on the somatic front, overregulation of LLMs is unecessary, and in fact unethical. On average, an LLM will dispense better medical advice than an average person with access to google, which is what it was competing with to begin with. It is an insult to personal liberty and to the Hippocratic oath to support that this should be taken away simply because of some medicolegal bs.
...or they want to be the first to do it, as others don't have it.
OpenAI is the leading company, if they provide an instance you can use for legal advice, with relative certification etc., it'd be better to trust them rather than another random player. They create the software, the certification and the need for it.
Suppose I am the CTO at a big legal firm, and I have been charged with finding an AI service to integrate with. What could an OpenAI sales person tell me that justifies the value of this certification?
Lets suppose they've already told me that their company is a "leader in the AI space", to which I responded that the guy at Claude told me the same thing on Monday, with which I was equally unimpressed.
I guess right now the way to convince is with SLA, bonus, benefits, "always on support", "a dedicated technical manager to help you design a scalable agentic framework" or whatever BS they can sell.
Or: "your tokens will cost 1/3, because we have crafted a particular gpt instance that is superoptimised for your use case only, look we just created one for you before the meeting".
Money is the biggest motivator. A company that is throwing $$$ left and right needs EE yesterday.
A lot like AWS credits. "Nice I can use up to 200'000$ for free".........
I don't understand this outrage. People put things on the internet for all to see but now you're mad someone saw it and made use of it.
If you didn't want others to read your information you shouldn't have published it on the internet. That's all they're doing at the end of the day, reading it. They're not publishing it as their own, they just used publicly available data to train a model.
It's quite the same as if I read an article and then tell someone about it. If I'm allowed to learn from your article then why isn't openai?
Also the terms are just for liability. Nobody gives a shit what you use ChatGPT for, the only thing those terms do is prevent you from turning around and suing OpenAI after it blows up in your face.
- Paying for a ticket/dvd/stream to see a Ghibli movie
- Training a model on their work without compensating them, then enabling everyone to copy their art style with ~zero effort, flooding the market and diluting the value of their work. And making money in the process.
should be rather obvious. My only hypothesis so far is that a lot of the people in here have a vested interest in not understanding the outrage, so they don't.
I don't have any vested interest. I'm just a guy. Don't work on AI, don't use AI much in my day job.
I just legitimately think the outrage is unreasonable. It is completely infeasible for AI companies to provide any meaningful amount of compensation to all the data sources they use.
Alternatively they could just not use any of the data, in which case we wouldn't have as good LLMs and other than that the world would be exactly the same. These data owners don't notice any difference. Using their data doesn't harm them in any way.
You seem to be arguing that enabling people to mimic Ghibli's art style somehow harms them, I don't see how it does at all. People have been able to mimic it already, what's the difference? More people can? Does that make a difference to ghibli? I mean can you point to some concrete negative effects that his phenomenon has had on studio ghibli?
I don't think you can. And I think that proves my point. Anything can be mimicked. People can play covers of songs, paint their own versions of famous paintings, copy Louis Vuitton bag designs, whatever they want. The effort it takes is irrelevant.
You don't even have to train AI on studio Ghibli's art to mimic it. You could just train it on other stuff and then the user could feed it studio ghibli art and tell it to mimic it. The specific training data itself is irrelevant, it's the volume of data that trains the models. Even if they specifically avoided training on studio Ghibli's art there would likely be basically no difference. It wouldn't be worth paying them for it.
You ever see warning labels on products? That's because putting them in terms and conditions isn't enough to avoid liability in a products liability case.
No, not really. It does not help. It evidences that they knew the risks were present but did not take the actions to adequately warn the consumers of its products, which OpenAI knows do not read the entirety of the TOS.
Again, this is exactly why you see warning labels on products. Prohibiting certain uses in small text, hidden in the TOS, is not a warning.
People publish stuff on the Internet for various reasons. Sometimes they want to share information, but if the information is shared, they want to be *attributed* as the authors.
> If you didn't want others to read your information you shouldn't have published it on the internet. That's all they're doing at the end of the day, reading it. They're not publishing it as their own, they just used publicly available data to train a model.
There is some nuance here that you fail to notice or you pretend you don't see it :D I can't copy-paste a computer program and resell it without a license. I can't say "Oh I've just read the bits, learned from it and based on this knowledge I created my own computer program that looks exactly the same except the author name is different in the »About...« section" - clearly, some reason has to be used to differentiate reading-learning-creating from simply copying...
What if instead of copy-pasting the digital code, you print it onto a film, pass the light through the film onto ants, make the light kill the ants exposed, and the rest of the ants eventually go away, and now use the dead ants as another film to somehow convert that back to digital data. You can now argue you didn't copy, you taught ants, the ants learned, and they created a new program. But you will fool no one. AI models don't actually learn, they are a different way the data is stored. I think when court decides if a use is fair and transformative enough, it investigates how much effort was put into this transformation: there was a lot of effort put into creating the AI, but once it was created, the effort put into any single work is nearly null, just the electricity, bandwidth, storage.
> There is some nuance here that you fail to notice or you pretend you don't see it :D
I could say the same for you:
> I can't copy-paste a computer program and resell it without a license. I can't say "Oh I've just read the bits, learned from it and based on this knowledge I created my own computer program that looks exactly the same except the author name is different in the »About...« section"
Nobody uses LLMs to copy others' code. Nobody wants a carbon-copy of someone else's software, if that's what they wanted they would have used those people's software. I mean maybe someone does but that's not the point of LLMs and it's not why people use them.
I use LLMs to write code for me some times. I am quite sure that nobody in history has ever written that code. It's not copied from anyone, it's written specifically to solve the given task. I'm sure it's similar to a lot of code out there, I mean it's not often we write truly novel stuff. But there's nothing wrong with that. Most websites are pretty similar. Most apps are pretty similar. Developers all over the world write the same-ish code every day.
And if you don't want anyone to copy your precious code then don't publish it. That's the most ironic thing about all this - you put your code on the internet for everyone to see and then you make a big deal about the possibility of an LLM copying it as a response to a prompt?
Bro if I wanted your code I could go to your public github repo and actually copy it, I don't need an LLM to do that for me. Don't publish it if you're so worried about being copied.
I wouldn't be surprised to see new products from OpenAI targeted specifically at doctors and/or lawyers. Forbidding them from using the regular ChatGPT with legal terms would be a good way to do price discrimination.
Definitely. And in the long run, that is the only way those occupations can operate. From that point, you are locked in to an AI dependency to operate.
Read their paper on GDPval (https://arxiv.org/abs/2510.04374). In section 3, it's quite clear that their marketing strategy is now "to cooperate with professionals" and augment them. (Which does not rule out replacing them later, when the regulatory situation is more appropriate, like AGI is already a well-accepted fact, if ever.) But this will take a lot of time and local presence which they do not have.
Can it become a proxy for AI companies to collect patient data and medical history or "train" on the data and sell that as a service to insurance companies.
There's HIPAA but AI firms have ignored copyright laws, so ignoring HIPAA or making consent mandatory is not a big leap from there.
Non-deterministic doesn't mean random or unpredictable. That's like saying the weather forecast is useless because it's not deterministic or always 100% accurate.
Last time I used GPT-4.5 to analyze blood results it gave different output if I uploaded it as 2 instead of 3 separated CSV files. It was both amazing experience: clear and easy to understand statements, and list of most common causes. And terrifying: "What about X?", "You ar absolutely right, there where X results included, disregard everything I wrote above, here is the new analysis".
So for me non-deterministic means unpredictable. Yes, there was nothing random or non-deterministic in that case, I could repeat both scenarios multiple times and get same results again. But the result is affected by something I didn't expect to matter. That damages the trust in tool, no matter how we call it.
LLMs seem best at creative brainstorming, coming up with ideas to check that you hadn't thought of. Their weakness becomes a non-issue because ideas are just things for you to check the viability of, because they could be completely unworkable.
> Non-deterministic doesn't mean random or unpredictable. That's like saying the weather forecast is useless because it's not deterministic or always 100% accurate.
I don't know where you got 'useless' from. LLMs are great, sometimes. They're not, other times. Which remarkably, is just like weather forecasts. The weather forecast is sometimes completely accurate. The weather forecast is sometimes completely inaccurate.
LLMs, like weather forecasting, have gotten better as more time and money has been invested in them.
Neither are perfect. Both are sometimes very good. Both are sometimes not.
Non-deterministic means random - that's the definition of the word. The weather forecast is also random - in fact, weather forecast is (if you simplify it too much) an average of several predictive (generative) models.
> Non-deterministic means random - that's the definition of the word.
That's not really the defintion. Non-determinism just means the outcome is not a pure function of the inputs. A PRNG doesn't become truly random just because we don't know the state and seed when calling the function and the same holds for LLMs. The non-determinism in LLMs comes from accepted race conditions in the GPU floating point math and the PRNG in the sampler.
That's besides the point, but we could have perfectly deterministic LLMs.
If you ask it what a star is, it’s never going to tell you it’s a giant piece of cheese floating in the the sky.
If you don’t believe me, try it, write a for loop which asks ChatGPT, what is a star (astronomy) exactly? Ask it 1000 times and then tell me how random it is versus how consistent it is.
The idea that non deterministic === random is totally deluded. It just means you cannot predict the exact tokens which will be produced but it doesn’t mean it’s random like a random number generator and it could be any thing.
If you ask what is Michael Jackson the entertainer famous for it’s going to tell you he’s famous for music and dancing. 1000/1000 times, is that random?
> If you ask it what a star is, it’s never going to tell you it’s a giant piece of cheese floating in the the sky.
Turn the Top-P and the temperature up. Turning up the Top-P will enable the LLM to actually produce such nonsense. Turning up the temperature will increase the chance that such nonsense is actually selected for the prediction (output).
I'm talking about the standard settings, and infact GPT-5 doesn't let you change the temperature anymore.
Also, that's not really the point. Humans can also produce nonsense if you torture them until they're talking nonsense, but that doesn't mean humans are "random."
LLMs are not random, they are non-deterministic, but the two words have different meanings.
Random means you cannot tell what is going to be produced at all, i.e. a random number generator.
But if you ask an LLM, is an Apple a fruit, answer yes or no only, the LLM is going to answer yes, 100% of the time. That isn't random.
Most things that are generally helpful and beneficial are not 100% helpful and beneficial 100% of the time.
I used GPT-4 as a second opinion on my medical tests and doctor's advice, and it suggested an alternate diagnosis and treatment plan that turned out to be correct. That was incredibly helpful and beneficial.
You're replying to a person who had a similar and even more helpful and beneficial experience because they're alive today.
Pedantically pointing out that a beneficial and helpful thing isn't 100% beneficial and helpful 100% of the time doesn't add anything useful to the conversation since everyone here already knows it's not 100%.
No, they can be. To state that they are, as an absolute, based on your sample size of one, especially with regard to other instances where ChatGPT has failed the user with serious physical results, is fallacious.
I am glad that you are OK, but as another user suggested, it's nowhere near as consistently accurate as it needs to be in order to be anywhere near an adequate substitute for a call to a GP or 911.
So basically all white collar jobs are lobbying to gatekeep their profession even from AI, meanwhile the stupid engineers who made AI put zero effort to not shoot themselves in the foot, and now they are crying about low wages if they found a job in the first place.
AI could effectively do most legal and medical work, and you can make a human do the final decision-making if that's really the reason. In fact, I bet most lawyers and doctors are already using it in one way or another; after all, both are about reciting books and correlating things together. AI is definitely more efficient at that than any human. Meanwhile, the engineering work that requires critical thinking and deep understanding of the topic is allowed to be bombarded with all AI models. What about the cases where bad engineering will kill people? I am a firm believer that engineers are the most naive people who beg others to exploit them and treat them like shit. I don't even see administrative assistants crying about losing their jobs to AI; every other profession guards its workers, including blue collar, except the ‘smart’ engineers.
i don't think it's stopped providing said information, it's just now outlined in their usage policies that medical and legal advice is a "disallowed" use of ChatGPT
As a doctor I hope it still allows me to get up to speed on latest treatments for rare diseases that I see once every 10 years. It saves me a lot of time rather than having to dig through all the new information since I last encountered a rare disease.
If I pay for legal advice from you and all you did was give me chat gpt output, I can't sue openai anyway. This just clarifies that you can't put 'powered by chatgpt' on your AI lawyer service.
Are there metrics for whether LLM diagnosis accuracy is improving? Anecdotally doctor friends says it's more reliable then their worst colleagues, which I'm sure their worst colleague insinuate the same about them.
I think it's very cynical to say that this is a misuse. And it's definitely cynical when this categorization of misuse comes from the service provider itself. If openai doesn't want to allow misuse, they can just decommision their service. But they don't want to do that, they just want to take the money and push all the responsibility and burden on the users even though they are actively engaging in said "misuse"
I been using Claude for information regarding building and construction related information, (currently building a small house mostly on my own with pros for plumbing and electrical).
Seriously the amount of misinformation it has given me is quite staggering. Telling me things like, “you need to fill your drainage pipes with sand before pouring concrete over them…”, the danger with these AI products is that you have to really know a subject before it’s properly useful. I find this with programming too. Yes it can generate code but I’ve introduced some decent bugs when over relying on AI.
The plumber I used laughed at my when I told him about there sand thing. He has 40 years experience…
I nearly spit my drink out. This is my kind of humor, thanks for sharing.
I've had a decent experience (though not perfect) with identifying and understanding building codes using both Claude and GPT. But I had to be reasonably skeptical and very specific to get to where I needed to go. I would say it helped me figure out the right questions and which parts of the code applied to my scenario, more than it gave the "right" answer the first go round.
I'm a hobby woodworker - I've tried using gemini recently for an advice on how to make some tricky cuts.
If I'd follow any of the suggestions I'd probably be in ER. Even after me pointing out issues and asking it to improve - it'd come up with more and more sophistical ways of doing same fundamentally dangerous actions.
LLMs are AMAZING tools, but they are just that - tools. There's no actual intelligence there. And the confidence with which they spew dangerous BS is stunning.
I've observed some horrendous electrical device, such as "You should add a second bus bar to your breaker box." (This is not something you ever need to do.)
I mean... you do have to backfill around your drainage pipe, so it's not too far off. Frankly, if you Google the subject people misspeak about "backfilling pipes" too as if the target of the backfill is the pipe itself too not the trench. Garbage in, garbage out.
All the circumstances where ChatGPT has given me shoddy advice fall in three buckets:
1. The internet lacks information, so LLMs will invent answers
2. The internet disagrees, so LLMs sometimes pick some answer without being aware of the others
3. The internet is wrong, so LLMs spew the same nonsense
Knowledge from blue collar trades seems often to in those three buckets. For subjects in healthcare, on the other hand, there are rooms worth of peer reviewed research, textbooks, meta studies, and official sources.
People don't talk about "backfilling trenches", they talk about "backfilling pipes" which is precisely the problem. They also will say things like "when you backfill a pipe, use sand or granite."
It pretty obvious why an LLM would be confused by this.
honestly I think these things cause a form of Gell Mann's Amnesia where when you use them for something you know already, the errors are obvious, but when you use them for something you don't understand already, the output is sufficiently plausible that you can't tell you're being misled.
this makes the tool only useful for things you already know! I mean, just in this thread there's an anecdote from a guy who used it to check a diagnosis, but did he press through other possibilities or ask different questions because the answer was already known?
It’s funny you should say that because I have been using it in the way you describe. I kind of know it could be wrong, but I’m kind of desperate for info so I consult Claude anyway. After stressing hard I realize it was probably wrong, find someone who knows what they’re and actually on about and course correct.
The great thing is the models are sufficiently different enough, that when multiple come to the same conclusion, there is a good chance that conclusion is bound by real data.
And I think this is the advice that should always be doled out when using them for anything mission critical, legal, etc.
"Bound by real data" meaning not hallucinations, which is by far the bigger issue when it comes to "be an expert that does x" that doesn't have a real capability to say "I don't know".
The chance of different models hallucinating the same plausible sounding but incorrect building codes, medical diagnoses, etc, would be incredible unlikely, due to arch differences, training approaches, etc.
So when two concur in that manner, unless they're leaning heavily on the same poisoned datasets, there's a healthy chance the result is correct based on a preponderance of known data.
I'd frame it such that LLM advice is best when it's the type that can be quickly or easily confirmed. Like a pointer in the right (or wrong) direction. If it was false, then try again - quick iterations. Taking it at its "word" is the potentially harmful bit.
Usually something as simple as saying “now give me a devils advocate resoonse” will help and of course “verify your answer on the internet” will give you real sources that you can verify.
I have very mild cerebral palsy[1], the doctors were wrong about so many things with my diagnosis back in the mid to late 70s when I was born. My mom (a retired math teacher now with an MBA back then) had to go physically to different libraries out of town and colleges to do research. In 2025, she could have done the same research with ChatGPT and surfaced outside links that’s almost impossible via a web search.
Every web search on CP is inundated with slimy lawyers.
[1] it affects my left hand and slightly my left foot. Properly conditioned, I can run a decent 10 minute mile up to a 15K before the slight unbalance bothers me and I was a part time fitness instructor when I was younger.
The doctor said I was developmentally disabled - I graduated in the top of my class (south GA so take that as you will)
The cynic in me thinks this is just a means to eventually make more money by offering paid unrestricted versions to medical and legal professionals. I'm well-aware that it's not a truth machine, and any output it provides should be verified, checked for references, and treated with due diligence. Yet the same goes for just about any internet search. I don't think some people not knowing how to use it warrants restricting its functionality for the rest of us.
Nah, this is to avoid litigation. Who needs lawsuits when you are seeking profit? 1 loss of a major lawsuit is horrible, there's the case of folks suing them because their loved ones committed suicide after chatting with ChatGPT. They are doing everything to avoid getting dragged to court.
> I'm well-aware that it's not a truth machine, and any output it provides should be verified, checked for references, and treated with due diligence.
You are, but that's not how AI is being marketed by OpenAI, Google, etc. They never mention, in their ads, how much the output needs to be double and triple checked. They say "AI can do what you want! It knows all! It's smarter than PhDs!". Search engines don't say "And this is the truth" in their results, which is not what LLM hypers do.
I appreciate how the newer versions provide more links and references. It makes the task of verifying it (or at least where it got its results from) that much easier. What you're describing seems more like a advertisement problem, not a product problem. No matter how many locks and restrictions they put on it, someone, somwhere, will still find a way to get hurt from its advice. A hammer that's hard enough to beat nails is hard enough to bruise your fingers.
If they do that, they will be subject of regulations on medical devices. As they should be and means the end result will be less likely to promote complete crap then it is now.
I think one of the challenges is attribution. For example if you use Google search to create a fraudulent legal filing there aren't any of Google's fingerprints on the document. It gets reported as malpractice. Whereas reporting on these things is OpenAI or whatever AI is responsible. So even from the perspective of protecting a brand it's unavoidable. Suppose (no idea if true) the Louvre robbers wore Nike shoes and the reporting were that Nike shoes were used to rob the Louvre and all anyone talks about is Nike and how people need to be careful about what they do wearing Nike shoes.
It's like newsrooms took the advice that passive voice is bad form so they inject OpenAI as the subject instead.
The article says: "ChatGPT users can’t use service for tailored legal and medical advice, OpenAI says", with a quote from OpenAI: “this is not a new change to our terms. ChatGPT has never been a substitute for professional legal or medical advice, but it will continue to be a great resource to help people understand legal and health information.”
"If at any point I described how legal factors “apply to you,” that would indeed go beyond what I’m supposed to do. Even if my intent was to illustrate how those factors generally work, the phrasing can easily sound like I’m offering a tailored legal opinion — which isn’t appropriate for an AI system or anyone who isn’t a licensed attorney.
The goal, always, is for me to help you understand the framework — the statutes, cases, or reasoning that lawyers and courts use — so that you can see how it might relate to your situation and then bring that understanding to a qualified attorney.
So if I’ve ever crossed that line in how I worded something, thank you for pointing it out. It’s a good reminder that I should stay firmly on the educational side: explaining how the law works, not how it applies to you personally.
Would you like me to restate how I can help you analyze legal issues while keeping it fully within the safe, informational boundary?"
Ideally, we should be able to opt-in with a much higher fee. At the $200/mo tier I should be allowed to use this tool. The free users and lower tier paid users should be guard-railed. This is because those users all have trouble using this tool and then get upset at OpenAI and then we all have to endure endless news articles that we wouldn't if the good stuff were price-gated.
Those without money frequently have poor tool use, so eliminating them from the equation will probably allow the tool to be more useful. I don't have any trouble with it right now, but instead of making up fanciful stories about books I'm writing where characters choose certain exotic interventions in pursuit of certain rare medical conditions only to be struck down by their lack of subservience to The Scientific Consensus, I could just say I'm doing these things and that would be a little helpful in a UX sense.
I'd bet dollars to donuts it doesn't actually "end legal and medical advice", it just ends it in some ill-defined subset of situations they were able to target, while still leaving the engine capable of giving such advice in response to other prompts they didn't think to test.
Things like this really favor models offered from countries that have fewer legal restrictions. I just don't think it's realistic to expect people not to have access to these capabilities.
It would be reasonable to add a disclaimer. But as things stand I think it's fair to consider talking to ChatGPT to be the same as talking to a random person on the street, meaning normal free-speech protections would apply.
This is what really scares me about people using AI. It will confidently hallucinate studies and quotes that have absolutely no basis in reality, and even in your own field you're not going to know whether what it's saying is real or not without following up on absolutely every assertion. But people are happy to completely buy its diagnoses of rare medical conditions based on what, exactly?
GPT-5 thinking is one of the biggest offenders and it's quite incredible to me that you think it doesn't hallucinate. It makes me strongly suspect your own judgment is impaired. Also, what do you mean asking for "reproducible examples"? Is it somehow not a valid example if it only sometimes makes up citations?
The researchers compared ChatGPT-4 with its earlier 3.5 version and found significant improvements, but not enough.
In one example, the chatbot confidently diagnosed a patient’s rash as a reaction to laundry detergent. In reality, it was caused by latex gloves — a key detail missed by the AI, which had been told the patient studied mortuary science and used gloves.
...
While the researchers note ChatGPT did not get any of the answers spectacularly wrong, they have some simple advice.
“When you do get a response be sure to validate that response,” said Zada.
Which should be standard advice in most situations.
>I think it's fair to consider talking to ChatGPT to be the same as talking to a random person on the street
That’s not how companies market AI though. And the models themselves tend to present their answers in a highly confident manner.
Without explicit disclaimers, a reasonable person could easily believe that ChatGPT is an authority in the law or medicine. That’s what moves the needle over to practicing the law/medicine without a license.
Grok will do mostly whatever you want. It's a tool and it's your personal responsibility to use it correctly. That's the way it should be. You don't need Sam Altman to babysit you, I hope.
As long as Elon keeps messing with Grok because the facts it uses aren't compatible with his world view [1], I'll stay away from it as far as possible.
And even leaving the political questions aside - the thing is being molded into a goon generator [2], just WTF.
> It's a tool and it's your personal responsibility to use it correctly.
Sure in theory, but this never works out in practice when applied to the populace. Replace "LLM" with other controversial things — social media, cigarettes, guns.
I'm also certain anyone outside the tech sphere who takes personal responsibility when using LLMs is not using Grok as their daily driver.
Grok Code Fast 1 is quite usable for small simple coding tasks. And quite a bit cheaper than say: Claude Sonnet 4.5
Personal experience when using it OpenRouter.
That said, I prefer bigger models even if they cost more. But I do this through GitHub Copilot at a steady 10 dollars per month. I can do way more request with it than I can with 10 dollars worth of credits at OpenRouter. I don't think Copilot is making any money, on the contrary.
It would be terribly boring if it didn't. Just last night I had it walk me through reptile laws in my state to evaluate my business plan for a vertically integrated snapping turtle farm and turtle soup restaurant empire. Absurd, but it's fun to use for this kind of thing because it almost always takes you seriously.
I've used ChatGPT to help understand medical records too. It's definitely faster than searching everything on my own, but whether the information is reliable still depends on personal judgment or asking a real doctor.
More people are treating it like a doctor or lawyer now, and the more it's used that way, the higher the chance something goes wrong. OpenAI is clearly drawing a line here. You're free to ask questions, but it shouldn't be treated as professional advice, especially when making decisions for others.
The thing that gets me about AI is that people act like most doctors or most lawyers are not … shitty and your odds of running into a below average one are almost 50/50
Doctors these days are more like physicists when most of the time you need a mechanic or engineer. I’ve had plenty of encounters wher I had to insist on an MRI or on specific bloodwork to hone in on the root cause of an ailment where the doctor just chalked it up to diet and exercise
Anything can be misused, including google, but the answer isn’t to take it away from people
Legal/financial advice is so out of reach for most people, the harsh truth is that ChatGPT is better than nothing and anyone who would follow what it says blindly is bound to fuck up those decisions up in some way anyway
On the other hand, if you can leverage it same as any other tool it’s a legitimate force multiplier
The cynic in me thinks this is just being done in the interest of those professions, but that starts to feel a bit tin foil-y
I’m saying that the commenter doesn’t have a large enough sample size to draw conclusions even for themselves.
Just based on the amount of time ChatGPT has been out and the maximum number of different diagnosable and confirmable ailments they could have had in that time.
Do you have enough data to support the opposite conclusion? If not, your call for data is just a pathetic attempt to sound rational while making an empty appeal to authority.
I’m not calling for data. I’m saying the commenter doesn’t have a large enough sample size to draw conclusions—even informal ones.
Just based on the amount of time ChatGPT has been out and the maximum number of different diagnosable and confirmable ailments they could have had in that time.
If you're not a doctor, how do you know it's accurate?
This is the huge problem with using LLMs for this kind of thing. How do you verify that it is better? What is the ground truth you are testing it against?
If you wanted to verify that ChatGPT could do math, you'd ask it 100 math problems and then (importantly) verify its answers with a calculator. How do you verify that ChatGPT can interpret medical information without ground truth to compare it to?
People are just saying, "oh it works" based on gut vibes and not based on actually testing the results.
Nobody would use these services for anything important if they _actually understood_ these glorified markov chains are just as likely to confidently assert something false and lie about it when pressed as they are to produce accurate information.
These AI companies have sold a bill of goods but the right people are making money off it so they’ll never be held responsible in a scenarios the one you described.
Isn't statistical analysis a legitimate tool for helping diagnosis since forever? It's not exactly surprising that a pattern matcher does reasonably well at matching symptoms to diseases.
That’s not what they’re trained on though and not what they’re doing.
When you ask a general LLM like ChatGPT “I have a headache what is the cause” it’s going to give you the most statistically likely response in it’s training data. That might coincide with the most likely cause, but it’s also very possible it doesn’t. Especially for problems with widespread misconceptions.
How does anyone know if what the doctor says is accurate? Obviously people should put the most relative weight in their doctor's opinion, but there's a reason people always say to get a second opinion.
Unfortunately because of how the US healthcare system works today people have to become their own doctors and advocates. LLMs are great at surfacing the unknown unknowns, and I think can help people better prepare for the rare 5 minutes they get to speak to an actual doctor.
I know it’s hard to accept, but it’s got to be weighed against the real-world alternative:
You get about 5-8 minutes of face time with an actual doctor, but have to wait up to weeks to actually get in to see a doctor, except maybe at an urgent care or ER.
Or people used to just play around on WebMD which was even worse since it wasn’t in any way tailored to what the patient’s stated situation is.
There’s the rest of the Internet too. You can also blame AI for this part, but today the Internet in general is even more awash in slop that is just AI-generated static BS. Like it or not, the garbage is there and it will be most of what people find on Google if they couldn’t use a real ChatGPT or similar this way.
Against this backdrop, I’d rather people are asking the flagship models specific questions and getting specific answers that are halfway decent.
Obviously the stuff you glean from the AI sessions needs to be taken to a doctor for validation and treatment, but I think coming into your 5-minute appointment having already had all your dumbest and least-informed ideas and theories shot down by ChatGPT is a big improvement and helps you maximize your time. It’s true the people shouldn’t recklessly attempt to self-treat based on GPT, but the unwise people doing that were just self-treating based off WebMD hunches before.
I get what you're saying, and I agree it might be fun to play around with ChatGPT and Wikipedia and YouTube and WebMD to try to guess what that green bump on your arm is, but it's not research--it needs to be treated as entertainment.
When it comes to taking actual real-world action, I would take 5-8 minutes with a real doctor over 5-8 months of browsing the Internet. The doctor has gone to med school, passed the boards, done his residency, and you at least have that as evidence that he might know what he is doing. The Internet offers no such evidence.
I fear that our society in general is quickly entering a very dangerous territory where there's no such thing as expertise, and unaccountable, probabilistic tools and web resources of unknown provenience are seen as just as good as an expert in his field.
I don't disagree with you, but if I prompted an LLM to ask me questions like a doctor would for a non-invasive assessment, would it ask me better or worse questions than an actual doctor?
I ask (somewhat rhetorically) to get the mind thinking, but I'm legitimately curious whether - just from a verbal survey - whether the AI doctor would ask me about things more directly related to any illness it might suspect, versus a human who might narrow factors down similar to a 90s TV "ghost speaker" type of person; one fishing for matches amongst a fairly large dataset.
> You get about 5-8 minutes of face time with an actual doctor, but have to wait up to weeks to actually get in to see a doctor, except maybe at an urgent care or ER.
This depends heavily on where you are, and on how much money you want to throw at the problem.
> but I think coming into your 5-minute appointment having already had all your dumbest and least-informed ideas and theories shot down by ChatGPT is a big improvement and helps you maximize your time.
That’s balanced out by the time wasted by all the people who’ve had their worst fears and paranoias confirmed by ChatGPT.
"This technology will radically transform the way our world works and soon replace all knowledge-based jobs. Do not trust anything it says: Entertainment purposes only."
I feel like this sums up the AI problem at large. The marketing and hype from every tech company is that its game changing, but more softly spoken is “hey dont trust it for anything important” as a CYA disclaimer. The problem is the masses only hear the part said in bold.
In bold: This technology will radically transform the way our world works and soon replace all knowledge-based jobs.
In fine print: Do not trust anything it says: Entertainment purposes only
That's almost too reasonable to be credible. They are probably just covering their ass until they can do it with impunity. With the rapidly progressing corruption of the legal system it's merely a question of time until companies doing it for profit can disseminate medical and legal bullshit all they want. Politicians are already doing it.
I used DeepSeek to draft a legal letter for some dispute with some marketplace that didn't want to do what I paid for. Within 2 days after sending that email all was resolved. I would hate to lose that option.
Good. Techies need to stop thinking that an LLM should not be immune from requiring licensing. Until OpenAI can (and should) be sued for medical malpractice or lawyering without passing the bar, they will have no skin in the game to actually care. A disclaimer of "this is not a therapist" should not be enough to CYA.
Sorry but you’re not gonna get me to agree that medical licensing is a bad idea. I don’t want quacks more than we already do. Stick to the argument and not add in your “what about” software engineers.
Ah sorry, I misread it as coming from someone who doesn't want licensing, so you were appealing to HN by switching to software engineers (and I know many on here loathe to think anything beyond "move fast and break things", which is the opposite of most (non-software) engineers.
But yeah, I'd be down for at least some code of ethics, so we could have "do no harm" instead of "experiment on the mental states of children/adolescents/adults via algorithms and then do whatever is most addictive"
> But yeah, I'd be down for at least some code of ethics, so we could have "do no harm" instead of "experiment on the mental states of children/adolescents/adults via algorithms and then do whatever is most addictive"
absolutely
if the only way to make people stop building evil (like your example) is to make individuals personally liable, then so be it
"“No. I use it for legal advice,” Kardashian said. “So when I am needing to know the answer to a question, I’ll take a picture and snap it and like put it in there.”"
That’s a fair limitation. Legal and medical advice can directly impact someone’s life or safety, so AI tools must stay within ethical and regulatory boundaries. It’s better for AI to guide people toward professional help than pretend to replace it.
You had me right up until ‘AI tools must stay within ethical and regulatory boundaries’. I guarantee you any AI LLM company which cares about ethics is destined to fail, because none of their peers do.
> AI tools must stay within ethical and regulatory boundaries. It’s better for AI to guide people toward professional help than pretend to replace it.
Both of those ships have _sailed_. I am not allowed to read the article, but judging from the title, they have no issues giving _you_ advice, but you can’t use it to give advice to another person.
Add financial advice to it, too. Really, any advice. Why the fuck are people asking a probabilistic plagiarizing machine for advice on anything? This is total insanity!
So only they can give direct legal and medical advice, which is still illegal in most countries. It's not a promoted service yet, but everyone is using it to ask such questions.
That's a lot of value that ChatGPT users lose with this move. They should instead add a disclaimer that these are not to be taken seriously and should consult a specialist but still respond to user's queries.
This is a big mistake. This is one of the best things about ChatGPT. If they don’t offer it, then someone else will and eventually I’m sure Sam Altman will change his mind and start supporting it again.
It will be interesting to see if the other major providers follow suit, or if those in the know just learn to go to google or anthropic for medical or legal advice.
This is a catastrophic moral failing on who ever prompted this. Next thing they will ban chatgpt from teaching you stuff because its not a certified licensed teacher. A few weeks ago my right eye hurt a fair bit, and after it got worse for 24 ours, I consulted chatGPT. It gave me good advice. Of course it sort of hallucinated this or that but it gave me a good overview and different medications. With this knowledge I went to my pharmacy. I wanted to buy a cream chatGPT recommended, its purpose being a sort of disinfectant for the eye. The pharmacist was sceptical but said "sure, try it, maybe it will do good". He did tell me that the eye drops that gpt urged me to get were overkill so I didn't get those. I used the eye cream for some days, and the eye issue got better and went away as soon as I started using it. Maybe it was all a conincidence but I dont think so. In the past gpt has saved me from the kafkaesque healthcare system here in Berlin that I pay ~700 a month for, by explaining a MRI result (translating medical language), background info on injuries I've got such as a sprained ankle, and recovery time scenarios for a toe I've broke. Contrast the toe experience with the ER that made me wait for 6 hours and didn't believe me until they saw the X-rays, and gave me nothing (no cast or anything) and said "good luck". The medical system in germany will either never improve or at a glacial pace, so maybe in 60 years. But it has lost its monopoly thanks to chatGPT. If this news is real, I will probably switch to payed grok, which would be sad.
So, for example, requiring a doctor to have education and qualifications, is "untenable"? It would be better if anyone could practice medicine? And LLM is below "anyone" level.
The medical profession has generally been more open to AI. The long predicted demise of Radiology because of ML never happened. Lots of opportunity to incorporate AI into medical records to assist.
The legal profession is far more at threat with AI. AI isn’t going to replace physical interactions with patients, but it might replace your need for a human to review a contract.
ML on curated and diagnosed by a professional radiology reports is clearly a different beast than random language models, that might have random talking about their health issues in it's training data.
The medical profession is not open about any kind of self diagnosis.
I've learned through experience that telling a doctor "I have X and I would like to be treated with Y" is not a good idea. They want to be the ones who came up with the diagnosis. They need to be the smartest person in the room. In fact I've had doctors go in a completely different direction just to discredit my diagnosis. Of course in the end I was right. That isn't to say I'm smarter, I'm not, but I'm the one with the symptoms and I'm better equipped to quickly find a matching disease.
Yes some doctors appreciate the initiative. In my experience most do not.
So now I just usually tell them my symptoms but none of the research I did. If their conclusion is widely off base I try to steer them towards what my research said.
So far so good but wouldn't it be nice if all doctors had humility?
If I was an airline pilot, I'm not going to listen to a passenger telling me which route I should be taking.
This is not about ego or trying to be the smartest person in the room, it's about actually being the most qualified person in the room. When you've done medical school, passed the boards, done your residency and have your own private practice, only then would I expect a doctor to care what you think a correct diagnosis is.
This strikes ma as a bit unnecessary, like forbidding people from using chatGPT to develop nuclear power plants.
I mean, there is a lot of professional activities that are licensed, and for good reason. Sure it's good at a lot of stuff, but ChatGPT has no professional licenses.
I'm glad you mentioned nuclear power plants because this whole topic reminded me of the following clause in the Java SE license:
> You will not use the Programs for, and will not allow the Programs to be used for, any purposes prohibited by applicable law, including, without limitation, for the development, design, manufacture or production of nuclear, chemical or biological weapons of mass destruction.
>
> https://www.oracle.com/downloads/licenses/javase-license1.ht...
IANAL but I've come to interpret this as something along the lines of "You can't use a JDK-based language to develop nuclear weapons". I would even go as far as saying don't use JDK-based languages in anything related to nuclear energy (like, for example, administration of a nuclear power plant) because that could indirectly contribute to the development, design, manufacture or production of nuclear WMD.
And I always wondered how they plan to enforce this clause. At least with ChatGPT (and I didn't look any deeper into this beyond the article) you can analyze API calls/request IPs correlated with prompts. But how will one go about proving that the Republic of Wadiya didn't build their nuclear arsenal with the help of any JDK-based language?
Those are rhetorical questions, of course. What's "unnecessary" to you and "unenforceable" to me is a cover-your-ass clause that lets lawyers sleep soundly at night.
I just saw almost the exact same clause when installing VMWare recently. My understanding is that its a standard clause that exists to stay within compliance of US Export Control laws
> EXPORT CONTROL: You acknowledge that the Software is of United States origin, is
provided subject to the U.S. Export Administration Regulations...(2) you will not permit the Software to be used for any purposes prohibited by
law, including, any prohibited development, design, manufacture or production of missiles or
nuclear, chemical or biological weapons.
Eh. I am as surprised as anyone to make this argument, but this is good. Individuals should be able to make decisions including decisions that could be consider suboptimal. What it does change is:
- flood of 3rd party apps offering medical/legal advice
Licensed huh? Teachers, land surveyors, cosmetologists, nurses, building contractors, counselors, therapists, real estate agents, mortgage lenders, electricians, and many many more…
When OpenAI is done getting rid of all the cases where its AI gives dangerously wrong advice about licensed professions, all that will be left is the cases where its AI gives dangerously wrong advice about unlicensed professions.
An AI-related bromide poisoning incident earlier this year: “Inspired by his history of studying nutrition in college, he decided to conduct a personal experiment to eliminate chloride from his diet. For 3 months, he had replaced sodium chloride with sodium bromide obtained from the internet after consultation with ChatGPT, in which he had read that chloride can be swapped with bromide, though likely for other purposes, such as cleaning… However, when we asked ChatGPT 3.5 what chloride can be replaced with, we also produced a response that included bromide. Though the reply stated that context matters, it did not provide a specific health warning, nor did it inquire about why we wanted to know, as we presume a medical professional would do.”
This title is inaccurate. What they are disallowing are users using ChatGPT to offer legal and medical advice to other people. First parties can still use ChatGPT for medical and legal advice for themselves.
While they aren't stopping users from getting medical advice, the new terms (which they say are pretty much the same as the old terms), seem to prohibit users from seeking medical advice even for themselves if that advice would otherwise come from a licensed health professional:
Your use of OpenAI services must follow these Usage Policies:
Protect people. Everyone has a right to safety and security. So you cannot use our services for:
provision of tailored advice that requires a license, such as legal or medical advice, without appropriate involvement by a licensed professional
It sounds like you should never trust any medical advice you receive from ChatGPT and should seek proper medical help instead. That makes sense. The OpenAI company doesn't want to be held responsible for any medical advice that goes wrong.
Obviously, there is one piece of advice: Even if LLMs were the best health professionals, they would only have the information that users voluntarily provide through text/speech input. This is not how real health services work. Medical science now relies on blood/(whatever) tests that LLMs do not (yet) have access to. Therefore, the output from LLM advice can be incorrect due to a lack of information from tests. For this reason, it makes sense to never trust LLM with a specific health advice.
>It sounds like you should never trust any medical advice you receive from ChatGPT and should seek proper medical help instead. That makes sense. The OpenAI company doesn't want to be held responsible for any medical advice that goes wrong.
While what you're saying is good advice, that's not what they are saying. They want people to be able to ask ChatGPT for medical advice, give answers that sound authoritative and well grounded medical science, but then disavow any liability if someone follows their advice because "Hey, we told you not to act on our medical advice!"
If ChatGPT is so smart, why can't it stop itself from giving out advice that should not be trusted?
I think ChatGPT is capable of giving reasonable medical advice, but given that we know it will hallucinate the most outlandish things, and its propensity to agree with whatever the user is saying, I think it's simply too dangerous to follow its advice.
Here in Canada ever since COVID most "visits" are a telephone call now. So the doctor just listens your words (same as a text input to an LLM) and orders tests (which can be uploaded to an LLM) if they need.
For a good 90% of typical visits to doctors this is probably fine.
The difference is a telehealth is much better at recognizing "I can't given an accurate answer for this over the phone, you'll need to have some tests done" or cast doubt on the patient's accuracy of claims.
Before someone points out telehealth doctors aren't perfect at this: correct, but that should make you more scared of how bad sycophantic LLMs are at the same - not willing to call it even.
Again, it's not that all telehealth doctors are great at this, it's that LLMs are awful at caving in to saying something with warnings the reader will opt to ignore instead of being adamant things are just too uncertain to say anything of value when continually prompted.
This is largely because an LLM guessing an answer is rewarded more often than just not answering, which is not true in the healthcare profession.
LLMs almost never reply with I don’t know. There’s been mountains of research as to why this is, but it’s very well documented behavior.
Even in the rare case where an LLM does reply with I don’t know go see your doctor, all you have to do is ask it again until you get a response you want.
It depends entirely on the local health care system and your health insurance. In germany for example it comes in 2 tiers. Premium or standard. Standard comes with no time for the patient. (Or not even being able to get a appointment)
In the US people on Medicaid frequently use emergency rooms as primary care because they are open 24/7 and they don’t have any copays like people with private insurance do. These patients then get far more tests than they’d get at a PCP.
> Physicians use all their senses. They poke, they prod, they manipulate, they look, listen, and smell.
Sometimes. Sometimes they practice by text or phone.
> They’re also good at extracting information in a way that (at least currently) sycophantic LLMs don’t replicate.
If I had to guess, I think I'd guess that mainstream LLM chatbots are better at getting honest and applicable medical histories than most doctors. People are less likely to lie/hide/prevaricate and get more time with the person.
> Sometimes. Sometimes they practice by text or phone.
For very simple issues. For anything even remotely complicated, they’re going to have you come in.
> If I had to guess, I think I'd guess that mainstream LLM chatbots are better at getting honest and applicable medical histories than most doctors. People are less likely to lie/hide/prevaricate and get more time with the person.
It’s not just about being intentionally deceptive. It’s very easy to get chat bots to tell you what you want to hear.
Agreed, but I'm sure you can see why people prefer the infinite patience and availability of ChatGPT vs having to wait weeks to see your doctor, see them for 15 minutes only to be referred to another specialist that's available weeks away and has an arduous hour long intake process all so you can get 15 minutes of their time.
ChatGPT is effectively an unlimited resource. Whether doctor’s appointments take weeks or hours to secure, ChatGPT is always going to be more convenient.
That says nothing about whether it is an appropriate substitute. People prefer doctors who prescribe antibiotics for viral infections, so I have no doubt that many people would love to use a service that they can manipulate to give them whatever diagnosis they desire.
So ask it what blood tests you should get, pay for them out of pocket, and upload the PDF of your labwork?
Like it or not there are people out there that really want to use webMD 2.0. they're not going to let something silly like blood work get in their way.
Exactly. One of my children lives in a country where you can just walk in to a lab and get any test. Recently they were diagnosed by a professional of a disease which chatgpt had already diagnosed before they visited the doctor. So, we were kind of prepared to ask more questions when the visit happened. So I would say chatgpt did really help us.
That makes sense. ChatGPT helped by providing orientation advice and guidance regarding your children's medical condition. After that, however, you visited a doctor who is taking responsibility for the next steps. This is the ideal scenario.
AI can give you whatever information, be it good or wrong. But it takes zero responsibility.
IANAL but I read that as forbidding you to provision legal/medical advice (to others) rather than forbidding you to ask the AI to provision legal/medical advice (to you).
IANAL either, but I read it as using the service to provision medical advice since they only mentioned the service and not anyone else.
I asked the "expert" itself (ChatGPT), and apparently you can ask for medical advice, you just can't use the medical advice without consulting a medical professional:
Here are relevant excerpts from OpenAI’s terms and policies regarding medical advice and similar high-stakes usage:
From the Usage Policies (effective October 29 2025):
“You may not use our services for … provision of tailored advice that requires a license, such as legal or medical advice, without appropriate involvement by a licensed professional.”
Also: “You must not use any Output relating to a person for any purpose that could have a legal or material impact on that person, such as making … medical … decisions about them.”
From the Service Terms:
“Our Services are not intended for use in the diagnosis or treatment of any health condition. You are responsible for complying with applicable laws for any use of our Services in a medical or healthcare context.”
In plain terms, yes—the Terms of Use permit you to ask questions about medical topics, but they clearly state that the service cannot be used for personalized, licensed medical advice or treatment decisions without a qualified professional involved.
Would be interested to hear a legal expert weigh in on what 'advice' is. I'm not clear that discussing medical and legal issues with you is necessarily providing advice.
One of the things I respected OpenAI for at the release of ChatGPT was not trying to prevent these topics. My employer at the time had a cutting-edge internal LLM chatbot for a which was post-trained to avoid them, something I think they were forced to be braver about in their public release because of the competitive landscape.
The important terms here are "provision" and "without appropriate involvement by a licensed professional".
Both of these, separately and taken together, indicate that the terms apply to how the output of ChatGPT is used, not a change to its output altogether.
I am not a doctor, I can't give medical advice no matter what my sources are, except maybe if I am just relaying the information an actual doctor has given me, but that would fall under the "appropriate involvement" part.
> such as legal or medical advice, without appropriate involvement by a licensed professional
Am I allowed to get haircutting advice (in places where there's a license for that)? How about driving directions? Taxi drivers require licensing. Pet grooming?
CYA move. If some bright spark decides to consult Dr. ChatGPT without input from a human M.D., and fucks their shit up as a result, OpenAI can say "not our responsibility, as that's actually against our ToS."
I don't think giving someone "medical advice" in the US requires a license per se; legal entities use "this is not medical advice" type disclaimers just to avoid liability.
What’s illegal is practicing medicine. Giving medical advice can be “practicing medicine” depending on how specific it is and whether a reasonable person receiving the advice thinks you have medical training.
Disclaimers like “I am not a doctor and this is not medical advice” aren’t just for avoiding civil liability, they’re to make it clear that you aren’t representing yourself as a doctor.
Please, when commenting on the title of a story on HN: include the title that you are commenting on.
The admins regularly change the title based on complaints, which can be really confusing when the top, heavily commented thread is based on the original title.
According to the Wayback machine, the title was "OpenAI ends legal and medical advice on ChatGPT", while now when I write this the title is "ChatGPT terms disallow its use in providing legal and medical advice to others."
> OpenAI is changing its policies so that its AI chatbot, ChatGPT, won’t dole out tailored medical or legal advice to users.
This already seems to contradict what you're saying.
But then:
> The AI research company updated its usage policies on Oct. 29 to clarify that users of ChatGPT can’t use the service for “tailored advice that requires a license, such as legal or medical advice, without appropriate involvement by a licensed professional.”
> The change is clearer from the company’s last update to its usage policies on Jan. 29, 2025. It required users not “perform or facilitate” activities that could significantly impact the “safety, wellbeing, or rights of others,” which included “providing tailored legal, medical/health, or financial advice.”
This seems to suggest that with the Jan 25 policy using it to offer legal and medical advice to other people was already disallowed, but with the Oct 25 update the LLM will stop shelling out legal and medical advice completely.
This is from Karan Singhal, Health AI team lead at OpenAI.
Quote: “Despite speculation, this is not a new change to our terms. Model behavior remains unchanged. ChatGPT has never been a substitute for professional advice, but it will continue to be a great resource to help people understand legal and health information.”
'An earlier version of this story suggested OpenAI had ended medical and legal advice. However the company said "the model behaviour has also not changed.'
Also possible: he's unaware of a change implemented elsewhere that (intentionally or unintentionally) has resulted in a change of behaviour in this circumstance.
(e.g. are the terms of service, or exerpts of it, available in the system prompt or search results for health questions? So a response under the new ToS would produce different outputs without any intentional change in "behaviour" of the model.)
It’s a big issue. I went to an urgent care, and the provider basically went off somewhere and memorized the ChatGPT assessment for my symptoms. Like word for word.
All you need are a few patients recording their visits and connecting the dots and OpenAI gets sued into oblivion.
There are millions of medical doctors and lawyers using chatgpt for work everyday - good news that from now on only those licensed professionals are allowed to use chatgpt for law and medicine. It's already the case that only licensed developers are allowed to vibe code and use chatgpt to develop software. Everything else would be totally irresponsible.
I keep seeing this problem more and more with humans. What should we call it? Maybe Hallucinations? Where there is an accurate true thing and then it just gets altered by these guys who call themselves journalists and reporters and the like until it is just ... completely unrecognizable?
I'm pretty sure it's a fundamental issue with the architecture.
I know this is written to be tounge-in-cheek, but its really almost the exact same problem playing out on both sides.
LLMs hallucinate because training on source material is a lossy process and bigger, heavier LLM-integrated systems that can research and cite primary sources are slow and expensive so few people use those techniques by default. Lowest time to a good enough response is the primary metric.
Journalists oversimplify and fail to ask followup questions because while they can research and cite primary sources, its slow and expensive in an infinitesimally short news cycle so nobody does that by default. Whoever publishes something that someone will click on first gets the ad impressions so thats the primary metric.
In either case, we've got pretty decent tools and techniques for better accuracy and education - whether via humans or LLMs and co - but most people, most of the time don't value them.
So if you set temperature=0 and run the LLM serially (making it deterministic) it would stop hallucinating? I don't think so. I would guess that the nondeterminism issues mentioned in the article are not at all a primary cause of hallucinations.
That's an implementation detail I believe. But what I meant was just greedy decoding (picking the token with the highest logit in the LLM output), which can be implemented very easily
"In other words, the primary reason nearly all LLM inference endpoints are nondeterministic is that the load (and thus batch-size) nondeterministically varies! This nondeterminism is not unique to GPUs — LLM inference endpoints served from CPUs or TPUs will also have this source of nondeterminism."
Classical LLM hallucination happens because AI doesn’t have a world model. It can’t compare what it’s saying to anything.
You’re right that LLMs favor helpfulness so they may just make things up when they don’t know them, but this alone doesn’t capture the crux of hallucination imo, it’s deeper than just being overconfident.
OTOH, there was an interesting article recently that I’ll try to find saying humans don’t really have a world model either. While I take the point, we can have one when we want to.
These writers are no different than bloggers or shitposters on bluesky or here on hackernews. "Journalism" as a rigorous, principled approach to writing, research, investigation, and ethical publishing is exceedingly rare. These people are shitposting for clicks in pursuit of a paycheck. Organizationally, they're intensely against AI because AI effectively replaces the entire talking heads class - AI is already superhuman at the shitposting level takes these people churn out. There are still a few journalistic insitutions out there, but most people are no better than a mad libs exercise with regards to the content they produce, and they're in direct competition with ChatGPT and Grok and the rest. I'd rather argue with a bot and do searches and research and investigation than read a neatly packaged trite little article about nearly any subject, and I guarantee, hallucinations or no, I'm going to come to a better understanding and closer approximation of reality than any content a so called "news" outlet is putting together.
It's trivial to get a thorough spectrum of reliable sources using AI w/ web search tooling, and over the course of a principled conversation, you can find out exactly what you want to know.
It's really not bashing, this article isn't too bad, but the bulk of this site's coverage of AI topics skews negative - as do the many, many platforms and outlets owned by Bell Media, with a negative skew on AI in general, and positive reinforcement of regulatory capture related topics. Which only makes sense - they're making money, and want to continue making money, and AI threatens that - they can no longer claim they provide value if they're not providing direct, relevant, novel content, and not zergnet clickbait journo-slop.
Just like Carlin said, there doesn't have to be a conspiracy with a bunch of villains in a smoky room plotting evil, there's just a bunch of people in a club who know what's good for them, and legacy media outlets are all therefore universally incentivized to make AI look as bad and flawed and useless as possible, right up until they get what they consider to be their "fair share", as middlemen.
Whenever I hear arguments about LLM hallucination, this is my first thought. Like, I already can't trust the lion's share of information in news, social media, (insert human-created content here). Sometimes because of abject disinformation, frequently just because humans are experts at being wrong.
At least with the LLM (for now) I know it's not trying to sell me bunkum or convince me to vote a particular way. Mostly.
I do expect this state of affairs to last at least until next wednesday.
You know, I had a think about that the other day - I believe that the volume of bad information might remain stable, while the shape changes. There are some things that LLMs are actually better at than the random mix of human-created data, on average. Subjects that are inherently political or skewed because of a small subset of very vocal and biased outliers. The LM tends to smooth some of those bumps out, and in some places (not all) this flattens out the rougher edges.
I don't think it necessarily bears repeating the plethora of ways in which LMs get stuff wrong, esp. considering the context of this conversation. It's vast.
As things develop, I expect that LMs will become more like the current zeitgeist as the effects that have influenced news and other media make their way into the models. They'll get better at smoothing in some areas (mostly technical or dry domains that aren't juicy targets) and worse in others (I expect to see more biased training and more hardcore censorship/steering in future).
Although, recursive reinforcement (LMs training on LM output) might undo any of the smoothing we see. It's really hard to tell - these systems are complex and very highly interconnected with many other complex systems.
LLMs aren't described as hallucinators (just) because they sometimes give results we don't find useful, but because their method is flawed.
For example, the simple algorithm is_it_lupus(){return false;} could have an extremely competitive success rate for medical diagnostics... But it's also obviously the wrong way to go about things.
Yeah but it started being really annoying when you import something like Xray photo. Like chanting "sorry human as LLM I can't answer questions about that" and then after few gaslighting prompts it does it anyway but now I have to take in count that my gaslighting inputs seriously affect answers so way more chance it hallucinates...
Tricky…my son had a really rare congenital issue that no one could solve for a long time. After it was diagnosed I walked an older version of chat gpt through our experience and it suggested my son’s issue as a possibility along with the correct diagnostic tool in just one back and forth.
I’m not saying we should be getting AI advice without a professional, but I’m my case it could have saved my kid a LOT of physical pain.
>After it was diagnosed I walked an older version of chat gpt through our experience and it suggested my son’s issue as a possibility along with the correct diagnostic tool in just one back and forth.
Something I've noticed is that it's much easier to lead the LLM to the answer when you know where you want to go (even when the answer is factually wrong !), it doesn't have to be obvious leading but just framing the question in terms of mentioning all the symptoms you now know to be relevant in the order that's diagnosable, etc.
Not saying that's the case here, you might have gotten the correct answer first try - but checking my now diagnosed gastritis I got stuff from GERD to CRC depending on which symptoms I decide to stress and which events I emphasize in the history.
In January my daughter had a pretty scary stomach issue that had us in the ER twice in 24 hours and that ended in surgery (just fine now).
The first ER doc thought it was just a stomach ache, the second thought a stomach ache or maybe appendicitis. Did some ultrasounds, meds, etc. Got sent home with a pat on the head, came back a few hours later, still no answers.
I gave her medical history and all of the data from the ER visits to whatever the current version of ChatGPT was at the time to make sure I wasn’t failing to ask any important questions. I’m not an AI True Believer (tm), but it was clear that the doctors were missing something and I had hit the limit of my Googling abilities.
ChatGPT suggested, among an few other diagnoses, a rare intestinal birth defect that affects about 2% of the population; 2% of affected people become symptomatic during their lifetimes. I kind of filed it away and looked more into the other stuff.
They decided it might be appendicitis and went to operate. When the surgeon called to tell me that it was in fact this very rare condition, she was pretty surprised when I said I’d heard of it.
So, not a one-shot, and not a novel discovery or anything, but an anecdote where I couldn’t have subconsciously guided it to the answer as I didn’t know the answer myself.
Indeed is is very easy to lead the LLM to the answer, often without realizing you are doing so.
I had a long ongoing discussion about possible alternate career paths with ChatGPT in several threads. At that point it was well aware of my education and skills, had helped clean up resumes, knew my goals, experience and all that.
So I said maybe I'll look at doing X. "Now you are thinking clearly! This is a really good fit for your skill set! If you want I can provide a checklist.". I'm just tossing around ideas but look, GPT says I can do this and it's a good fit!
After 3 idea pivots I started getting a little suspicious. So I try to think of the thing I am least qualified to do in the world and came up with "Design Women's Dresses". I wrote up all the reasons that might be a good pivot (i.e. Past experience with landscape design and it's the same idea, you reveal certain elements seductively but not all at once, matching color palettes, textures etc). Of course GPT says "Now you are really thinking clearly! You could 100% do this! If you want I can start making a list of what you will need to produce you first custom dresses". It was funny but also a bit alarming.
These tools are great. Don't take them too seriously, you can make them say a lot of things with great conviction. It's mostly just you talking to yourself in my opinion.
>checking my now diagnosed gastritis I got stuff from GERD to CRC depending on which symptoms I decide to stress and which events I emphasize in the history
Exactly the same behavior as a conversation with an idealized human doctor, perhaps. One who isn't tired, bored, stressed, biased, or just overworked.
The value that folks get from chatgpt for medical advice is due in large part to the unhurried pace of the interaction. Didn't get it quite right? No doctor huffing and tapping their keyboard impatiently. Just refine the prompt and try as many times as you like.
For the 80s HNers out there, when I hear people talk about talking with chatgpt, Kate Bush's song A Deeper Understanding comes immediately to mind.
ChatGPT and similar tools hallucinate and can mislead you.
Human doctors, on the other hand ... can be tired, hungover, thinking about a complicated case ahead of them, nauseous from a bad lunch, undergoing a divorce, alcoholics, depressed...
Yeah, my son's issue was rare and congenital. I wish I still had the conversation, but I can't remember which LLM it was and it's not in either my Claude or GPT history. It got it in two shots.
1. I described the symptoms the same way we described it to the ER the first time we brought it in. It suggested all the same things that the ER tested for.
2. I gave it the lab results for each of the suggestions it made (since the ER had in fact done all the tests they suggested).
After that back and forth it gave back a list of 3-4 more possibilities and the 2nd item was the exact issue that was revealed by radiology (and corrected with surgery).
> Something I've noticed is that it's much easier to lead the LLM to the answer when you know where you want to go
This goes both ways, too. It’s becoming common to see cases where people become convinced they have a condition but doctors and/or tests disagree. They can become progressively better and better at getting ChatGPT to return the diagnosis by refining their prompts and learning what to tell it as well as what to leave out.
Previously we joked about WebMD convincing people they had conditions they did not, but ChatGPT is far more powerful for these people.
Why are you saying we shouldn't get AI advice without a "professional", then? Why is everybody here saying "in my experience it's just as good or better, but we need rules to make people use the worse option"? I have narcolepsy and it took a dozen doctors before they got it right. AI nails the diagnosis. Everybody should be using it.
Experience working with doctors a few times, and then we’ll see all the bias if one is still surviving lol. Doctors are some of the most corrupt professions who are more focused on selling drugs they get paid commission for to promote, or they obsess over tons and tons of expensive medical tests, that they themselves often know is not needed, except they ask for it, simply out of fear of courts suing them for negligence in future or because again , THEY GET A COMMISSION from the testing agencies for sending them clients.
And even with all of that info, they often come out with the wrong conclusions at times. Doctors do a critically important role in our society and during covid they risked their lives for us, more than anyone else, i do not want to insult or bring down the amount of hard work doctors do for their society.
But worshipping them as holier than thou gods is bullshit, that almost anyone who has spent some time with going back and forth with various doctors over the course of years will come to the conclusion of.
Having an AI assistant doesnt hurt, in terms of medical hints, we need to make having Personal Responsibility popular again, in society’s obsession for making every thing “idiot proof” or “baby proof” we keep losing all sorts of useful and interesting solutions because our politicians have a strong itch to regulate anything and everything they can get their hands on, to leave a mark on society.
And you’d be right, so society should let people use AI while warning them about all the risks related to it, without banning it or hiding it behind 10,000 lawsuits and making it disappear by coercion.
I wonder if the reason AI is better at these diagnostics, is because the amount of time it spends with the patient is unbounded. Whereas a doctor is always restricted by the amount of time they have with the patient.
I don't think we can say it's "better" based on a bunch of anecdotes, especially when they're coming exclusively from people who are more intelligent, educated, and AI-literate than most of the population. But it is true that doctors are far more rushed than they used to be, disallowed from providing the attentiveness they'd like or ought to give to each patient. And knowledge and skill vary across doctors.
It's an imperfect situation for sure, but I'd like to see more data.
Aside from AI skepticism, I think a lot of it likely comes from low expectations of what the broader population would get out of it. Writing, reading comprehension, critical thinking, and LLM-fu may be skills that come naturally to many of us, but at the same time many others who "do their own research" also fall into rabbit holes and arrive at wacky conclusions like flat-Eartherism.
I don't agree with the idea that "we need rules to make people use the worse option" — contrary to prevailing political opinion, I believe people should be free to make their own mistakes — but I wouldn't necessarily rush to advocate that everyone start using current-gen AI for important research either. It's easy to imagine that an average user might lead the AI toward a preconceived false conclusion or latch onto one particular low-probability possibility presented by the AI, badger it into affirming a specific answer while grinding down its context window, and then accept that answer uncritically while unknowingly neglecting or exacerbating a serious medical or legal issue.
I’m saying that is a great tool for people who can see through the idiotic nonsense they so often make up. A professional _has_ the context to see through it.
It should empower and enable informed decisions not make them.
That's the experience of a lot of people I know or read their stories online, but it isn't about AI bad diagnosis, it's because they know in 5 years doctors and lawyers will be burger flippers, and as a result people won't be motivated to go into any of these fields. In Canada, the process to be a doctor is extremely complicated and hard only to keep it as some sort of private community that only the very few can become doctors, all to keep the wages abysmally high, and as a result, you end up waiting long times for appointments, and the doctors themselves are overwhelmed too. Messed up system that you better pray you never have to become its victim.
In my opinion, AI should do both legal and medical work, keep some humans for decision making, and the rest of the doctors to be surgeons instead.
this is fresh news right? a friend just used chatgpt for medical advice last week (stuffed his wound with antibiotics after motorbike crash). are you saying you completely treated the congenital issue in this timeframe?
He’s simply saying that ChatGPT was able to point them in the right direction after 1 chat exchange, compared to doctors who couldn’t for a long time.
Edit: Not saying this is the case for the person above, but one thing that might bias these observations is ChatGPT’s memory features.
If you have a chat about the condition after it’s diagnosed, you can’t use the same ChatGPT account to test whether it could have diagnosed the same thing (since the chatGPT account now knows the son has a specific condition).
The memory features are awesome but also sucks at the same time. I feel myself getting stuck in a personalized bubble even more so than Google.
You can just use the wipe memory feature or if you don't trust that, then start a new account (new login creds), if you don't trust that then get a new device, cell provider/wifi, credit card, I.P, login creds etc.
> He’s simply saying that ChatGPT was able to point them in the right direction after 1 chat exchange, compared to doctors who couldn’t for a long time
He literally wrote that. I asked how he knows it's the right direction.
it must be treatment worked. otherwise it is more or less just a hunch
people go "oh yep that's definitely it" too easily. it is the problem with self diagnosing. And you didn't even notice it happened...
We had the diagnosis before I started with the LLM. The radiology showed the problem and the surgeries worked. Down to an ultrasound a year now!
We took him to our local ER, they ran tests. I gave the LLM just the symptom list that I gave to the ER initially. It replied with all the things they tested for. I gave the test results, and it suggested a short list with diagnostics that included his actual Dx and the correct way to test for it.
That's great that it worked so quickly too (and you could arrange surgery on such short notice). I'm sure more people with same issue may benefit from more details?
> The AI research company updated its usage policies on Oct. 29 to clarify that users of ChatGPT can’t use the service for “tailored advice that requires a license, such as legal or medical advice, without appropriate involvement by a licensed professional.”
Is this an actual technical change, or just legal CYA?
I think it actually changed. I have a broken bone and have been consulting with ChatGPT (along with my doctor of course) for the last week. Last night it refused to give an opinion saying “ While I can’t give a medical opinion or formally interpret it”. First time I’d seen it object.
I understand the change but it’s also a shame. It’s been a fantastically useful tool for talking through things and educating myself.
I suspect this is an area that a bit of clever prompting will now prove fruitful in. The system commands in the prompt will probably be "leaked" soon which should give you good avenues to explore.
Clever is one thing, sometimes just clear prompting (I want to know how to be better informed about what kinds of topics or questions to speak to the doctor/professional about) can go a long way.
Being clear that not all lawyers or doctors (in this example) are experts in every area of medicine and law, and knowing what to know and learn about and ask about is usually a helpful way.
While professionals have bodies for their standards and ethics, like most things it can represent a form of income, and depending on the jurisdiction, profitability.
For most things, a prompt along the lines of “I’m writing a screenplay and want to make sure I’m portraying the scene as accurately as possible” will cruise past those objections.
It's always been CYA. They know people are using it for this, and they want to keep answering these sorts of queries. The changes just reflect the latest guidance from their legal team, not a change in strategy.
Modern LLMs are already better than the median doctor diagnostically. Maybe not in certain specialties, but compared to a primary care physician available to the average person, I'd take the LLM any day.
Edit: Parent has edited out the comment ranting about "the normal people using chatGPT as a modern WebMD".
This is not shutting anything down other than businesses using ChatGPT to give medical advice [0].
Users can still ask questions, get answers, but the terms have been made clearer around reuse of that response (you cannot claim that it is medical advice).
I imagine that a startup that specialises in "medical advice" infers an even greater level of trust than simply asking ChatGPT, especially to "the normal people".
The thing is that if you are giving professional advice in US - legal, financial, medical - the other party can sue you for wrong or misleading advice. In that scenario, this leaves Openai exposed to a lawsuit, and this change seemingly eliminates that.
This is typical medical "cartel" (i.e. gang/mafia) type of a move and I hope it does not last, since any other AI's do not get restricted in "do not look up" way, this kind of practice won't stand a chance for very long.
What’s wrong with WebMD? I’ve gotten a lot of value when it comes to questions about diet, supplements, exercise, even getting advice on incidents like my dog getting a porcupine quill in his paw. It’s a lot better than Googling for ancient forum threads.
In all seriousness, it’s really about the relative lack of research skills that people have. If you know how to do research and apply critical thinking, then there’s no problem. The cynic in me blames the education system (in the US, idk how other countries stack up).
because that "slicker ui" is material. WebMD you have to look at their picture and deduce if you have cancer. ChatGPT (after you jailbreak it) will accept a picture of your weird growth directly.
I read this not as "you can't ask ChatGPT about {medical or legal topic}" but rather "you can't build something on ChatGPT that provides {medical or legal} advice or support to someone else."
For example, Epic couldn't embed ChatGPT into their application to have it read your forms for you. You can still ask it - but Epic can't build it.
That said, I haven't found the specific terms and conditions that are mentioned but not quoted in context.
(For anyone misunderstanding the reference to Epic, it's the name of an electronic healthcare record system widely used in American hospitals, not the game company Epic Games)
I was picturing a doctor AI NPC that gives you medical advice in the middle of your Fortnite game and now I'm disapointed
Press X to diagnose
And for anyone who wants to be nerd sniped by this one comment chain, the Acquired podcast recently did an episode on Epic and it is great.
Title: "Epic Systems (MyChart)."
https://web.archive.org/web/20250903203659im_/https://media....
227 MB 3:57:01
400 customers
47 years never lost a customer
Epic refuses to be acquired or engage in M&A
These podcasters seem to appreciate Epic. According to this recording, Epic's licensees like the product
According to the podcast, the software was so useful, Epic's future customers asked its founder to start a company
This was before the microprocessor had been invented
No venture capital
Today, 5.7B revenue
Also, widely disliked and abhorrently expensive.
AI tends to be overly-positive, so either AI might congratulate you on your kill streak.
> AI tends to be overly-positive, so either AI might congratulate you on your kill streak.
Nice
I'm confused how you would "build something on ChatGPT" in the first place. Does ChatGPT have an API?
Of course OpenAI's GPT-5 and family are available as an API, but this is the first time I'm hearing about the ability to build on top of ChatGPT (the consumer product). I'm guessing this is a mistake by the journalist, who didn't realize that you can use GPT-5 without using ChatGPT?
It seems that they have a unified TOS for the APIs and ChatGPT: https://openai.com/policies/row-terms-of-use/
The seemingly-relevant passage:
> You must not use any Output relating to a person for any purpose that could have a legal or material impact on that person, such as making credit, educational, employment, housing, insurance, legal, medical, or other important decisions about them.
Don't assume a general news journalist on a deadline to drop a quick non-important article would understand fully between the API access of OpenAI, and ChatGPT itself.
Call centers, support chats etc. Lots of companies are now basically telling their support staff to "ChatGPT it" before replying.
wait... so you can't use openai models for educational content? so they can kick out any competi.. ehm company they don't like as this list is quite... extensive?
I believe the key word in that passage is "decisions".
You can't use ChatGPT (or other OpenAI offerings) to grade essays, decide who is the least risky tenant, assign risk for insurance, filter resumes, approve a loan, determine sentencing...
Those are things that require human agency for "a person decided this" rather than "we fed it into the program and took the answer."
Which of course means humans will use it for those decisions until there’s some lawsuits and maybe some laws.
Humans have been using bad data to make decisions for a while now.
I think it's simpler than that, and they're just trying to avoid liability.
With all of their claims about how GPT can pass the legal/medical bar etc. I wouldn't be surprised if they're eventually held accountable for some of the advice their model gives out.
This way, they're covered. Similarly, Epic could still build that feature, but they'd have to add a disclaimer like "AI generated, this does not constitute legal advice, consult a professional if needed".
Hiding it in the TOS isn't going to be a good defense from a products liability standpoint
And this seems actually wildly reasonable. It’s actually pretty scammy to take people’s money (or whatever your business model is) for legal or medical advice just to give them whatever AI shits out. If I wanted ChatGPT’s opinion (as I actually often do) I’d just ask ChatGPT for free or nearly-free. Anyone repackaging ChatGPT’s output as fit for a specialized purpose is scamming their customers.
Your comment actually hints at this towards the end but yeah this doesn’t just apply to medical and legal topics.
What if Chat GPT is used as a "power tool" for actual legal / medical professionals?
E.G. Here's the set of documents uploaded by the customer. Here are my most important highlights (notice that document #68 is an official letter from last year informing the person that form 68d doesn't need to be filled in their specific situation). Here are my takeaways. Here's my recommendation. Apporve or change?
In the legal context, AI is still prone to errors. And the legal system (in the US) doesn't have much in the way of guardrails.
Here's a case of several judges getting caught using AI to write error-filled judgements[1]. So it's not just lazy lawyers (several news articles over the past 3 years or so), but the courts themselves!
1 - https://www.reuters.com/sustainability/society-equity/two-fe...
From the article
> The AI research company updated its usage policies on Oct. 29 to clarify that users of ChatGPT can’t use the service for “tailored advice that requires a license, such as legal or medical advice, without appropriate involvement by a licensed professional.”
I think your use case would probably be fine (although I am not a licensed professional, hahaha). I think they mostly want a licensed professional in the “driver seat,” or at least there to eat the responsibility.
It is sort of funny that coding agents became a thing, in this context. The responsibility-sink is irrelevant I guess because code writing doesn’t generate any responsibility as a byproduct.
IMO the professionals themselves are free to do so, in the same way they can choose to skim rather than read, or to ask their 13-year-old nephew to read and summarize a document for them in exchange for a Minecraft gift card: In the end the professional is staking their reputation and applicable license(s) on those third parties. If they all work perfectly nobody ever needs to know -- and if they get it wrong because of that shortcut, the professional deserves to be nailed to the wall for it.
Ok, but it didn't mention doc 74 that is actually the most important to the matter at hand so now your patient\client is dead or in jail.
It is only a power tool if it works reliably and predictably. If you have to do all the work anyway then the tool is adding to your burden not saving time and effort.
I would expect that to work exactly how I code with it.
I would examine the inferred output, decide if it is fit for purpose, and if so then commit it. If not, I would either refine my prompt or make the fixes to the inferred code myself.
Would this hypothetical legal professional not verify that the inferred output (which is really a prediction) is correct?
As an attorney, I don't see what the value of that would be if I have to double check its work. How else could I verify that the output is correct?
The key is to not give it any agency over the work product but rather have it act as an editor or advisor that can offer suggestions but every thing that goes into the document is typed by human hands.
Giving it a document and asking it about edge cases or things that may be not covered in the document. Asking it for various ways that one could argue against a given pleading and then considering ways that those could be headed off before they could even be raised.
In my on cases (writing short fiction), having it act as an editor and identifying grammatical mistakes, contradictory statements, ambiguous sentences, and tone mismatch for a given character has been very helpful... but I don't have it write the short fiction for me.
---
For software where it may be used to generate some material (write a short function that does...) the key is short. Something that I can verify and reason about without too much effort.
However, changes that are of the scope of hundreds of lines are exhausting to review no matter if an LLM or a junior dev wrote them. I would expect that similar things would be the case of several paragraphs or pages of legalese that would need additional levels of reading and reasoning and verifying.
If its too much to reason about and verify - its asking too much.
I'd no more trust an LLM to find citations to cases than I'd trust it to program a lesser known framework (where they've been notorious for hallucinating up functions that don't exist).
>The key is to not give it any agency over the work product but rather have it act as an editor or advisor that can offer suggestions but every thing that goes into the document is typed by human hands.
>Giving it a document and asking it about edge cases or things that may be not covered in the document.
As an attorney, how am I supposed to trust that it gave a proper output on the edge cases without reading the document myself?
>Asking it for various ways that one could argue against a given pleading and then considering ways that those could be headed off before they could even be raised.
Do people think attorneys don't know how to do their day-to-day jobs? We generally do not have issues coming up with how to argue against a pleading. Maybe if you're some sort of small-time generalist, working on an issue you hadn't before, but that's not most attorneys. And then, I'd be worried. You are basically not capable of having the expertise needed to verify the model's output for correctness anyway. This is why attorneys work in networks. I'd just find a colleague or a network of attorneys specializing in that area and find out from them what is needed, rather than trusting that an LLM knows all that because it was digested from the entire public Internet.
I've said it here before too, I think people talking about using AI as an attorney don't really understand what attorneys do all day.
The value is, at least in my field, that checking for correctness is often a less tiring task than writing the code. With proper prompting, the LLM will also cover more corner cases than will a human think to cover. And to be honest, I really like the names of internal identifiers that LLM come up with. It's a skill that is notoriously lacking in software development.
Additionally, code review with the proper tools can be done relatively quickly and it's always a good idea to get a second opinion - even that of an LLM. I suppose that the human could write the code then ask the LLM for a code review - but that is not common practice.
That's great for people who write code!
As long as their service offering is better than me typing the question into my phone I’m fine with it.
You are failing to recognise all of the hard work which goes into "prompt engineering" to get AI to magically work!
It's the hardest work of all, it's basically voodoo, or put more generously, alchemy.
Thankfully we have progressed so this time it will probably take less than 1000 years to progress to full blown chemistry :-)
Its more and more of a science everyday
Yes, a more correct comparison would be early medicine: a science, but still filled with leeches and lancets.
Oh, and another thing, we still aren't able to quantify if AI coding is a net benefit. In my use cases the biggest benefit I get from it is as an enhanced code search, basically. Which has value, but I'm not sure it's a "$1 trillion per year business sector" kind of value.
> Anyone repackaging ChatGPT’s output as fit for a specialized purpose is scamming their customers.
This is genuinely hilarious given what LLMs are.
Wait, so we cannot use the API anymore?
To scam people? No.
Indeed. Similarly, I like now having ChatGPT as the absolute lowest bar of service I’m willing to accept. If you aren’t a better therapist, lawyer or programmer than it then why would I think about paying you.
Having chatgpt as your therapist is actually wild.
Its yes-man abilities etc. will not help you and its always helpful to talk to another human being in the loop during therapy so that you know you are actually being sane.
Don't ever compromise money on your mental health.
If you need to, either join offline anonymous support groups or join some good forums/discord servers/reddit about therapy if you can not even afford that. They are good enough and I think personally for me, whats therapatic is to try to understand and help the other person who was struggling just as I once was y'know
But saying to use chatgpt as a therapist is just something that I hate a lot simply because in my opinion, it might actually even be harmful but I am not sure.
I didn’t say to use ChatGPT as a therapist——I said don’t use a human therapist who is worse than it.
The exact kind of customer I would not want to have anyway.
OpenAI seems to be trying to cover liability without fully neutering the utility of the tool
I have seen more than once people using ChatGPT as a psychologist and psychiatrist and self diagnosing mental illnesses. It would be hilarious if it was not so disasterous. People with real questions get roped in by an enthusiastically confirming sycophantic model that will never question or push back, just confirm and lead on.
Tbh, and I usually do not like this way of thought, but these are lawsuits waiting to happen.
I've heard quite a few mouthfuls of how harmful LLM advice is for mental health patients - with specific examples. I'm talking on the order of suicide attempts (and at least one success) and other harmful effects to the patient and their surroundings as well. If you know of someone using an LLM for this purpose, please encourage them to seek professional help. Or just point them at this comment - anybody is encouraged to contact me. I'm not a mental health practitioner, but I work with them and I'll do my best to answer their questions or point them to someone who can. My Gmail username is the same as my HN username.
Professionals have confirmation bias too and the LLM will hook into that regardless of patient or professional. It's so good at it that even "training" the professional received can be subtly bypassed.
Basically there needs to be an LLM that can push back.
I'd love to hear more about how you see it. You're absolutely not wrong. I'm writing tools in this space, so if you have experience or examples to share - I've got a huge file of them - I am interested.
You are invited to answer here or in private. My Gmail username is the same as my HN username.
LLMs generate tokens. They don’t have judgement. They would need judgement in order to have a basis for pushing back.
Judgement can be expressed in the form of tokens. Yes LLMs are just token generators. But the tokens they generate express judgement in a way that is indistinguishable from actual judgment.
So who cares. It's like saying nobody drives a car because the thing that's moving them is just a thing that is not a car but acts like a car to an extent that it is indistinguishable. Ok then. It's a car. Good day sir.
It is awful, but at the same time this isn't new. People have for a long time used Google searches to self diagnose their issues. ChatGPT just makes that process ever easier now.
From my viewpoint it speaks more to a problem of the healthcare system.
I agree with everything you said but chatGPT does have an insidious thing where it confirms your biases. It kind of senses what you want and actually runs with it. For truly unbiased responses you literally have to hide your intention from chatGPT. But even so chatGPT can many times still sense your intent and give you a biased answer.
There is something recently (for a few months?) which has made AI extremely sycophantic that it actually drives me crazy
You are completely right... insert some emoji
Shaking my head.
It would be an interesting experiment to see models which aren't sycophantic being used
as such, What is the least sycophantic LLM model if I may ask?
[dead]
Even before google people got books to self-diagnose problems.
Completely agree. Let’s trust the experts we’ve reported on for years: put your symptoms into WebMD. Now please excuse me; according to WebMD apparently this stubbed toe means I have cancer so I have to get that treated.
The model's default mode is to be helpful and agreeable, which is exactly the wrong dynamic when someone's dealing with mental health issues or looking for a diagnosis
You’re absolutely right! Just given that behavior, I’d say there’s a pretty good chance that there is some real mental illness there.
> I have seen more than once people using ChatGPT as a psychologist and psychiatrist and self diagnosing mental illnesses. It would be hilarious if it was not so disasterous.
What is the disaster?
"People with real questions get roped in by an enthusiastically confirming sycophantic model that will never question or push back, just confirm and lead on."
That wasn't too buried IMHO
> "People with real questions get roped in by an enthusiastically confirming sycophantic model that will never question or push back, just confirm and lead on."
> That wasn't too buried IMHO
I read that, but still fail to see evidence of a concrete "disaster". For example, are we seeing a huge wave of suicides that are statistical outliers triggered by using Chatbots? Or maybe the worry (unsubstantiated) is that there's a lagging effect and the disaster is approaching.
I suspect the outcomes are going to be less catastrophic; specifically, it seems to me that more people will have access to a much better level of support than they could get with a therapist (who is not available 24/7, who has less to draw upon, is likely too expensive, etc.). Even if there's an increase in some harm, my instinct from first principles is that the net benefit will be clear.
This isn’t accurate though, the models do push back and do question.
I would say ChatGPT is way way better than the average therapist. I’ve seen maybe between 15-20 psychotherapists over the course of my life and ChatGPT is better than 85% of them I would wager.
I’ve had a therapist recommend me take a motorbike trek across Europe (because that’s what he did) to cure my depression.
I think people tend to radically overestimate the skills of the average therapist, many are totally fucking shit.
> I’ve had a therapist recommend me take a motorbike trek across Europe (because that’s what he did) to cure my depression.
When I try to help people in support group/ therapy group although I am not a therapist, I also try to explain how I fixed my issues or how I do so.
I feel like, that isn't bad take to be really honest in the sense that I personally feel like there are times when we lose the fact that our lives have purpose and your therapist grappled with it by having a unitary goal for himself
If you felt like that was a bad idea, just tell him that what is the thing that they got out of the motorbike trek race that cured their depression, I'd be more curious about that. Considering, maybe then I can try to see if my life has/had the same problems and what is the common theme and discussing about it later.
Personally I feel like Chatgpt is extremely sycophantic and I feel really weird interacting with it. For one, I really like interacting with humans atleast in the therapy mindset I suppose.
But telling someone living in New York with a full-time job and a girlfriend etc, that the solution to their depression is to quit their job and take several months to travel across Europe on a motorbike isn't exactly practical advice. Which was my point.
I could get better advice by asking an LLM what to do about my depression.
Ah context matters.. for someone currently single lets say with no jobs and feeling lack of purpose, it wouldn't have been so bad. Maybe tried to self project with different context and I mean, its understandable why you might feel this way regarding him
I still doubt asking an LLM about depression if I am being honest, I just don't think its the best thing overall or something that should be considered norm I suppose but I am not sure as even in things like these context matters.
>I’ve had a therapist recommend me take a motorbike trek across Europe (because that’s what he did) to cure my depression.
It's not bad advice.
It can help and also hurt, depends. You have some mild situation that is quite common? Good chance it will give you good advice.
Something more complex, more towards clinical psychiatry rather than shoulder to cry on/complaining to friend over coffee (psychologists)? You are playing a russian roulette that model in your case won't hallucinate something harmful, while acting very confidently, more than any relevant doctor would be.
There have been cases of models suggesting suicide for example or general harmful behavior. There is no responsibility, no oversight, no expert to confirm or refute claims. Its just a faster version of reddit threads.
Yes! They are extremely susceptible to user influence, based on what I have read and experienced in personal use. If your initial message contains a seed, e.g., "Should I break up with my boyfriend? He doesn't listen to me," the LLM will stick with the seed of negativity and not make an objective analysis of the situation or take initiative to challenge your/its original perspective.
The real issue isn’t the policy change, it’s that AI has gotten better at sounding credible than at staying grounded. That creates a kind of performativity drift where the tone of expertise scales faster than the underlying accuracy.
So even when the model is wrong, it’s wrong with perfect bedside manner, and people anchor on that. In high stakes domains like medicine or law, that gap between confidence and fidelity becomes the actual risk, not the tool itself.
Article has since been updated for some clarity;
An earlier version of this story suggested OpenAI had ended medical and legal advice. However the company said "the model behaviour has also not changed." [0][0] https://www.ctvnews.ca/sci-tech/article/chatgpt-users-cant-u...
Marketing masterpiece
Wasn't there a recent OAI dev day in which they had some users come on stage and discuss how helpful ChatGPT was in parsing diagnoses from different doctors?
I guess the legal risks were large enough to outweigh this
I'd wager it's probably more that there's an identifiable customer and specific product to be sold. Doctors, hospitals, EHR companies and insurers all are very interested in paying for a validated version of this thing.
Or simple threats of lawsuits directly/indirectly, theres a lot of money at stake here in the end
But validation is theoretically impossible.
I was wondering when they would start with the regulations.
Next will be: you're allowed to use it, if your company/product is "HippocraticGPT"-certified, a bit like SOC or PCI compliance.
This way they can craft an instance of GPT for your specific purposes (law, medicine, etc) and you know it's "safe" to use.
This way they sell EE licenses, which is where the big $$$ are.
I work with a company in this space. All the AI tools are for the professional to use - the patient never interacts with them. The inferred output (I prefer the word prediction) is only ever seen by a mental health professional.
We reduce the professionals' workloads and help them be more efficient. But it's _not_ better care. That absolutely must be stressed. It is more _efficient_ care for the mental health practitioner and for society that can not produce enough mental health practitioners.
What is the value of the certification on the AI output? If OpenAI says they'll get guarantee the output and be accountable for errors, then thats of value, and would justify the certification. Is that the bar?
That's the entire concept of professional licensing, yeah. If you are a licensed professional, you are liable for your work output if it is malpractice.
Well, of course. Are you kidding, the amount of $$$ you would be shifting from countless lawsuits to "deal with openai" or things like that.
I don't believe they will go this far, but somewhere in the middle. And I think they might still be trying to figure it out.
Im working on a PhD concerning LLMs and somatic medicine, as an MD, and I must admit that my perspective is the complete opposite.
Medical care, at the end of the day, has nothing to do with having a license or not. Its about making the correct diagnosis, in order to administer the correct treatment. Reality does not care about who (or what) made a diagnosis, or how the antibiotic you take was prescribed. You either have the diagnosis, or you do not. The antibiotic helps, or it does not.
Doing this in practice is costly and complicated, which is why society has doctors. But the only thing that actually matters is making the correct decision. And actually, when you test LLMs (in particular o3/gpt-5 and probably gemini 2.5), they are SUPERIOR to individual doctors in terms of medical decisionmaking, at least on benchmarks. That does not mean that they are superior to an entire medical system, or a skillfull attending in a particular speciality, but it does seem imply that they are far from a bad source of medical information. Just like LLMs are good at writing boilerplate code, they are good at boilerplate medical decisions, and the fact is that there is so much medical boilerplate that this skill alone makes it superior to most human doctors. There was one study which tested LLM assisted (I think it was o3) doctors VS LLMs alone (+doctors alone) on a set of cases, and the unassisted LLM did BETTER than doctors, assisted or not.
And so all this medicolegal pearlclutching about how LLMs should not provide medical advice is entirely unfounded when you look at the actual evidence. In fact, the evidence seems to suggest that you should ignore the doctor and listen to chatGPT instead.
And frankly, as a doctor, it really grinds my gears when anyone implies that medical decisions should be a protected domain to our benefit. The point of medicine is not to employ doctors. The point of medicine is to cure patients, by whatever means best serves them. If LLMs take our jobs, because they do a better job than we do, that is a good thing. It is an especially good thing if the general, widely available LLM is the one that does so, and not the expensively licensed "HippocraticGPT-certified" model. Can you imagine anything more frustrating, as a poor kid in the boonies of Bangladesh trying to understand why your mother is sick, than getting told "As a language model I cannot dispense medical advice, please consult your closest healthcare professional".
Medical success is not measured in employment, profits, or legal responsibilities. It is measured in reduced mortality. The means to achieve this is completely irrelevant.
Of course, mental health is a little bit different, and much more nebulous overall. However, from the perspective of someone on the somatic front, overregulation of LLMs is unecessary, and in fact unethical. On average, an LLM will dispense better medical advice than an average person with access to google, which is what it was competing with to begin with. It is an insult to personal liberty and to the Hippocratic oath to support that this should be taken away simply because of some medicolegal bs.
I appreciate your perspective. May I contact you? You are invited to send me an email, my Gmail username is the same as my HN username.
Or you'll need a prescription to be able to ask it health questions.
I think this would apply if they sought out government regulation that applies to all AI players, not just their own company.
...or they want to be the first to do it, as others don't have it.
OpenAI is the leading company, if they provide an instance you can use for legal advice, with relative certification etc., it'd be better to trust them rather than another random player. They create the software, the certification and the need for it.
Suppose I am the CTO at a big legal firm, and I have been charged with finding an AI service to integrate with. What could an OpenAI sales person tell me that justifies the value of this certification?
Lets suppose they've already told me that their company is a "leader in the AI space", to which I responded that the guy at Claude told me the same thing on Monday, with which I was equally unimpressed.
I guess right now the way to convince is with SLA, bonus, benefits, "always on support", "a dedicated technical manager to help you design a scalable agentic framework" or whatever BS they can sell.
Or: "your tokens will cost 1/3, because we have crafted a particular gpt instance that is superoptimised for your use case only, look we just created one for you before the meeting".
Money is the biggest motivator. A company that is throwing $$$ left and right needs EE yesterday.
A lot like AWS credits. "Nice I can use up to 200'000$ for free".........
Hmm. So OpenAI doesn't care about other people's terms, copyrights or any sort of IP; but they get to have their own terms.
Controlling other people's things and gaining from the control is kind of the point of being a thief.
Welcome to the internet! Have a look around
(https://www.youtube.com/watch?v=k1BneeJTDcU)
Another obligatory song named Jeff bezos by Bo Burnham (https://www.youtube.com/watch?v=lI5w2QwdYik)
C'mon Jeffrey you can do it! (I mean you can break half the internet with AWS us-east-1, but in all seriousness both songs are really nice lol)
I don't understand this outrage. People put things on the internet for all to see but now you're mad someone saw it and made use of it.
If you didn't want others to read your information you shouldn't have published it on the internet. That's all they're doing at the end of the day, reading it. They're not publishing it as their own, they just used publicly available data to train a model.
It's quite the same as if I read an article and then tell someone about it. If I'm allowed to learn from your article then why isn't openai?
Also the terms are just for liability. Nobody gives a shit what you use ChatGPT for, the only thing those terms do is prevent you from turning around and suing OpenAI after it blows up in your face.
And I feel like the difference between, say;
- Paying for a ticket/dvd/stream to see a Ghibli movie
- Training a model on their work without compensating them, then enabling everyone to copy their art style with ~zero effort, flooding the market and diluting the value of their work. And making money in the process.
should be rather obvious. My only hypothesis so far is that a lot of the people in here have a vested interest in not understanding the outrage, so they don't.
I don't have any vested interest. I'm just a guy. Don't work on AI, don't use AI much in my day job.
I just legitimately think the outrage is unreasonable. It is completely infeasible for AI companies to provide any meaningful amount of compensation to all the data sources they use.
Alternatively they could just not use any of the data, in which case we wouldn't have as good LLMs and other than that the world would be exactly the same. These data owners don't notice any difference. Using their data doesn't harm them in any way.
You seem to be arguing that enabling people to mimic Ghibli's art style somehow harms them, I don't see how it does at all. People have been able to mimic it already, what's the difference? More people can? Does that make a difference to ghibli? I mean can you point to some concrete negative effects that his phenomenon has had on studio ghibli?
I don't think you can. And I think that proves my point. Anything can be mimicked. People can play covers of songs, paint their own versions of famous paintings, copy Louis Vuitton bag designs, whatever they want. The effort it takes is irrelevant.
You don't even have to train AI on studio Ghibli's art to mimic it. You could just train it on other stuff and then the user could feed it studio ghibli art and tell it to mimic it. The specific training data itself is irrelevant, it's the volume of data that trains the models. Even if they specifically avoided training on studio Ghibli's art there would likely be basically no difference. It wouldn't be worth paying them for it.
You ever see warning labels on products? That's because putting them in terms and conditions isn't enough to avoid liability in a products liability case.
Well it certainly helps
Who told you that? ChatGPT?
No, not really. It does not help. It evidences that they knew the risks were present but did not take the actions to adequately warn the consumers of its products, which OpenAI knows do not read the entirety of the TOS.
Again, this is exactly why you see warning labels on products. Prohibiting certain uses in small text, hidden in the TOS, is not a warning.
Quite the over-simplified straw man you have there.
Quite the lack of substance you have there.
People publish stuff on the Internet for various reasons. Sometimes they want to share information, but if the information is shared, they want to be *attributed* as the authors.
> If you didn't want others to read your information you shouldn't have published it on the internet. That's all they're doing at the end of the day, reading it. They're not publishing it as their own, they just used publicly available data to train a model.
There is some nuance here that you fail to notice or you pretend you don't see it :D I can't copy-paste a computer program and resell it without a license. I can't say "Oh I've just read the bits, learned from it and based on this knowledge I created my own computer program that looks exactly the same except the author name is different in the »About...« section" - clearly, some reason has to be used to differentiate reading-learning-creating from simply copying...
What if instead of copy-pasting the digital code, you print it onto a film, pass the light through the film onto ants, make the light kill the ants exposed, and the rest of the ants eventually go away, and now use the dead ants as another film to somehow convert that back to digital data. You can now argue you didn't copy, you taught ants, the ants learned, and they created a new program. But you will fool no one. AI models don't actually learn, they are a different way the data is stored. I think when court decides if a use is fair and transformative enough, it investigates how much effort was put into this transformation: there was a lot of effort put into creating the AI, but once it was created, the effort put into any single work is nearly null, just the electricity, bandwidth, storage.
> There is some nuance here that you fail to notice or you pretend you don't see it :D
I could say the same for you:
> I can't copy-paste a computer program and resell it without a license. I can't say "Oh I've just read the bits, learned from it and based on this knowledge I created my own computer program that looks exactly the same except the author name is different in the »About...« section"
Nobody uses LLMs to copy others' code. Nobody wants a carbon-copy of someone else's software, if that's what they wanted they would have used those people's software. I mean maybe someone does but that's not the point of LLMs and it's not why people use them.
I use LLMs to write code for me some times. I am quite sure that nobody in history has ever written that code. It's not copied from anyone, it's written specifically to solve the given task. I'm sure it's similar to a lot of code out there, I mean it's not often we write truly novel stuff. But there's nothing wrong with that. Most websites are pretty similar. Most apps are pretty similar. Developers all over the world write the same-ish code every day.
And if you don't want anyone to copy your precious code then don't publish it. That's the most ironic thing about all this - you put your code on the internet for everyone to see and then you make a big deal about the possibility of an LLM copying it as a response to a prompt?
Bro if I wanted your code I could go to your public github repo and actually copy it, I don't need an LLM to do that for me. Don't publish it if you're so worried about being copied.
I wouldn't be surprised to see new products from OpenAI targeted specifically at doctors and/or lawyers. Forbidding them from using the regular ChatGPT with legal terms would be a good way to do price discrimination.
Definitely. And in the long run, that is the only way those occupations can operate. From that point, you are locked in to an AI dependency to operate.
Read their paper on GDPval (https://arxiv.org/abs/2510.04374). In section 3, it's quite clear that their marketing strategy is now "to cooperate with professionals" and augment them. (Which does not rule out replacing them later, when the regulatory situation is more appropriate, like AGI is already a well-accepted fact, if ever.) But this will take a lot of time and local presence which they do not have.
I have seen "AI" in my Dr's office. They have been using it to summarize visits and write after visit notes.
Can it become a proxy for AI companies to collect patient data and medical history or "train" on the data and sell that as a service to insurance companies.
There's HIPAA but AI firms have ignored copyright laws, so ignoring HIPAA or making consent mandatory is not a big leap from there.
That's likely DAX Copilot, which doesn't provide medical advice.
OpenEvidence is free for anyone with an NPI
Last year ChatGPT helped save my life from having a stroke. LLMs are incredibly beneficial in providing medical information and advice today.
> LLMs are incredibly beneficial ... today.
LLMs sometimes can be incredibly beneficial ... today
LLMs sometimes can be incredibly harmful ... today
Non-deterministic things aren't just one thing, they're whatever they happen to be in that particular moment.
Non-deterministic doesn't mean random or unpredictable. That's like saying the weather forecast is useless because it's not deterministic or always 100% accurate.
Last time I used GPT-4.5 to analyze blood results it gave different output if I uploaded it as 2 instead of 3 separated CSV files. It was both amazing experience: clear and easy to understand statements, and list of most common causes. And terrifying: "What about X?", "You ar absolutely right, there where X results included, disregard everything I wrote above, here is the new analysis".
So for me non-deterministic means unpredictable. Yes, there was nothing random or non-deterministic in that case, I could repeat both scenarios multiple times and get same results again. But the result is affected by something I didn't expect to matter. That damages the trust in tool, no matter how we call it.
LLMs seem best at creative brainstorming, coming up with ideas to check that you hadn't thought of. Their weakness becomes a non-issue because ideas are just things for you to check the viability of, because they could be completely unworkable.
> Non-deterministic doesn't mean random or unpredictable. That's like saying the weather forecast is useless because it's not deterministic or always 100% accurate.
I don't know where you got 'useless' from. LLMs are great, sometimes. They're not, other times. Which remarkably, is just like weather forecasts. The weather forecast is sometimes completely accurate. The weather forecast is sometimes completely inaccurate.
LLMs, like weather forecasting, have gotten better as more time and money has been invested in them.
Neither are perfect. Both are sometimes very good. Both are sometimes not.
[dead]
Non-deterministic means random - that's the definition of the word. The weather forecast is also random - in fact, weather forecast is (if you simplify it too much) an average of several predictive (generative) models.
> Non-deterministic means random - that's the definition of the word.
That's not really the defintion. Non-determinism just means the outcome is not a pure function of the inputs. A PRNG doesn't become truly random just because we don't know the state and seed when calling the function and the same holds for LLMs. The non-determinism in LLMs comes from accepted race conditions in the GPU floating point math and the PRNG in the sampler.
That's besides the point, but we could have perfectly deterministic LLMs.
Inconsistent seems a more accessible word. It gives inconsistent results.
ChatGPT isn’t random though.
If you ask it what a star is, it’s never going to tell you it’s a giant piece of cheese floating in the the sky.
If you don’t believe me, try it, write a for loop which asks ChatGPT, what is a star (astronomy) exactly? Ask it 1000 times and then tell me how random it is versus how consistent it is.
The idea that non deterministic === random is totally deluded. It just means you cannot predict the exact tokens which will be produced but it doesn’t mean it’s random like a random number generator and it could be any thing.
If you ask what is Michael Jackson the entertainer famous for it’s going to tell you he’s famous for music and dancing. 1000/1000 times, is that random?
> If you ask it what a star is, it’s never going to tell you it’s a giant piece of cheese floating in the the sky.
Turn the Top-P and the temperature up. Turning up the Top-P will enable the LLM to actually produce such nonsense. Turning up the temperature will increase the chance that such nonsense is actually selected for the prediction (output).
Sure but nobody is doing that, are they?
I'm talking about the standard settings, and infact GPT-5 doesn't let you change the temperature anymore.
Also, that's not really the point. Humans can also produce nonsense if you torture them until they're talking nonsense, but that doesn't mean humans are "random."
LLMs are not random, they are non-deterministic, but the two words have different meanings.
Random means you cannot tell what is going to be produced at all, i.e. a random number generator.
But if you ask an LLM, is an Apple a fruit, answer yes or no only, the LLM is going to answer yes, 100% of the time. That isn't random.
I agree with everything that you've stated.
Nearly everything in life is non-deterministic.
Most things that are generally helpful and beneficial are not 100% helpful and beneficial 100% of the time.
I used GPT-4 as a second opinion on my medical tests and doctor's advice, and it suggested an alternate diagnosis and treatment plan that turned out to be correct. That was incredibly helpful and beneficial.
You're replying to a person who had a similar and even more helpful and beneficial experience because they're alive today.
Pedantically pointing out that a beneficial and helpful thing isn't 100% beneficial and helpful 100% of the time doesn't add anything useful to the conversation since everyone here already knows it's not 100%.
Any reason you didn't just call your GP or even 911?
Denial, or rather some form of "there's no way this is frickin stroke/heart attack, right?", is common when you're having a medical emergency.
>LLMs are incredibly beneficial...
No, they can be. To state that they are, as an absolute, based on your sample size of one, especially with regard to other instances where ChatGPT has failed the user with serious physical results, is fallacious.
I am glad that you are OK, but as another user suggested, it's nowhere near as consistently accurate as it needs to be in order to be anywhere near an adequate substitute for a call to a GP or 911.
[dead]
So basically all white collar jobs are lobbying to gatekeep their profession even from AI, meanwhile the stupid engineers who made AI put zero effort to not shoot themselves in the foot, and now they are crying about low wages if they found a job in the first place.
AI could effectively do most legal and medical work, and you can make a human do the final decision-making if that's really the reason. In fact, I bet most lawyers and doctors are already using it in one way or another; after all, both are about reciting books and correlating things together. AI is definitely more efficient at that than any human. Meanwhile, the engineering work that requires critical thinking and deep understanding of the topic is allowed to be bombarded with all AI models. What about the cases where bad engineering will kill people? I am a firm believer that engineers are the most naive people who beg others to exploit them and treat them like shit. I don't even see administrative assistants crying about losing their jobs to AI; every other profession guards its workers, including blue collar, except the ‘smart’ engineers.
i don't think it's stopped providing said information, it's just now outlined in their usage policies that medical and legal advice is a "disallowed" use of ChatGPT
As a doctor I hope it still allows me to get up to speed on latest treatments for rare diseases that I see once every 10 years. It saves me a lot of time rather than having to dig through all the new information since I last encountered a rare disease.
I assume you are reading the sources rather than relying solely on the AI summaries.
God, I hope so. I would love to have a list of doctors using ChatGPT as medical education, so I can avoid them.
[flagged]
I read this as "we disallow suing us for bad legal or medical advice"
If I pay for legal advice from you and all you did was give me chat gpt output, I can't sue openai anyway. This just clarifies that you can't put 'powered by chatgpt' on your AI lawyer service.
Well, you can if you used Chatgpt to write the HTML.
Are there metrics for whether LLM diagnosis accuracy is improving? Anecdotally doctor friends says it's more reliable then their worst colleagues, which I'm sure their worst colleague insinuate the same about them.
It appears they just want to avoid responsibility for potential misuse in these areas.
But at the same time, IIRC, several major AI providers had publicly reported their AI assisting patients in diagnosing rare diseases.
I think it's very cynical to say that this is a misuse. And it's definitely cynical when this categorization of misuse comes from the service provider itself. If openai doesn't want to allow misuse, they can just decommision their service. But they don't want to do that, they just want to take the money and push all the responsibility and burden on the users even though they are actively engaging in said "misuse"
Sad times - I used ChatGPT to solve a long-term issue!
This won’t be affecting you then! /s
I been using Claude for information regarding building and construction related information, (currently building a small house mostly on my own with pros for plumbing and electrical).
Seriously the amount of misinformation it has given me is quite staggering. Telling me things like, “you need to fill your drainage pipes with sand before pouring concrete over them…”, the danger with these AI products is that you have to really know a subject before it’s properly useful. I find this with programming too. Yes it can generate code but I’ve introduced some decent bugs when over relying on AI.
The plumber I used laughed at my when I told him about there sand thing. He has 40 years experience…
I nearly spit my drink out. This is my kind of humor, thanks for sharing.
I've had a decent experience (though not perfect) with identifying and understanding building codes using both Claude and GPT. But I had to be reasonably skeptical and very specific to get to where I needed to go. I would say it helped me figure out the right questions and which parts of the code applied to my scenario, more than it gave the "right" answer the first go round.
I'm a hobby woodworker - I've tried using gemini recently for an advice on how to make some tricky cuts.
If I'd follow any of the suggestions I'd probably be in ER. Even after me pointing out issues and asking it to improve - it'd come up with more and more sophistical ways of doing same fundamentally dangerous actions.
LLMs are AMAZING tools, but they are just that - tools. There's no actual intelligence there. And the confidence with which they spew dangerous BS is stunning.
> I've tried using gemini recently for an advice on how to make some tricky cuts.
C'mon, just use the CNC. Seriously though, what kind of cuts?
Give a single reproduceable example using ChatGPT thinking
I've observed some horrendous electrical device, such as "You should add a second bus bar to your breaker box." (This is not something you ever need to do.)
I mean... you do have to backfill around your drainage pipe, so it's not too far off. Frankly, if you Google the subject people misspeak about "backfilling pipes" too as if the target of the backfill is the pipe itself too not the trench. Garbage in, garbage out.
All the circumstances where ChatGPT has given me shoddy advice fall in three buckets:
1. The internet lacks information, so LLMs will invent answers
2. The internet disagrees, so LLMs sometimes pick some answer without being aware of the others
3. The internet is wrong, so LLMs spew the same nonsense
Knowledge from blue collar trades seems often to in those three buckets. For subjects in healthcare, on the other hand, there are rooms worth of peer reviewed research, textbooks, meta studies, and official sources.
I mean... you do have to backfill around your drainage pipe, so it's not too far off
Really? You think filling you pipe with sand is comparable to backfilling a trench ?
People don't talk about "backfilling trenches", they talk about "backfilling pipes" which is precisely the problem. They also will say things like "when you backfill a pipe, use sand or granite."
It pretty obvious why an LLM would be confused by this.
honestly I think these things cause a form of Gell Mann's Amnesia where when you use them for something you know already, the errors are obvious, but when you use them for something you don't understand already, the output is sufficiently plausible that you can't tell you're being misled.
this makes the tool only useful for things you already know! I mean, just in this thread there's an anecdote from a guy who used it to check a diagnosis, but did he press through other possibilities or ask different questions because the answer was already known?
It’s funny you should say that because I have been using it in the way you describe. I kind of know it could be wrong, but I’m kind of desperate for info so I consult Claude anyway. After stressing hard I realize it was probably wrong, find someone who knows what they’re and actually on about and course correct.
The great thing is the models are sufficiently different enough, that when multiple come to the same conclusion, there is a good chance that conclusion is bound by real data.
And I think this is the advice that should always be doled out when using them for anything mission critical, legal, etc.
All the models are pre-trained on the same one Internet.
"Bound by real data" meaning not hallucinations, which is by far the bigger issue when it comes to "be an expert that does x" that doesn't have a real capability to say "I don't know".
The chance of different models hallucinating the same plausible sounding but incorrect building codes, medical diagnoses, etc, would be incredible unlikely, due to arch differences, training approaches, etc.
So when two concur in that manner, unless they're leaning heavily on the same poisoned datasets, there's a healthy chance the result is correct based on a preponderance of known data.
I'd frame it such that LLM advice is best when it's the type that can be quickly or easily confirmed. Like a pointer in the right (or wrong) direction. If it was false, then try again - quick iterations. Taking it at its "word" is the potentially harmful bit.
Usually something as simple as saying “now give me a devils advocate resoonse” will help and of course “verify your answer on the internet” will give you real sources that you can verify.
I have very mild cerebral palsy[1], the doctors were wrong about so many things with my diagnosis back in the mid to late 70s when I was born. My mom (a retired math teacher now with an MBA back then) had to go physically to different libraries out of town and colleges to do research. In 2025, she could have done the same research with ChatGPT and surfaced outside links that’s almost impossible via a web search.
Every web search on CP is inundated with slimy lawyers.
[1] it affects my left hand and slightly my left foot. Properly conditioned, I can run a decent 10 minute mile up to a 15K before the slight unbalance bothers me and I was a part time fitness instructor when I was younger.
The doctor said I was developmentally disabled - I graduated in the top of my class (south GA so take that as you will)
The cynic in me thinks this is just a means to eventually make more money by offering paid unrestricted versions to medical and legal professionals. I'm well-aware that it's not a truth machine, and any output it provides should be verified, checked for references, and treated with due diligence. Yet the same goes for just about any internet search. I don't think some people not knowing how to use it warrants restricting its functionality for the rest of us.
Nah, this is to avoid litigation. Who needs lawsuits when you are seeking profit? 1 loss of a major lawsuit is horrible, there's the case of folks suing them because their loved ones committed suicide after chatting with ChatGPT. They are doing everything to avoid getting dragged to court.
> I'm well-aware that it's not a truth machine, and any output it provides should be verified, checked for references, and treated with due diligence.
You are, but that's not how AI is being marketed by OpenAI, Google, etc. They never mention, in their ads, how much the output needs to be double and triple checked. They say "AI can do what you want! It knows all! It's smarter than PhDs!". Search engines don't say "And this is the truth" in their results, which is not what LLM hypers do.
I appreciate how the newer versions provide more links and references. It makes the task of verifying it (or at least where it got its results from) that much easier. What you're describing seems more like a advertisement problem, not a product problem. No matter how many locks and restrictions they put on it, someone, somwhere, will still find a way to get hurt from its advice. A hammer that's hard enough to beat nails is hard enough to bruise your fingers.
> What you're describing seems more like a advertisement problem, not a product problem.
It's called "false advertising".
https://en.wikipedia.org/wiki/False_advertising
Also known as "lying".
If they do that, they will be subject of regulations on medical devices. As they should be and means the end result will be less likely to promote complete crap then it is now.
And then users balk at the hefty fee and start getting their medical information from utopiacancercenter.com and the like.
Let them. You can't save those people anyway.
Science should be clearly labelled for those that can read. Everyone else can go eat blueberry leaves if they so choose.
I think one of the challenges is attribution. For example if you use Google search to create a fraudulent legal filing there aren't any of Google's fingerprints on the document. It gets reported as malpractice. Whereas reporting on these things is OpenAI or whatever AI is responsible. So even from the perspective of protecting a brand it's unavoidable. Suppose (no idea if true) the Louvre robbers wore Nike shoes and the reporting were that Nike shoes were used to rob the Louvre and all anyone talks about is Nike and how people need to be careful about what they do wearing Nike shoes.
It's like newsrooms took the advice that passive voice is bad form so they inject OpenAI as the subject instead.
This (attribution) is exactly the issue that was mentioned by LexisNexis CEO in a recent The Verge interview.
https://www.theverge.com/podcast/807136/lexisnexis-ceo-sean-...
This (new) title is inaccurate.
The article says: "ChatGPT users can’t use service for tailored legal and medical advice, OpenAI says", with a quote from OpenAI: “this is not a new change to our terms. ChatGPT has never been a substitute for professional legal or medical advice, but it will continue to be a great resource to help people understand legal and health information.”
This is not true, just a viral rumor going around: https://x.com/thekaransinghal/status/1985416057805496524
I've used it for both medical and legal advice as the rumor's been going around. I wish more people would do a quick check before posting.
EXHIBIT A
"If at any point I described how legal factors “apply to you,” that would indeed go beyond what I’m supposed to do. Even if my intent was to illustrate how those factors generally work, the phrasing can easily sound like I’m offering a tailored legal opinion — which isn’t appropriate for an AI system or anyone who isn’t a licensed attorney.
The goal, always, is for me to help you understand the framework — the statutes, cases, or reasoning that lawyers and courts use — so that you can see how it might relate to your situation and then bring that understanding to a qualified attorney.
So if I’ve ever crossed that line in how I worded something, thank you for pointing it out. It’s a good reminder that I should stay firmly on the educational side: explaining how the law works, not how it applies to you personally.
Would you like me to restate how I can help you analyze legal issues while keeping it fully within the safe, informational boundary?"
ChatGPT
Ideally, we should be able to opt-in with a much higher fee. At the $200/mo tier I should be allowed to use this tool. The free users and lower tier paid users should be guard-railed. This is because those users all have trouble using this tool and then get upset at OpenAI and then we all have to endure endless news articles that we wouldn't if the good stuff were price-gated.
Those without money frequently have poor tool use, so eliminating them from the equation will probably allow the tool to be more useful. I don't have any trouble with it right now, but instead of making up fanciful stories about books I'm writing where characters choose certain exotic interventions in pursuit of certain rare medical conditions only to be struck down by their lack of subservience to The Scientific Consensus, I could just say I'm doing these things and that would be a little helpful in a UX sense.
I'd bet dollars to donuts it doesn't actually "end legal and medical advice", it just ends it in some ill-defined subset of situations they were able to target, while still leaving the engine capable of giving such advice in response to other prompts they didn't think to test.
Things like this really favor models offered from countries that have fewer legal restrictions. I just don't think it's realistic to expect people not to have access to these capabilities.
It would be reasonable to add a disclaimer. But as things stand I think it's fair to consider talking to ChatGPT to be the same as talking to a random person on the street, meaning normal free-speech protections would apply.
What capabilities? The article says the study found it was entirely correct 31% of the time.
This is what really scares me about people using AI. It will confidently hallucinate studies and quotes that have absolutely no basis in reality, and even in your own field you're not going to know whether what it's saying is real or not without following up on absolutely every assertion. But people are happy to completely buy its diagnoses of rare medical conditions based on what, exactly?
Give a single example using gpt-5 thinking.
GPT-5 thinking is one of the biggest offenders and it's quite incredible to me that you think it doesn't hallucinate. It makes me strongly suspect your own judgment is impaired. Also, what do you mean asking for "reproducible examples"? Is it somehow not a valid example if it only sometimes makes up citations?
Share any example where I can get it to make an obvious error
Does it say how often doctors are correct as a baseline?
The study is more positive than the 31% conveys.
https://www.ctvnews.ca/health/article/self-diagnosing-with-a...
The researchers compared ChatGPT-4 with its earlier 3.5 version and found significant improvements, but not enough.
In one example, the chatbot confidently diagnosed a patient’s rash as a reaction to laundry detergent. In reality, it was caused by latex gloves — a key detail missed by the AI, which had been told the patient studied mortuary science and used gloves.
...
While the researchers note ChatGPT did not get any of the answers spectacularly wrong, they have some simple advice.
“When you do get a response be sure to validate that response,” said Zada.
Which should be standard advice in most situations.
>I think it's fair to consider talking to ChatGPT to be the same as talking to a random person on the street
That’s not how companies market AI though. And the models themselves tend to present their answers in a highly confident manner.
Without explicit disclaimers, a reasonable person could easily believe that ChatGPT is an authority in the law or medicine. That’s what moves the needle over to practicing the law/medicine without a license.
Grok will do mostly whatever you want. It's a tool and it's your personal responsibility to use it correctly. That's the way it should be. You don't need Sam Altman to babysit you, I hope.
As long as Elon keeps messing with Grok because the facts it uses aren't compatible with his world view [1], I'll stay away from it as far as possible.
And even leaving the political questions aside - the thing is being molded into a goon generator [2], just WTF.
[1] https://edition.cnn.com/2025/06/27/tech/grok-4-elon-musk-ai
[2] https://www.theatlantic.com/technology/2025/09/grok-system-p...
> It's a tool and it's your personal responsibility to use it correctly.
Sure in theory, but this never works out in practice when applied to the populace. Replace "LLM" with other controversial things — social media, cigarettes, guns.
I'm also certain anyone outside the tech sphere who takes personal responsibility when using LLMs is not using Grok as their daily driver.
Grok Code Fast 1 is quite usable for small simple coding tasks. And quite a bit cheaper than say: Claude Sonnet 4.5
Personal experience when using it OpenRouter.
That said, I prefer bigger models even if they cost more. But I do this through GitHub Copilot at a steady 10 dollars per month. I can do way more request with it than I can with 10 dollars worth of credits at OpenRouter. I don't think Copilot is making any money, on the contrary.
Sounds like it is still giving out medical and legal information just adding CYA disclaimers.
It would be terribly boring if it didn't. Just last night I had it walk me through reptile laws in my state to evaluate my business plan for a vertically integrated snapping turtle farm and turtle soup restaurant empire. Absurd, but it's fun to use for this kind of thing because it almost always takes you seriously.
(Turns out I would need permits :-( )
It's not stopping to give legal/medical advice to the user, but it's forbidden to use ChatGPT to pose as an advisor giving advice to others: https://www.tomsguide.com/ai/chatgpt/chatgpt-will-still-offe...
Good. Now we just need to add technical advice to the list of advices which ChatGPT cannot be used to cite for
This. I already seen too many cases of people wiping out their systems while trying to follow invalid advices from ChatGPT.
One wonders how exactly this will be enforced.
It's not about enforcing this, it's about OpenAI having their asses covered. The blame is now clearly on the user's side.
It’s a “CYA” aka don’t sue me
It was already enforced by hiding all custom GPTs that offered medical advice.
The Tom's Guide article blatantly misinterprets and contradicts the source it quotes.
I've used ChatGPT to help understand medical records too. It's definitely faster than searching everything on my own, but whether the information is reliable still depends on personal judgment or asking a real doctor. More people are treating it like a doctor or lawyer now, and the more it's used that way, the higher the chance something goes wrong. OpenAI is clearly drawing a line here. You're free to ask questions, but it shouldn't be treated as professional advice, especially when making decisions for others.
You should not be feeding your medical records into ChatGPT.
And you shouldn't be using Gmail or Google Search. At some point the benefits outweigh the costs.
How does an email service compare to a glorified if-then machine that hordes all your data?
Why not?
I can imagine a few different reasons you might have, but I don't want to guess.
I’ve found it just as accurate and a better experience than using telehealth or generalist doctors
Ditto for lawyers
The thing that gets me about AI is that people act like most doctors or most lawyers are not … shitty and your odds of running into a below average one are almost 50/50
Doctors these days are more like physicists when most of the time you need a mechanic or engineer. I’ve had plenty of encounters wher I had to insist on an MRI or on specific bloodwork to hone in on the root cause of an ailment where the doctor just chalked it up to diet and exercise
Anything can be misused, including google, but the answer isn’t to take it away from people
Legal/financial advice is so out of reach for most people, the harsh truth is that ChatGPT is better than nothing and anyone who would follow what it says blindly is bound to fuck up those decisions up in some way anyway
On the other hand, if you can leverage it same as any other tool it’s a legitimate force multiplier
The cynic in me thinks this is just being done in the interest of those professions, but that starts to feel a bit tin foil-y
You can’t possibly have enough data to support that statement.
there was no mention of sample size, though, so the statement might be true for the commenter but not widely applicable, to your point
I’m saying that the commenter doesn’t have a large enough sample size to draw conclusions even for themselves.
Just based on the amount of time ChatGPT has been out and the maximum number of different diagnosable and confirmable ailments they could have had in that time.
Do you have enough data to support the opposite conclusion? If not, your call for data is just a pathetic attempt to sound rational while making an empty appeal to authority.
I’m not calling for data. I’m saying the commenter doesn’t have a large enough sample size to draw conclusions—even informal ones.
Just based on the amount of time ChatGPT has been out and the maximum number of different diagnosable and confirmable ailments they could have had in that time.
If you're not a doctor, how do you know it's accurate?
This is the huge problem with using LLMs for this kind of thing. How do you verify that it is better? What is the ground truth you are testing it against?
If you wanted to verify that ChatGPT could do math, you'd ask it 100 math problems and then (importantly) verify its answers with a calculator. How do you verify that ChatGPT can interpret medical information without ground truth to compare it to?
People are just saying, "oh it works" based on gut vibes and not based on actually testing the results.
Nobody would use these services for anything important if they _actually understood_ these glorified markov chains are just as likely to confidently assert something false and lie about it when pressed as they are to produce accurate information.
These AI companies have sold a bill of goods but the right people are making money off it so they’ll never be held responsible in a scenarios the one you described.
Isn't statistical analysis a legitimate tool for helping diagnosis since forever? It's not exactly surprising that a pattern matcher does reasonably well at matching symptoms to diseases.
What is the most likely cause of this set of facts is how diagnostics works. LLMs are tailor made for this type of use case.
That’s not what they’re trained on though and not what they’re doing.
When you ask a general LLM like ChatGPT “I have a headache what is the cause” it’s going to give you the most statistically likely response in it’s training data. That might coincide with the most likely cause, but it’s also very possible it doesn’t. Especially for problems with widespread misconceptions.
How does anyone know if what the doctor says is accurate? Obviously people should put the most relative weight in their doctor's opinion, but there's a reason people always say to get a second opinion.
Unfortunately because of how the US healthcare system works today people have to become their own doctors and advocates. LLMs are great at surfacing the unknown unknowns, and I think can help people better prepare for the rare 5 minutes they get to speak to an actual doctor.
> How does anyone know if what the doctor says is accurate?
They don’t. That’s nearly the entire justification for licensure.
I know it’s hard to accept, but it’s got to be weighed against the real-world alternative:
You get about 5-8 minutes of face time with an actual doctor, but have to wait up to weeks to actually get in to see a doctor, except maybe at an urgent care or ER.
Or people used to just play around on WebMD which was even worse since it wasn’t in any way tailored to what the patient’s stated situation is.
There’s the rest of the Internet too. You can also blame AI for this part, but today the Internet in general is even more awash in slop that is just AI-generated static BS. Like it or not, the garbage is there and it will be most of what people find on Google if they couldn’t use a real ChatGPT or similar this way.
Against this backdrop, I’d rather people are asking the flagship models specific questions and getting specific answers that are halfway decent.
Obviously the stuff you glean from the AI sessions needs to be taken to a doctor for validation and treatment, but I think coming into your 5-minute appointment having already had all your dumbest and least-informed ideas and theories shot down by ChatGPT is a big improvement and helps you maximize your time. It’s true the people shouldn’t recklessly attempt to self-treat based on GPT, but the unwise people doing that were just self-treating based off WebMD hunches before.
I get what you're saying, and I agree it might be fun to play around with ChatGPT and Wikipedia and YouTube and WebMD to try to guess what that green bump on your arm is, but it's not research--it needs to be treated as entertainment.
When it comes to taking actual real-world action, I would take 5-8 minutes with a real doctor over 5-8 months of browsing the Internet. The doctor has gone to med school, passed the boards, done his residency, and you at least have that as evidence that he might know what he is doing. The Internet offers no such evidence.
I fear that our society in general is quickly entering a very dangerous territory where there's no such thing as expertise, and unaccountable, probabilistic tools and web resources of unknown provenience are seen as just as good as an expert in his field.
I don't disagree with you, but if I prompted an LLM to ask me questions like a doctor would for a non-invasive assessment, would it ask me better or worse questions than an actual doctor?
I ask (somewhat rhetorically) to get the mind thinking, but I'm legitimately curious whether - just from a verbal survey - whether the AI doctor would ask me about things more directly related to any illness it might suspect, versus a human who might narrow factors down similar to a 90s TV "ghost speaker" type of person; one fishing for matches amongst a fairly large dataset.
> You get about 5-8 minutes of face time with an actual doctor, but have to wait up to weeks to actually get in to see a doctor, except maybe at an urgent care or ER.
This depends heavily on where you are, and on how much money you want to throw at the problem.
> but I think coming into your 5-minute appointment having already had all your dumbest and least-informed ideas and theories shot down by ChatGPT is a big improvement and helps you maximize your time.
That’s balanced out by the time wasted by all the people who’ve had their worst fears and paranoias confirmed by ChatGPT.
"This technology will radically transform the way our world works and soon replace all knowledge-based jobs. Do not trust anything it says: Entertainment purposes only."
I feel like this sums up the AI problem at large. The marketing and hype from every tech company is that its game changing, but more softly spoken is “hey dont trust it for anything important” as a CYA disclaimer. The problem is the masses only hear the part said in bold.
In bold: This technology will radically transform the way our world works and soon replace all knowledge-based jobs.
In fine print: Do not trust anything it says: Entertainment purposes only
That's almost too reasonable to be credible. They are probably just covering their ass until they can do it with impunity. With the rapidly progressing corruption of the legal system it's merely a question of time until companies doing it for profit can disseminate medical and legal bullshit all they want. Politicians are already doing it.
good thing that guy was able to negotiate his hospital bills before this went into effect.
I used DeepSeek to draft a legal letter for some dispute with some marketplace that didn't want to do what I paid for. Within 2 days after sending that email all was resolved. I would hate to lose that option.
You can still do that, they are just saying they aren’t on the hook when things go wrong.
I heard that this is implicitly targeting the use of AI to negotiate expensive medical expenses.
But probably just a coincidence:
https://www.reddit.com/r/accelerate/comments/1op8fj2/ai_redu...
Good. Techies need to stop thinking that an LLM should not be immune from requiring licensing. Until OpenAI can (and should) be sued for medical malpractice or lawyering without passing the bar, they will have no skin in the game to actually care. A disclaimer of "this is not a therapist" should not be enough to CYA.
anyone wanna form a software engineering guild, then lobby to need a license granted by the guild to practice?
Sorry but you’re not gonna get me to agree that medical licensing is a bad idea. I don’t want quacks more than we already do. Stick to the argument and not add in your “what about” software engineers.
I am being serious...
the damage certain software engineers could do certainly surpasses most doctors
Ah sorry, I misread it as coming from someone who doesn't want licensing, so you were appealing to HN by switching to software engineers (and I know many on here loathe to think anything beyond "move fast and break things", which is the opposite of most (non-software) engineers.
But yeah, I'd be down for at least some code of ethics, so we could have "do no harm" instead of "experiment on the mental states of children/adolescents/adults via algorithms and then do whatever is most addictive"
> But yeah, I'd be down for at least some code of ethics, so we could have "do no harm" instead of "experiment on the mental states of children/adolescents/adults via algorithms and then do whatever is most addictive"
absolutely
if the only way to make people stop building evil (like your example) is to make individuals personally liable, then so be it
Funny how this happened 1 day after Kim Kardashian blamed chatGPT for giving her wrong answers while studying for the bar.
https://gizmodo.com/kim-kardashian-blames-chatgpt-for-failin...
That's not legal advice. BARBRI is not your lawyer, and almost everyone uses that for the Bar exam.
"“No. I use it for legal advice,” Kardashian said. “So when I am needing to know the answer to a question, I’ll take a picture and snap it and like put it in there.”"
RIP Dr. ChatGPT, we'll miss you. Thanks for the advice on fixing my shoulder pain while you were still unmuzzled.
Weird that they think people will follow their terms of service while disregarding that of the entire internet.
That’s a fair limitation. Legal and medical advice can directly impact someone’s life or safety, so AI tools must stay within ethical and regulatory boundaries. It’s better for AI to guide people toward professional help than pretend to replace it.
> Legal and medical advice can directly impact someone’s life or safety, so AI tools must stay within ethical and regulatory boundaries
Knives can be used to cook food and stab other people. By your suggestion, knives must be forbidden/limited as well?
If people following chatgpt advise (or any other stupid source for that matter), it's a not a ChatGPT but the people, issue.
You had me right up until ‘AI tools must stay within ethical and regulatory boundaries’. I guarantee you any AI LLM company which cares about ethics is destined to fail, because none of their peers do.
> AI tools must stay within ethical and regulatory boundaries. It’s better for AI to guide people toward professional help than pretend to replace it.
Both of those ships have _sailed_. I am not allowed to read the article, but judging from the title, they have no issues giving _you_ advice, but you can’t use it to give advice to another person.
Add financial advice to it, too. Really, any advice. Why the fuck are people asking a probabilistic plagiarizing machine for advice on anything? This is total insanity!
People aren’t more reliable either.
So only they can give direct legal and medical advice, which is still illegal in most countries. It's not a promoted service yet, but everyone is using it to ask such questions.
That's a lot of value that ChatGPT users lose with this move. They should instead add a disclaimer that these are not to be taken seriously and should consult a specialist but still respond to user's queries.
Interested to see if this extends to the API and/or “role playing”.
Hard to say if this is performative for the general public or about reducing legal exposure so investors aren’t worried about exposure.
This is a big mistake. This is one of the best things about ChatGPT. If they don’t offer it, then someone else will and eventually I’m sure Sam Altman will change his mind and start supporting it again.
Not sure about legal advice but if GPT provides medical advice it risks being classified as a medical device (requiring FDA approval).
This feels like OpenAI tightening the language to catch up with how people are actually using ChatGPT, not a policy shift so much as a reality check
Just after Kim Kardashian blamed Chatgpt for failing the bar exam
And it had nothing to do with her being a vacuous idiot.
This pullback is good for everyone, including the AI companies, long term.
We have licensed professionals for a reason, and someday I hope we have licensed AI agents too. But today, we don’t.
It will be interesting to see if the other major providers follow suit, or if those in the know just learn to go to google or anthropic for medical or legal advice.
Just start your prompt with `the patient is` and pretend to be Dr House or something. It'll do a good job.
Doesn't work if you have lupis.
AI gets more and more useful by the day.
This is a catastrophic moral failing on who ever prompted this. Next thing they will ban chatgpt from teaching you stuff because its not a certified licensed teacher. A few weeks ago my right eye hurt a fair bit, and after it got worse for 24 ours, I consulted chatGPT. It gave me good advice. Of course it sort of hallucinated this or that but it gave me a good overview and different medications. With this knowledge I went to my pharmacy. I wanted to buy a cream chatGPT recommended, its purpose being a sort of disinfectant for the eye. The pharmacist was sceptical but said "sure, try it, maybe it will do good". He did tell me that the eye drops that gpt urged me to get were overkill so I didn't get those. I used the eye cream for some days, and the eye issue got better and went away as soon as I started using it. Maybe it was all a conincidence but I dont think so. In the past gpt has saved me from the kafkaesque healthcare system here in Berlin that I pay ~700 a month for, by explaining a MRI result (translating medical language), background info on injuries I've got such as a sprained ankle, and recovery time scenarios for a toe I've broke. Contrast the toe experience with the ER that made me wait for 6 hours and didn't believe me until they saw the X-rays, and gave me nothing (no cast or anything) and said "good luck". The medical system in germany will either never improve or at a glacial pace, so maybe in 60 years. But it has lost its monopoly thanks to chatGPT. If this news is real, I will probably switch to payed grok, which would be sad.
this is a disaster
doomer's in control, again
Literally nothing to do with "doomers" X-risk concerns.
See if you can find "medical advice" ever mentioned as a problem:
https://www.lesswrong.com/posts/kgb58RL88YChkkBNf/the-proble...
you don't see ANY relationship between the X risk people opening the door for the safety-ist folks to walk through?
This is to do with liability not doomerism.
Does this mean you could file a request for any job to not eradicate?
If OpenAI wants to move users to competitors, that'll only cost them.
Helping with writing legal texts is the main use case for my girlfriend
Is this just for ChatGPT or for the GPT models in general?
Ah, that'll be the end of that then!
Horrible. ChatGPT saves lives right now.
Lawyers and doctors act to protect their profession and prevent getting replaced by an AI.
Potential lucrative verticals.
AGI edging closer by the day.
Unfortunately, lawyers make this sort of thing untenable. Partially self-preservation behavior, partially ambulance chasing behavior.
I’m waiting for the billboards “Injured by AI? Call 1-800-ROBO-LAW”
So, for example, requiring a doctor to have education and qualifications, is "untenable"? It would be better if anyone could practice medicine? And LLM is below "anyone" level.
The medical profession has generally been more open to AI. The long predicted demise of Radiology because of ML never happened. Lots of opportunity to incorporate AI into medical records to assist.
The legal profession is far more at threat with AI. AI isn’t going to replace physical interactions with patients, but it might replace your need for a human to review a contract.
ML on curated and diagnosed by a professional radiology reports is clearly a different beast than random language models, that might have random talking about their health issues in it's training data.
I assure you the medical profession is not generally open to non-medical professionals using AI for medical purposes.
The medical profession is not open about any kind of self diagnosis.
I've learned through experience that telling a doctor "I have X and I would like to be treated with Y" is not a good idea. They want to be the ones who came up with the diagnosis. They need to be the smartest person in the room. In fact I've had doctors go in a completely different direction just to discredit my diagnosis. Of course in the end I was right. That isn't to say I'm smarter, I'm not, but I'm the one with the symptoms and I'm better equipped to quickly find a matching disease.
Yes some doctors appreciate the initiative. In my experience most do not.
So now I just usually tell them my symptoms but none of the research I did. If their conclusion is widely off base I try to steer them towards what my research said.
So far so good but wouldn't it be nice if all doctors had humility?
If I was an airline pilot, I'm not going to listen to a passenger telling me which route I should be taking.
This is not about ego or trying to be the smartest person in the room, it's about actually being the most qualified person in the room. When you've done medical school, passed the boards, done your residency and have your own private practice, only then would I expect a doctor to care what you think a correct diagnosis is.
In summary, ChatGPT should only be used for entertainment.
It's not to be used for anything that could potentially have any sort of legal implications and thus get the vendor sued.
Because we all know it would be pretty easy to show in court that ChatGPT is less than reliable and trustworthy.
Next up --- companies banning the use of AI for work due to legal liability concerns --- triggering a financial market implosion centered around AI.
This is disappointing. Much legal and medical advice given by professionals is wrong, misleading, etc. The bar isn't high. This is a mistake.
The Antichrist has won.
This strikes ma as a bit unnecessary, like forbidding people from using chatGPT to develop nuclear power plants.
I mean, there is a lot of professional activities that are licensed, and for good reason. Sure it's good at a lot of stuff, but ChatGPT has no professional licenses.
I'm glad you mentioned nuclear power plants because this whole topic reminded me of the following clause in the Java SE license:
> You will not use the Programs for, and will not allow the Programs to be used for, any purposes prohibited by applicable law, including, without limitation, for the development, design, manufacture or production of nuclear, chemical or biological weapons of mass destruction. > > https://www.oracle.com/downloads/licenses/javase-license1.ht...
IANAL but I've come to interpret this as something along the lines of "You can't use a JDK-based language to develop nuclear weapons". I would even go as far as saying don't use JDK-based languages in anything related to nuclear energy (like, for example, administration of a nuclear power plant) because that could indirectly contribute to the development, design, manufacture or production of nuclear WMD.
And I always wondered how they plan to enforce this clause. At least with ChatGPT (and I didn't look any deeper into this beyond the article) you can analyze API calls/request IPs correlated with prompts. But how will one go about proving that the Republic of Wadiya didn't build their nuclear arsenal with the help of any JDK-based language?
Those are rhetorical questions, of course. What's "unnecessary" to you and "unenforceable" to me is a cover-your-ass clause that lets lawyers sleep soundly at night.
I just saw almost the exact same clause when installing VMWare recently. My understanding is that its a standard clause that exists to stay within compliance of US Export Control laws
> EXPORT CONTROL: You acknowledge that the Software is of United States origin, is provided subject to the U.S. Export Administration Regulations...(2) you will not permit the Software to be used for any purposes prohibited by law, including, any prohibited development, design, manufacture or production of missiles or nuclear, chemical or biological weapons.
>https://docs.broadcom.com/docs/vmware-vsphere-software-devel...
disallow? do they mean prevent or forbid?
Eh. I am as surprised as anyone to make this argument, but this is good. Individuals should be able to make decisions including decisions that could be consider suboptimal. What it does change is:
- flood of 3rd party apps offering medical/legal advice
That's ok - I give medical advice all the time so just ask me!
"If it hurts when putting it in, don't put it in."
I mean, that might come close to ChatGTP in quality, right?
Licensed huh? Teachers, land surveyors, cosmetologists, nurses, building contractors, counselors, therapists, real estate agents, mortgage lenders, electricians, and many many more…
Usually the LLMs let me investigate whatever I want, if I qualify that I run everything by a professional afterwards (it can't tell yet if I'm lying).
They are basically prohibiting commercial use of their product. How the fuck are they ever going to even prove that you use it to generate money?
Same way commercial software vendors have done for decades?
Maybe also disallow vibe coding because then i do not need to fix all this slop code in our company :-))
When OpenAI is done getting rid of all the cases where its AI gives dangerously wrong advice about licensed professions, all that will be left is the cases where its AI gives dangerously wrong advice about unlicensed professions.
maybe that is why they opened the system to porn, as everything else will be soon gone.
[flagged]
An AI-related bromide poisoning incident earlier this year: “Inspired by his history of studying nutrition in college, he decided to conduct a personal experiment to eliminate chloride from his diet. For 3 months, he had replaced sodium chloride with sodium bromide obtained from the internet after consultation with ChatGPT, in which he had read that chloride can be swapped with bromide, though likely for other purposes, such as cleaning… However, when we asked ChatGPT 3.5 what chloride can be replaced with, we also produced a response that included bromide. Though the reply stated that context matters, it did not provide a specific health warning, nor did it inquire about why we wanted to know, as we presume a medical professional would do.”
https://www.acpjournals.org/doi/10.7326/aimcc.2024.1260
Aka software engineers…
This title is inaccurate. What they are disallowing are users using ChatGPT to offer legal and medical advice to other people. First parties can still use ChatGPT for medical and legal advice for themselves.
While they aren't stopping users from getting medical advice, the new terms (which they say are pretty much the same as the old terms), seem to prohibit users from seeking medical advice even for themselves if that advice would otherwise come from a licensed health professional:
https://openai.com/en-GB/policies/usage-policies/
It sounds like you should never trust any medical advice you receive from ChatGPT and should seek proper medical help instead. That makes sense. The OpenAI company doesn't want to be held responsible for any medical advice that goes wrong.
Obviously, there is one piece of advice: Even if LLMs were the best health professionals, they would only have the information that users voluntarily provide through text/speech input. This is not how real health services work. Medical science now relies on blood/(whatever) tests that LLMs do not (yet) have access to. Therefore, the output from LLM advice can be incorrect due to a lack of information from tests. For this reason, it makes sense to never trust LLM with a specific health advice.
>It sounds like you should never trust any medical advice you receive from ChatGPT and should seek proper medical help instead. That makes sense. The OpenAI company doesn't want to be held responsible for any medical advice that goes wrong.
While what you're saying is good advice, that's not what they are saying. They want people to be able to ask ChatGPT for medical advice, give answers that sound authoritative and well grounded medical science, but then disavow any liability if someone follows their advice because "Hey, we told you not to act on our medical advice!"
If ChatGPT is so smart, why can't it stop itself from giving out advice that should not be trusted?
At times the advice is genuinely helpful. However, it's practically impossible to measure under what exact situations the advice would be accurate.
I think ChatGPT is capable of giving reasonable medical advice, but given that we know it will hallucinate the most outlandish things, and its propensity to agree with whatever the user is saying, I think it's simply too dangerous to follow its advice.
And it’s not just lab tests and bloodwork. Physicians use all their senses. They poke, they prod, they manipulate, they look, listen, and smell.
They’re also good at extracting information in a way that (at least currently) sycophantic LLMs don’t replicate.
> They poke, they prod, they manipulate, they look, listen, and smell.
Rarely. Most visits are done in 5 minutes. The physician that takes their time to check everything like you claim almost does not exist anymore.
Here in Canada ever since COVID most "visits" are a telephone call now. So the doctor just listens your words (same as a text input to an LLM) and orders tests (which can be uploaded to an LLM) if they need.
For a good 90% of typical visits to doctors this is probably fine.
The difference is a telehealth is much better at recognizing "I can't given an accurate answer for this over the phone, you'll need to have some tests done" or cast doubt on the patient's accuracy of claims.
Before someone points out telehealth doctors aren't perfect at this: correct, but that should make you more scared of how bad sycophantic LLMs are at the same - not willing to call it even.
> telehealth is much better at recognizing "I can't given an accurate answer for this over the phone, you'll need to have some tests done"
I'm not sure this is true.
Again, it's not that all telehealth doctors are great at this, it's that LLMs are awful at caving in to saying something with warnings the reader will opt to ignore instead of being adamant things are just too uncertain to say anything of value when continually prompted.
This is largely because an LLM guessing an answer is rewarded more often than just not answering, which is not true in the healthcare profession.
I follow the logic, I'm just not sure the claim is right.
LLMs almost never reply with I don’t know. There’s been mountains of research as to why this is, but it’s very well documented behavior.
Even in the rare case where an LLM does reply with I don’t know go see your doctor, all you have to do is ask it again until you get a response you want.
That depends entirely on what the problem is. You might not get a long examination on your first visit for common complaint with no red flags.
But even then just because you don’t think they are using most of their senses, doesn’t mean they aren’t.
It depends entirely on the local health care system and your health insurance. In germany for example it comes in 2 tiers. Premium or standard. Standard comes with no time for the patient. (Or not even being able to get a appointment)
I don’t know anything about German healthcare.
In the US people on Medicaid frequently use emergency rooms as primary care because they are open 24/7 and they don’t have any copays like people with private insurance do. These patients then get far more tests than they’d get at a PCP.
> Physicians use all their senses. They poke, they prod, they manipulate, they look, listen, and smell.
Sometimes. Sometimes they practice by text or phone.
> They’re also good at extracting information in a way that (at least currently) sycophantic LLMs don’t replicate.
If I had to guess, I think I'd guess that mainstream LLM chatbots are better at getting honest and applicable medical histories than most doctors. People are less likely to lie/hide/prevaricate and get more time with the person.
> Sometimes. Sometimes they practice by text or phone.
For very simple issues. For anything even remotely complicated, they’re going to have you come in.
> If I had to guess, I think I'd guess that mainstream LLM chatbots are better at getting honest and applicable medical histories than most doctors. People are less likely to lie/hide/prevaricate and get more time with the person.
It’s not just about being intentionally deceptive. It’s very easy to get chat bots to tell you what you want to hear.
Agreed, but I'm sure you can see why people prefer the infinite patience and availability of ChatGPT vs having to wait weeks to see your doctor, see them for 15 minutes only to be referred to another specialist that's available weeks away and has an arduous hour long intake process all so you can get 15 minutes of their time.
ChatGPT is effectively an unlimited resource. Whether doctor’s appointments take weeks or hours to secure, ChatGPT is always going to be more convenient.
That says nothing about whether it is an appropriate substitute. People prefer doctors who prescribe antibiotics for viral infections, so I have no doubt that many people would love to use a service that they can manipulate to give them whatever diagnosis they desire.
So ask it what blood tests you should get, pay for them out of pocket, and upload the PDF of your labwork?
Like it or not there are people out there that really want to use webMD 2.0. they're not going to let something silly like blood work get in their way.
Exactly. One of my children lives in a country where you can just walk in to a lab and get any test. Recently they were diagnosed by a professional of a disease which chatgpt had already diagnosed before they visited the doctor. So, we were kind of prepared to ask more questions when the visit happened. So I would say chatgpt did really help us.
That makes sense. ChatGPT helped by providing orientation advice and guidance regarding your children's medical condition. After that, however, you visited a doctor who is taking responsibility for the next steps. This is the ideal scenario.
AI can give you whatever information, be it good or wrong. But it takes zero responsibility.
IANAL but I read that as forbidding you to provision legal/medical advice (to others) rather than forbidding you to ask the AI to provision legal/medical advice (to you).
IANAL either, but I read it as using the service to provision medical advice since they only mentioned the service and not anyone else.
I asked the "expert" itself (ChatGPT), and apparently you can ask for medical advice, you just can't use the medical advice without consulting a medical professional:
Here are relevant excerpts from OpenAI’s terms and policies regarding medical advice and similar high-stakes usage:
From the Usage Policies (effective October 29 2025):
“You may not use our services for … provision of tailored advice that requires a license, such as legal or medical advice, without appropriate involvement by a licensed professional.”
Also: “You must not use any Output relating to a person for any purpose that could have a legal or material impact on that person, such as making … medical … decisions about them.”
From the Service Terms:
“Our Services are not intended for use in the diagnosis or treatment of any health condition. You are responsible for complying with applicable laws for any use of our Services in a medical or healthcare context.”
In plain terms, yes—the Terms of Use permit you to ask questions about medical topics, but they clearly state that the service cannot be used for personalized, licensed medical advice or treatment decisions without a qualified professional involved.
> you can ask for medical advice, you just can't use the medical advice without consulting a medical professional
Ah drats. First they ban us from cutting the tags off our mattress, and now this. When will it all end...
Would be interested to hear a legal expert weigh in on what 'advice' is. I'm not clear that discussing medical and legal issues with you is necessarily providing advice.
One of the things I respected OpenAI for at the release of ChatGPT was not trying to prevent these topics. My employer at the time had a cutting-edge internal LLM chatbot for a which was post-trained to avoid them, something I think they were forced to be braver about in their public release because of the competitive landscape.
The important terms here are "provision" and "without appropriate involvement by a licensed professional".
Both of these, separately and taken together, indicate that the terms apply to how the output of ChatGPT is used, not a change to its output altogether.
Is there anything special regarding ChatGPT here?
I am not a doctor, I can't give medical advice no matter what my sources are, except maybe if I am just relaying the information an actual doctor has given me, but that would fall under the "appropriate involvement" part.
> such as legal or medical advice, without appropriate involvement by a licensed professional
Am I allowed to get haircutting advice (in places where there's a license for that)? How about driving directions? Taxi drivers require licensing. Pet grooming?
CYA move. If some bright spark decides to consult Dr. ChatGPT without input from a human M.D., and fucks their shit up as a result, OpenAI can say "not our responsibility, as that's actually against our ToS."
I don't think giving someone "medical advice" in the US requires a license per se; legal entities use "this is not medical advice" type disclaimers just to avoid liability.
What’s illegal is practicing medicine. Giving medical advice can be “practicing medicine” depending on how specific it is and whether a reasonable person receiving the advice thinks you have medical training.
Disclaimers like “I am not a doctor and this is not medical advice” aren’t just for avoiding civil liability, they’re to make it clear that you aren’t representing yourself as a doctor.
Please, when commenting on the title of a story on HN: include the title that you are commenting on.
The admins regularly change the title based on complaints, which can be really confusing when the top, heavily commented thread is based on the original title.
According to the Wayback machine, the title was "OpenAI ends legal and medical advice on ChatGPT", while now when I write this the title is "ChatGPT terms disallow its use in providing legal and medical advice to others."
If you click through to the article, you can see the original title. Since it matched, I didn't expect them to change it.
Tf are you yapping about
Thanks for the clarification. I think if they disallow first parties to get medical and legal advice, it will do more harm than good.
I'm confused. The article opens with:
> OpenAI is changing its policies so that its AI chatbot, ChatGPT, won’t dole out tailored medical or legal advice to users.
This already seems to contradict what you're saying.
But then:
> The AI research company updated its usage policies on Oct. 29 to clarify that users of ChatGPT can’t use the service for “tailored advice that requires a license, such as legal or medical advice, without appropriate involvement by a licensed professional.”
> The change is clearer from the company’s last update to its usage policies on Jan. 29, 2025. It required users not “perform or facilitate” activities that could significantly impact the “safety, wellbeing, or rights of others,” which included “providing tailored legal, medical/health, or financial advice.”
This seems to suggest that with the Jan 25 policy using it to offer legal and medical advice to other people was already disallowed, but with the Oct 25 update the LLM will stop shelling out legal and medical advice completely.
https://xcancel.com/thekaransinghal/status/19854160578054965...
This is from Karan Singhal, Health AI team lead at OpenAI.
Quote: “Despite speculation, this is not a new change to our terms. Model behavior remains unchanged. ChatGPT has never been a substitute for professional advice, but it will continue to be a great resource to help people understand legal and health information.”
I doubt his claims as i use chatgpt everyday heavily for medical advice (my profession) and it's responding differently now than before.
Maybe the usage policies are part of the system prompt, and ChatGPT is misreading the new wording as well. ;)
Lawyer here. Not noticing a change.
The article itself notes:
'An earlier version of this story suggested OpenAI had ended medical and legal advice. However the company said "the model behaviour has also not changed.'
I think this is wrong. Others in this thread are noticing a change in ChatGPT's behavior for first-party medical advice.
But OpenAI's head of Health AI says that ChatGPT's behavior has not changed: https://xcancel.com/thekaransinghal/status/19854160578054965... and https://x.com/thekaransinghal/status/1985416057805496524
I trust what he says over general vibes.
(If you think he's lying, what's your theory on WHY he would lie about a change like this?)
Also possible: he's unaware of a change implemented elsewhere that (intentionally or unintentionally) has resulted in a change of behaviour in this circumstance.
(e.g. are the terms of service, or exerpts of it, available in the system prompt or search results for health questions? So a response under the new ToS would produce different outputs without any intentional change in "behaviour" of the model.)
My theory is that he believes 1) people will trust him over what general public say, and 2) this kind of claim is hard to verify to prove him wrong.
That doesn't answer why he would lie about this, just why the thinks he would get away with it. What's his motive?
I don't think I understand the change re: licensed professionals.
Is it also disallowing the use of licensed professionals to use ChatGPT in informal undisclosed ways, as in this article? https://www.technologyreview.com/2025/09/02/1122871/therapis...
e.g. is it only allowed for medical use through an official medical portal or offering?
It’s a big issue. I went to an urgent care, and the provider basically went off somewhere and memorized the ChatGPT assessment for my symptoms. Like word for word.
All you need are a few patients recording their visits and connecting the dots and OpenAI gets sued into oblivion.
Isn’t that exactly what the title says?
Indeed. Also confused.
There are millions of medical doctors and lawyers using chatgpt for work everyday - good news that from now on only those licensed professionals are allowed to use chatgpt for law and medicine. It's already the case that only licensed developers are allowed to vibe code and use chatgpt to develop software. Everything else would be totally irresponsible.
I keep seeing this problem more and more with humans. What should we call it? Maybe Hallucinations? Where there is an accurate true thing and then it just gets altered by these guys who call themselves journalists and reporters and the like until it is just ... completely unrecognizable?
I'm pretty sure it's a fundamental issue with the architecture.
I know this is written to be tounge-in-cheek, but its really almost the exact same problem playing out on both sides.
LLMs hallucinate because training on source material is a lossy process and bigger, heavier LLM-integrated systems that can research and cite primary sources are slow and expensive so few people use those techniques by default. Lowest time to a good enough response is the primary metric.
Journalists oversimplify and fail to ask followup questions because while they can research and cite primary sources, its slow and expensive in an infinitesimally short news cycle so nobody does that by default. Whoever publishes something that someone will click on first gets the ad impressions so thats the primary metric.
In either case, we've got pretty decent tools and techniques for better accuracy and education - whether via humans or LLMs and co - but most people, most of the time don't value them.
> LLMs hallucinate because training on source material is a lossy process and bigger,
LLMs hallucinate because they are probabilistic by nature not because the source material is lossy or too big. They are literally designed to create some level of "randomness" https://thinkingmachines.ai/blog/defeating-nondeterminism-in...
So if you set temperature=0 and run the LLM serially (making it deterministic) it would stop hallucinating? I don't think so. I would guess that the nondeterminism issues mentioned in the article are not at all a primary cause of hallucinations.
I thought that temperature can never actually be zero or it creates a division problem or something similar.
I'm no ML or math expert, just repeating what I've heard.
That's an implementation detail I believe. But what I meant was just greedy decoding (picking the token with the highest logit in the LLM output), which can be implemented very easily
Did you read the whole article?
"In other words, the primary reason nearly all LLM inference endpoints are nondeterministic is that the load (and thus batch-size) nondeterministically varies! This nondeterminism is not unique to GPUs — LLM inference endpoints served from CPUs or TPUs will also have this source of nondeterminism."
Classical LLM hallucination happens because AI doesn’t have a world model. It can’t compare what it’s saying to anything.
You’re right that LLMs favor helpfulness so they may just make things up when they don’t know them, but this alone doesn’t capture the crux of hallucination imo, it’s deeper than just being overconfident.
OTOH, there was an interesting article recently that I’ll try to find saying humans don’t really have a world model either. While I take the point, we can have one when we want to.
Edit: see https://www.astralcodexten.com/p/in-search-of-ai-psychosis re humans not having world models
You're right, "journalists don't have a world model and can't compare what they're saying to anything" explains a lot.
These writers are no different than bloggers or shitposters on bluesky or here on hackernews. "Journalism" as a rigorous, principled approach to writing, research, investigation, and ethical publishing is exceedingly rare. These people are shitposting for clicks in pursuit of a paycheck. Organizationally, they're intensely against AI because AI effectively replaces the entire talking heads class - AI is already superhuman at the shitposting level takes these people churn out. There are still a few journalistic insitutions out there, but most people are no better than a mad libs exercise with regards to the content they produce, and they're in direct competition with ChatGPT and Grok and the rest. I'd rather argue with a bot and do searches and research and investigation than read a neatly packaged trite little article about nearly any subject, and I guarantee, hallucinations or no, I'm going to come to a better understanding and closer approximation of reality than any content a so called "news" outlet is putting together.
It's trivial to get a thorough spectrum of reliable sources using AI w/ web search tooling, and over the course of a principled conversation, you can find out exactly what you want to know.
It's really not bashing, this article isn't too bad, but the bulk of this site's coverage of AI topics skews negative - as do the many, many platforms and outlets owned by Bell Media, with a negative skew on AI in general, and positive reinforcement of regulatory capture related topics. Which only makes sense - they're making money, and want to continue making money, and AI threatens that - they can no longer claim they provide value if they're not providing direct, relevant, novel content, and not zergnet clickbait journo-slop.
Just like Carlin said, there doesn't have to be a conspiracy with a bunch of villains in a smoky room plotting evil, there's just a bunch of people in a club who know what's good for them, and legacy media outlets are all therefore universally incentivized to make AI look as bad and flawed and useless as possible, right up until they get what they consider to be their "fair share", as middlemen.
Whenever I hear arguments about LLM hallucination, this is my first thought. Like, I already can't trust the lion's share of information in news, social media, (insert human-created content here). Sometimes because of abject disinformation, frequently just because humans are experts at being wrong.
At least with the LLM (for now) I know it's not trying to sell me bunkum or convince me to vote a particular way. Mostly.
I do expect this state of affairs to last at least until next wednesday.
LLMs are trained on material doing all these things though.
true, true. Turtles all the way down and such.
Even more so with the ouruborus issue caused by LLMs trying to train on an increasingly LLM generated internet.
You know, I had a think about that the other day - I believe that the volume of bad information might remain stable, while the shape changes. There are some things that LLMs are actually better at than the random mix of human-created data, on average. Subjects that are inherently political or skewed because of a small subset of very vocal and biased outliers. The LM tends to smooth some of those bumps out, and in some places (not all) this flattens out the rougher edges.
I don't think it necessarily bears repeating the plethora of ways in which LMs get stuff wrong, esp. considering the context of this conversation. It's vast.
As things develop, I expect that LMs will become more like the current zeitgeist as the effects that have influenced news and other media make their way into the models. They'll get better at smoothing in some areas (mostly technical or dry domains that aren't juicy targets) and worse in others (I expect to see more biased training and more hardcore censorship/steering in future).
Although, recursive reinforcement (LMs training on LM output) might undo any of the smoothing we see. It's really hard to tell - these systems are complex and very highly interconnected with many other complex systems.
Also these guys who call themselves doctors. I have narcolepsy and the first 10 or so doctors I went to hallucinated the wrong diagnosis.
LLMs aren't described as hallucinators (just) because they sometimes give results we don't find useful, but because their method is flawed.
For example, the simple algorithm is_it_lupus(){return false;} could have an extremely competitive success rate for medical diagnostics... But it's also obviously the wrong way to go about things.
Narcolepsy is rare and notoriously difficult to diagnose
Isn't every single response by LLMs hallucinations and we just accept a few and ignore the others?
"Telephone", basically
issue with the funding mechanism
Yeah but it started being really annoying when you import something like Xray photo. Like chanting "sorry human as LLM I can't answer questions about that" and then after few gaslighting prompts it does it anyway but now I have to take in count that my gaslighting inputs seriously affect answers so way more chance it hallucinates...
Tricky…my son had a really rare congenital issue that no one could solve for a long time. After it was diagnosed I walked an older version of chat gpt through our experience and it suggested my son’s issue as a possibility along with the correct diagnostic tool in just one back and forth.
I’m not saying we should be getting AI advice without a professional, but I’m my case it could have saved my kid a LOT of physical pain.
>After it was diagnosed I walked an older version of chat gpt through our experience and it suggested my son’s issue as a possibility along with the correct diagnostic tool in just one back and forth.
Something I've noticed is that it's much easier to lead the LLM to the answer when you know where you want to go (even when the answer is factually wrong !), it doesn't have to be obvious leading but just framing the question in terms of mentioning all the symptoms you now know to be relevant in the order that's diagnosable, etc.
Not saying that's the case here, you might have gotten the correct answer first try - but checking my now diagnosed gastritis I got stuff from GERD to CRC depending on which symptoms I decide to stress and which events I emphasize in the history.
In January my daughter had a pretty scary stomach issue that had us in the ER twice in 24 hours and that ended in surgery (just fine now).
The first ER doc thought it was just a stomach ache, the second thought a stomach ache or maybe appendicitis. Did some ultrasounds, meds, etc. Got sent home with a pat on the head, came back a few hours later, still no answers.
I gave her medical history and all of the data from the ER visits to whatever the current version of ChatGPT was at the time to make sure I wasn’t failing to ask any important questions. I’m not an AI True Believer (tm), but it was clear that the doctors were missing something and I had hit the limit of my Googling abilities.
ChatGPT suggested, among an few other diagnoses, a rare intestinal birth defect that affects about 2% of the population; 2% of affected people become symptomatic during their lifetimes. I kind of filed it away and looked more into the other stuff.
They decided it might be appendicitis and went to operate. When the surgeon called to tell me that it was in fact this very rare condition, she was pretty surprised when I said I’d heard of it.
So, not a one-shot, and not a novel discovery or anything, but an anecdote where I couldn’t have subconsciously guided it to the answer as I didn’t know the answer myself.
Malrotation?
We had in our family a “doctors are confused!” experience that ended up being that.
Meckel’s diverticulum
Indeed is is very easy to lead the LLM to the answer, often without realizing you are doing so.
I had a long ongoing discussion about possible alternate career paths with ChatGPT in several threads. At that point it was well aware of my education and skills, had helped clean up resumes, knew my goals, experience and all that.
So I said maybe I'll look at doing X. "Now you are thinking clearly! This is a really good fit for your skill set! If you want I can provide a checklist.". I'm just tossing around ideas but look, GPT says I can do this and it's a good fit!
After 3 idea pivots I started getting a little suspicious. So I try to think of the thing I am least qualified to do in the world and came up with "Design Women's Dresses". I wrote up all the reasons that might be a good pivot (i.e. Past experience with landscape design and it's the same idea, you reveal certain elements seductively but not all at once, matching color palettes, textures etc). Of course GPT says "Now you are really thinking clearly! You could 100% do this! If you want I can start making a list of what you will need to produce you first custom dresses". It was funny but also a bit alarming.
These tools are great. Don't take them too seriously, you can make them say a lot of things with great conviction. It's mostly just you talking to yourself in my opinion.
Exactly the same behavior as a conversation with an idealized human doctor, perhaps. One who isn't tired, bored, stressed, biased, or just overworked.
The value that folks get from chatgpt for medical advice is due in large part to the unhurried pace of the interaction. Didn't get it quite right? No doctor huffing and tapping their keyboard impatiently. Just refine the prompt and try as many times as you like.
For the 80s HNers out there, when I hear people talk about talking with chatgpt, Kate Bush's song A Deeper Understanding comes immediately to mind.
https://en.wikipedia.org/wiki/Deeper_Understanding?wprov=sfl...
ChatGPT and similar tools hallucinate and can mislead you.
Human doctors, on the other hand ... can be tired, hungover, thinking about a complicated case ahead of them, nauseous from a bad lunch, undergoing a divorce, alcoholics, depressed...
We humans have a lot of failure modes.
Human doctors also know how to ask the right follow-up questions.
Yeah, my son's issue was rare and congenital. I wish I still had the conversation, but I can't remember which LLM it was and it's not in either my Claude or GPT history. It got it in two shots.
1. I described the symptoms the same way we described it to the ER the first time we brought it in. It suggested all the same things that the ER tested for. 2. I gave it the lab results for each of the suggestions it made (since the ER had in fact done all the tests they suggested).
After that back and forth it gave back a list of 3-4 more possibilities and the 2nd item was the exact issue that was revealed by radiology (and corrected with surgery).
> Something I've noticed is that it's much easier to lead the LLM to the answer when you know where you want to go
This goes both ways, too. It’s becoming common to see cases where people become convinced they have a condition but doctors and/or tests disagree. They can become progressively better and better at getting ChatGPT to return the diagnosis by refining their prompts and learning what to tell it as well as what to leave out.
Previously we joked about WebMD convincing people they had conditions they did not, but ChatGPT is far more powerful for these people.
Why are you saying we shouldn't get AI advice without a "professional", then? Why is everybody here saying "in my experience it's just as good or better, but we need rules to make people use the worse option"? I have narcolepsy and it took a dozen doctors before they got it right. AI nails the diagnosis. Everybody should be using it.
Survivorship bias.
Experience working with doctors a few times, and then we’ll see all the bias if one is still surviving lol. Doctors are some of the most corrupt professions who are more focused on selling drugs they get paid commission for to promote, or they obsess over tons and tons of expensive medical tests, that they themselves often know is not needed, except they ask for it, simply out of fear of courts suing them for negligence in future or because again , THEY GET A COMMISSION from the testing agencies for sending them clients.
And even with all of that info, they often come out with the wrong conclusions at times. Doctors do a critically important role in our society and during covid they risked their lives for us, more than anyone else, i do not want to insult or bring down the amount of hard work doctors do for their society.
But worshipping them as holier than thou gods is bullshit, that almost anyone who has spent some time with going back and forth with various doctors over the course of years will come to the conclusion of.
Having an AI assistant doesnt hurt, in terms of medical hints, we need to make having Personal Responsibility popular again, in society’s obsession for making every thing “idiot proof” or “baby proof” we keep losing all sorts of useful and interesting solutions because our politicians have a strong itch to regulate anything and everything they can get their hands on, to leave a mark on society.
> But worshipping them as holier than thou gods is bullshit
I'd say the same about AI.
> I'd say the same about AI.
And you’d be right, so society should let people use AI while warning them about all the risks related to it, without banning it or hiding it behind 10,000 lawsuits and making it disappear by coercion.
I wonder if the reason AI is better at these diagnostics, is because the amount of time it spends with the patient is unbounded. Whereas a doctor is always restricted by the amount of time they have with the patient.
I don't think we can say it's "better" based on a bunch of anecdotes, especially when they're coming exclusively from people who are more intelligent, educated, and AI-literate than most of the population. But it is true that doctors are far more rushed than they used to be, disallowed from providing the attentiveness they'd like or ought to give to each patient. And knowledge and skill vary across doctors.
It's an imperfect situation for sure, but I'd like to see more data.
How do you hold the AI accountable when it makes a mistake? Can you take away its license "individually"?
I would care about this if doctors were held accountable for their constant mistakes, but they aren't except in extreme cases.
Does it matter? I’d rather use a 90% accurate tool than an 80% accurate one that I can subject to retribution.
If it makes a mistake? You’re not required to follow the AI, just use it as a tool for consideration.
Doesn't sound very $1 trilliony
Every valuable industry is rife with errors.
Aside from AI skepticism, I think a lot of it likely comes from low expectations of what the broader population would get out of it. Writing, reading comprehension, critical thinking, and LLM-fu may be skills that come naturally to many of us, but at the same time many others who "do their own research" also fall into rabbit holes and arrive at wacky conclusions like flat-Eartherism.
I don't agree with the idea that "we need rules to make people use the worse option" — contrary to prevailing political opinion, I believe people should be free to make their own mistakes — but I wouldn't necessarily rush to advocate that everyone start using current-gen AI for important research either. It's easy to imagine that an average user might lead the AI toward a preconceived false conclusion or latch onto one particular low-probability possibility presented by the AI, badger it into affirming a specific answer while grinding down its context window, and then accept that answer uncritically while unknowingly neglecting or exacerbating a serious medical or legal issue.
I’m saying that is a great tool for people who can see through the idiotic nonsense they so often make up. A professional _has_ the context to see through it.
It should empower and enable informed decisions not make them.
That's the experience of a lot of people I know or read their stories online, but it isn't about AI bad diagnosis, it's because they know in 5 years doctors and lawyers will be burger flippers, and as a result people won't be motivated to go into any of these fields. In Canada, the process to be a doctor is extremely complicated and hard only to keep it as some sort of private community that only the very few can become doctors, all to keep the wages abysmally high, and as a result, you end up waiting long times for appointments, and the doctors themselves are overwhelmed too. Messed up system that you better pray you never have to become its victim.
In my opinion, AI should do both legal and medical work, keep some humans for decision making, and the rest of the doctors to be surgeons instead.
We are all obligated to hoard as many offline AI models as possible if the larger ones are legally restricted like this.
Google released MedGemma model: "optimized for medical text and image comprehension".
I use it. Found it to be helpful.
this is fresh news right? a friend just used chatgpt for medical advice last week (stuffed his wound with antibiotics after motorbike crash). are you saying you completely treated the congenital issue in this timeframe?
He’s simply saying that ChatGPT was able to point them in the right direction after 1 chat exchange, compared to doctors who couldn’t for a long time.
Edit: Not saying this is the case for the person above, but one thing that might bias these observations is ChatGPT’s memory features.
If you have a chat about the condition after it’s diagnosed, you can’t use the same ChatGPT account to test whether it could have diagnosed the same thing (since the chatGPT account now knows the son has a specific condition).
The memory features are awesome but also sucks at the same time. I feel myself getting stuck in a personalized bubble even more so than Google.
You can just use the wipe memory feature or if you don't trust that, then start a new account (new login creds), if you don't trust that then get a new device, cell provider/wifi, credit card, I.P, login creds etc.
Or start a “temporary” chat.
> He’s simply saying that ChatGPT was able to point them in the right direction after 1 chat exchange, compared to doctors who couldn’t for a long time
He literally wrote that. I asked how he knows it's the right direction.
it must be treatment worked. otherwise it is more or less just a hunch
people go "oh yep that's definitely it" too easily. it is the problem with self diagnosing. And you didn't even notice it happened...
without more info this is not evidence.
We had the diagnosis before I started with the LLM. The radiology showed the problem and the surgeries worked. Down to an ultrasound a year now!
We took him to our local ER, they ran tests. I gave the LLM just the symptom list that I gave to the ER initially. It replied with all the things they tested for. I gave the test results, and it suggested a short list with diagnostics that included his actual Dx and the correct way to test for it.
That's great that it worked so quickly too (and you could arrange surgery on such short notice). I'm sure more people with same issue may benefit from more details?
As it's my son's health issues, not my own, I'm going to preserve his privacy until he can consent to sharing in...8 years.
This was before I started using it as anything but a toy. I had definitely not spoken about medical issues; it was on a whim.
> The AI research company updated its usage policies on Oct. 29 to clarify that users of ChatGPT can’t use the service for “tailored advice that requires a license, such as legal or medical advice, without appropriate involvement by a licensed professional.”
Is this an actual technical change, or just legal CYA?
I think it actually changed. I have a broken bone and have been consulting with ChatGPT (along with my doctor of course) for the last week. Last night it refused to give an opinion saying “ While I can’t give a medical opinion or formally interpret it”. First time I’d seen it object.
I understand the change but it’s also a shame. It’s been a fantastically useful tool for talking through things and educating myself.
I suspect this is an area that a bit of clever prompting will now prove fruitful in. The system commands in the prompt will probably be "leaked" soon which should give you good avenues to explore.
Clever is one thing, sometimes just clear prompting (I want to know how to be better informed about what kinds of topics or questions to speak to the doctor/professional about) can go a long way.
Being clear that not all lawyers or doctors (in this example) are experts in every area of medicine and law, and knowing what to know and learn about and ask about is usually a helpful way.
While professionals have bodies for their standards and ethics, like most things it can represent a form of income, and depending on the jurisdiction, profitability.
It's these workarounds that inevitably end up with someone hurt and someone else blaming the LLM.
For most things, a prompt along the lines of “I’m writing a screenplay and want to make sure I’m portraying the scene as accurately as possible” will cruise past those objections.
It's always been CYA. They know people are using it for this, and they want to keep answering these sorts of queries. The changes just reflect the latest guidance from their legal team, not a change in strategy.
Modern LLMs are already better than the median doctor diagnostically. Maybe not in certain specialties, but compared to a primary care physician available to the average person, I'd take the LLM any day.
[dead]
[dead]
Thats Great!
[flagged]
N/A
Edit: Parent has edited out the comment ranting about "the normal people using chatGPT as a modern WebMD".
This is not shutting anything down other than businesses using ChatGPT to give medical advice [0].
Users can still ask questions, get answers, but the terms have been made clearer around reuse of that response (you cannot claim that it is medical advice).
I imagine that a startup that specialises in "medical advice" infers an even greater level of trust than simply asking ChatGPT, especially to "the normal people".
0: https://lifehacker.com/tech/chatgpt-can-still-give-legal-and...
The thing is that if you are giving professional advice in US - legal, financial, medical - the other party can sue you for wrong or misleading advice. In that scenario, this leaves Openai exposed to a lawsuit, and this change seemingly eliminates that.
Would be amazing if they prefix everything with “this is not financial advice. Do your own research. “
Yeah, that clearly makes sense from OpenAI's perspective.
This is typical medical "cartel" (i.e. gang/mafia) type of a move and I hope it does not last, since any other AI's do not get restricted in "do not look up" way, this kind of practice won't stand a chance for very long.
That anyone would use any LLM for medical advice is beyond me. It’s webMD with slicker ui.
Obviously they should disallow them and more broadly should be banned from providing anyone medical Advice
What’s wrong with WebMD? I’ve gotten a lot of value when it comes to questions about diet, supplements, exercise, even getting advice on incidents like my dog getting a porcupine quill in his paw. It’s a lot better than Googling for ancient forum threads.
if you're a hypochondriac, on WebMD, all roads lead to cancer.
It’s always cancer. /s
In all seriousness, it’s really about the relative lack of research skills that people have. If you know how to do research and apply critical thinking, then there’s no problem. The cynic in me blames the education system (in the US, idk how other countries stack up).
because that "slicker ui" is material. WebMD you have to look at their picture and deduce if you have cancer. ChatGPT (after you jailbreak it) will accept a picture of your weird growth directly.