FWIW I know of a pharmacy network that's implementing AI. Not only does the AI make egregious errors like speaking in Spanish when the conversation is in English. It also has a limited context window for ground rules of 5000 tokens or less. So correcting all the various errors goes over the instructions limit. As a result if the implementation goes through it will be of poor quality. All I can think is that people with no technical expertise are making these decisions, it is easy for any techie to predict that AI of this level won't be capable.
This is probably (if the business here is large enough to support it) an excellent area for a tailored model instead of a general purpose LLM. A lot of the context around medication comprehension is going to be rather universal to pharmacies and other HCPs so ideally that information would be baked into the model itself with the actual user supplied context mostly being focused on business specific logic. The commodity LLMs are just an amalgam of a hammer welded to a screwdriver, a car, a compass and a toothbrush - more specialized models, with more specialized testing and offered by a company specialized in the field would possibly offer a decent customer experience here?
With something like healthcare though I'll remain eternally skeptical that anything other than a licensed expert will be palatable to the public.
This is an underappreciated point when people talk about AI/chat integrations. If they put a claude-level model behind it, I'm sure it could resolve 99% of the situations faster and better than a human could, or at least realize its limitations when it can't. But of course that is too expensive so they often use extremely weak models that are literally worse than a good incremental full text search box.
So - I run a company that does AI agents for pharmacy. We are at Series B now and growing well - series C in 9-12 months given growth. We just acquired a competitor of that vendor so very much an insider.
First, The technology works, and it scales, but the whole bottleneck is domain expertise and implementation. These are expensive, and hard to scale. We hire pharmacists as project managers, that's how important domain expertise and implementations are.
Second, the amount of noise of "Voice AI for <industry>" is incredible. Most of them are completely clueless about the industry & basically "YC-striver" type who can only sell to other yc companies. They fail hard the moment they touch critical functions of the real world.
We have hundreds of pharmacies with us now and they mostly successful and happy. Our biggest friction is customers being burned by vendors like this and writing off AI altogether as "not ready".
> We have hundreds of pharmacies with us now and they mostly successful and happy. Our biggest friction is customers being burned by vendors like this and writing off AI altogether as "not ready".
This is such a great graph. Your company feels successful because your customer is happy yet the rest of the world would be just happy if your company died in a fire. This seems to also be just fine for companies like yours.
Very much clueless commentary. Retail pharmacies in the US are dying dozens a day. They don't have margin to support people answering the phone like tech companies with 80% margin do. Welcome to the real world where it is survival for these pharmacies not greed or shareholder value.
They literally lose money filling your branded prescription even with no labor baked in. We are their lifeline.
I know because I am not some spoiled tech worker who stumbled into this industry. I ran these pharmacies for over a decade before ever writing a line of code.
First, most tech companies of comparable size to your customers have negative margins and survive on investment cycles, and the tech companies that are bigger haven't offered human support to anyone in decades. (And indeed, everyone already resents this practice and those tech companies. Generally, sentiment is abyssmal.)
I know you're just trying to illustrate the real plight of your customers, but it's a bad illustration that confuses the discussion.
Second, out of sincere curiosity: what was the future of the industry -- and these particular pharmacies -- if AI hadn't arrived to offer salvation? The role they fill in society is non-optional, so how the market restructuring around these infeasibly narrow margins before?
If my experience is common, most health plans are moving to mail-order for routine, generic, recurring prescriptions. They only want you to visit a retail pharmacy for stuff that is urgently needed or strictly controlled.
Most pharmacies are having financial issues basically due to being stuck in the middle of the battle between insurers, drug manufacturers, and PBMs over reimbursement rates, and having the weakest negotiating position of all of the parties.
AI probably will save the pharmacies some money, but IMO this will probably cause reimbursement rates to decline further, leaving the pharmacies in the same position as before.
Weakest negotiating position other than the customers.
The funny part is probably a majority of pharmacies are owned by CVS or Walgreens each of which are insurance companies in their own right, and often people with company health insurance will ONLY allow reimbursements through their company pharmacy.
Tbh, this is most industries. Very few have much of a moat and have some wedge/pressure from their suppliers (land/labor/capital), their clients and competition.
Insurance companies and PBMs would probably say the same they have competitors (ie:replaceable) and can’t just pay more for this or that, or reject any cost saving measure available.
Clueless? You sound tone deaf. If you came here thinking everyone would love your product becase AI, well, sorry about that.
Just because you think it is an amazing product does not mean the rest of us will agree with you. It's right up there with "I've never had a problem with it so anyone claiming to have a problem is wrong" mentality. It was your comment acknowledging the public does not like your product or products like yours. You just think your product is better than others. That may be true but it again does not mean it is so good that the public enjoys interacting with your product.
NPS typically improves because no one stays on hold for 20+ mins anymore.
People have no ideas have pharmacies work. Pharmacies don't control how much they pay for drug, or how much they get paid. The margins are terrible and you can't hire enough techs to answer the phone at peak call volume. So - most people experience are terrible as they wait on hold for a really long time.
EVEN if the AI is a little rough around the edges with relational calls, just solving for transactional calls with no hold is a massive win.
I'll gladly take a 20min hold after having to resort to screaming "give me a fucking human" or driving the the pharmacy JUST TO TALK TO a human in the exceptions, which seem to be about 75% of my interactions with CVS/Walgreens AI agents.
> or driving the the pharmacy JUST TO TALK TO a human
It’s nice that this remains an option. It’s a dying one. So much is better handled and more easily handled in person than over the phone/mail/web/fax/email/blah, even if they’re 1:1 with a human.
It’s why I stick with my credit union instead of moving everything over to an online bank even tho I’ve managed to get 99% of everything done with the CU without going to a branch.
I recently had an AI chatbot lie to me. SiriusXM's ai bot INSISTED that the weather notifications were NOT a siriusxm feature. But, I had called ~4 months previously and the human confirmed it WAS a feature and she very kindly turned it off
Just as you said, I spent 20 minutes YELLING at the AI before it would transfer me to a human. Oh, and the entire time it was spewing ads for their programming. When I got to the human she, once again, confirmed it WAS a feature and it re-activated during 'free weekends'.
My father had cancer. After he was diagnosed he retired. His full time job became managing his health: medicare, state benefits, doctors appointments, medications, talking with billing people. He was on firstname basis with multiple providers. He invited some nurses and their families to the first 4th of july when he was cancer-free (they politely declined to keep the relationship professional).
I just imagine my father, or my 75yr old mother (whose health is beginning to decline) trying to navigate this new AI world. It must be an absolute nightmare. I can imagine how frustrated and disempowered they'd feel after a lifetime of reasoning with other humans.
EVEN if the AI is a little rough around the edges with relational calls, just solving for transactional calls with no hold is a massive win.
You keep saying that reducing call time is the "win." It seems like that the only metric you're measuring. If so, your product is a failure, not a success.
This is HEALTHCARE. You don't get to do "rough around the edges." You don't get to dismiss edge cases.
I also work in healthcare tech. My company doesn't permit such things because people die. I'll write it again because it's a point that tech fetishes can't get into their heads:
Do you know what NPS is? That's how the comment started. Patients like it. Because the alternative is to stay on hold for 20+ minutes, and then get it wrong. (Or have no pharmacies except Optum's and Express Scripts).
Good for you - I want in the ICU delivering care during COVID as a frontline clinician. You don't get to pull this line.
I find it hard to believe that patients like it given how sales work for these sorts of services - customer satisfaction is often the lowest priority given it’s ultimately a cost cutting service.
I live in a major metro and there are no pharmacies left except the largest ones. All my problems have to do with insurance not covering things, and insurance asks me to ask the pharmacy, pharmacy asks me to ask the insurance, and often insurance then asks me to ask my doctor to adjust the dosage cause they won’t cover that many days. Customers are batted around and I don’t see any AI that will solve this problem which is structural. The only value I see in AI is cost savings to replace humans, which is to say practically none as a customer over a website to check status and try to refill things.
NPS is notoriously unreliable. When I’m frustrated with a stupid support clanker, I don’t wait for another different clanker to ask me how it did. I hang up.
If that’s how you’re measuring patient satisfaction, you’re completely clueless.
Maybe “rock solid” could be understood as slightly hyperbolic (though it is metaphorical, so hard to say whether it is really hyperbole), but, it is possible to predict some things probabilistically with a high degree of accuracy. Depending on the task, a high enough degree of accuracy sounds doable, and if it offers a sufficiently convenient fallback to an actual human when it fails, that sounds like it could be quite a bit nicer than navigating a phone tree menu?
Classical voice to text combined with NN based voice to text, I would imagine can be highly accurate, and that’s probabilistic.
Of course, you're right. I didn't intend to criticise him.
I just think that it's not a good idea to transfer this megatrend to healthcare.
Yes, those systems can be highly accurate, but as we all have experienced, this is not a stable or consistent property. A very good model can give you an ingenious answer one minute and an utterly dumb one the next. My guess is that it's simply a consequence of the extremely high complexity of the 'plant' (natural language, language interfacing with the real world), leading to some highly non linear behaviour.
Good for you and wishing you more success. You probably going to be down voted as sentiment here in HN is going against AI due to the pressure on jobs, which is completely understandable from emotional point. People tend to compare AI to perfection for some reasons, which is not the case with humans. How many errors happened due to human phone operators? In the last few days there was an article where humans scored 66% on human evals.
I’ve been working with AI for years now and it’s not an easy task to get it right, but negative news go viral faster especially when it’s what people want to hear.
With pharmacies, people are emotional because their perscriptions getting filled can literally mean their livelihood, their health or their life. I hoard medications because my insurance will only cover 30 day fills, and often it takes 2-4 days to call enough people for insurance to cover a refill. Its a common problem for older folks to take half of their perscribed medication from either cost, or inability to get through to the pharmacy.
You effectively have AI gatekeeping healthcare, as companies roll these out and also take away any escalation path that a human call center, or local pharmacy staff might've been able to handle.
I've been also working with AI for years now, and it is absolutely horrifying to me that any company would feel comfortable rolling it out - you're basically saying a 5% error rate is acceptable to you when you're providing a gatekeeping service that is life or death.
> People tend to compare AI to perfection for some reasons, which is not the case with humans.
So I mostly avoid commenting on or pontificating about the value or lack thereof of AI, but the problem I have with companies rolling out "AI for whatever" isn't that sometimes AI gets "whatever" wrong, it's that the companies simultaneous gut the processes around correcting these mistakes.
I'm currently dealing with a months-long banking error where the bank has incorrectly linked my credit card account to my wife's account and/or an account of mine in a different region, so it variously doesn't show up in my online account, shows the wrong number, shows the purchases from the wrong account, etc.
Every time I call in in a 5-minute rigamarole of:
"Please describe what you're calling about."
"Representative."
"In order to connect to a representative, please describe what you're calling about."
"There's an error with my account."
"Would you like to know your account balance?"
"No, my credit card account is not correctly configured and isn't showing up in my online banking account."
"I didn't understand. Please describe what you're calling about."
Eventually the bot reaches some failure state and then allows me to wait on hold for an hour to talk to an actual rep.
When I eventually get to talk to a rep, I get "Yes, I see that your account isn't correctly configured, I'll submit a back-end ticket and it should get fixed within 5-10 business days."
The next week I check my online banking, my credit card still isn't showing up, and I call and repeat the entire process.
So yes, it's not AI that's directly causing this Kafkaesque nightmare, but it's AI that's empowered the leaders at every utility/retailer/bank/insurer/etc. that I deal with turn their support systems into this slop.
With that experience, why have you stayed with that provider? It would seem to be no harder, and quite satisfying, to open new accounts with another provider for you and your wife, and then cancel these accounts.
Basically every provider seems to have a similar level of customer support.
For banking in particular, I already bank with a half-dozen banks because they have different products/services that are optimal for some specific use cases (i.e. I min-max my bank accounts, credit cards, and investment accounts.) — however, customer support seems to be universally bad.
If you're curious about why I'm using that particular bank, I'm forced to use them for some investment accounts because my employer matching benefits uses them. For the credit card, they're one of the few banks that offer proper co-applicants (rather than just supplemental users) which really works best for the way my wife and I have our finances organized. And flights in my city are dominated by a single airline where the only way to avoid paying for baggage is to get the co-branded credit card from this bank.
Anecdotal evidence to support this - on a much less important platform.
We have two cars from our local car dealership. When we first set up the account, we did it through what I assumed were some kind of AI voice prompts. First time we take my wifes car in? I'm getting the calls from the dealership about whats wrong. Then I start getting texts telling me the car is ready. Same thing with her, but she's getting all the calls and notifications when we take my car in.
This has gone on for THREE YEARS. Last year we just gave up after telling people in person and going online and changing it several times. It did work for one visit, then reverted right back to mixing them up.
This isn't nearly as important as your banking issues, but its in a completely different industry and the experience is nearly identical.
That's been my observation of AI customer support. They're pretty much all glorified FAQ reciters unable to handle any unusual or uncommon situation not in the FAQs.
The one AI customer support experience I’ve actually enjoyed was last winter in a cold snap when my pipes froze. Drove to a Walmart to find a space heater and of course they were sold out. Called my local Lowe’s, got an AI agent, asked if any space heaters were in stock, it told me the one model left and the aisle to find it. Drove there and got what I needed without a fuss.
That’s the exception of course, literally every other time I’ve had to deal with an AI agent it can’t deal with the slightest deviation from its assumptions. Most of the time I call support, it’s because I already tried the automated options and they didn’t work for me. The fuzzy, nonstandard situations AI customer support agents suck are the one thing customer support is most needed for. I would’ve just used the website to check space heaters in stock if I hadn’t been driving.
Right, of the faster/better/cheaper triad, these things are overwhelmingly being implemented towards cheaper, and the idea that owners can gut the support structure for errors and edge-cases.
I realize quoting Doctorow on this is... rather low-hanging fruit for this topic, but I like how this scenario suggests the AI company makes more money when the implementation is hostile to the consumer:
> We’ve heard some pretty credible stuff about how AI can be used to spot solid mass tumors that sometimes humans miss. And, you know, if there was a sales call right now at your local hospital, where there was a pitchman for an AI company telling the hospital administrator, “Here’s what we’re going to do. Right now you have 10 radiologists. They cost $3 million a year. They review a hundred X-rays a day each. And I tell you what: I’m going to sell you a chatbot for a million bucks a year. And it’s going to sit in the shadows, and a couple of times a day, it’s going to tap your radiologist on the shoulder and say, 'Why don't you take another look at that one? I’d be very happy,’” that would seem like a real advance on medicine. But that’s not how the pitch is going.
> [...] But what the AI companies want to sell you is: fire nine-tenths of your radiologists, save $2.7 million a year, split that between the hospital shareholders and Sam Altman, take that remaining radiologist and put them in charge of marking the AI’s homework, put them in charge of clicking OK a hundred times a minute for the radiology reports that are coming out of the chatbot, and then, when it misses something and someone dies, blame that guy, make him what Dan Davies calls the “accountability sink” for the AI.
I actually made another comment about lack of leadership causing many issues including exhausting dev teams.
What you describe is an execution failure not AI, these are basic NLP systems that could be easily configured with some dataset to make them better, but i will make a wild guess that the bank has incompetent manager who was pushing for some numbers to put in a slide to say “I” did this.
The parent commenter's experience pretty much sums up my experience with every IVR system, AI or not. The IVR cannot do anything I can't do on the website or the app. I wouldn't be calling if I could get the job done without calling them, so my experience is simply that the IVR is simply an obstacle to talking to a human. Very often the first human I talk to is simply an obstacle to talking to the second human, because the first human is so limited in their knowledge and authority that they can't do anything I can't do with the website or app either.
I don't really think this is an implementation problem or incompetence. AI is just not reliable enough yet to deal with long tail situations, especially in cases where the most common procedures don't apply. You can't give it the authority to override common-case business rules or make financially consequential decisions, because it will be manipulated/prompt injected by fraudsters.
I suspect a lot of big orgs don’t care if they lose high-need/low-margin/negative-margin clients that need a couple chains of humans to fix their issue.
If they were otherwise high profit customers, they wouldn’t/shouldn’t have put them into IVR hell in the first place.
Anywho, I’m off to acquiring a DID that suggests I’m from a high-net-worth area. Until then, I’ll forge my caller id.
Yes, down to the false equivalence of "human hallucination" with AI hallucination. Except that a human who hallucinates too much will eventually be permanently discredited and especially so if that human is trying to be an expert in too many things. Yet we are asked over and over again to forgive the LLM's mistakes because of what it can do. Oh it can do this right, this was just one situation where it did something wrong. So tiresome.
As someone who has worked with companies in the space more than a decade ago, I'm glad that you're doing this! Genuinely love the "embedded not installed" approach.
I suspect that applying an aerospace approach of dissimilar redundancy and root cause analysis to this will yield a much better experience for everyone (i.e. you, the pharmacies and the patients) than what would have been possible with humans alone.
I've ranted about this before, but medicine as it stands isn't a serious field. And I can say that with a straight face, because medicine has until now, been the only field of modern scientific endeavor (or rather cloaked under modern scientific endeavor) that has fought tooth and nail against gathering more data points for improving understanding.
Good to see info on how critical domain expertise is for success.
I've seen AI used well in a few support situations, and it is great - faster to get to the answer vs a human (usually faster than even getting to a human, which is another issue)
When my case can be handled by the automated system, it is great, when it is not, it is usually misery, leading to anger at the customer and whatever company sold them that automated isht.
Especially for intelligent knowledgeable users of anything, by the time we call, it is an issue likely to need escalation.
So, the one absolutely critical factor for me and everyone I know is: how fast and easy is it to get to a human when we find the automated assistance does not work for our particular issue? It should be "I need to speak to a human now", and the immediate response should be "OK, let me get you one... Hello, this is [person], how can I help?" with the prior transcript already on the human's screen.
If this works, it is good because the humans are operating a level up, not having to deal with endless monotonous minutiae, and dealing with interesting issues all day, which means they get good at it and have a better attitude — a win for all.
How, and how well do you handle that situation — how close are you to the proper response I described?
Cases like this raise a question. As an engineer, the errors that coding AIs make are an annoyance, and reassuring in terms of my job prospect.
As a consumer, the errors that AIs make are more than an annoyance. I have yet to meet an AI phone assistant that can do anything more than an explicitly programmed phone tree, and several, including the one for the pharmacy I use, that do far less. They are undoubtedly much more expensive to create and test.
Clearly there's a bubble going on, and unprincipled people chasing funding and promotions as usual, but why are companies like CVS paying more to get less?
I think it is literally to waste our time. It's one more layer of defense to stop you from talking to a person.
There are two big wins in health care. One is to increase the scope of health care that gets paid for. More and more expensive treatments, to bring the money in. The second big win is to stop people from getting treatment. Deny coverage, confuse them and wear them out with bureaucracy, and in this case stop them from talking to an expensively educated human pharmacist. That's what an AI does better than a phone tree, because it more effectively creates a sense of helplessness and valuelessness in the customer.
Expanding the scope of care you're entitled to and then fighting against your ability to get it isn't a contradiction any more than it's a contradiction when a person breathes in and breathes out. It's two synergistic parts of a system whose goal ultimately has nothing to do with health care.
Companies will pawn off thier customers to AI at their peril.
In many ways this is a repeat of the India call center train wrecks of the 00s. On paper, letting someone in Bangalore vs onshore handle incoming customer service calls looked like a path to amazing savings. In practice the customer experience was horrendous and companies CTRL-Zed these decisions and rapidly brought customer service back onshore again. AI is just that story of shortsighted decisions by weak leadership playing out all over again.
The bigger question is that given all the consolidation and monopolies, if the top 2/3 companies do this, what is the alternative? Even for something like car insurance in the US, all the ELT at those want EBITDA at all costs. I'm not sure what fall out will be.
This, I suspect, is exactly like offshoring - and a lot of companies died during the offshoring fad because their software products become full of low quality merges and feature improvements that the company couldn't afford to fix or remove by the time they realized there was a problem. That remains my primary concern with AI code generation - well established codebases might become full of slop to the point of being unusable without a clear path back to maintainability leading to company death.
There's precedence for new versions of software being abandoned for older code bases... usually the new version doesn't ship or gets forked relatively quickly though, e.g. the Windows Longhorn reset or the Gnome 3/MATE fork.
It can happen - and companies can recover from that. But that's wasting a whole lot of money and a lot of executives will sunk cost fallacy themselves into continuing when that may be the much saner decision.
I didn't mean to imply that all companies that went hard into AI will die - just that a lot of companies that follow the trend may not realize the true maintenance costs and of those that did misguidedly follow the trend I'm sure some will luck or manage their way out of disaster for various reasons.
Pharmacy phone lines are a terrible first use case for voice AI. Older callers, drug names, insurance mess, zero patience. Automating "store hours" is fine. Automating refill exceptions is how you get this outcome.
We build text-based (e.g. SMS rather than voice) agents for various medical conversations (e.g. preparing people for complex appointments).
Our user base skews elderly. Going in, I was really skeptical about how much engagement and value we could provide to these older patients. I'm blown away how well it all turned out and how valuable it's been for our patients and client clinics.
For example, we were able to reduce no-shows for some high-cost appointments by 80%. Turns out that the reasons why people don't show up are a complicated and diffuse — everything form anxiety, to not speaking english, to being confused about how to get into the building — and properly configured LLM-based systems are really, really good at dealing with exactly this sort of problem.
I just don't get why we're not doing it vertically like you said here. Implement a part of the cognitive work using AI but let the hard parts be done by a person. They're still gonna be more rich, just less so.
Explain to me then why pharmacies have almost exclusively used IVR based phone systems for pharmacy refills and prescription management for the last decade?
Like they don’t even say the full name of the prescription they just tell you the first three letters and they’ll say “for the patient born on…”
Current pharmacy IT systems are currently almost 100% automated so how are you claiming it’s the first use?
Do you actually use these systems? They can’t at all handle my case where half the time the drug is on back order or there some insurance non sense in the way. I need to talk to a human every time
Correct they fail in any out of boundary condition which is like 50% of cases in my experience
Even worse is that the human is usually just as worthless because the problem is often upstream at the insurance provider or regulation
This is something I’m finding with people complaining about AI systems the AI system in this case is not any worse than the previous IVR system was it just expanded the use cases and now people say oh it doesn’t work for those expanded use cases
There’s no mention whatsoever of the current absolute devastatingly bad state of all software across the world
It’s like people are mad when there’s an application that already sucks, someone attempts to use genAI, that fails and then they blame AI
I'm testing it in my business. I'm in a unique position of running a business totally unrelated to software dev, handled customer service personally for 15+ years. I developed a AI backed live chat app first a little over a year ago. It has handled ~6,000 chats since then so I have a ton of valuable data on what does and doesn't work (spoiler alert, it works really well).
I'm taking that information and building a voice version. I'm taking a lot of time to make sure the phone version works naturally and that is can address most common things people call in about. Just like my chat version, GPT 4o was barely good enough to do what I needed. Since GPT 5 and on it has been more than adequate. I expect the same thing over time with voice, Open AI is going to release the GPT-live to the API which is a noticeable improvement.
Some people are just offended at the idea of talking to an AI chatbot voice or phone, no matter how good it is. I don't know how good this Kinney Drugs bot was, but it's very easy to make something that works but ends up being very frustrating for customers.
LOL, pleased to read that Air Canada got smacked down.
Yeah, if I'm calling or chatting with a company, I expect them to stand by whatever commitments they make, and I will judge them harshly if they don't. This is true whether an AI or a human is speaking on the company's behalf.
Not Pharmacy, but I literally had Shopify checkout button clickjack me and then AI agent and human agents (chat, email, and phone) refuse to cancel the resulting order today. Despite contacting within 5 minutes. Given how hostile so many companies are, I question why they even bother appearing to have support.
But this doesn't seem likely to translate well to pharmacy, where there will be consequences for treating customers like vermin, as opposed to actual incentives to do it.
I wouldn't be shocked if the tech needed in places like pharmacy or other sectors where there's some level forced responsibility leads to the version that's actually palatable elsewhere.
Actual personal service from a human being is the main reason that I'd choose a small regional pharmacy over a local CVS or mail-order. It seems to me like personal service is the ONLY advantage a regional chain like this has. If I were them, I'd lean into it rather than burning customer's goodwill with AI.
my experience with AI assistants has not been that great so far, they ask for customer id numbers which are long as ... with several digits and if you group them they don't understand, for example if the number begins with two 1s and you say eleven instead of one one they don't understand you, then they are usually not trained for the local pronunciation or dialect which makes it very hard communicating, my blood pressure rises into unhealthy ranges, but if I use expletives enough I get a human on the line
if you've tried to legitimately productionize LLMs to deliver a specific repeatable outcome, rather than human editable output(text, code), you know the models still aren't there yet and the whole product experience feels hacky
My partner got trapped in an email loop with a support agent and my goodness was she pissed.
Still, I got it to send a chocolate chip cookies recipe to us.
These things are horrible, but then coding agents were dumb as dogshite not too long ago, so maybe in a few years AI support agents will become useful.
Do we think that as a society there exists some meaningful lever by which we can apply force against this type of carelessness and exploitation?
I think we need to take a good hard look at what we want out of this world and ask ourselves. Do we really need to automate this? Like really. What benefit does the world receive having an AI chatbot tell grandma the wrong doage for her meds? This is the naked extraction of value from a service that's already been offshored. Companies like these are gleeful to make their service as shitty as they can because they know their users are doing so against their will. Infact, that's become the goal.
wonder how much of the ai CX hate is availability bias where people really notice it when it breaks but you never see the whole picture on total impact
it seems very implementation dependant, I know a startup with incredibly realistic voice agents that most people don’t detect is an ai that outperforms humans. But ai for CX seems like an easy project to do for companies looking to cut costs so the average agent a user encounters in the wild is poor quality
> wrong dosages, and missed prescription notifications.
BS. How is the AI touching that information?
Prescription notification is outbound communication. It’s almost a recorded message. A little text to speech if it is supposed to read the drug name.
Wrong dosages?? This must be handled by the “enroll a new scrip” functionality, right? So basically B2B doctor interaction. You let AI touch that? Aren’t the vast majority of the costs from customers not doctors? None of this makes sense.
The desired speed of AI adoption is what is hindering AI adoption. Big companies are trying to sell this as magic, and it's not. It requires proper use to get anything useful out of this system. But that takes away the magic of it and the investors can't have that as the eventual share price only works if it is magic. Frontier labs will smother their product with their timelines.
This is probably the most pragmatic take here and the overall responses support it. There's a handful of practitioners who are seeing some implementation success. They probably had significant domain expertise and are several iterations deep. Then there's a bunch of people who are experiencing the "AI iS mAgIc" approach where LLMs are being shoehorned into workflows with very little effort given to make the process actually… work (well)… to satisfy investor-driven mania. This growing-pains stage will pass eventually and we'll hit some equilibrium.
And even if the AI worked well enough (a claim to be proven by its proponents), companies wouldn’t give it access to be actually helpful. It’s all to save costs on routine calls, not enhance problem solving, the same way they’ve outsourced CS operators to body shops with a strict script.
I think of this as the kiosk economy. It's like how businesses introduce kiosks even when having a human employee would be more convenient, just for the sake of greater profit.
The biggest problem with AI customer service is that a human employee would've let a minor issue slide without escalating it. But a chatbot often inflames the situation, and by the time the customer reaches a human agent, they're already furious.
Most people aren't rational or logical. Non verbal feedback, like acknowledging someone's anger and showing empathy, is incredibly important.
If an AI representing a company I must do business with is trying to “empathize” with me to calm me down about not being able to access the services I pay them for, you better believe it’s going to make me and other customers much more upset.
Then the poor wretched line workers have to handle people for whom the companies are screwing over - there is no effective line upwards to anyone making the decisions to use these things other than an impotent “feedback survey”. You can’t “ask for AI’s manager”. This accountability sink is not only horrible customer service, when it’s life and death like “can I get the perscription that allows me to live” wild stuff starts to happen.
AI customer service doesn't fucking listen. A few years ago, I noticed that my bank had an old address on file for only one of my accounts. When I clicked edit next to the old address, it would bring me to a page that had the correct address.
At the time, they had a real person handling their support. The real person told me this was just a glitch and that my correct address was on file and everything is fine.
That wasn't actually true. A new debit card to replace my expiring one ended up being sent to the old address a couple of months ago.
So I go back to the website to try to edit it one last time. Same problem.
I call them and get their fucking stupid AI slopbot.
Still, I explain the whole situation:
> I'm trying to update my address on the website. When I click the edit link next to the old address, it shows my new address instead. I need to remove the old address.
What did the slopbot decide to say? Oh well, I should go on the website and edit my address of course! I gave it two more chances by trying to explain it a different way and how this didn't work.
Did it accept that and escalate the issue to a human? No. It kept telling me to go to the website.
I had to say
> Hey you fucking bitch, how many times do I have to tell you that doesn't fucking work? Escalate this to a fucking human before I get the attorney general involved.
to finally get a live person to fix it (I actually had to do this twice, the first set of fucks, the slopbot forgot what problem I was having and needed a refresher!). Suffice it to say, I will never trust any large amount of money with this bank again. I'm highly skeptical of any company using AI CS now, because this is one of my many experiences where the slopbot is incapable of recognizing a complex situation outside of its capabilities.
I mean look at this situation: their AI slopbot turned this one mistake into 5 mistakes instead of just allowing it to be one that another human corrects.
It's gotten to the point where whenever I see someone lauding AI customer support, I just assume they have an investment/job that hinges on people liking it and/or their only experiences are using it as a glorified refund button.
As someone who's been doing AI call centres for a while, I'm not surprised. It's really, really hard to build voice AI that works. There are no open source solutions that work out of the box.
But the biggest problem is ASR. WER is still atrocious even with SOTA models. When you add drug names and regional accents, it's a recipe for disaster.
Drug names are frequently BS loaded with strange consonants without singular pronunciation guidelines (sadly English isn't German) so humans already have a lot of variation in how things are pronounced - models then need to comprehend those variations, regional dialects (like those fuggin' ahs from Boston).
Wow, those names are actually real. They sound like fly-by-night compliance "brand" names on amazon sellers. How did those get approved by a multi-billion dollar marketing team?
They need a name they can trademark, and common words can't be trademarked. It's sort of a .com domain name problem, all the good names are already taken so you just go for something you hope is pronouncable.
I happened to be in Vermont last week and was unaware of this change. I heard complaints from three separate people unprompted. It was a total dumpster fire put on top of a critical service.
The idea, within regulated circles, is that it lacks provenance - there’s no vendor to sue. Many open source licenses also carry an explicit disclaimer of merchantability and fitness for purpose. These are taken seriously, and open-source software is viewed akin to car parts found on the side of the road. They could work but it’s better to let someone else verify.
But in the context of privacy it makes no sense though right? Open source and close source models (depending on vendor) both can be configured in a way that protects privacy of the user. The entire quote is wonky despite your point about companies preferring vendor-based system rather than raw dog oss
It will probably sound nice to laypersons but you're right, it has to have access to, at a minimum, a data extract with relevant information to perform it's tasks, although in my experience in the pharmacy IT world, a database view.
It's important to note while people are pumping the AI bubble that a near-moron with a checklist can do this job. They replaced AI with a regular old phone tree.
AI couldn't do this job with a checklist. It would forget the entire checklist after two responses, then get fixated on the words that the customer used to complain about how dumb the AI was and loop until they hung up and called back to hopefully clear the context.
edit: Unless Kinney had been sold a product that used caller ID to carry over context, then the customer's best chances would be to scream "Agent!" over and over again (to get put on hold for 45 minutes, then transferred to the most screamed-at Philippine minimum-wage gig worker on the planet), or to show up to the drugstore and start screaming at people.
FWIW I know of a pharmacy network that's implementing AI. Not only does the AI make egregious errors like speaking in Spanish when the conversation is in English. It also has a limited context window for ground rules of 5000 tokens or less. So correcting all the various errors goes over the instructions limit. As a result if the implementation goes through it will be of poor quality. All I can think is that people with no technical expertise are making these decisions, it is easy for any techie to predict that AI of this level won't be capable.
This is probably (if the business here is large enough to support it) an excellent area for a tailored model instead of a general purpose LLM. A lot of the context around medication comprehension is going to be rather universal to pharmacies and other HCPs so ideally that information would be baked into the model itself with the actual user supplied context mostly being focused on business specific logic. The commodity LLMs are just an amalgam of a hammer welded to a screwdriver, a car, a compass and a toothbrush - more specialized models, with more specialized testing and offered by a company specialized in the field would possibly offer a decent customer experience here?
With something like healthcare though I'll remain eternally skeptical that anything other than a licensed expert will be palatable to the public.
This is an underappreciated point when people talk about AI/chat integrations. If they put a claude-level model behind it, I'm sure it could resolve 99% of the situations faster and better than a human could, or at least realize its limitations when it can't. But of course that is too expensive so they often use extremely weak models that are literally worse than a good incremental full text search box.
I wonder what the price level (per token) needs to be, to match a minimum wage employee.
So - I run a company that does AI agents for pharmacy. We are at Series B now and growing well - series C in 9-12 months given growth. We just acquired a competitor of that vendor so very much an insider.
First, The technology works, and it scales, but the whole bottleneck is domain expertise and implementation. These are expensive, and hard to scale. We hire pharmacists as project managers, that's how important domain expertise and implementations are.
Second, the amount of noise of "Voice AI for <industry>" is incredible. Most of them are completely clueless about the industry & basically "YC-striver" type who can only sell to other yc companies. They fail hard the moment they touch critical functions of the real world.
We have hundreds of pharmacies with us now and they mostly successful and happy. Our biggest friction is customers being burned by vendors like this and writing off AI altogether as "not ready".
> We have hundreds of pharmacies with us now and they mostly successful and happy. Our biggest friction is customers being burned by vendors like this and writing off AI altogether as "not ready".
This is such a great graph. Your company feels successful because your customer is happy yet the rest of the world would be just happy if your company died in a fire. This seems to also be just fine for companies like yours.
Very much clueless commentary. Retail pharmacies in the US are dying dozens a day. They don't have margin to support people answering the phone like tech companies with 80% margin do. Welcome to the real world where it is survival for these pharmacies not greed or shareholder value.
They literally lose money filling your branded prescription even with no labor baked in. We are their lifeline.
I know because I am not some spoiled tech worker who stumbled into this industry. I ran these pharmacies for over a decade before ever writing a line of code.
First, most tech companies of comparable size to your customers have negative margins and survive on investment cycles, and the tech companies that are bigger haven't offered human support to anyone in decades. (And indeed, everyone already resents this practice and those tech companies. Generally, sentiment is abyssmal.)
I know you're just trying to illustrate the real plight of your customers, but it's a bad illustration that confuses the discussion.
Second, out of sincere curiosity: what was the future of the industry -- and these particular pharmacies -- if AI hadn't arrived to offer salvation? The role they fill in society is non-optional, so how the market restructuring around these infeasibly narrow margins before?
Microsoft offers human chat support for consumers, once you try their self-help tools: https://support.microsoft.com/en-us/contactus?ContactUsExper...
Apple offers online, telephone, and of course in-person support for hardware and software: https://support.apple.com/?cid=mc-ols-general-homepage-marco...
Amazon offers online and telephone customer support on your “my orders” page.
Netflix offers phone and chat support: https://help.netflix.com/en/contactus
NONE of those is comparable size.
I was replying to this:
> the tech companies that are bigger haven't offered human support to anyone in decades.
> what was the future of the industry
If my experience is common, most health plans are moving to mail-order for routine, generic, recurring prescriptions. They only want you to visit a retail pharmacy for stuff that is urgently needed or strictly controlled.
This is not accurate in states where a licensed pharmacist must own the business and PBMs have been forced out.
Why were pharmacies viable before? Presumably they didn't lose money for their whole history.
Most pharmacies are having financial issues basically due to being stuck in the middle of the battle between insurers, drug manufacturers, and PBMs over reimbursement rates, and having the weakest negotiating position of all of the parties.
AI probably will save the pharmacies some money, but IMO this will probably cause reimbursement rates to decline further, leaving the pharmacies in the same position as before.
Weakest negotiating position other than the customers.
The funny part is probably a majority of pharmacies are owned by CVS or Walgreens each of which are insurance companies in their own right, and often people with company health insurance will ONLY allow reimbursements through their company pharmacy.
Tbh, this is most industries. Very few have much of a moat and have some wedge/pressure from their suppliers (land/labor/capital), their clients and competition.
Insurance companies and PBMs would probably say the same they have competitors (ie:replaceable) and can’t just pay more for this or that, or reject any cost saving measure available.
https://www.healthaffairs.org/doi/abs/10.1377/hlthaff.2024.0...
Of the 88,930 retail pharmacies operating during 2010–20, 29.4 percent had closed by 2021.
and that was not due to answering phones .. adding a nondeterministic black box of bullshit will change nothing.
see: https://en.wikipedia.org/wiki/Pharmacy_benefit_management
> We are their lifeline.
Yeaaaah, I don't think that's true at all but I am glad it makes you feel virtuous and helpful.
Clueless? You sound tone deaf. If you came here thinking everyone would love your product becase AI, well, sorry about that.
Just because you think it is an amazing product does not mean the rest of us will agree with you. It's right up there with "I've never had a problem with it so anyone claiming to have a problem is wrong" mentality. It was your comment acknowledging the public does not like your product or products like yours. You just think your product is better than others. That may be true but it again does not mean it is so good that the public enjoys interacting with your product.
> We have hundreds of pharmacies with us now and they mostly successful and happy
How happy are their patients?
NPS typically improves because no one stays on hold for 20+ mins anymore.
People have no ideas have pharmacies work. Pharmacies don't control how much they pay for drug, or how much they get paid. The margins are terrible and you can't hire enough techs to answer the phone at peak call volume. So - most people experience are terrible as they wait on hold for a really long time.
EVEN if the AI is a little rough around the edges with relational calls, just solving for transactional calls with no hold is a massive win.
I'll gladly take a 20min hold after having to resort to screaming "give me a fucking human" or driving the the pharmacy JUST TO TALK TO a human in the exceptions, which seem to be about 75% of my interactions with CVS/Walgreens AI agents.
> or driving the the pharmacy JUST TO TALK TO a human
It’s nice that this remains an option. It’s a dying one. So much is better handled and more easily handled in person than over the phone/mail/web/fax/email/blah, even if they’re 1:1 with a human.
It’s why I stick with my credit union instead of moving everything over to an online bank even tho I’ve managed to get 99% of everything done with the CU without going to a branch.
I recently had an AI chatbot lie to me. SiriusXM's ai bot INSISTED that the weather notifications were NOT a siriusxm feature. But, I had called ~4 months previously and the human confirmed it WAS a feature and she very kindly turned it off
Just as you said, I spent 20 minutes YELLING at the AI before it would transfer me to a human. Oh, and the entire time it was spewing ads for their programming. When I got to the human she, once again, confirmed it WAS a feature and it re-activated during 'free weekends'.
My father had cancer. After he was diagnosed he retired. His full time job became managing his health: medicare, state benefits, doctors appointments, medications, talking with billing people. He was on firstname basis with multiple providers. He invited some nurses and their families to the first 4th of july when he was cancer-free (they politely declined to keep the relationship professional).
I just imagine my father, or my 75yr old mother (whose health is beginning to decline) trying to navigate this new AI world. It must be an absolute nightmare. I can imagine how frustrated and disempowered they'd feel after a lifetime of reasoning with other humans.
> EVEN if the AI is a little rough around the edges with relational calls, just solving for transactional calls
So "AI" treats relatonal as excluding transactional??
It's 100% this for me too. If I hear an AI voice, I hang up and try somewhere else.
EVEN if the AI is a little rough around the edges with relational calls, just solving for transactional calls with no hold is a massive win.
You keep saying that reducing call time is the "win." It seems like that the only metric you're measuring. If so, your product is a failure, not a success.
This is HEALTHCARE. You don't get to do "rough around the edges." You don't get to dismiss edge cases.
I also work in healthcare tech. My company doesn't permit such things because people die. I'll write it again because it's a point that tech fetishes can't get into their heads:
People die.
Do you know what NPS is? That's how the comment started. Patients like it. Because the alternative is to stay on hold for 20+ minutes, and then get it wrong. (Or have no pharmacies except Optum's and Express Scripts).
Good for you - I want in the ICU delivering care during COVID as a frontline clinician. You don't get to pull this line.
I find it hard to believe that patients like it given how sales work for these sorts of services - customer satisfaction is often the lowest priority given it’s ultimately a cost cutting service.
I live in a major metro and there are no pharmacies left except the largest ones. All my problems have to do with insurance not covering things, and insurance asks me to ask the pharmacy, pharmacy asks me to ask the insurance, and often insurance then asks me to ask my doctor to adjust the dosage cause they won’t cover that many days. Customers are batted around and I don’t see any AI that will solve this problem which is structural. The only value I see in AI is cost savings to replace humans, which is to say practically none as a customer over a website to check status and try to refill things.
NPS is notoriously unreliable. When I’m frustrated with a stupid support clanker, I don’t wait for another different clanker to ask me how it did. I hang up.
If that’s how you’re measuring patient satisfaction, you’re completely clueless.
It’s probably how their payor measures performance.
But I thought the issue with customer surveys was how biased they could be toward negative experiences cuz nobody else is angry enough to respond.
They have so many possible biases that are hard to control for.
They can be biased toward positive experiences too, if customers feel the job of the person that helped them is on the line.
As if the techs making $15/hr answering the phone don't make mistakes?
But at least he's responsible. And we accept that humans make mistakes.
However, when the mistake is made by the replacement that cost one of your community members their job for a few pennies gain...
The bar for these systems is higher than the bar for humans.
> You keep saying that reducing call time is the "win."
"AI" reduces call time here... as I hang up and look for an alternative supplier.
Why would he care about that? They're not the ones paying him.
Former patients
Starting with your funding and who you've acquired tells me whatever follows is not really worth reading.
> First, The technology works
God I would love it if this comment is read out at a hearing about defrauding investors or massively leaking PII
Make it so the parts that are done by AI are rock solid, until then, they shouldn't be near a customer and be done by humans. Resist the greed!
So you want him to make a piece of software that models a probability distribution rock solid.
Maybe “rock solid” could be understood as slightly hyperbolic (though it is metaphorical, so hard to say whether it is really hyperbole), but, it is possible to predict some things probabilistically with a high degree of accuracy. Depending on the task, a high enough degree of accuracy sounds doable, and if it offers a sufficiently convenient fallback to an actual human when it fails, that sounds like it could be quite a bit nicer than navigating a phone tree menu?
Classical voice to text combined with NN based voice to text, I would imagine can be highly accurate, and that’s probabilistic.
> could be understood as slightly hyperbolic
Of course, you're right. I didn't intend to criticise him.
I just think that it's not a good idea to transfer this megatrend to healthcare.
Yes, those systems can be highly accurate, but as we all have experienced, this is not a stable or consistent property. A very good model can give you an ingenious answer one minute and an utterly dumb one the next. My guess is that it's simply a consequence of the extremely high complexity of the 'plant' (natural language, language interfacing with the real world), leading to some highly non linear behaviour.
Yes
Good for you and wishing you more success. You probably going to be down voted as sentiment here in HN is going against AI due to the pressure on jobs, which is completely understandable from emotional point. People tend to compare AI to perfection for some reasons, which is not the case with humans. How many errors happened due to human phone operators? In the last few days there was an article where humans scored 66% on human evals.
I’ve been working with AI for years now and it’s not an easy task to get it right, but negative news go viral faster especially when it’s what people want to hear.
With pharmacies, people are emotional because their perscriptions getting filled can literally mean their livelihood, their health or their life. I hoard medications because my insurance will only cover 30 day fills, and often it takes 2-4 days to call enough people for insurance to cover a refill. Its a common problem for older folks to take half of their perscribed medication from either cost, or inability to get through to the pharmacy.
You effectively have AI gatekeeping healthcare, as companies roll these out and also take away any escalation path that a human call center, or local pharmacy staff might've been able to handle.
I've been also working with AI for years now, and it is absolutely horrifying to me that any company would feel comfortable rolling it out - you're basically saying a 5% error rate is acceptable to you when you're providing a gatekeeping service that is life or death.
> People tend to compare AI to perfection for some reasons, which is not the case with humans.
So I mostly avoid commenting on or pontificating about the value or lack thereof of AI, but the problem I have with companies rolling out "AI for whatever" isn't that sometimes AI gets "whatever" wrong, it's that the companies simultaneous gut the processes around correcting these mistakes.
I'm currently dealing with a months-long banking error where the bank has incorrectly linked my credit card account to my wife's account and/or an account of mine in a different region, so it variously doesn't show up in my online account, shows the wrong number, shows the purchases from the wrong account, etc.
Every time I call in in a 5-minute rigamarole of:
"Please describe what you're calling about."
"Representative."
"In order to connect to a representative, please describe what you're calling about."
"There's an error with my account."
"Would you like to know your account balance?"
"No, my credit card account is not correctly configured and isn't showing up in my online banking account."
"I didn't understand. Please describe what you're calling about."
Eventually the bot reaches some failure state and then allows me to wait on hold for an hour to talk to an actual rep.
When I eventually get to talk to a rep, I get "Yes, I see that your account isn't correctly configured, I'll submit a back-end ticket and it should get fixed within 5-10 business days."
The next week I check my online banking, my credit card still isn't showing up, and I call and repeat the entire process.
So yes, it's not AI that's directly causing this Kafkaesque nightmare, but it's AI that's empowered the leaders at every utility/retailer/bank/insurer/etc. that I deal with turn their support systems into this slop.
With that experience, why have you stayed with that provider? It would seem to be no harder, and quite satisfying, to open new accounts with another provider for you and your wife, and then cancel these accounts.
Basically every provider seems to have a similar level of customer support.
For banking in particular, I already bank with a half-dozen banks because they have different products/services that are optimal for some specific use cases (i.e. I min-max my bank accounts, credit cards, and investment accounts.) — however, customer support seems to be universally bad.
If you're curious about why I'm using that particular bank, I'm forced to use them for some investment accounts because my employer matching benefits uses them. For the credit card, they're one of the few banks that offer proper co-applicants (rather than just supplemental users) which really works best for the way my wife and I have our finances organized. And flights in my city are dominated by a single airline where the only way to avoid paying for baggage is to get the co-branded credit card from this bank.
Cancelling a long term credit car can profoundly effect ones credit score sadly.
Anecdotal evidence to support this - on a much less important platform.
We have two cars from our local car dealership. When we first set up the account, we did it through what I assumed were some kind of AI voice prompts. First time we take my wifes car in? I'm getting the calls from the dealership about whats wrong. Then I start getting texts telling me the car is ready. Same thing with her, but she's getting all the calls and notifications when we take my car in.
This has gone on for THREE YEARS. Last year we just gave up after telling people in person and going online and changing it several times. It did work for one visit, then reverted right back to mixing them up.
This isn't nearly as important as your banking issues, but its in a completely different industry and the experience is nearly identical.
That's been my observation of AI customer support. They're pretty much all glorified FAQ reciters unable to handle any unusual or uncommon situation not in the FAQs.
The one AI customer support experience I’ve actually enjoyed was last winter in a cold snap when my pipes froze. Drove to a Walmart to find a space heater and of course they were sold out. Called my local Lowe’s, got an AI agent, asked if any space heaters were in stock, it told me the one model left and the aisle to find it. Drove there and got what I needed without a fuss.
That’s the exception of course, literally every other time I’ve had to deal with an AI agent it can’t deal with the slightest deviation from its assumptions. Most of the time I call support, it’s because I already tried the automated options and they didn’t work for me. The fuzzy, nonstandard situations AI customer support agents suck are the one thing customer support is most needed for. I would’ve just used the website to check space heaters in stock if I hadn’t been driving.
Right, of the faster/better/cheaper triad, these things are overwhelmingly being implemented towards cheaper, and the idea that owners can gut the support structure for errors and edge-cases.
I realize quoting Doctorow on this is... rather low-hanging fruit for this topic, but I like how this scenario suggests the AI company makes more money when the implementation is hostile to the consumer:
> We’ve heard some pretty credible stuff about how AI can be used to spot solid mass tumors that sometimes humans miss. And, you know, if there was a sales call right now at your local hospital, where there was a pitchman for an AI company telling the hospital administrator, “Here’s what we’re going to do. Right now you have 10 radiologists. They cost $3 million a year. They review a hundred X-rays a day each. And I tell you what: I’m going to sell you a chatbot for a million bucks a year. And it’s going to sit in the shadows, and a couple of times a day, it’s going to tap your radiologist on the shoulder and say, 'Why don't you take another look at that one? I’d be very happy,’” that would seem like a real advance on medicine. But that’s not how the pitch is going.
> [...] But what the AI companies want to sell you is: fire nine-tenths of your radiologists, save $2.7 million a year, split that between the hospital shareholders and Sam Altman, take that remaining radiologist and put them in charge of marking the AI’s homework, put them in charge of clicking OK a hundred times a minute for the radiology reports that are coming out of the chatbot, and then, when it misses something and someone dies, blame that guy, make him what Dan Davies calls the “accountability sink” for the AI.
-- https://www.democracynow.org/2026/6/25/reverse_centaur_docto...
I actually made another comment about lack of leadership causing many issues including exhausting dev teams.
What you describe is an execution failure not AI, these are basic NLP systems that could be easily configured with some dataset to make them better, but i will make a wild guess that the bank has incompetent manager who was pushing for some numbers to put in a slide to say “I” did this.
The parent commenter's experience pretty much sums up my experience with every IVR system, AI or not. The IVR cannot do anything I can't do on the website or the app. I wouldn't be calling if I could get the job done without calling them, so my experience is simply that the IVR is simply an obstacle to talking to a human. Very often the first human I talk to is simply an obstacle to talking to the second human, because the first human is so limited in their knowledge and authority that they can't do anything I can't do with the website or app either.
I don't really think this is an implementation problem or incompetence. AI is just not reliable enough yet to deal with long tail situations, especially in cases where the most common procedures don't apply. You can't give it the authority to override common-case business rules or make financially consequential decisions, because it will be manipulated/prompt injected by fraudsters.
I suspect a lot of big orgs don’t care if they lose high-need/low-margin/negative-margin clients that need a couple chains of humans to fix their issue.
If they were otherwise high profit customers, they wouldn’t/shouldn’t have put them into IVR hell in the first place.
Anywho, I’m off to acquiring a DID that suggests I’m from a high-net-worth area. Until then, I’ll forge my caller id.
>;People tend to compare AI to perfection for some reasons, which is not the case with humans.
Every single time AI or FSD tesla booster say these ... reality is that AI is given all benefits of the doubt that people are not.
It is always selling the thing way sooner then it is capable, maligning everyone else and then have everyone else deal with the fallout.
Yes, down to the false equivalence of "human hallucination" with AI hallucination. Except that a human who hallucinates too much will eventually be permanently discredited and especially so if that human is trying to be an expert in too many things. Yet we are asked over and over again to forgive the LLM's mistakes because of what it can do. Oh it can do this right, this was just one situation where it did something wrong. So tiresome.
> People tend to compare AI to perfection for some reasons, which is not the case with humans.
No, people compare so-called AI with computer programs that are designed to work properly.
As someone who has worked with companies in the space more than a decade ago, I'm glad that you're doing this! Genuinely love the "embedded not installed" approach.
I suspect that applying an aerospace approach of dissimilar redundancy and root cause analysis to this will yield a much better experience for everyone (i.e. you, the pharmacies and the patients) than what would have been possible with humans alone.
I've ranted about this before, but medicine as it stands isn't a serious field. And I can say that with a straight face, because medicine has until now, been the only field of modern scientific endeavor (or rather cloaked under modern scientific endeavor) that has fought tooth and nail against gathering more data points for improving understanding.
More detail here, https://news.ycombinator.com/item?id=35762650
I'm glad that you're doing this!
Please apologize for making the world a worse place and explain how you will close your company down.
I've been building agentic voice workflows with nearly 2 decades of healthcare experience. Would love to talk if you're available.
We have hundreds of pharmacies with us now and they mostly successful and happy.
Is your pharmacy one of your customers? Do you use your own product regularly?
The pharmacies are happy because you are giving them higher profits.
Meanwhile you are royally fucking over the end user with your AI shit.
“You are absolutely right! You should indeed not give your 5 year old child 1000 milligrams of ibuprofen per hour”
Good to see info on how critical domain expertise is for success.
I've seen AI used well in a few support situations, and it is great - faster to get to the answer vs a human (usually faster than even getting to a human, which is another issue)
When my case can be handled by the automated system, it is great, when it is not, it is usually misery, leading to anger at the customer and whatever company sold them that automated isht.
Especially for intelligent knowledgeable users of anything, by the time we call, it is an issue likely to need escalation.
So, the one absolutely critical factor for me and everyone I know is: how fast and easy is it to get to a human when we find the automated assistance does not work for our particular issue? It should be "I need to speak to a human now", and the immediate response should be "OK, let me get you one... Hello, this is [person], how can I help?" with the prior transcript already on the human's screen.
If this works, it is good because the humans are operating a level up, not having to deal with endless monotonous minutiae, and dealing with interesting issues all day, which means they get good at it and have a better attitude — a win for all.
How, and how well do you handle that situation — how close are you to the proper response I described?
Cases like this raise a question. As an engineer, the errors that coding AIs make are an annoyance, and reassuring in terms of my job prospect.
As a consumer, the errors that AIs make are more than an annoyance. I have yet to meet an AI phone assistant that can do anything more than an explicitly programmed phone tree, and several, including the one for the pharmacy I use, that do far less. They are undoubtedly much more expensive to create and test.
Clearly there's a bubble going on, and unprincipled people chasing funding and promotions as usual, but why are companies like CVS paying more to get less?
I think it is literally to waste our time. It's one more layer of defense to stop you from talking to a person.
There are two big wins in health care. One is to increase the scope of health care that gets paid for. More and more expensive treatments, to bring the money in. The second big win is to stop people from getting treatment. Deny coverage, confuse them and wear them out with bureaucracy, and in this case stop them from talking to an expensively educated human pharmacist. That's what an AI does better than a phone tree, because it more effectively creates a sense of helplessness and valuelessness in the customer.
Expanding the scope of care you're entitled to and then fighting against your ability to get it isn't a contradiction any more than it's a contradiction when a person breathes in and breathes out. It's two synergistic parts of a system whose goal ultimately has nothing to do with health care.
Companies will pawn off thier customers to AI at their peril.
In many ways this is a repeat of the India call center train wrecks of the 00s. On paper, letting someone in Bangalore vs onshore handle incoming customer service calls looked like a path to amazing savings. In practice the customer experience was horrendous and companies CTRL-Zed these decisions and rapidly brought customer service back onshore again. AI is just that story of shortsighted decisions by weak leadership playing out all over again.
Its still a huge and growing business and one that AI helps https://tech.yahoo.com/ai/articles/call-centres-using-ai-whi...
lol at the title
The bigger question is that given all the consolidation and monopolies, if the top 2/3 companies do this, what is the alternative? Even for something like car insurance in the US, all the ELT at those want EBITDA at all costs. I'm not sure what fall out will be.
This, I suspect, is exactly like offshoring - and a lot of companies died during the offshoring fad because their software products become full of low quality merges and feature improvements that the company couldn't afford to fix or remove by the time they realized there was a problem. That remains my primary concern with AI code generation - well established codebases might become full of slop to the point of being unusable without a clear path back to maintainability leading to company death.
There's precedence for new versions of software being abandoned for older code bases... usually the new version doesn't ship or gets forked relatively quickly though, e.g. the Windows Longhorn reset or the Gnome 3/MATE fork.
It can happen - and companies can recover from that. But that's wasting a whole lot of money and a lot of executives will sunk cost fallacy themselves into continuing when that may be the much saner decision.
I didn't mean to imply that all companies that went hard into AI will die - just that a lot of companies that follow the trend may not realize the true maintenance costs and of those that did misguidedly follow the trend I'm sure some will luck or manage their way out of disaster for various reasons.
> companies CTRL-Zed these decisions and rapidly brought customer service back onshore again.
Not here in UK. They rerouted to the Phillipines, Egypt etc.
Pharmacy phone lines are a terrible first use case for voice AI. Older callers, drug names, insurance mess, zero patience. Automating "store hours" is fine. Automating refill exceptions is how you get this outcome.
We build text-based (e.g. SMS rather than voice) agents for various medical conversations (e.g. preparing people for complex appointments).
Our user base skews elderly. Going in, I was really skeptical about how much engagement and value we could provide to these older patients. I'm blown away how well it all turned out and how valuable it's been for our patients and client clinics.
For example, we were able to reduce no-shows for some high-cost appointments by 80%. Turns out that the reasons why people don't show up are a complicated and diffuse — everything form anxiety, to not speaking english, to being confused about how to get into the building — and properly configured LLM-based systems are really, really good at dealing with exactly this sort of problem.
I just don't get why we're not doing it vertically like you said here. Implement a part of the cognitive work using AI but let the hard parts be done by a person. They're still gonna be more rich, just less so.
You figured it out, there, at the end!
Explain to me then why pharmacies have almost exclusively used IVR based phone systems for pharmacy refills and prescription management for the last decade?
Like they don’t even say the full name of the prescription they just tell you the first three letters and they’ll say “for the patient born on…”
Current pharmacy IT systems are currently almost 100% automated so how are you claiming it’s the first use?
Do you actually use these systems? They can’t at all handle my case where half the time the drug is on back order or there some insurance non sense in the way. I need to talk to a human every time
Correct they fail in any out of boundary condition which is like 50% of cases in my experience
Even worse is that the human is usually just as worthless because the problem is often upstream at the insurance provider or regulation
This is something I’m finding with people complaining about AI systems the AI system in this case is not any worse than the previous IVR system was it just expanded the use cases and now people say oh it doesn’t work for those expanded use cases
There’s no mention whatsoever of the current absolute devastatingly bad state of all software across the world
It’s like people are mad when there’s an application that already sucks, someone attempts to use genAI, that fails and then they blame AI
The whole thing is absurd
I'm testing it in my business. I'm in a unique position of running a business totally unrelated to software dev, handled customer service personally for 15+ years. I developed a AI backed live chat app first a little over a year ago. It has handled ~6,000 chats since then so I have a ton of valuable data on what does and doesn't work (spoiler alert, it works really well).
I'm taking that information and building a voice version. I'm taking a lot of time to make sure the phone version works naturally and that is can address most common things people call in about. Just like my chat version, GPT 4o was barely good enough to do what I needed. Since GPT 5 and on it has been more than adequate. I expect the same thing over time with voice, Open AI is going to release the GPT-live to the API which is a noticeable improvement.
Some people are just offended at the idea of talking to an AI chatbot voice or phone, no matter how good it is. I don't know how good this Kinney Drugs bot was, but it's very easy to make something that works but ends up being very frustrating for customers.
This is my favorite AI customer service story:
https://arstechnica.com/tech-policy/2024/02/air-canada-must-...
> Air Canada essentially argued that “the chatbot is a separate legal entity that is responsible for its own actions”
LOL, pleased to read that Air Canada got smacked down.
Yeah, if I'm calling or chatting with a company, I expect them to stand by whatever commitments they make, and I will judge them harshly if they don't. This is true whether an AI or a human is speaking on the company's behalf.
That event happened in 2022. It keeps being tagged as "AI" when it was nothing more than a dumb front end for a document search.
Yes, it still screwed up. And yes, Air Canada tried to weasel out of it, but even though it was a chatbot it had no LLM behind it.
It wasn't an LLM (nobody said it was) but it was definitely more than just a document search.
Not Pharmacy, but I literally had Shopify checkout button clickjack me and then AI agent and human agents (chat, email, and phone) refuse to cancel the resulting order today. Despite contacting within 5 minutes. Given how hostile so many companies are, I question why they even bother appearing to have support.
But this doesn't seem likely to translate well to pharmacy, where there will be consequences for treating customers like vermin, as opposed to actual incentives to do it.
I wouldn't be shocked if the tech needed in places like pharmacy or other sectors where there's some level forced responsibility leads to the version that's actually palatable elsewhere.
Actual personal service from a human being is the main reason that I'd choose a small regional pharmacy over a local CVS or mail-order. It seems to me like personal service is the ONLY advantage a regional chain like this has. If I were them, I'd lean into it rather than burning customer's goodwill with AI.
my experience with AI assistants has not been that great so far, they ask for customer id numbers which are long as ... with several digits and if you group them they don't understand, for example if the number begins with two 1s and you say eleven instead of one one they don't understand you, then they are usually not trained for the local pronunciation or dialect which makes it very hard communicating, my blood pressure rises into unhealthy ranges, but if I use expletives enough I get a human on the line
I guess everything is ok these days as long as the President says "We screwed up [our most basic responsibility to customers], and we own that."
if you've tried to legitimately productionize LLMs to deliver a specific repeatable outcome, rather than human editable output(text, code), you know the models still aren't there yet and the whole product experience feels hacky
Not to mention insecure.
My partner got trapped in an email loop with a support agent and my goodness was she pissed.
Still, I got it to send a chocolate chip cookies recipe to us.
These things are horrible, but then coding agents were dumb as dogshite not too long ago, so maybe in a few years AI support agents will become useful.
This doesn’t surprise me.
Set up whatever online portal or app you want. I’d prefer to just sign in and click a button.
If I’m picking up the phone or sending an email it’s because I have a weird problem and I need to talk to a person.
It's interesting that I had to close 3 popups just to read this article, two of them were for tracking cookies
GDPR popups remind me of the spam popups of the 90s.
No, maliciously compliant scumbag companies did that.
You don’t need a cookie banner for functional cookies, and opt out should be a single button.
GDPR requires websites to get user consent before doing sketchy shit. You don’t need a cookie popup if you’re not a scumbag.
Agreed. It's horrible and stupid. I actually run a service in the EU and I don't present that nonsense even with the risk of being sued.
What is your service and what data are you collecting?
Do we think that as a society there exists some meaningful lever by which we can apply force against this type of carelessness and exploitation?
I think we need to take a good hard look at what we want out of this world and ask ourselves. Do we really need to automate this? Like really. What benefit does the world receive having an AI chatbot tell grandma the wrong doage for her meds? This is the naked extraction of value from a service that's already been offshored. Companies like these are gleeful to make their service as shitty as they can because they know their users are doing so against their will. Infact, that's become the goal.
Related: A pharmacy chain in Vermont implemented AI for efficiency (11 days ago, 40 points) https://news.ycombinator.com/item?id=49105190
Wow, it lasted over a week.
wonder how much of the ai CX hate is availability bias where people really notice it when it breaks but you never see the whole picture on total impact
it seems very implementation dependant, I know a startup with incredibly realistic voice agents that most people don’t detect is an ai that outperforms humans. But ai for CX seems like an easy project to do for companies looking to cut costs so the average agent a user encounters in the wild is poor quality
How open source makes it less private?!?
> wrong dosages, and missed prescription notifications.
BS. How is the AI touching that information?
Prescription notification is outbound communication. It’s almost a recorded message. A little text to speech if it is supposed to read the drug name.
Wrong dosages?? This must be handled by the “enroll a new scrip” functionality, right? So basically B2B doctor interaction. You let AI touch that? Aren’t the vast majority of the costs from customers not doctors? None of this makes sense.
The desired speed of AI adoption is what is hindering AI adoption. Big companies are trying to sell this as magic, and it's not. It requires proper use to get anything useful out of this system. But that takes away the magic of it and the investors can't have that as the eventual share price only works if it is magic. Frontier labs will smother their product with their timelines.
This is probably the most pragmatic take here and the overall responses support it. There's a handful of practitioners who are seeing some implementation success. They probably had significant domain expertise and are several iterations deep. Then there's a bunch of people who are experiencing the "AI iS mAgIc" approach where LLMs are being shoehorned into workflows with very little effort given to make the process actually… work (well)… to satisfy investor-driven mania. This growing-pains stage will pass eventually and we'll hit some equilibrium.
And even if the AI worked well enough (a claim to be proven by its proponents), companies wouldn’t give it access to be actually helpful. It’s all to save costs on routine calls, not enhance problem solving, the same way they’ve outsourced CS operators to body shops with a strict script.
I think of this as the kiosk economy. It's like how businesses introduce kiosks even when having a human employee would be more convenient, just for the sake of greater profit.
The biggest problem with AI customer service is that a human employee would've let a minor issue slide without escalating it. But a chatbot often inflames the situation, and by the time the customer reaches a human agent, they're already furious.
Most people aren't rational or logical. Non verbal feedback, like acknowledging someone's anger and showing empathy, is incredibly important.
If an AI representing a company I must do business with is trying to “empathize” with me to calm me down about not being able to access the services I pay them for, you better believe it’s going to make me and other customers much more upset.
Then the poor wretched line workers have to handle people for whom the companies are screwing over - there is no effective line upwards to anyone making the decisions to use these things other than an impotent “feedback survey”. You can’t “ask for AI’s manager”. This accountability sink is not only horrible customer service, when it’s life and death like “can I get the perscription that allows me to live” wild stuff starts to happen.
AI customer service doesn't fucking listen. A few years ago, I noticed that my bank had an old address on file for only one of my accounts. When I clicked edit next to the old address, it would bring me to a page that had the correct address.
At the time, they had a real person handling their support. The real person told me this was just a glitch and that my correct address was on file and everything is fine.
That wasn't actually true. A new debit card to replace my expiring one ended up being sent to the old address a couple of months ago.
So I go back to the website to try to edit it one last time. Same problem.
I call them and get their fucking stupid AI slopbot.
Still, I explain the whole situation:
> I'm trying to update my address on the website. When I click the edit link next to the old address, it shows my new address instead. I need to remove the old address.
What did the slopbot decide to say? Oh well, I should go on the website and edit my address of course! I gave it two more chances by trying to explain it a different way and how this didn't work.
Did it accept that and escalate the issue to a human? No. It kept telling me to go to the website.
I had to say
> Hey you fucking bitch, how many times do I have to tell you that doesn't fucking work? Escalate this to a fucking human before I get the attorney general involved.
to finally get a live person to fix it (I actually had to do this twice, the first set of fucks, the slopbot forgot what problem I was having and needed a refresher!). Suffice it to say, I will never trust any large amount of money with this bank again. I'm highly skeptical of any company using AI CS now, because this is one of my many experiences where the slopbot is incapable of recognizing a complex situation outside of its capabilities.
I mean look at this situation: their AI slopbot turned this one mistake into 5 mistakes instead of just allowing it to be one that another human corrects.
It's gotten to the point where whenever I see someone lauding AI customer support, I just assume they have an investment/job that hinges on people liking it and/or their only experiences are using it as a glorified refund button.
As someone who's been doing AI call centres for a while, I'm not surprised. It's really, really hard to build voice AI that works. There are no open source solutions that work out of the box.
But the biggest problem is ASR. WER is still atrocious even with SOTA models. When you add drug names and regional accents, it's a recipe for disaster.
ASR = Automatic Speech Recognition
WER = Word Error Rate
Too many acronyms; not everybody is in your field.
Just curious, why are drug names such a disaster? Do LLMs not understand pronunciation keys?
Trademarks.
And the patients generally don't have a pronunciation key. Doctors prescribing might not, either.
Drug names are frequently BS loaded with strange consonants without singular pronunciation guidelines (sadly English isn't German) so humans already have a lot of variation in how things are pronounced - models then need to comprehend those variations, regional dialects (like those fuggin' ahs from Boston).
It ends up being quite a lot of variation.
As some examples: Wegovy, Ixempra, Qvar, Keflex
Wow, those names are actually real. They sound like fly-by-night compliance "brand" names on amazon sellers. How did those get approved by a multi-billion dollar marketing team?
They need a name they can trademark, and common words can't be trademarked. It's sort of a .com domain name problem, all the good names are already taken so you just go for something you hope is pronouncable.
People pronounce drug names all sorts of ways, and there's a lot of them (with a steady trickle of new ones)
I happened to be in Vermont last week and was unaware of this change. I heard complaints from three separate people unprompted. It was a total dumpster fire put on top of a critical service.
> Marraffa says Burt is fully HIPAA compliant and that the AI is not open-source and does not generate or manipulate data.
We need to fight the talking point that open source is bad!
How could something like open source ever be perceived as bad?
The idea, within regulated circles, is that it lacks provenance - there’s no vendor to sue. Many open source licenses also carry an explicit disclaimer of merchantability and fitness for purpose. These are taken seriously, and open-source software is viewed akin to car parts found on the side of the road. They could work but it’s better to let someone else verify.
But in the context of privacy it makes no sense though right? Open source and close source models (depending on vendor) both can be configured in a way that protects privacy of the user. The entire quote is wonky despite your point about companies preferring vendor-based system rather than raw dog oss
how can it not manipulate data and still work? does it do it by magic? absolute L quote by the spokesperson
It will probably sound nice to laypersons but you're right, it has to have access to, at a minimum, a data extract with relevant information to perform it's tasks, although in my experience in the pharmacy IT world, a database view.
but if the source is open, the ay-eyes can find the zero days! /s
It's important to note while people are pumping the AI bubble that a near-moron with a checklist can do this job. They replaced AI with a regular old phone tree.
AI couldn't do this job with a checklist. It would forget the entire checklist after two responses, then get fixated on the words that the customer used to complain about how dumb the AI was and loop until they hung up and called back to hopefully clear the context.
edit: Unless Kinney had been sold a product that used caller ID to carry over context, then the customer's best chances would be to scream "Agent!" over and over again (to get put on hold for 45 minutes, then transferred to the most screamed-at Philippine minimum-wage gig worker on the planet), or to show up to the drugstore and start screaming at people.