Can AI answer medical questions better than your doctor?
Last year, headlines describing a study about artificial intelligence (AI) were eye-catching, to say the least:
- ChatGPT Rated as Better Than Real Doctors for Empathy, Advice
- The AI will see you now: ChatGPT provides higher quality answers and is more empathetic than a real doctor, study finds
- Is AI Better Than A Doctor? ChatGPT Outperforms Physicians In Compassion And Quality Of Advice
At first glance, the idea that a chatbot using AI might be able to generate good answers to patient questions isn’t surprising. After all, ChatGPT boasts that it passed a final exam for a Wharton MBA, wrote a book in a few hours, and composed original music.
But showing more empathy than your doctor? Ouch. Before assigning final honors on quality and empathy to either side, let’s take a second look.
What tasks is AI taking on in health care?
Already, a rapidly growing list of medical applications of AI includes drafting doctor’s notes, suggesting diagnoses, helping to read x-rays and MRI scans, and monitoring real-time health data such as heart rate or oxygen level.
But the idea that AI-generated answers might be more empathetic than actual physicians struck me as amazing — and sad. How could even the most advanced machine outperform a physician in demonstrating this important and particularly human virtue?
Can AI deliver good answers to patient questions?
It’s an intriguing question.
Imagine you’ve called your doctor’s office with a question about one of your medications. Later in the day, a clinician on your health team calls you back to discuss it.
Now, imagine a different scenario: you ask your question by email or text, and within minutes receive an answer generated by a computer using AI. How would the medical answers in these two situations compare in terms of quality? And how might they compare in terms of empathy?
To answer these questions, researchers collected 195 questions and answers from anonymous users of an online social media site that were posed to doctors who volunteer to answer. The questions were later submitted to ChatGPT and the chatbot’s answers were collected.
A panel of three physicians or nurses then rated both sets of answers for quality and empathy. Panelists were asked “which answer was better?” on a five-point scale. The rating options for quality were: very poor, poor, acceptable, good, or very good. The rating options for empathy were: not empathetic, slightly empathetic, moderately empathetic, empathetic, and very empathetic.
What did the study find?
The results weren’t even close. For nearly 80% of answers, ChatGPT was considered better than the physicians.
- Good or very good quality answers: ChatGPT received these ratings for 78% of responses, while physicians only did so on 22% of responses.
- Empathetic or very empathetic answers: ChatGPT scored 45% and physicians 4.6%.
Notably, the length of the answers was much shorter for physicians (average of 52 words) than for ChatGPT (average of 211 words).
Like I said, not even close. So, were all those breathless headlines appropriate after all?
Not so fast: Important limitations of this AI research
The study wasn’t designed to answer two key questions:
- Do AI responses offer accurate medical information and improve patient health while avoiding confusion or harm?
- Will patients accept the idea that questions they pose to their doctor might be answered by a bot?
And it had some serious limitations:
- Evaluating and comparing answers: The evaluators applied untested, subjective criteria for quality and empathy. Importantly, they did not assess actual accuracy of the answers. Nor were answers assessed for fabrication, a problem that has been noted with ChatGPT.
- The difference in length of answers: More detailed answers might seem to reflect patience or concern. So, higher ratings for empathy might be related more to the number of words than true empathy.
- Incomplete blinding: To minimize bias, the evaluators weren’t supposed to know whether an answer came from a physician or ChatGPT. This is a common research technique called “blinding.” But AI-generated communication does not always sound exactly like a human, and the AI answers were significantly longer. So, it’s likely that for at least some answers, the evaluators were not blinded.
The bottom line
Could physicians learn something about expressions of empathy from AI-generated answers? Possibly. Might AI work well as a collaborative tool, generating responses that a physician reviews and revises? Actually, some medical systems already use AI in this way.
But it seems premature to rely on AI answers to patient questions without solid proof of their accuracy and actual supervision by healthcare professionals. This study wasn’t designed to provide either.
And by the way, ChatGPT agrees: I asked it if it could answer medical questions better than a doctor. Its answer was no.
We’ll need more research to know when it’s time to set the AI genie free to answer patients’ questions. We may not be there yet — but we’re getting closer.
Want more information about the research? Read responses composed by doctors and a chatbot, such as answers to a concern about consequences after swallowing a toothpick.
About the Author
Robert H. Shmerling, MD, Senior Faculty Editor, Harvard Health Publishing; Editorial Advisory Board Member, Harvard Health Publishing
Dr. Robert H. Shmerling is the former clinical chief of the division of rheumatology at Beth Israel Deaconess Medical Center (BIDMC), and is a current member of the corresponding faculty in medicine at Harvard Medical School. … See Full Bio View all posts by Robert H. Shmerling, MD
Stepping up activity if winter slowed you down
If you've been cocooning due to winter’s cold, who can blame you? But a lack of activity isn't good for body or mind during any season. And whether you're deep in the grip of winter or fortunate to be basking in signs of spring, today is a good day to start exercising. If you’re not sure where to start — or why you should — we’ve shared tips and answers below.
Moving more: What’s in it for all of us?
We’re all supposed to strengthen our muscles at least twice a week and get a total at least 150 minutes of weekly aerobic activity (the kind that gets your heart and lungs working). But fewer than 18% of U.S. adults meet those weekly recommendations, according to the CDC.
How can choosing to become more active help? A brighter mood is one benefit: physical activity helps ease depression and anxiety, for example. And being sufficiently active — whether in short or longer chunks of time — also lowers your risk for health problems like
- heart disease
- stroke
- diabetes
- cancer
- brain shrinkage
- muscle loss
- weight gain
- poor posture
- poor balance
- back pain
- and even premature death.
What are your exercise obstacles?
Even when we understand these benefits, a range of obstacles may keep us on the couch.
Don’t like the cold? Have trouble standing, walking, or moving around easily? Just don’t like exercise? Don’t let obstacles like these stop you anymore. Try some workarounds.
- If it’s cold outside: It’s generally safe to exercise when the mercury is above 32° F and the ground is dry. The right gear for cold doesn’t need to be fancy. A warm jacket, a hat, gloves, heavy socks, and nonslip shoes are a great start. Layers of athletic clothing that wick away moisture while keeping you warm can help, too. Consider going for a brisk walk or hike, taking part in an orienteering event, or working out with battle ropes ($25 and up) that you attach to a tree.
- If you have mobility issues: Most workouts can be modified. For example, it might be easier to do an aerobics or weights workout in a pool, where buoyancy makes it easier to move and there’s little fear of falling. Or try a seated workout at home, such as chair yoga, tai chi, Pilates, or strength training. You’ll find an endless array of free seated workout videos on YouTube, but look for those created by a reliable source such as Silver Sneakers, or a physical therapist, certified personal trainer, or certified exercise instructor. Another option is an adaptive sports program in your community, such as adaptive basketball.
- If you can’t stand formal exercise: Skip a structured workout and just be more active throughout the day. Do some vigorous housework (like scrubbing a bathtub or vacuuming) or yard work, climb stairs, jog to the mailbox, jog from the parking lot to the grocery store, or do any activity that gets your heart and lungs working. Track your activity minutes with a smartphone (most devices come with built-in fitness apps) or wearable fitness tracker ($20 and up).
- If you’re stuck indoors: The pandemic showed us there are lots of indoor exercise options. If you’re looking for free options, do a body-weight workout, with exercises like planks and squats; follow a free exercise video online; practice yoga or tai chi; turn on music and dance; stretch; or do a resistance band workout. Or if it’s in the budget, get a treadmill, take an online exercise class, or work online with a personal trainer. The American Council on Exercise has a tool on its website to locate certified trainers in your area.
Is it hard to find time to exercise?
The good news is that any amount of physical activity is great for health. For example, a 2022 study found that racking up 15 to 20 minutes of weekly vigorous exercise (less than three minutes per day) was tied to lower risks of heart disease, cancer, and early death.
"We don't quite understand how it works, but we do know the body's metabolic machinery that imparts health benefits can be turned on by short bouts of movement spread across days or weeks," says Dr. Aaron Baggish, founder of Harvard-affiliated Massachusetts General Hospital's Cardiovascular Performance Program and an associate professor of medicine at Harvard Medical School.
And the more you exercise, Dr. Baggish says, the more benefits you accrue, such as better mood, better balance, and reduced risks of diabetes, high blood pressure, high cholesterol, and cognitive decline.
What’s the next step to take?
For most people, increasing activity is doable. If you have a heart condition, poor balance, muscle weakness, or you’re easily winded, talk to your doctor or get an evaluation from a physical therapist.
And no matter which activity you select, ease into it. When you’ve been inactive for a while, your muscles are vulnerable to injury if you do too much too soon.
“Your muscles may be sore initially if they are being asked to do more,” says Dr. Sarah Eby, a sports medicine specialist at Harvard-affiliated Spaulding Rehabilitation Hospital. “That’s normal. Just be sure to start low, and slowly increase your duration and intensity over time. Pick activities you enjoy and set small, measurable, and attainable goals, even if it’s as simple as walking five minutes every day this week.”
Remember: the aim is simply exercising more than you have been. And the more you move, the better.
About the Author
Heidi Godman, Executive Editor, Harvard Health Letter
Heidi Godman is the executive editor of the Harvard Health Letter. Before coming to the Health Letter, she was an award-winning television news anchor and medical reporter for 25 years. Heidi was named a journalism fellow … See Full Bio View all posts by Heidi Godman
About the Reviewer
Howard E. LeWine, MD, Chief Medical Editor, Harvard Health Publishing
Dr. Howard LeWine is a practicing internist at Brigham and Women’s Hospital in Boston, Chief Medical Editor at Harvard Health Publishing, and editor in chief of Harvard Men’s Health Watch. See Full Bio View all posts by Howard E. LeWine, MD
Will miscarriage care remain available?
When you first learned the facts about pregnancy — from a parent, perhaps, or a friend — you probably didn’t learn that up to one in three ends in a miscarriage.
What causes miscarriage? How is it treated? And why is appropriate health care for miscarriage under scrutiny — and in some parts of the US, getting harder to find?
What is miscarriage?
Many people who come to us for care are excited and hopeful about building their families. It’s devastating when a hoped-for pregnancy ends early.
Miscarriage is a catch-all term for a pregnancy loss before 20 weeks, counting from the first day of the last menstrual period. Miscarriage happens in as many as one in three pregnancies, although the risk gradually decreases as pregnancy progresses. By 20 weeks, it occurs in fewer than one in 100 pregnancies.
What causes miscarriage?
Usually, there is no obvious or single cause for miscarriage. Some factors raise risk, such as:
- Pregnancy at older ages. Chromosome abnormalities are a common cause of pregnancy loss. As people age, this risk rises.
- Autoimmune disorders. While many pregnant people with autoimmune disorders like lupus or Sjogren’s syndrome have successful pregnancies, their risk for pregnancy loss is higher.
- Certain illnesses. Diabetes or thyroid disease, if poorly controlled, can raise risk.
- Certain conditions in the uterus. Uterine fibroids, polyps, or malformations may contribute to miscarriage.
- Previous miscarriages. Having a miscarriage slightly increases risk for miscarriage in the next pregnancy. For instance, if a pregnant person’s risk of miscarriage is one in 10, it may increase to 1.5 in 10 after their first miscarriage, and four in 10 after having three miscarriages.
- Certain medicines. A developing pregnancy may be harmed by certain medicines. It’s safest to plan pregnancy and receive pre-pregnancy counseling if you have a chronic illness or condition.
How is miscarriage diagnosed?
Before ultrasounds in early pregnancy became widely available, many miscarriages were diagnosed based on symptoms like bleeding and cramping. Now, people may be diagnosed with a miscarriage or early pregnancy loss on a routine ultrasound before they notice any symptoms.
How is miscarriage treated?
Being able to choose the next step in treatment may help emotionally. When there are no complications and the miscarriage occurs during the first trimester (up to 13 weeks of pregnancy), the options are:
Take no action. Passing blood and pregnancy tissue often occurs at home naturally, without need for medications or a procedure. Within a week, 25% to 50% will pass pregnancy tissue; more than 80% of those who experience bleeding as a sign of miscarriage will pass the pregnancy tissue within two weeks.
What to know: This can be a safe option for some people, but not all. For example, heavy bleeding would not be safe for a person who has anemia (lower than normal red blood cell counts).
Take medication. The most effective option uses two medicines: mifepristone is taken first, followed by misoprostol. Using only misoprostol is a less effective option. The two-step combination is 90% successful in helping the body pass pregnancy tissue; taking misoprostol alone is 70% to 80% successful in doing so.
What to know: Bleeding and cramping typically start a few hours after taking misoprostol. If bleeding does not start, or there is pregnancy tissue still left in the uterus, a surgical procedure may be necessary: this happens in about one in 10 people using both medicines and one in four people who use only misoprostol.
Use a procedure. During dilation and curettage (D&C), the cervix is dilated (widened) so that instruments can be inserted into the uterus to remove the pregnancy tissue. This procedure is nearly 99% successful.
What to know: If someone is having life-threatening bleeding or has signs of infection, this is the safest option. This procedure is typically done in an operating room or surgery center. In some instances, it is offered in a doctor’s office.
If you have a miscarriage during the second trimester of pregnancy (after 13 weeks), discuss the safest and best plan with your doctor. Generally, second trimester miscarriages will require a procedure and cannot be managed at home.
Red flags: When to ask for help during a miscarriage
During the first 13 weeks of pregnancy: Contact your health care provider or go to the emergency department immediately if you experience
- heavy bleeding combined with dizziness, lightheadedness, or feeling faint
- fever above 100.4° F
- severe abdominal pain not relieved by over-the-counter pain medicine, such as acetaminophen (Tylenol) or ibuprofen (Motrin, Advil). Please note: ibuprofen is not recommended during pregnancy, but is safe to take if a miscarriage has been diagnosed.
After 13 weeks of pregnancy: Contact your health care provider or go to the emergency department immediately if you experience
- any symptoms listed above
- leakage of fluid (possibly your water may have broken)
- severe abdominal or back pain (similar to contractions).
How is care for miscarriages changing?
Unfortunately, political interference has had significant impact on safe, effective miscarriage care:
- Some states have banned a procedure used to treat second trimester miscarriage. Called dilation and evacuation (D&E), this removes pregnancy tissue through the cervix without making any incisions. A D&E can be lifesaving in instances when heavy bleeding or infection is complicating a miscarriage.
- Federal and state lawsuits, or laws banning or seeking to ban mifepristone for abortion care, directly limit access to a safe, effective drug approved for miscarriage care. This could affect miscarriage care nationwide.
- Many laws and lawsuits that interfere with miscarriage care offer an exception to save the life of a pregnant patient. However, miscarriage complications may develop unexpectedly and worsen quickly, making it hard to ensure that people will receive prompt care in life-threatening situations.
- States that ban or restrict abortion are less likely to have doctors trained to perform a full range of miscarriage care procedures. What’s more, clinicians in training, such as resident physicians and medical students, may never learn how to perform a potentially lifesaving procedure.
Ultimately, legislation or court rulings that ban or restrict abortion care will decrease the ability of doctors and nurses to provide the highest quality miscarriage care. We can help by asking our lawmakers not to pass laws that prevent people from being able to get reproductive health care, such as restricting medications and procedures for abortion and miscarriage care.
About the Authors
Sara Neill, MD, MPH, Contributor
Dr. Sara Neill is a physician-researcher in the department of obstetrics & gynecology at Beth Israel Deaconess Medical Center and Harvard Medical School. She completed a fellowship in complex family planning at Brigham and Women's Hospital, and … See Full Bio View all posts by Sara Neill, MD, MPH
Scott Shainker, DO, MS, Contributor
Scott Shainker, D.O, M.S., is a maternal-fetal medicine specialist in the Department of Obstetrics and Gynecology at Beth Israel Deaconess Medical Center (BIDMC). He is also a member of the faculty in the Department of Obstetrics, … See Full Bio View all posts by Scott Shainker, DO, MS
How well do you worry about your health?
Don’t worry. It’s good advice if you can take it. Of course that’s not always easy, especially for health concerns.
The truth is: it’s impossible (and ill-advised) to never worry about your health. But are you worrying about the right things? Let’s compare a sampling of common worries to the most common conditions that actually shorten lives. Then we can think about preventing the biggest health threats.
Dangerous but rare health threats
The comedian John Mulaney says the cartoons he watched as a child gave him the impression that quicksand, anvils falling from the sky, and lit sticks of dynamite represented major health risks. For him (as is true for most of us), none of these turned out to be worth worrying about.
While harm can befall us in many ways, some of our worries are not very likely to occur:
- Harm by lightning: In the US, lightning strikes kill about 25 people each year. Annually, the risk for the average person less than one in a million. There are also several hundred injuries due to nonfatal lightning strikes. Even though lightning strikes the earth millions of times each year, the chances you’ll be struck are quite low.
- Dying in a plane crash: The yearly risk of being killed in a plane crash for the average American is about one in 11 million. Of course, the risk is even lower if you never fly, and higher if you regularly fly on small planes in bad weather with inexperienced pilots. By comparison, the average yearly risk of dying in a car accident is approximately 1 in 5,000.
- Snakebite injuries and deaths: According to the Centers for Disease Control and Prevention, an estimated 7,000 to 8,000 people are bit by poisonous snakes each year in the US. Lasting injuries are uncommon, and deaths are quite rare (about five per year). In parts of the country where no poisonous snakes live, the risk is essentially zero.
- Shark attacks: As long as people aren’t initiating contact with sharks, attacks are fairly uncommon. Worldwide, about 70 unprovoked shark attacks occur in an average year, six of which are fatal. In 2022, 41 attacks occurred in the US, two of which were fatal.
- Public toilet seats: They may appear unclean (or even filthy), but they pose little or no health risk to the average person. While it’s reasonable to clean off the seat and line it with paper before touching down, health fears should not discourage you from using a public toilet.
I’m not suggesting that these pose no danger, especially if you’re in situations of increased risk. If you’re on a beach where sharks have been sighted and seals are nearby, it’s best not to swim there. When in doubt, it’s a good idea to apply common sense and err on the side of safety.
What do Google and TikTok tell us about health concerns?
Analyzing online search topics can tell us a lot about our health worries.
The top Google health searches in 2023 were:
- How long is strep throat contagious?
- How contagious is strep throat?
- How to lower cholesterol?
- What helps with bloating?
- What causes low blood pressure?
Really? Cancer, heart disease and stroke, or COVID didn’t reach the top five? High blood pressure didn’t make the list, but low blood pressure did?
Meanwhile, on TikTok the most common topics searched were exercise, diet, and sexual health, according to one study. Again, no top-of-the-list searches on the most common and deadly diseases.
How do our worries compare with the top causes of death?
In the US, these five conditions took the greatest number of lives in 2022:
- heart disease
- cancer
- unintentional injury (including motor vehicle accidents, drug overdoses, and falls)
- COVID-19
- stroke.
This list varies by age. For example, guns are the leading cause of death among children and teenagers (ages 1 to 19). For older teens (ages 15 to 19), the top three causes of death were accidents, homicide, and suicide.
Perhaps the lack of overlap between leading causes of death and most common online health-related searches isn’t surprising. Younger folks drive more searches and may not have heart disease, cancer, or stroke at top of mind. In addition, online searches might reflect day-to-day concerns (how soon can my child return to school after having strep throat?) rather than long-term conditions, such as heart disease or cancer. And death may not be the most immediate health outcome of interest.
But the disconnect suggests to me that we may be worrying about the wrong things — and focusing too little on the biggest health threats.
Transforming worry into action
Most of us can safely worry less about catching something from a toilet seat or shark attacks. Instead, take steps to reduce the risks you face from our biggest health threats. Chipping away at these five goals could help you live longer and better while easing unnecessary worry:
- Choose a heart-healthy diet.
- Get routinely recommended health care, including blood pressure checks and cancer screens, such as screening for colorectal cancer.
- Drive more safely. Obey the speed limit, drive defensively, always wear a seatbelt, and don’t drive if you’ve been drinking.
- Don’t smoke. If you need to quit, find help.
- Get regular exercise.
The bottom line
Try not to focus too much on health risks that are unlikely to affect you. Instead, think about common causes of poor health. Then take measures to reduce your risk. Moving more and adding healthy foods to your meals is a great start.
And in case you’re curious, the average number of annual deaths due to quicksand is zero in the US. Still a bit worried? Fine, here’s a video that shows you how to save yourself from quicksand even though you’ll almost certainly never need it.
About the Author
Robert H. Shmerling, MD, Senior Faculty Editor, Harvard Health Publishing; Editorial Advisory Board Member, Harvard Health Publishing
Dr. Robert H. Shmerling is the former clinical chief of the division of rheumatology at Beth Israel Deaconess Medical Center (BIDMC), and is a current member of the corresponding faculty in medicine at Harvard Medical School. … See Full Bio View all posts by Robert H. Shmerling, MD