Doximity’s Response to the Nature Medicine Paper (well, kinda 😉)

Doximity’s new white paper on Doximity Ask came out today

Welcome to AI Health Uncut, a brutally honest newsletter on AI, innovation, and the state of the healthcare market. If you’d like to sign up to receive issues over email, you can do so here.

Before I begin, let me reiterate my standard disclaimer: I do not own equity, hold any financial interest, or have any business relationship with Doximity, OpenEvidence, or any other companies discussed in this article. The views expressed here are solely my own and are provided for informational and analytical purposes only. Nothing in this article should be interpreted as investment, financial, medical, or professional advice.

Let me be clear: I’m being a bit sensational with the framing here. I highly doubt Doximity’s new paper was intended as a response to the Nature Medicine study, the OpenEvidence debate, or OpenEvidence’s subsequent “twittorial” after the startling findings of “General-purpose large language models outperform specialized clinical AI tools on medical benchmarks,” which I covered in detail in my research article this morning.

That said, I do think Doximity’s paper fits squarely into the same broader conversation. So here we are: you have to endure two articles from me in one day. 🤦‍♂️

So, Doximity’s new white paper on Doximity Ask, formerly DocGPT, came out today, courtesy of Dr. Louis-Antoine Mullie, head of Doximity’s Clinical AI.

And yes, you already know what I’m going to say: it’s Doximity’s own paper, so read it with a giant grain of salt.

Especially the comparison between Doximity Ask and Claude Opus 4.7. Honestly, I didn’t see a dramatic difference there. Maybe that should be the headline. 😉

At the end of the day, I’d rather be discussing peer-reviewed papers, like the Nature Medicine study that OpenEvidence completely freaked out about, and that I covered in detail in today’s piece.

I wrote almost two years ago that specialized fine-tuned models as a business moat were in trouble. You can make the “logical” argument: “If we start with a frontier model and add clinical data, retrieval, and fine-tuning, shouldn’t the result be at least as good?”

Maybe. In theory.

But in practice, fine-tuning and “RAGing” are becoming weaker and weaker as standalone business propositions. That was obvious to me in August 2024 from my own startup work. It is painfully obvious now.

If you are OpenEvidence (or Hippocratic AI), you can keep being mad at the messenger. But the messenger is not the problem.

The point is simple: honest information needs to be out there for the benefit of the medical community. Business leaders should take criticism seriously, absorb it, and use it to make their companies and the industry better.

Not to take sides, but Doximity seems to have a more rational and frankly more adult approach to business.

1. They seem to listen to customers. Their moves into DocFax, Dialer, and now Scribe, Prescribe, and Ask look like real customer-driven iteration. I usually hate the word “pivot,” but here I mean it as a compliment.

2. They seem open to criticism. They don’t appear to freak out and start sending discrediting messages to people’s friends and coworkers, like others do. They are willing to have a conversation, including tomorrow’s interview with their Chief Clinical Experience Officer, Amit Phull, and hopefully later this summer with Jeff Tangney. (Get ready, Jeff. Tough questions are coming. I leave no prisoners. 😉) This will be part of my Substack and our podcast, “Digital Health Inside Out,” co-hosted with Alex Koshykov and available on all your favorite podcast platforms.

3. Doximity has had ups and downs in earnings and, therefore, its stock price. But unlike companies like OpenEvidence, it has actual earnings. Only 32% of ~200 publicly traded companies in my dataset can say that. Among private healthtech companies, that number is probably in the single digits.

To me, that shows business and fiscal discipline.

And now, to kill this kumbaya moment, here is a preview of the questions I’ll be asking Doximity 😊

• “The core marketing claim for Ask, echoing your own blog language, is that “unlike a large language model,” it is “grounded in peer-reviewed, clinical-quality evidence” via Pathway’s knowledge graph. But Ask is still built on an LLM, historically through Doximity’s OpenAI partnership, plus Pathway’s datasets and a retrieval layer. So architecturally, how is that fundamentally different from any other retrieval-augmented system running on top of an LLM?”

• “Let’s talk about PeerCheck, because it is the centerpiece of your trust pitch. It is co-edited by Eric Topol and former Surgeon General Regina Benjamin, and draws on “10,000+ physician authors.” But as of today, as far as I know, Doximity has published no scoring rubric, no inter-rater reliability data, no sample sizes, no specialty breakdown, no peer-reviewed white paper, and no conflict-of-interest disclosures for Topol or Benjamin. For something called “peer review,” that is a lot of missing methodology. When does that get published?”

If you have your own questions for Doximity’s leadership, send them my way. 😉

Hope everyone is having a great summer. 🏖️ But as you’re sipping that piña colada, here is your summer homework: think about what kind of framework or policy would let all of us in healthcare have open, honest, constructive debate without fear of mob-like tactics or repercussions.

---

Like what you’re reading in this newsletter? Want more in-depth investigations and research? Alright then, go tell your friends!

---

👉👉👉👉👉 Hi! My name is Sergei Polevikov. I’m an AI researcher and a healthcare AI startup founder. In my newsletter ‘AI Health Uncut,’ I combine my knowledge of AI models with my unique skills in analyzing the financial health of digital health companies. Why “Uncut”? Because I never sugarcoat or filter the hard truth. I don’t play games, I don’t work for anyone, and therefore, with your support, I produce the most original, the most unbiased, the most unapologetic research in AI, innovation, and healthcare. Thank you for your support of my work. You’re part of a vibrant community of healthcare AI enthusiasts! Your engagement matters. 🙏🙏🙏🙏🙏