From Unreliable to Real-Life: How Tufts Built Digital Patients with Letta

From a struggling lab exercise with derailing conversations to a realistic clinical scenario used by over 60 physical therapy students, Dr. Ben Stern transformed clinical education at Tufts University with a digital patient simulation built on Letta. His journey reveals how the right agent architecture can solve both technical and educational challenges.

The Challenge: Unreliable Simulations and Context Limitations

"People have personalities and they have other issues. They don't come in with a checklist of things they know you're going to ask, and then just read it off to you."

Dr. Ben Stern, Assistant Professor at Tufts University School of Medicine’s Doctor of Physical Therapy Program, pinpointed a fundamental issue in physical therapist education. The traditional method involved, among other approaches, students reading published patient cases and answering questions. However, this static approach failed to capture the dynamic, unpredictable nature of real clinical encounters and often left students unprepared for the complexities of actual patient interaction.

The core motivations for exploring AI-driven solutions were compelling: the need for flexibility, accessibility, and cost-effectiveness in educational tools, alongside the potential to create engaging experiences for students while reducing faculty workload. More importantly, AI offered the promise of infinite, customizable variations in patient scenarios, allowing for an extraordinary range of adjustments to meet diverse learning objectives.

Dr. Stern imagined a more interactive approach: "Wouldn't it be cool if we could just develop this stuff using software? What if I could have the students read a published case study and then actually interact with the patient who's the subject of that case study?" The goal was to use LLMs to help bridge gaps in medical and DPT education:

• The Knowledge-Practice Gap: Helping students navigate the social and environmental factors inherent in patient care. 
• Clinical Reasoning Skills: Better equipping students to make sound clinical judgments, especially in ambiguous situations. ‍
• Practical Experiences: Providing more opportunities to develop empathy, effective communication skills, and exposure to diverse patient populations and conditions.

Early Hurdles: Unreliable Simulations and Context Limitation

The idea was simple - use AI to create simulated patients that students could interview—but the implementation proved challenging. Stern initially tried ChatGPT, but quickly hit frustrating limitations.

"I would run into these issues where once the context got to a certain length, we'd get these really weird things. All of a sudden the patient - the model - would start acting like the healthcare provider," Stern recalled. These derailments occurred after only a few interactions, as each exchange was packed with extensive information from patient cases and layered with personality traits for realism, quickly exhausting the context window.

He tried building a RAG system by attaching case documents to ChatGPT and creating prompts to extract information in formats like JSON, markdown, or XML. But the system still used up the context window too quickly, and worse, it often made things up.

"I tried to set up a system where the model would respond with a citation," Stern said. "But what I was finding is it would cite stuff that wasn't real. So instead of interacting with the patient, I'm chasing down these citations—'Are you sure this is real?'"

Even attempts with other agent frameworks led to frustrating loops, with agents sometimes endlessly exchanging pleasantries: "Thank you, have a nice day. Oh, thank you, have a nice day. Thank you, have a nice day."

For a medical education lab needing to serve multiple student groups with consistent and reliable scenarios, this unpredictability was a major roadblock. Dr. Stern required a system that could reliably present realistic patient scenarios across numerous sessions without breaking character or inventing medical details.

The Turning Point: From RAG to Memory-Augmented Agents

The turning point came when Stern read about memory-augmented language models and started experimenting with early versions of Letta.

"I was reading papers and I read the MemGPT paper," Stern recalled. "What if we have a student who's working on this and they're working on my class, and then we come back the next week and we can continue working with the same patient? It remembers all the interactions with the students."

What surprised him wasn't just the memory capabilities but the reliability of the system. "The thing that struck me was the accuracy of the responses and how reliable it was. Over and over, I would get the correct information back. I was like, 'Wait a minute, I'm using the same OpenAI model, but this one's giving me the right information.'"

This reliability was crucial because Stern knew the case studies intimately, making it immediately obvious when the model fabricated information. With Letta, the information stayed accurate and consistent.

Stern's process evolved through several stages, focusing first on ensuring the system could reliably extract information from uploaded documents. He then structured this extraction around the standard requirements, the CARE Checklist for writing healthcare case reports. This ensured that important details weren’t missed and allowed for comparable information across cases. Finally, he added personality layers that could be adjusted based on educational needs.

"Faculty were talking about how when they're testing students in lab, you give them a basic case and the idea is to pull information from that case," Stern explained. "But they said, 'The problem with this is when a student goes to a clinic, that's not the way real life works. You have patients who are irritated or don't want to be there in the first place, or they're depressed and just start crying.'"

Adding Complexity: Realistic Personalities and Dynamic Scenarios

The ability to infuse digital patients with believable personality traits transformed the educational experience. The system exceeded Stern's expectations, successfully simulating patients with complex conditions, emotional states, and social situations.

"We had cases where I had a patient with a history of bipolar disorder and a patient who is schizophrenic, a patient who presents with their kid and their kid's running around all over the place, or a patient whose second language is English," Stern noted. "Time after time, even after we flushed the conversation history, we'd come back and get the same patient, same personality. It was just really good." This even extended to building a simulated rapport with the patient over time, a unique outcome of the specialized context memory.

Initially developing with Python (using the Letta Python SDK) and later switching to Letta's Agent Development Environment (ADE), Stern found that the cloud-based platform offered an unexpected advantage for classroom settings.

"The piece that I hadn't thought about that turned out to be really useful was the ability to flush all that information from the ADE and then reset everything for the next group of students," Stern said. "I could have a bunch of stations, each station had a unique patient, and each time the students rotated, they'd be going to the same patient, same everything, but all fresh, no memories, nothing."

Beyond Technical Success: Meaningful Educational Impact

For Stern, the first measure of success was simple: reliability. "Selfishly, success was, did this approach perform reliably? Regardless of the student experience, if it performed reliably, I was like, 'Yes, this is going to be awesome,' because you can build on that."

But the educational impact went beyond technical performance. "The student interaction was really good because their ability to come up with the correct diagnosis and ask the appropriate questions totally changed," Stern observed. "When they're confronted with this new dimension of a patient with a personality or a patient with a caregiver with them or a patient with a kid, all of a sudden they're distracted and they forget about the really pertinent information they're trying to dig out of this case."

This distraction wasn't a bug - it was a feature, mirroring real-world clinical challenges. "I got a few students who felt that it was too distracting. But from my perspective, that's the point. In real life, when you're with patients, that's the nature of the beast."

Other faculty responded positively to the approach, and informal student responses highlighted the value of the realistic experience. The goal was to facilitate deeper engagement through these interactive and dynamic AI patient models, ultimately transforming teaching and learning by bridging the gap between educational outcomes and industry needs.

Looking Ahead: Integrating Digital Patients Across Programs

Following the successful initial implementation with over 60 students, Dr. Stern aims to expand the use of these digital patients into Tufts' Accelerated Development of Excellence in Physical Therapy (ADEPT) Program. The ADEPT Program is a 12-week online certificate offered through the Department of Rehabilitation Sciences, designed to support students pursuing a Doctor of Physical Therapy (DPT) degree. It specifically aims to help students develop key study skills, enhance their academic preparedness.

"Our ADEPT program is for students who want to go to the accelerated DPT program but may need or want additional preparation," Stern explained. "The idea is within the ADEPT program, allowing these students to practice with that digital patient simulation would be super useful." Given that the ADEPT program is designed for students planning to apply to or enroll in a DPT program, the integration of these realistic digital patient simulations aligns perfectly with its objectives.

The simulated patients have also attracted interest from other departments, including the Physician Assistant program, suggesting a broader potential impact across medical education at Tufts.

What began as an attempt to create more dynamic patient interactions has evolved into a powerful educational tool. By immersing students in the challenging, unpredictable, and complex realities of patient encounters, complete with diverse personalities and real-life distractions, Stern and his collaborators, including Peter Nadel (natural language processing specialist at Tufts) and the team at Letta, have crafted a learning environment that better prepares future healthcare professionals for the realities of clinical practice.

Try Letta today, or request a demo for your team.