GUILDERLAND — Daniel Joseph, a Guilderland High School student, created an app simulating a doctor or nurse’s real-life interactions with patients.

He explains in a video that he was inspired to create the application when his sister, a nursing student, was frustrated by an assignment.

“The problem was that it didn’t reflect what actually happens in nurse-patient interactions,” says Joseph. “That’s when I realized something very important: Most medical training software is static. There’s no real conversation or interaction. So I wanted to change that.”

His medical training application was chosen as the Congressional App Challenge winner for New York’s District 20. Congressman Paul Tonko visited Guilderland High school on Thursday, Jan. 29, to present Joseph with a Congressional Certificate of Achievement and a letter of recognition.

“With his innovative MedMate app, Daniel Joseph demonstrated not only his excellent coding skills, but also his admirable commitment to helping our health care workers deliver high-quality care to every patient,” Tonko said in a statement. “I’m proud to welcome Daniel into the storied ranks of Capital Region innovators, and I’m grateful to him for his hard work.”

MedMate was chosen as the winner by a panel of local judges including Dr. Mohammed J. Zaki, professor and department head of Computer Science at Rensselaer Polytechnic Institute; Annmarie Lanesey, chief executive officer and founder of CanCode Communities; and Tobi Saulnier, chief executive officer of 1st Playable Productions.

“I come from a family full of medical professionals such as doctors and nurses,” said Joseph in his video. His mother, sister, grandmother, and aunts all worked as nurses.

Joseph explained how his MedMate app works, letting users talk to virtual patents, view charts, and experience hospital-like scenarios.

“It’s designed to be dynamic rather than robotic like most patient simulations are,” he says.

When a user speaks to the app, OpenAI’s Whisper model converts their speech to text, which is then processed by OpenAI’s language model, creating a patient response. An AI avatar is animated, with the likeness of a patient that can talk back in real time.

OpenAI is open-source artificial intelligence, meaning it uses a collaborative development model where the software's source code is publicly accessible rather than being proprietary with restricted code access.

Joseph’s app has three modes: a practice mode to build confidence, a challenge mode to test expertise and experience clinical decision-making, and a quiz mode to create personalized study sets.

Joseph, in his video, uses the practice mode to demonstrate the process. A patient-intake form allows the user to ask questions, run a lab, or make a diagnosis.

When he sees the chief complaint is abdominal pain, Joseph asks a question: “When did the pain start and how bad is it?”

A girl’s face appears on the screen and “she” replies: “The pain started a couple of days ago. At first it felt like cramps, but now it’s a constant ache that’s getting worse.”

He orders an MRI scan. “It looks like there’s something wrong with the appendix,” says Joseph, guessing the ailment is appendicitis.

“After I get it right, it gives a performance evaluation. And OpenAI’s API actually gives me an analysis of how well I did. And now the score goes into my stats menu.”

Joseph concludes his video by displaying pages of the code he used to create the MedMate app that includes D-iDAPI, OpenAI API, and OpenAI’s Whisper model.

The Congressional App Challenge was created by Congress in 2013 to encourage innovation and engagement in coding, computer science, and STEM (science, technology, engineering, math) education.