Use and Application of Large Language Models for Patient Questions Following Total Knee Arthroplasty.

A consumer-focused health care model not only allows unprecedented access to information, but equally warrants consideration of the appropriateness of providing accurate patient health information. Nurses play a large role in influencing patient satisfaction following total knee arthroplasty (TKA), but they come at a cost. A specific natural language artificial intelligence (AI) model, ChatGPT (Chat Generative Pre-trained Transformer), has accumulated over 100 million users within months of launching. As such, we aimed to compare: (1) orthopaedic surgeons' evaluation of the appropriateness of the answers to the most frequently asked patient questions after TKA; and (2) patients' comfort level in answering their postoperative questions by using answers provided by arthroplasty-trained nurses and ChatGPT. We prospectively created 60 questions based on the most commonly asked patient questions following TKA. There were 3 fellowship-trained surgeons who assessed the answers provided by arthroplasty-trained nurses and ChatGPT-4 to each of the questions. The surgeons graded each set of responses based on clinical judgment as: (1) "appropriate," (2) "inappropriate" if the response contained inappropriate information, or (3) "unreliable," if the responses provided inconsistent content. Patients' comfort level and trust in AI were assessed using Research Electronic Data Capture (REDCap) hosted at our local hospital. The surgeons graded 44 out of 60 (73.3%) responses for the arthroplasty-trained nurses and 44 out of 60 (73.3%) for ChatGPT to be "appropriate." There were 4 responses graded "inappropriate" and one response graded "unreliable" provided by the nurses. For the ChatGPT response, there were 5 responses graded "inappropriate" and no responses graded "unreliable." There were 136 patients (53.8%) who were more comfortable with the answers provided by ChatGPT compared to 86 patients (34.0%) who preferred the answers from arthroplasty-trained nurses. Of the 253 patients, 233 (92.1%) were uncertain if they would trust AI to answer their postoperative questions. There were 127 patients (50.2%) who answered that if they knew the previous answer was provided by ChatGPT, their comfort level in trusting the answer would change. One potential use of ChatGPT can be found in providing appropriate answers to patient questions after TKA. At our institution, cost expenditures can potentially be minimized while maintaining patient satisfaction. Inevitably, successful implementation is dependent on the ability to provide information that is credible and in accordance with the objectives of both physicians and patients. III.

Bains SS ，Dubin JA ，Hameed D ，Sax OC ，Douglas S ，Mont MA ，Nace J ，Delanois RE ... -  《-》

Appropriateness of Frequently Asked Patient Questions Following Total Hip Arthroplasty From ChatGPT Compared to Arthroplasty-Trained Nurses.

The use of ChatGPT (Generative Pretrained Transformer), which is a natural language artificial intelligence model, has gained unparalleled attention with the accumulation of over 100 million users within months of launching. As such, we aimed to compare the following: 1) orthopaedic surgeons' evaluation of the appropriateness of the answers to the most frequently asked patient questions after total hip arthroplasty; and 2) patients' evaluation of ChatGPT and arthroplasty-trained nurses responses to answer their postoperative questions. We prospectively created 60 questions to address the most commonly asked patient questions following total hip arthroplasty. We obtained answers from arthroplasty-trained nurses and from the ChatGPT-3.5 version for each of the questions. Surgeons graded each set of responses based on clinical judgment as 1) "appropriate," 2) "inappropriate" if the response contained inappropriate information, or 3) "unreliable" if the responses provided inconsistent content. Each patient was given a randomly selected question from the 60 aforementioned questions, with responses provided by ChatGPT and arthroplasty-trained nurses, using a Research Electronic Data Capture survey hosted at our local hospital. The 3 fellowship-trained surgeons graded 56 out of 60 (93.3%) responses for the arthroplasty-trained nurses and 57 out of 60 (95.0%) for ChatGPT to be "appropriate." There were 175 out of 252 (69.4%) patients who were more comfortable following the ChatGPT responses and 77 out of 252 (30.6%) who preferred arthroplasty-trained nurses' responses. However, 199 out of 252 patients (79.0%) responded that they were "uncertain" with regard to trusting AI to answer their postoperative questions. ChatGPT provided appropriate answers from a physician perspective. Patients were also more comfortable with the ChatGPT responses than those from arthroplasty-trained nurses. Inevitably, its successful implementation is dependent on its ability to provide credible information that is consistent with the goals of the physician and patient alike.

Dubin JA ，Bains SS ，DeRogatis MJ ，Moore MC ，Hameed D ，Mont MA ，Nace J ，Delanois RE ... -  《-》

Assessing Ability for ChatGPT to Answer Total Knee Arthroplasty-Related Questions.

Artificial intelligence in the field of orthopaedics has been a topic of increasing interest and opportunity in recent years. Its applications are widespread both for physicians and patients, including use in clinical decision-making, in the operating room, and in research. In this study, we aimed to assess the quality of ChatGPT answers when asked questions related to total knee arthroplasty. ChatGPT prompts were created by turning 15 of the American Academy of Orthopaedic Surgeons Clinical Practice Guidelines into questions. An online survey was created, which included screenshots of each prompt and answers to the 15 questions. Surgeons were asked to grade ChatGPT answers from 1 to 5 based on their characteristics: (1) relevance, (2) accuracy, (3) clarity, (4) completeness, (5) evidence-based, and (6) consistency. There were 11 Adult Joint Reconstruction fellowship-trained surgeons who completed the survey. Questions were subclassified based on the subject of the prompt: (1) risk factors, (2) implant/intraoperative, and (3) pain/functional outcomes. The average and standard deviation for all answers, as well as for each subgroup, were calculated. Inter-rater reliability (IRR) was also calculated. All answer characteristics were graded as being above average (ie, a score > 3). Relevance demonstrated the highest scores (4.43 ± 0.77) by surgeons surveyed, and consistency demonstrated the lowest scores (3.54 ± 1.10). ChatGPT prompts in the Risk Factors group demonstrated the best responses, while those in the Pain/Functional Outcome group demonstrated the lowest. The overall IRR was found to be 0.33 (poor reliability), with the highest IRR for relevance (0.43) and the lowest for evidence-based (0.28). ChatGPT can answer questions regarding well-established clinical guidelines in total knee arthroplasty with above-average accuracy but demonstrates variable reliability. This investigation is the first step in understanding large language model artificial intelligence like ChatGPT and how well they perform in the field of arthroplasty.

Magruder ML ，Rodriguez AN ，Wong JCJ ，Erez O ，Piuzzi NS ，Scuderi GR ，Slover JD ，Oh JH ，Schwarzkopf R ，Chen AF ，Iorio R ，Goodman SB ，Mont MA ... -  《-》

Using a Google Web Search Analysis to Assess the Utility of ChatGPT in Total Joint Arthroplasty.

Rapid technological advancements have laid the foundations for the use of artificial intelligence in medicine. The promise of machine learning (ML) lies in its potential ability to improve treatment decision making, predict adverse outcomes, and streamline the management of perioperative healthcare. In an increasing consumer-focused health care model, unprecedented access to information may extend to patients using ChatGPT to gain insight into medical questions. The main objective of our study was to replicate a patient's internet search in order to assess the appropriateness of ChatGPT, a novel machine learning tool released in 2022 that provides dialogue responses to queries, in comparison to Google Web Search, the most widely used search engine in the United States today, as a resource for patients for online health information. For the 2 different search engines, we compared i) the most frequently asked questions (FAQs) associated with total knee arthroplasty (TKA) and total hip arthroplasty (THA) by question type and topic; ii) the answers to the most frequently asked questions; as well as iii) the FAQs yielding a numerical response. A Google web search was performed with the following search terms: "total knee replacement" and "total hip replacement." These terms were individually entered and the first 10 FAQs were extracted along with the source of the associated website for each question. The following statements were inputted into ChatGPT: 1) "Perform a google search with the search term 'total knee replacement' and record the 10 most FAQs related to the search term" as well as 2) "Perform a google search with the search term 'total hip replacement' and record the 10 most FAQs related to the search term." A Google web search was repeated with the same search terms to identify the first 10 FAQs that included a numerical response for both "total knee replacement" and "total hip replacement." These questions were then inputted into ChatGPT and the questions and answers were recorded. There were 5 of 20 (25%) questions that were similar when performing a Google web search and a search of ChatGPT for all search terms. Of the 20 questions asked for the Google Web Search, 13 of 20 were provided by commercial websites. For ChatGPT, 15 of 20 (75%) questions were answered by government websites, with the most frequent one being PubMed. In terms of numerical questions, 11 of 20 (55%) of the most FAQs provided different responses between a Google web search and ChatGPT. A comparison of the FAQs by a Google web search with attempted replication by ChatGPT revealed heterogenous questions and responses for open and discrete questions. ChatGPT should remain a trending use as a potential resource to patients that needs further corroboration until its ability to provide credible information is verified and concordant with the goals of the physician and the patient alike.

Dubin JA ，Bains SS ，Chen Z ，Hameed D ，Nace J ，Mont MA ，Delanois RE ... -  《-》

Do ChatGPT and Google differ in answers to commonly asked patient questions regarding total shoulder and total elbow arthroplasty?

Tharakan S ，Klein B ，Bartlett L ，Atlas A ，Parada SA ，Cohn RM ... -  《-》