Guideline Sheds Light on Best Practices for Reporting AI-Powered Health Chatbot Studies

Hamilton, ON, November 2025

A new reporting guideline, the Chatbot Assessment Reporting Tool (CHART), has been launched to standardize how studies evaluating AI-driven chatbots for health advice report their findings. With the rapid advancement of AI technology in healthcare, particularly the proliferation of large language models (LLMs) like ChatGPT, it's critical to have consistent and comprehensive reporting to ensure reliability and transparency.

With healthcare experiencing a digital revolution, AI-driven chatbots demand robust evaluation standards. To address this need, the CHART guideline was meticulously developed through a multi-phase process, including a comprehensive systematic review and modified Delphi consensus involving 531 global stakeholders. The methodology of CHART consists of 12 key items and 39 subitems crafted to enhance the detail and clarity of chatbot health advice studies. This includes essential elements such as prompt engineering details, query strategies, model specifics, and performance evaluations.

The development of CHART responds to the growing inconsistency in how chatbot health advice studies are conducted and reported. Prior to this guideline, many studies lacked transparency in critical areas, such as the development of prompts and the query strategies employed, which are fundamental to assessing the AI model's output. By involving a broad array of experts—from clinicians to AI researchers—the CHART guideline seeks to elevate the rigor and reliability of research in this burgeoning field.

Key components of CHART were curated through a systematic review followed by a Delphi consensus to ensure it covers all necessary reporting aspects. The guideline mandates clear documentation of model identifiers, descriptions of the performance evaluation process, and explicit state of both sample size and data analysis techniques. By adhering to CHART, authors can facilitate better understanding and reproduction of their studies, advancing scientific discourse in AI-related healthcare solutions.

"As a collaborative effort with significant input from various experts, CHART represents a vital step forward in ensuring that AI health solutions are developed and evaluated to the highest standards, according to the EQUATOR Network recommendations." reported Bright Huo, MD, one lead author involved in the development of the guideline. "It not only promotes transparency but also builds trust with the public and stakeholders who rely on these technologies for health advice."

Looking ahead, the introduction of CHART is expected to set a benchmark for future research, encouraging high methodological standards and open science practices. As AI continues to evolve, the framework will undergo regular updates to reflect technological advancements. This places CHART at the forefront of guiding robust and transparent health AI research which could translate into safer, more reliable applications in clinical settings.

The CHART Collaborative. (2025). Reporting Guideline for Chatbot Health Advice Studies: Chatbot Assessment Reporting Tool (CHART) Statement. Annals of Family Medicine, 23, online.

https://bit.ly/CHARTbmjmed

https://bit.ly/CHARTbjs

https://bit.ly/CHARTbmc

https://bit.ly/CHARTjamao

https://bit.ly/CHARTannfam

https://bit.ly/CHARTartmed