Using Combinatorial Testing for Prompt Engineering of LLMs in Medicine

Alexander Perko; Iulia-Dana Nica; Franz Wotawa

doi:10.70314/is.2024.chtm.10

Using Combinatorial Testing for Prompt Engineering of LLMs in Medicine

Alexander Perko, Iulia-Dana Nica, Franz Wotawa

Institute of Software Engineering and Artificial Intelligence (7160)

Publikation: Beitrag in Buch/Bericht/Konferenzband › Beitrag in einem Konferenzband › Begutachtung

Abstract

Large Language Models (LLMs) like GPT-4o are of growing interest. Interfaces such as ChatGPT invite an ever-growing number of people to ask questions, including health advice, which brings in additional risks for harm. It is well known that tools based on LLMs tend to hallucinate or deliver different answers for the same or similar questions. In both cases, the outcome might be wrong or incomplete, possibly leading to safety issues. In this paper, we investigate the outcome of ChatGPT when we ask similar questions in the medical domain. In particular, we suggest using combinatorial testing to generate variants of questions aimed at identifying wrong or misleading answers. In detail, we discuss the general framework and its parts and present a proof-of-concept utilizing a medical query and ChatGPT.

Originalsprache	englisch
Titel	Proceedings of the 27th International Multiconference Information Society – IS 2024
Kapitel	K
Seiten	930-935
Seitenumfang	6
DOIs	https://doi.org/10.70314/is.2024.chtm.10
Publikationsstatus	Veröffentlicht - Okt. 2024
Veranstaltung	27th International Multiconference Information Society – IS 2024 - Ljubljana, Slowenien Dauer: 7 Okt. 2024 → 11 Okt. 2024

Konferenz

Konferenz	27th International Multiconference Information Society – IS 2024
Kurztitel	IS 2024
Land/Gebiet	Slowenien
Ort	Ljubljana
Zeitraum	7/10/24 → 11/10/24

ASJC Scopus subject areas

Artificial intelligence

Fields of Expertise

Information, Communication & Computing

Zugriff auf Dokument

10.70314/is.2024.chtm.10

EU - ChatMED - Brückenschlag zwischen Forschungseinrichtungen, um die Übernahme generativer KI durch den Gesundheitssektor in den Erweiterungsländern voranzutreiben
Wotawa, F. (Teilnehmer (Co-Investigator))
1/09/24 → 31/08/27
Projekt: Forschungsprojekt

Using Combinatorial Testing for Prompt Engineering of LLMs in Medicine
Perko, A. (Redner/in)
9 Okt. 2024
Aktivität: Vortrag oder Präsentation › Vortrag bei Konferenz oder Fachtagung › Science to science

Best Paper Award
Perko, A. (Empfänger/-in), Nica, I.-D. (Empfänger/-in) & Wotawa, F. (Empfänger/-in), Okt. 2024
Auszeichnung: Preise / Medaillen / Ehrungen

Dieses zitieren

@inproceedings{db8bf0486704444fa19c5d1e7f29a717,

title = "Using Combinatorial Testing for Prompt Engineering of LLMs in Medicine",

abstract = "Large Language Models (LLMs) like GPT-4o are of growing interest. Interfaces such as ChatGPT invite an ever-growing number of people to ask questions, including health advice, which brings in additional risks for harm. It is well known that tools based on LLMs tend to hallucinate or deliver different answers for the same or similar questions. In both cases, the outcome might be wrong or incomplete, possibly leading to safety issues. In this paper, we investigate the outcome of ChatGPT when we ask similar questions in the medical domain. In particular, we suggest using combinatorial testing to generate variants of questions aimed at identifying wrong or misleading answers. In detail, we discuss the general framework and its parts and present a proof-of-concept utilizing a medical query and ChatGPT.",

keywords = "Large Language Models, ChatGPT, Prompt Engineering, Combinatorial Testing, Validation",

author = "Alexander Perko and Iulia-Dana Nica and Franz Wotawa",

year = "2024",

month = oct,

doi = "10.70314/is.2024.chtm.10",

language = "English",

pages = "930--935",

booktitle = "Proceedings of the 27th International Multiconference Information Society – IS 2024",

note = "27th International Multiconference Information Society – IS 2024, IS 2024 ; Conference date: 07-10-2024 Through 11-10-2024",

}

TY - GEN

T1 - Using Combinatorial Testing for Prompt Engineering of LLMs in Medicine

AU - Perko, Alexander

AU - Nica, Iulia-Dana

AU - Wotawa, Franz

PY - 2024/10

Y1 - 2024/10

N2 - Large Language Models (LLMs) like GPT-4o are of growing interest. Interfaces such as ChatGPT invite an ever-growing number of people to ask questions, including health advice, which brings in additional risks for harm. It is well known that tools based on LLMs tend to hallucinate or deliver different answers for the same or similar questions. In both cases, the outcome might be wrong or incomplete, possibly leading to safety issues. In this paper, we investigate the outcome of ChatGPT when we ask similar questions in the medical domain. In particular, we suggest using combinatorial testing to generate variants of questions aimed at identifying wrong or misleading answers. In detail, we discuss the general framework and its parts and present a proof-of-concept utilizing a medical query and ChatGPT.

AB - Large Language Models (LLMs) like GPT-4o are of growing interest. Interfaces such as ChatGPT invite an ever-growing number of people to ask questions, including health advice, which brings in additional risks for harm. It is well known that tools based on LLMs tend to hallucinate or deliver different answers for the same or similar questions. In both cases, the outcome might be wrong or incomplete, possibly leading to safety issues. In this paper, we investigate the outcome of ChatGPT when we ask similar questions in the medical domain. In particular, we suggest using combinatorial testing to generate variants of questions aimed at identifying wrong or misleading answers. In detail, we discuss the general framework and its parts and present a proof-of-concept utilizing a medical query and ChatGPT.

KW - Large Language Models

KW - ChatGPT

KW - Prompt Engineering

KW - Combinatorial Testing

KW - Validation

U2 - 10.70314/is.2024.chtm.10

DO - 10.70314/is.2024.chtm.10

M3 - Conference paper

SP - 930

EP - 935

BT - Proceedings of the 27th International Multiconference Information Society – IS 2024

T2 - 27th International Multiconference Information Society – IS 2024

Y2 - 7 October 2024 through 11 October 2024

ER -

Using Combinatorial Testing for Prompt Engineering of LLMs in Medicine

Abstract

Konferenz

ASJC Scopus subject areas

Fields of Expertise

Zugriff auf Dokument

Fingerprint

Projekte

EU - ChatMED - Brückenschlag zwischen Forschungseinrichtungen, um die Übernahme generativer KI durch den Gesundheitssektor in den Erweiterungsländern voranzutreiben

Aktivitäten

Using Combinatorial Testing for Prompt Engineering of LLMs in Medicine

Auszeichnungen

Best Paper Award

Dieses zitieren