TY - GEN
T1 - On the Variations of ChatGPT’s Response Quality for Generating Source Code Across Programming Languages
AU - González de Diego, Ángela
AU - Wotawa, Franz
N1 - Publisher Copyright:
© IFIP International Federation for Information Processing 2025.
PY - 2025/1/25
Y1 - 2025/1/25
N2 - The rise of Large Language Models, particularly the ChatGPT model, has transformed the field of natural language information processing and has led to widespread adoption in a diverse range of applications and across a multitude of industries. In this paper, we focus on assessing the quality of the responses generated by Chat-GPT for the code generation tasks using seven different programming languages. We selected the languages considering diversity in terms of the fields of application, philosophies, and popularity. We carried out an experimental evaluation utilizing different introductory coding examples for each of the programming languages using the pass@k metric for evaluation. The results indicate a correlation between the effectiveness of the model and the popularity of programming languages.
AB - The rise of Large Language Models, particularly the ChatGPT model, has transformed the field of natural language information processing and has led to widespread adoption in a diverse range of applications and across a multitude of industries. In this paper, we focus on assessing the quality of the responses generated by Chat-GPT for the code generation tasks using seven different programming languages. We selected the languages considering diversity in terms of the fields of application, philosophies, and popularity. We carried out an experimental evaluation utilizing different introductory coding examples for each of the programming languages using the pass@k metric for evaluation. The results indicate a correlation between the effectiveness of the model and the popularity of programming languages.
KW - ChatGPT for programming
KW - Experimentally evaluating code generation using LLMs
KW - Large language models for programming
UR - http://www.scopus.com/inward/record.url?scp=85218455336&partnerID=8YFLogxK
U2 - 10.1007/978-3-031-80889-0_5
DO - 10.1007/978-3-031-80889-0_5
M3 - Conference paper
AN - SCOPUS:85218455336
SN - 9783031808883
T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
SP - 63
EP - 78
BT - Testing Software and Systems - 36th IFIP WG 6.1 International Conference, ICTSS 2024, Proceedings
A2 - Menéndez, Héctor D.
A2 - Bello-Orgaz, Gema
A2 - Barnard, Pepita
A2 - Bautista, John Robert
A2 - Farahi, Arya
A2 - Dash, Santanu
A2 - Han, DongGyun
A2 - Fortz, Sophie
A2 - Rodriguez-Fernandez, Victor
PB - Springer Science and Business Media Deutschland GmbH
T2 - 36th IFIP WG 6.1 International Conference on Testing Software and Systems, ICTSS 2024
Y2 - 30 October 2024 through 1 November 2024
ER -