On the Variations of ChatGPT’s Response Quality for Generating Source Code Across Programming Languages

Ángela González de Diego; Franz Wotawa

doi:10.1007/978-3-031-80889-0_5

On the Variations of ChatGPT’s Response Quality for Generating Source Code Across Programming Languages

Ángela González de Diego
, Franz Wotawa^*

^*Corresponding author for this work

Institute of Software Engineering and Artificial Intelligence (7160)

Research output: Chapter in Book/Report/Conference proceeding › Conference paper › peer-review

Abstract

The rise of Large Language Models, particularly the ChatGPT model, has transformed the field of natural language information processing and has led to widespread adoption in a diverse range of applications and across a multitude of industries. In this paper, we focus on assessing the quality of the responses generated by Chat-GPT for the code generation tasks using seven different programming languages. We selected the languages considering diversity in terms of the fields of application, philosophies, and popularity. We carried out an experimental evaluation utilizing different introductory coding examples for each of the programming languages using the pass@k metric for evaluation. The results indicate a correlation between the effectiveness of the model and the popularity of programming languages.

Original language	English
Title of host publication	Testing Software and Systems - 36th IFIP WG 6.1 International Conference, ICTSS 2024, Proceedings
Editors	Héctor D. Menéndez, Gema Bello-Orgaz, Pepita Barnard, John Robert Bautista, Arya Farahi, Santanu Dash, DongGyun Han, Sophie Fortz, Victor Rodriguez-Fernandez
Publisher	Springer Science and Business Media Deutschland GmbH
Pages	63-78
Number of pages	16
ISBN (Print)	9783031808883
DOIs	https://doi.org/10.1007/978-3-031-80889-0_5
Publication status	Published - 25 Jan 2025
Event	36th IFIP WG 6.1 International Conference on Testing Software and Systems, ICTSS 2024 - London, United Kingdom Duration: 30 Oct 2024 → 1 Nov 2024

Publication series

Name	Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume	15383 LNCS
ISSN (Print)	0302-9743
ISSN (Electronic)	1611-3349

Conference

Conference	36th IFIP WG 6.1 International Conference on Testing Software and Systems, ICTSS 2024
Country/Territory	United Kingdom
City	London
Period	30/10/24 → 1/11/24

Keywords

ChatGPT for programming
Experimentally evaluating code generation using LLMs
Large language models for programming

ASJC Scopus subject areas

Theoretical Computer Science
General Computer Science

Access to Document

10.1007/978-3-031-80889-0_5

Cite this

González de Diego, Á., & Wotawa, F. (2025). On the Variations of ChatGPT’s Response Quality for Generating Source Code Across Programming Languages. In H. D. Menéndez, G. Bello-Orgaz, P. Barnard, J. R. Bautista, A. Farahi, S. Dash, D. Han, S. Fortz, & V. Rodriguez-Fernandez (Eds.), Testing Software and Systems - 36th IFIP WG 6.1 International Conference, ICTSS 2024, Proceedings (pp. 63-78). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 15383 LNCS). Springer Science and Business Media Deutschland GmbH. https://doi.org/10.1007/978-3-031-80889-0_5

On the Variations of ChatGPT’s Response Quality for Generating Source Code Across Programming Languages. / González de Diego, Ángela; Wotawa, Franz.
Testing Software and Systems - 36th IFIP WG 6.1 International Conference, ICTSS 2024, Proceedings. ed. / Héctor D. Menéndez; Gema Bello-Orgaz; Pepita Barnard; John Robert Bautista; Arya Farahi; Santanu Dash; DongGyun Han; Sophie Fortz; Victor Rodriguez-Fernandez. Springer Science and Business Media Deutschland GmbH, 2025. p. 63-78 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 15383 LNCS).

Research output: Chapter in Book/Report/Conference proceeding › Conference paper › peer-review

González de Diego, Á & Wotawa, F 2025, On the Variations of ChatGPT’s Response Quality for Generating Source Code Across Programming Languages. in HD Menéndez, G Bello-Orgaz, P Barnard, JR Bautista, A Farahi, S Dash, D Han, S Fortz & V Rodriguez-Fernandez (eds), Testing Software and Systems - 36th IFIP WG 6.1 International Conference, ICTSS 2024, Proceedings. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 15383 LNCS, Springer Science and Business Media Deutschland GmbH, pp. 63-78, 36th IFIP WG 6.1 International Conference on Testing Software and Systems, ICTSS 2024, London, United Kingdom, 30/10/24. https://doi.org/10.1007/978-3-031-80889-0_5

González de Diego Á, Wotawa F. On the Variations of ChatGPT’s Response Quality for Generating Source Code Across Programming Languages. In Menéndez HD, Bello-Orgaz G, Barnard P, Bautista JR, Farahi A, Dash S, Han D, Fortz S, Rodriguez-Fernandez V, editors, Testing Software and Systems - 36th IFIP WG 6.1 International Conference, ICTSS 2024, Proceedings. Springer Science and Business Media Deutschland GmbH. 2025. p. 63-78. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)). doi: 10.1007/978-3-031-80889-0_5

González de Diego, Ángela ; Wotawa, Franz. / On the Variations of ChatGPT’s Response Quality for Generating Source Code Across Programming Languages. Testing Software and Systems - 36th IFIP WG 6.1 International Conference, ICTSS 2024, Proceedings. editor / Héctor D. Menéndez ; Gema Bello-Orgaz ; Pepita Barnard ; John Robert Bautista ; Arya Farahi ; Santanu Dash ; DongGyun Han ; Sophie Fortz ; Victor Rodriguez-Fernandez. Springer Science and Business Media Deutschland GmbH, 2025. pp. 63-78 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)).

@inproceedings{0750e553acf9444583750cf99b5d962e,

title = "On the Variations of ChatGPT{\textquoteright}s Response Quality for Generating Source Code Across Programming Languages",

abstract = "The rise of Large Language Models, particularly the ChatGPT model, has transformed the field of natural language information processing and has led to widespread adoption in a diverse range of applications and across a multitude of industries. In this paper, we focus on assessing the quality of the responses generated by Chat-GPT for the code generation tasks using seven different programming languages. We selected the languages considering diversity in terms of the fields of application, philosophies, and popularity. We carried out an experimental evaluation utilizing different introductory coding examples for each of the programming languages using the pass@k metric for evaluation. The results indicate a correlation between the effectiveness of the model and the popularity of programming languages.",

keywords = "ChatGPT for programming, Experimentally evaluating code generation using LLMs, Large language models for programming",

author = "\{Gonz{\'a}lez de Diego\}, {\'A}ngela and Franz Wotawa",

note = "Publisher Copyright: {\textcopyright} IFIP International Federation for Information Processing 2025.; 36th IFIP WG 6.1 International Conference on Testing Software and Systems, ICTSS 2024 ; Conference date: 30-10-2024 Through 01-11-2024",

year = "2025",

month = jan,

day = "25",

doi = "10.1007/978-3-031-80889-0\_5",

language = "English",

isbn = "9783031808883",

series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",

publisher = "Springer Science and Business Media Deutschland GmbH",

pages = "63--78",

editor = "Men{\'e}ndez, \{H{\'e}ctor D.\} and Gema Bello-Orgaz and Pepita Barnard and Bautista, \{John Robert\} and Arya Farahi and Santanu Dash and DongGyun Han and Sophie Fortz and Victor Rodriguez-Fernandez",

booktitle = "Testing Software and Systems - 36th IFIP WG 6.1 International Conference, ICTSS 2024, Proceedings",

address = "Germany",

}

TY - GEN

T1 - On the Variations of ChatGPT’s Response Quality for Generating Source Code Across Programming Languages

AU - González de Diego, Ángela

AU - Wotawa, Franz

N1 - Publisher Copyright: © IFIP International Federation for Information Processing 2025.

PY - 2025/1/25

Y1 - 2025/1/25

N2 - The rise of Large Language Models, particularly the ChatGPT model, has transformed the field of natural language information processing and has led to widespread adoption in a diverse range of applications and across a multitude of industries. In this paper, we focus on assessing the quality of the responses generated by Chat-GPT for the code generation tasks using seven different programming languages. We selected the languages considering diversity in terms of the fields of application, philosophies, and popularity. We carried out an experimental evaluation utilizing different introductory coding examples for each of the programming languages using the pass@k metric for evaluation. The results indicate a correlation between the effectiveness of the model and the popularity of programming languages.

AB - The rise of Large Language Models, particularly the ChatGPT model, has transformed the field of natural language information processing and has led to widespread adoption in a diverse range of applications and across a multitude of industries. In this paper, we focus on assessing the quality of the responses generated by Chat-GPT for the code generation tasks using seven different programming languages. We selected the languages considering diversity in terms of the fields of application, philosophies, and popularity. We carried out an experimental evaluation utilizing different introductory coding examples for each of the programming languages using the pass@k metric for evaluation. The results indicate a correlation between the effectiveness of the model and the popularity of programming languages.

KW - ChatGPT for programming

KW - Experimentally evaluating code generation using LLMs

KW - Large language models for programming

UR - https://www.scopus.com/pages/publications/85218455336

U2 - 10.1007/978-3-031-80889-0_5

DO - 10.1007/978-3-031-80889-0_5

M3 - Conference paper

AN - SCOPUS:85218455336

SN - 9783031808883

T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

SP - 63

EP - 78

BT - Testing Software and Systems - 36th IFIP WG 6.1 International Conference, ICTSS 2024, Proceedings

A2 - Menéndez, Héctor D.

A2 - Bello-Orgaz, Gema

A2 - Barnard, Pepita

A2 - Bautista, John Robert

A2 - Farahi, Arya

A2 - Dash, Santanu

A2 - Han, DongGyun

A2 - Fortz, Sophie

A2 - Rodriguez-Fernandez, Victor

PB - Springer Science and Business Media Deutschland GmbH

T2 - 36th IFIP WG 6.1 International Conference on Testing Software and Systems, ICTSS 2024

Y2 - 30 October 2024 through 1 November 2024

ER -