CreAItive Collaboration?: Users' Misjudgment of AI-Creativity Affects Their Collaborative Performance

Mia Magdalena Bangerl; Leonie Disch; Tamara David; Viktoria Pammer-Schindler

doi:10.1145/3706598.3713886

CreAItive Collaboration? Users' Misjudgment of AI-Creativity Affects Their Collaborative Performance

Mia Magdalena Bangerl
, Leonie Disch
, Tamara David
, Viktoria Pammer-Schindler

Research output: Chapter in Book/Report/Conference proceeding › Conference paper › peer-review

Abstract

How does generative AI affect collaborative creative work and humans' capability to carry it out? We tested 52 participant pairs in a standard creativity test, the Alternate Uses Test. The experimental AI group had access to ChatGPT-4, while the control group did not. The intervention did not lead to an improved performance overall. Further, the AI group elaborated their ideas significantly less. This effect carried over to the unaided post-test, pointing to longer-term effects of AI be(com)ing everyday technology, as how people perform a task with a tool shapes how they (learn to) perform the task without it. Analysis of the human-AI collaboration process revealed that participants were selective in using ChatGPT-4 output for the experimental task, misjudging and falsely assessing its output. This actually reduced their number of created ideas and underscores that users need to understand a (generative AI-based) tool's capability for the specific task to support effective performance.

Translated title of the contribution	Kreative Kollaboration?: Die Fehleinschätzung von KI-Kreativität durch User:innen beeinträchtigt ihre kollaborative Leistung.
Original language	English
Title of host publication	CHI 2025 - Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems
Place of Publication	New York
Publisher	Association for Computing Machinery (ACM)
Number of pages	17
ISBN (Electronic)	979-8-4007-1394-1
DOIs	https://doi.org/10.1145/3706598.3713886
Publication status	Published - 26 Apr 2025
Event	ACM CHI conference on Human Factors in Computing Systems - Pacifico Yokohama, Yokohama, Japan Duration: 26 Apr 2025 → 1 May 2025 https://chi2025.acm.org/

Publication series

Name	Conference on Human Factors in Computing Systems - Proceedings

Conference

Conference	ACM CHI conference on Human Factors in Computing Systems
Abbreviated title	CHI'25
Country/Territory	Japan
City	Yokohama
Period	26/04/25 → 1/05/25
Internet address	https://chi2025.acm.org/

Keywords

Alternate Uses Test
ChatGPT
collaboration
creativity
experiment
generative AI
learning

ASJC Scopus subject areas

Human-Computer Interaction
Computer Graphics and Computer-Aided Design
Software

Fields of Expertise

Information, Communication & Computing

Access to Document

10.1145/3706598.3713886Licence: CC BY 4.0

3706598.3713886Final published version, 3.71 MBLicence: CC BY 4.0

Cite this

Bangerl, M. M., Disch, L., David, T., & Pammer-Schindler, V. (2025). CreAItive Collaboration? Users' Misjudgment of AI-Creativity Affects Their Collaborative Performance. In CHI 2025 - Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems Article 195 (Conference on Human Factors in Computing Systems - Proceedings). Association for Computing Machinery (ACM). https://doi.org/10.1145/3706598.3713886

CreAItive Collaboration? Users' Misjudgment of AI-Creativity Affects Their Collaborative Performance. / Bangerl, Mia Magdalena; Disch, Leonie; David, Tamara et al.
CHI 2025 - Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems. New York: Association for Computing Machinery (ACM), 2025. 195 (Conference on Human Factors in Computing Systems - Proceedings).

Research output: Chapter in Book/Report/Conference proceeding › Conference paper › peer-review

Bangerl, MM, Disch, L, David, T & Pammer-Schindler, V 2025, CreAItive Collaboration? Users' Misjudgment of AI-Creativity Affects Their Collaborative Performance. in CHI 2025 - Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems., 195, Conference on Human Factors in Computing Systems - Proceedings, Association for Computing Machinery (ACM), New York, ACM CHI conference on Human Factors in Computing Systems, Yokohama, Japan, 26/04/25. https://doi.org/10.1145/3706598.3713886

Bangerl MM, Disch L, David T, Pammer-Schindler V. CreAItive Collaboration? Users' Misjudgment of AI-Creativity Affects Their Collaborative Performance. In CHI 2025 - Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems. New York: Association for Computing Machinery (ACM). 2025. 195. (Conference on Human Factors in Computing Systems - Proceedings). doi: 10.1145/3706598.3713886

Bangerl, Mia Magdalena ; Disch, Leonie ; David, Tamara et al. / CreAItive Collaboration? Users' Misjudgment of AI-Creativity Affects Their Collaborative Performance. CHI 2025 - Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems. New York : Association for Computing Machinery (ACM), 2025. (Conference on Human Factors in Computing Systems - Proceedings).

@inproceedings{2bfb87dd3b5b4be1924a0759fdcb1fcf,

title = "CreAItive Collaboration?: Users' Misjudgment of AI-Creativity Affects Their Collaborative Performance",

abstract = "How does generative AI affect collaborative creative work and humans' capability to carry it out? We tested 52 participant pairs in a standard creativity test, the Alternate Uses Test. The experimental AI group had access to ChatGPT-4, while the control group did not. The intervention did not lead to an improved performance overall. Further, the AI group elaborated their ideas significantly less. This effect carried over to the unaided post-test, pointing to longer-term effects of AI be(com)ing everyday technology, as how people perform a task with a tool shapes how they (learn to) perform the task without it. Analysis of the human-AI collaboration process revealed that participants were selective in using ChatGPT-4 output for the experimental task, misjudging and falsely assessing its output. This actually reduced their number of created ideas and underscores that users need to understand a (generative AI-based) tool's capability for the specific task to support effective performance.",

keywords = "Mensch-Maschine-Interaktion, Large Language Model, Kreativit{\"a}t, Alternate Uses Test, ChatGPT, collaboration, creativity, experiment, generative AI, learning",

author = "Bangerl, \{Mia Magdalena\} and Leonie Disch and Tamara David and Viktoria Pammer-Schindler",

year = "2025",

month = apr,

day = "26",

doi = "10.1145/3706598.3713886",

language = "English",

series = "Conference on Human Factors in Computing Systems - Proceedings",

publisher = "Association for Computing Machinery (ACM)",

booktitle = "CHI 2025 - Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems",

address = "United States",

note = "ACM CHI conference on Human Factors in Computing Systems, CHI'25 ; Conference date: 26-04-2025 Through 01-05-2025",

url = "https://chi2025.acm.org/",

}

TY - GEN

T1 - CreAItive Collaboration?

T2 - ACM CHI conference on Human Factors in Computing Systems

AU - Bangerl, Mia Magdalena

AU - Disch, Leonie

AU - David, Tamara

AU - Pammer-Schindler, Viktoria

PY - 2025/4/26

Y1 - 2025/4/26

N2 - How does generative AI affect collaborative creative work and humans' capability to carry it out? We tested 52 participant pairs in a standard creativity test, the Alternate Uses Test. The experimental AI group had access to ChatGPT-4, while the control group did not. The intervention did not lead to an improved performance overall. Further, the AI group elaborated their ideas significantly less. This effect carried over to the unaided post-test, pointing to longer-term effects of AI be(com)ing everyday technology, as how people perform a task with a tool shapes how they (learn to) perform the task without it. Analysis of the human-AI collaboration process revealed that participants were selective in using ChatGPT-4 output for the experimental task, misjudging and falsely assessing its output. This actually reduced their number of created ideas and underscores that users need to understand a (generative AI-based) tool's capability for the specific task to support effective performance.

AB - How does generative AI affect collaborative creative work and humans' capability to carry it out? We tested 52 participant pairs in a standard creativity test, the Alternate Uses Test. The experimental AI group had access to ChatGPT-4, while the control group did not. The intervention did not lead to an improved performance overall. Further, the AI group elaborated their ideas significantly less. This effect carried over to the unaided post-test, pointing to longer-term effects of AI be(com)ing everyday technology, as how people perform a task with a tool shapes how they (learn to) perform the task without it. Analysis of the human-AI collaboration process revealed that participants were selective in using ChatGPT-4 output for the experimental task, misjudging and falsely assessing its output. This actually reduced their number of created ideas and underscores that users need to understand a (generative AI-based) tool's capability for the specific task to support effective performance.

KW - Mensch-Maschine-Interaktion

KW - Large Language Model

KW - Kreativität

KW - Alternate Uses Test

KW - ChatGPT

KW - collaboration

KW - creativity

KW - experiment

KW - generative AI

KW - learning

UR - https://www.scopus.com/pages/publications/105005737696

U2 - 10.1145/3706598.3713886

DO - 10.1145/3706598.3713886

M3 - Conference paper

T3 - Conference on Human Factors in Computing Systems - Proceedings

BT - CHI 2025 - Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems

PB - Association for Computing Machinery (ACM)

CY - New York

Y2 - 26 April 2025 through 1 May 2025

ER -