Addressing global challenges such as the United Nations Sustainable Development Goals (SDG) requires individuals to critically evaluate problems and generate viable solutions. However, assessing performance on such open-ended and complex tasks is often time-consuming and methodologically challenging. The rapid development of generative artificial intelligence (GAI), such as ChatGPT, offers a potential solution to this assessment challenge. In addition, critical thinking has been identified as a key factor influencing creative problem-solving. Accordingly, this study examined the role of critical thinking self-efficacy (CTSE) in solving ten online SDG-related problems among 82 college students, with ChatGPT employed as an automated scoring tool. Based on CTSE scores, participants were categorized into high- and low-CTSE groups. A mixed-design analysis of variance revealed a significant main effect of time, F(1, 80) = 20.541, p < .001, ηp² = .204, as well as a significant Time × Group interaction, F(1, 80) = 6.340, p = .014, ηp² = .073. Follow-up simple main effects analyses indicated that students in the high-CTSE group demonstrated significantly greater improvements in GAI-rated CPS performance, F(1, 40) = 22.364, p < .001, ηp² = .359. Notably, repeated trials revealed that ChatGPT was less effective in evaluating complex problem-solving responses involving large volumes of information. Accurate and consistent assessment was achieved only when responses were constrained to smaller, well-structured inputs. Collectively, these findings suggest that CT self-efficacy plays a critical role in creative problem-solving and underscore the importance of cautiously and strategically employing generative AI as an assessment tool for complex, open-ended tasks.

Authors:
Bethany C. Y. Wu, National Chengchi University, Taiwan
Rayen Jui-Yen Chang, National Chengchi University, Taiwan
Yu-Chu Yeh, National Chengchi University, Taiwan
Min-Wen Jao, National Chengchi University, Taiwan

About the Presenter(s)
With training in both social work and psychology, Dr. Bethany C. Y. Wu is dedicated to promoting public awareness of mental health issues and fostering the well-being of children and youth.

See this presentation on the full schedule – On Demand Schedule

Bookmark