A Corpus-Based Quantitative Analysis of Generative AI Outputs in Second Language Learning: An Applied Linguistics Perspective

Abstract

The advent of generative AI (GenAI) systems like large language models (LLMs) has brought a fresh and fast-changing landscape to the field of second language learning (SLL), raising critical concerns regarding the linguistic quality, pedagogical suitability, and evaluation consequences of AI-generated texts. This study is based on an original corpus of 1,240 AI-generated productions that were obtained using three prominent Generative AI (GenAI) tools: GPT-4o, Claude 3.5 Sonnet, and Gemini 1.5 Pro, and for five types of tasks typically employed in L2 English classrooms. The research compares the outputs of AI with a set of benchmark corpora of proficient L2 learner writing and native-speaker expert writing in terms of lexical density, syntactic complexity, discourse cohesion, pragmatic appropriateness and phraseological patterning using computational tools from natural language processing (NLP). The results indicate statistically significant differences in lexical sophistication and in the distribution of formulaic items, as well as in the coherence of discourse, between the texts produced by AI and authentic L2 learner output. The deviations have serious consequences for the study of instructed second language acquisition (ISLA), especially when it comes to providing writing feedback with AI, automated assessment, and the integrity of L2 learning tasks. The study provides a replicable approach to assess the outputs of GenAI and suggests an analytical approach in applied linguistics toward the use of GenAI in language education that is principled and takes into account the benefits of technological innovation and the needs to develop communicative competence.