INTERNAL RELIABILITY OF AUTOMATED LARGE LANGUAGE MODEL SPEAKING ASSESSMENT: A FOUR-RUN FEASIBILITY STUDY OF SPEAKAUDITOR.COM
Contributors
Kushardiyanti Novinda
Hendi Pratama
Izzati Gemi Seinsiani
Stephanie Putri
Keywords
Proceeding
Track
General Track
License
Copyright (c) 2026 International Conference on Cultures & Languages

This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.
Abstract
Automated speaking assessment increasingly relies on generative language models, yet the measurement stability of such systems is rarely reported, and instruments developed elsewhere are routinely applied to speakers of English in the Expanding Circle without evidence that they behave consistently. This feasibility study examined the internal reliability of SpeakAuditor, an assessment platform scoring public speaking on nine dimensions and twenty-seven sub-variables through a pipeline combining Automatic Speech Recognition with Large Language Model scoring. One hundred English-language speech videos produced by Indonesian speakers were scored four times on a single day under identical pipeline conditions. Agreement was analysed with repeated-measures Analysis of Variance, the Intraclass Correlation Coefficient, the Standard Error of Measurement, Fleiss Kappa and weighted Cohen Kappa. The first run completed without technical fault yet differed systematically from the later three by 2.65 points, whereas those three were statistically indistinguishable from one another. A label-based explanation was tested and rejected, leaving provider-side non-stationarity as the most plausible remaining account. Reliability was therefore estimated on the three homogeneous runs, yielding an Intraclass Correlation Coefficient of 0.835 for a single run and 0.938 for the mean of three, with a Standard Error of Measurement of 3.26 points. Both sets are reported side by side: excluding the first run raises the single-measure coefficient by 0.009 while slightly worsening four other indices, so the exclusion cannot be read as improving the result. Dimension reliability ranged from 0.867 for visually anchored confidence to 0.422 for humour, and level classification proved unstable because the reporting band is narrower than the minimal detectable change. Generative scoring appears stable enough for developmental use when scores are aggregated, and transparent reliability reporting is argued to be a precondition for deploying such systems on speakers already judged against imported norms.