INTERNAL RELIABILITY OF AUTOMATED LARGE LANGUAGE MODEL SPEAKING ASSESSMENT: A FOUR-RUN FEASIBILITY STUDY OF SPEAKAUDITOR.COM


Date Published : 7 November 2026

Contributors

Kushardiyanti Novinda

Universitas Negeri Semarang
Third Author

Hendi Pratama

Universitas Negeri Semarang
Main Author

Izzati Gemi Seinsiani

Universitas Negeri Semarang
Second Author

Stephanie Putri

Transformasi Pendidikan Indonesia
Fourth Author

Keywords

automated speaking assessment; large language model; test-retest reliability; intraclass correlation coefficient; decolonial assessment ethics

Proceeding

Track

General Track

License

Copyright (c) 2026 International Conference on Cultures & Languages

Creative Commons License

This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.

Abstract

Automated speaking assessment increasingly relies on generative language models, yet the measurement stability of such systems is rarely reported, and instruments developed elsewhere are routinely applied to speakers of English in the Expanding Circle without evidence that they behave consistently. This feasibility study examined the internal reliability of SpeakAuditor, an assessment platform scoring public speaking on nine dimensions and twenty-seven sub-variables through a pipeline combining Automatic Speech Recognition with Large Language Model scoring. One hundred English-language speech videos produced by Indonesian speakers were scored four times on a single day under identical pipeline conditions. Agreement was analysed with repeated-measures Analysis of Variance, the Intraclass Correlation Coefficient, the Standard Error of Measurement, Fleiss Kappa and weighted Cohen Kappa. The first run completed without technical fault yet differed systematically from the later three by 2.65 points, whereas those three were statistically indistinguishable from one another. A label-based explanation was tested and rejected, leaving provider-side non-stationarity as the most plausible remaining account. Reliability was therefore estimated on the three homogeneous runs, yielding an Intraclass Correlation Coefficient of 0.835 for a single run and 0.938 for the mean of three, with a Standard Error of Measurement of 3.26 points. Both sets are reported side by side: excluding the first run raises the single-measure coefficient by 0.009 while slightly worsening four other indices, so the exclusion cannot be read as improving the result. Dimension reliability ranged from 0.867 for visually anchored confidence to 0.422 for humour, and level classification proved unstable because the reporting band is narrower than the minimal detectable change. Generative scoring appears stable enough for developmental use when scores are aggregated, and transparent reliability reporting is argued to be a precondition for deploying such systems on speakers already judged against imported norms.

References

REFERENCES
Bavaresco, A., Bernardi, R., Bertolazzi, L., Elliott, D., Fernández, R., Gatt, A., ... Testoni, A. (2025). LLMs instead of human judges? A large-scale empirical study across 20 NLP evaluation tasks. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics.
Canagarajah, A. S. (1999). Interrogating the “native speaker fallacy”: Non-linguistic roots, non-pedagogical results. In G. Braine (Ed.), Non-native educators in English language teaching (pp. 77–92). Lawrence Erlbaum.
Cohen, J. (1960). A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20(1), 37–46. https://doi.org/10.1177/001316446002000104
Cook, V. (1999). Going beyond the native speaker in language teaching. TESOL Quarterly, 33(2), 185–209. https://doi.org/10.2307/3587717
Fleiss, J. L. (1971). Measuring nominal scale agreement among many raters. Psychological Bulletin, 76(5), 378–382. https://doi.org/10.1037/h0031619
Holliday, A. (2006). Native-speakerism. ELT Journal, 60(4), 385–387. https://doi.org/10.1093/elt/ccl030
Kachru, B. B. (1985). Standards, codification and sociolinguistic realism: The English language in the outer circle. In R. Quirk & H. G. Widdowson (Eds.), English in the world: Teaching and learning the language and literatures (pp. 11–30). Cambridge University Press.
Koenecke, A., Nam, A., Lake, E., Nudell, J., Quartey, M., Mengesha, Z., Toups, C., Rickford, J. R., Jurafsky, D., & Goel, S. (2020). Racial disparities in automated speech recognition. Proceedings of the National Academy of Sciences, 117(14), 7684–7689. https://doi.org/10.1073/pnas.1915768117
Koo, T. K., & Li, M. Y. (2016). A guideline of selecting and reporting intraclass correlation coefficients for reliability research. Journal of Chiropractic Medicine, 15(2), 155–163. https://doi.org/10.1016/j.jcm.2016.02.012
Phillipson, R. (1992). Linguistic imperialism. Oxford University Press.
Schreiber, L. M., Paul, G. D., & Shibley, L. R. (2012). The development and test of the public speaking competence rubric. Communication Education, 61(3), 205–233. https://doi.org/10.1080/03634523.2012.670709
Shrout, P. E., & Fleiss, J. L. (1979). Intraclass correlations: Uses in assessing rater reliability. Psychological Bulletin, 86(2), 420–428. https://doi.org/10.1037/0033-2909.86.2.420
Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., ... Stoica, I. (2023). Judging LLM-as-a-judge with MT-Bench and Chatbot Arena. Advances in Neural Information Processing Systems, 36. https://doi.org/10.48550/arXiv.2306.05685

Downloads

How to Cite

INTERNAL RELIABILITY OF AUTOMATED LARGE LANGUAGE MODEL SPEAKING ASSESSMENT: A FOUR-RUN FEASIBILITY STUDY OF SPEAKAUDITOR.COM. (2026). International Conference on Cultures & Languages, 4(1), 902-917. https://conferences.uinsaid.ac.id/iccl/paper/view/1258