Language Teachers’ Assessment Literacy in AI-aided Adaptive Learning Environments

Document Type : Research Article

Author

Karlsruhe Institute of Technology, Karlsruhe, Germany

Abstract

AI-aided Language Assessment Literacy (LAL), is a pivotal aspect of modern teaching, posing both an indispensable asset and a formidable challenge. Despite its paramount importance, research on teachers’ LAL, especially in the context of AI-aided tools, has been notably scarce. This paper addresses the research gap by conducting a practical investigation, emphasizing the application of LAL concepts with a special focus on integrating Artificial Intelligence in language assessment. For this purpose, 261 EFL teachers of three groups (i.e. novice teachers with one to three years of experience, senior teachers with three to five years of experience, and experienced teachers with more than five years of experience) participated in the research. Then the process of pre-test, 50-hour teacher training course, 16-session teaching in Magic School or Edapp environment, post-test, and interview were conducted. The reasons for choosing these platforms were that they offer Adaptive Learning Environments (ALE), are integrated with AI and accessible, and have a simple user interface. The mixed methods analyses of the experiment data showed that: AI-aided ALE training has led to the development of teachers’ LAL; regarding teaching quality, the teachers’ performance decreased from G3 to G2 and finally G1 respectively; while in the case of the LAL variable, G2 outperformed the others.  

Keywords


Akgun, S., & Greenhow, C. (2022). Artificial intelligence in education: Addressing ethical challenges in K‑12 settings. AI and Ethics, 2, 431-440 https://doi.org/10.1007/s43681-021-00096-7
Andujar, A., & Spratt, M. (2023). Using AI to support CLIL teacher language. Journal of Research in Applied Linguistics, 14(2), 7-19. https://doi.org/10.22055/rals.2023.45267.3177 
Assunção Flores, M. (2016). Teacher education curriculum. In J. Loughran, & M. L. Hamilton (Eds). International Handbook of Teacher Education (Vol. 1, pp. 187-231). Springer.
Baker, B. A., & Riches, C. (2018). The development of EFL examinations in Haiti: Collaboration and language assessment literacy development. Language Testing, 35(4), 557-581. https://doi.org/10.1177/02655322177167
Bannister, P. (2024). English medium instruction educator language assessment literacy and the test of generative AI in online higher education. Journal of Research in Applied Linguistics, 15(2), 55-72. https://doi.org/10.22055/rals.2024.45862.3214
Barnes-Hawkins, C. (2016). English language learners’ perspectives of the communicative language approach. [Unpublished Doctoral Dissertation]. Walden University, Minnesota, US.
Berry, V., Sheehan, S., & Munro, S. (2017). Exploring teachers’ language assessment literacy: A social constructivist approach to understanding effective practices. In E. G. Eugenio (Ed.), Learning and assessment: Making the connections (pp. 201-207). Cambridge.
Borge, N. (2016). White paper – Artificial Intelligence to Improve Education/Learning Challenges. International Journal of Advanced Engineering & Innovative Technology, 2(6), 10-13. https://doi.org/10.32595/iirjet.org/v2i2.2016.29 
Christodoulou, A., & Angeli, C. (2022). Adaptive learning techniques for a personalized educational software in developing teachers’ technological pedagogical content knowledge. Frontiers in Education, 7, 789397. https://doi.org/10.3389/feduc.2022.789397
Coombe, C., Vafadar, H., & Mohebbi, H. (2020). Language assessment literacy: What do we need to learn, unlearn, and relearn? Language Testing in Asia, 10(3), 1-16. https://doi.org/10.1186/s40468-020-00101-6
 Delgado, J. Z., & Rodriguez, C. (2022). Language assessment literacy of language teachers in the context of adult education in Spain. Studies in Language Assessment, 11(1), 64-91. https://doi.org/10.58379/RRDV2344
Demartini, C.G., Sciascia, L., Bosso, A., & Manuri, F. (2024). Artificial intelligence bringing improvements to adaptive learning in education: A case study. Sustainability, 16, 1347. https://doi.org/10.3390/su16031347
Díez-Arcón, P., & Martin-Monje, E. (2023). Language Teacher development in computer-mediated collaborative work and digital peer assessment: An innovative proposal. Journal of Research in Applied Linguistics, 4(2), 40-54. https://doi.org/10.22055/RALS.2023.44054.3080
Firoozi, T., Razavipour, K., & Ahmadi, A. (2019). The language assessment literacy needs of Iranian EFL teachers with a focus on reformed assessment policies. Language Testing in Asia, 9(2), 1-14. https://doi.org/10.1186/s40468-019-0078-7 
Fulcher, G. (2012). Assessment literacy for the language classroom. Language Assessment Quarterly, 9(2), 113-132. https://doi.org/10.1080/15434303.2011.642041
Gore, J., Rosser, B., Jaremus, F., Miller, A., & Harris, J. (2023). Fresh evidence on the relationship between years of experience and teaching quality. The Australian Educational Researcher, 51(2), 547-570. https://doi.org/10.1007/s13384-023-00612-0
Graham, L. J., White, S. L. J., Cologon, K., & Pianta, R. C. (2020). Do teachers’ years of experience make a difference in the quality of teaching? Teaching and Teacher Education, 96, 103190. https://doi.org/10.1016/j.tate.2020.103190
Harding, L., & Kremmel, B. (2016). Teacher assessment literacy and professional development. In D. Tsagari & J. Banerjee (Eds.), Handbook of second language assessment (pp. 413-427). De Gruyter
Hwang, G. J., Xie, H., Wah, B. W., & Gašević, D. (2020). Vision, challenges, roles and research issues of artificial intelligence in education. Computers in Education, 1,100001. https://doi.org/10.1016/j.caeai.2020.100001
Jaeger, G. J. (2016). Preparations for the new: Reaching and teaching creativity with adaptive technologies. Educational Technology, 56(6), 24-31. http://www.jstor.org/stable/44430504
Jeong, H. (2013). Defining assessment literacy: Is it different for language testers and non-language testers? Language Testing, 30(3), 345-362. https://doi.org/10.1177/0265532213480334
Jin, Y., & Fan, J. (2023). Test-taker engagement in AI technology-mediated language assessment. Language Assessment Quarterly, 20(4-5), 488-500. https://doi.org/10.1080/15434303.2023.2291731
Khodashenas, M. R., Khodabakhshzade, H., Baghaei, P., & Motallebzade, K. (2022). EFL teachers assessment literacy needs inventory: A case of Fulcher’s assessment literacy framework. Issues in Language Teaching, 11(1), 131-156. https://doi.org/10.22054/ilt.2022.63731.639
Khosravi, H., Sadiq, S., & Gasevic, D. (2020). Development and adoption of an adaptive learning system reflections and lessons learned. In Proceedings of the 51st ACM technical symposium on computer science education (pp. 58–64). Portland, OR. https://doi.org/10.1145/3328778.3366900
Khosravi, H., Shum, S. B., Chen, G., Conati, C., Tsai, Y. S., Kay, J., Knight, S., Martinez-Maldonado, R., Sadiq, S., & Gašević, D. (2022). Explainable artificial intelligence in education. Computers and Education: Artificial Intelligence, 3, 100074. https://doi.org/10.1016/j.caeai.2022.100074
Kremmel, B., & Harding, L. (2020). Towards a comprehensive, empirical model of language assessment literacy across stakeholder groups: Developing the Language Assessment Literacy Survey. Language Assessment Quarterly, 17(1), 100-120. https://doi.org/10.1080/15434303.2019.1674855
Ladd, H. F., & Sorensen, L. C. (2017). Returns to teacher experience: Student achievement and motivation in middle school. Education Finance and Policy, 12(2), 241-279. https://doi.org/10.1162/EDFPa_00194
Larenas S. V., & Brunfaut T. (2023). But who trains the language teacher educator who trains the language teacher? An empirical investigation of Chilean EFL teacher educators’ language assessment literacy. Language Testing, 40(3), 463-492. https://doi.org/10.1177/02655322221134218
Lee, S. W. (2016). Pulling back the curtain: The relationship between teacher quality and students’ educational outcomes [Unpublished Doctoral Dissertation]. University of Wisconsin-Madison, US.
Manasia, L., Ianos, M. G., & Chicioreanu, T. D. (2019). Pre-service teacher preparedness for fostering education for sustainable development: An empirical analysis of central dimensions of teaching readiness. Sustainability, 12, 166. https://doi.org/10.3390/su1201016 
Mansouri, B., Molana, K., & Nazari, M. (2021). The interconnection between second language teachers’ language assessment literacy and professional agency: The mediating role of institutional policies. System, 103, 102674. https://doi.org/10.1016/j.system.2021.102674
Marandi, S., Janatifar, M., & Nafisi, Z. (2021). Using virtual learning teams (VLTs) to enhance EFL teachers’ language assessment literacy (LAL). Foreign Language Research Journal, 11(3), 571-604. https://doi.org/10.22059/JFLR.2021.332638.904
Peng, H., Ma, S., & Spector, J. M. (2019). Personalized adaptive learning: An emerging pedagogical approach enabled by a smart learning environment. Smart Learning Environment, 6, 9. https://doi.org/10.1186/s40561-019-0089-y
Penn, J. (2020). Inventing Intelligence: On the history of complex information processing and artificial intelligence in the United States in the mid-twentieth century [Unpublished Doctoral Dissertation]. Apollo - University of Cambridge Repository, England. https://doi.org/10.17863/CAM.63087
Perrotta, C., & Selwyn, N. (2020). Deep learning goes to school: Toward a relational understanding of AI in education. Learning, Media and Technology, 45(3), 251-269. https://doi.org/10.1080/17439884.2020.1686017
Pill, J., & Harding, L. (2013). Defining the language assessment literacy gap: Evidence from a parliamentary inquiry. Language Testing, 30(3), 381-402. https://doi.org/10.1177/0265532213480337
Razavipour, K. (2014). Assessing assessment literacy: Insights from a high-stakes test. Journal of Research in Applied Linguistics, 4(1), 111-131.
Riazi, A. M., & Candlin, C. N. (2014). Mixed-methods research in language teaching and learning: Opportunities, issues and challenges. Language Teaching, 47(2), 135-173. https://doi.org/10.1017/S0261444813000505
Romero, C., & Ventura, S. (2020). Educational data mining and learning analytics: An updated survey. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, 10(3), e1355. https://doi.org/10.1002/widm.1355
Salas-Pilco, S. Z., Xiao, K., & Hu, X. (2022). Artificial intelligence and learning analytics in teacher education: A systematic review. Education Sciences, 12, 569. https://doi.org/10.3390/educsci12080569 
Schmid, R., & Petko, D. (2019). Does the use of educational technology in personalized learning environments correlate with self-reported digital skills and beliefs of secondary-school students? Computers and Education, 136, 75-86. https://doi.org/10.1016/j.compedu.2019.03.006  
Schmidhuber, J. (2015). Deep learning in neural networks: An overview. Neural Networks, 61(117), 85-117. https://doi.org/10.1016/j.neunet.2014.09.003 
Schmidt, T., & Strasser, T. (2022). Artificial intelligence in foreign language learning and teaching: A call for intelligent practice. Anglistik: International Journal of English Studies, 33(1), 165-184. https://doi.org/10.33675/ANGL/2022/1/14
Seo, K., Tang, J., Roll, I., Fells, S., & Yoon, D. (2021). The impact of artificial intelligence on learner–instructor interaction in online learning. International Journal of Educational Technology in Higher Education, 18(54), 1-23. https://doi.org/10.1186/s41239-021-00292-9
Tammets, K., & Ley, T. (2023). Integrating AI tools in teacher professional learning: A conceptual model and illustrative case. Frontiers in Artificial Intelligence, 6, 1255089. https://doi.org/10.3389/frai.2023.1255089
Taylor, L. (2013). Communicating the theory, practice and principles of language testing to test stakeholders: Some reflections. Language Testing, 30(3), 403-412. https://doi.org/10.1177/0265532213480338
Tsai, F.-H., Tsai, C.-C., & Lin, K.-Y. (2015). The evaluation of different gaming modes and feedback types on game-based formative assessment in an online learning environment. Computers & Education, 81, 259-269. https://doi.org/10.1016/j.compedu.2014.10.013
Vogt, K., & Tsagari, D. (2014). Assessment literacy of foreign language teachers: Findings of a European study. Language Assessment Quarterly, 11(4), 374-402. https://doi.org/10.1080/15434303.2014.960046
Voss, E., Cushing, S. T., Ockey, G. J., & Yan, X. (2023). The use of assistive technologies including generative I by test takers in language assessment: A debate of theory and practice. Language Assessment Quarterly, 20(4-5), 520-532. https://doi.org/10.1080/15434303.2023.2288256
Walkington, C., & Bernacki, M. L. (2020). Appraising research on personalized learning: Definitions, theoretical alignment, advancements, and future directions. Journal of Research on Technology in Education, 52, 235-252. https://doi.org/10.1080/15391523.2020.1747757
Wang, L., Lai, M., & Lo, L.N. (2014). Teacher professionalism under the recent reform of performance pay in Mainland China. Prospects, 44, 429-443. https://doi.org/10.1007/s11125-014-9315-0
Wei, L. (2023). Artificial intelligence in language instruction: Impact on English learning achievement, L2 motivation, and self-regulated learning. Frontiers in Psychology, 14, 1261955. https://doi.org/10.3389/fpsyg.2023.1261955
Weng, F., & Shen, B. (2022). Language assessment literacy of teachers. Frontiers in Psychology, 13, 864582. https://doi.org/10.3389/fpsyg.2022.864582
Wicking, P. (2022). Learning-oriented assessment in an international virtual exchange. In S. Hilliker (Ed.). Second language teaching and learning through virtual exchange (pp. 9-33). Walter de Gruyter GmbH.
Xi., X. (2023). Advancing language assessment with AI and ML-leaning into AI is inevitable, but can theory keep up? Language Assessment Quarterly, 20(4-5), 357-376. https://doi.org/10.1080/15434303.2023.2291488
Xie, H., Chu, H. C., Hwang, G. J., & Wang, C. C. (2019). Trends and development in technology-enhanced adaptive/personalized learning: A systematic review of journal publications from 2007 to 2017. Computers and. Education, 140, 103599. https://doi.org/10.1016/j.compedu.2019.103599
Xu, Y., & Brown, G.T.L. (2016). Teacher assessment literacy in practice: Reconceptualization. Teaching and Teacher Education, 58, 149-162. https://doi.org/10.1016/j.tate.2016.05.010
Yan, X., Zhang, C., & Fan, J. J. (2018). Assessment knowledge is important, but…: How contextual and experiential factors mediate assessment practice and training needs of language teachers. System, 74, 158-168. https://doi.org/10.1016/j.system.2018.03.003
Yudi Cahyono, B., Ardi, P., Nurak Siwa, Y., Sari, R., & Andria Gestanti, R. (2023). EFL teachers’ technological pedagogical knowledge (TPK) and ecological agency in responding to the differentiated learning policy in Indonesia. Journal of Research in Applied Linguistics, 14(2), 84-100. https://doi.org/10.22055/RALS.2023.44055.3081
Zhong, J., Xie, H., & Wang, F.L. (2019). The research trends in recommender systems for e-learning. Asian Association of Open Universities Journal, 14(1), 12-27. https://doi.org/10.1108/AAOUJ-03-2019-0015