Issues in the Design and Implementation of Chatbots for Oral Language Assessment

Document Type : Research Article

Authors

1 Filologia Moderna-Intituto Franklin, College of Education, Universidad de Alcala, Alcala Henares, Spain

2 Linguistica Aplicada, E.T.S. de Ing. de Caminos Canales, Universidad Politacnica de Madrid, Madrid, Spain

3 Department, E.T.S.I. Aeroespacial y Diseño Industrial, Universidad Politecnica de Valencia, Valencia, Spain

Abstract

Oral assessment in computer-assisted language learning is one of the best-known challenges at a technical and implementation level in an official language certification. The case of Spain is especially critical since the government has delayed for years the completion of the listening comprehension and oral expression test in the University Access Test (EVAU). This article presents first the evolution of oral tests at a general level, then a SWOT analysis of the potential of such implementation, and, third, how to implement them and the paper concludes that there is evidence that chatbots adapted to language learning can also be used for evaluation. Chatbot-assisted language learning with artificial intelligence adapted to voice recognition and its processing to obtain semi-automatic assessment supervised by the teacher can become a tool to be implemented within the language learning processes and / or included in the language certification methodology. Another interesting aspect could be the development and design of the interfaces of future adapted chatbots that must consider multimodality in the interaction between man-machine so that communication is effective and can be validated based on the knowledge available to the student.

Keywords

References

Ayedoun E., Hayashi Y., & Seta, K. (2015). A Conversational agent to encourage willingness to communicate in the context of English as a foreign language. Procedia Computer Science, 60(1), 1433-1442. https://www.doi.org/10.1016/j.procs.2015.08.219
Ayedoun, E., Hayashi, Y., & Seta, K. (2019). L2 learners’ preferences of dialogue agents: A key to achieve adaptive motivational support? In Isotani, S., Millán, E., Ogan, A., Hastings, P., McLaren, B., & Luckin, R. (Eds.), Artificial Intelligence in Education. AIED 2019. Lecture Notes in Computer Science, 11626 (pp. 19-23). Springer, Cham. https://doi.org/10.1007/978-3-030-23207-8_4
Börsting, I., & Hesenius, M. (2021). Towards a systematic approach for chatbot development in digital work environments. In Klumpp, M., & Ruiner, C. (Eds.), Digital Supply Chains and the Human Factor. Lecture Notes in Logistics (pp. 79-94). Springer, Cham. https://www.doi.org/10.1007/978-3-030-58430-6_5
Darabi Bazvand, A., & Ahmadi, A. (2020). Interpreting the validity of a high-stakes test in light of the argument-based framework: Implications for test improvement. Journal of Research in Applied Linguistics, 11(1), 66-88. https://www.doi.org/10.22055/rals.2020.15417  
Cerdeira J. M., Catela Nunes L., Balcão Reis A., & Seabra, C. (2018). Predictors of student success in higher education: Secondary school internal scores versus national exams. Higher Education Quarterly, 72(4), 303-313. https://doi.org/10.1111/hequ.12158
Díez-Arcón, P., & Martin-Monje, E. (2023). Language teacher development in computer-mediated collaborative work and digital peer assessment: An innovative proposal. Journal of Research in Applied Linguistics, 14(2), 40-54. doi: 10.22055/rals.2023.44054.3080
European Comission (2021). Proposal for a regulation of the European Parliament and of the Council amending Regulation (EU) No 910/2014 as regards establishing a framework for a European Digital Identity. EUR-Lex, European Digital Identity Regulation. https://eur-lex.europa.eu/legal-content/EN/ALL/?uri=CELEX:52021PC0281
Fernandez Alvarez, M., García Laborda, J., & Magal-Royo, T. (2022). Subrepresentación del constructo en exámenes estandarizados de lengua extranjera en España: propuesta de examen asistido por ordenador. Porta Linguarum Revista Interuniversitaria De Didáctica De Las Lenguas Extranjeras, Monográfico 2022, 27–45. https://doi.org/10.30827/portalin.vi.21393
Finney, S. J., Myers, A. J., & Mathers, C. E. (2018) Test instructions do not moderate the indirect effect of perceived test importance on test performance in low-stakes testing contexts. International Journal of Testing, 18(4), 297-322. https://doi.org/10.1080/15305058.2017.1396466
Fryer, L. K., Coniam, D., Carpenter, R., & Lăpușneanu, D. (2020). Bots for language learning now: Current and future directions. Language, Learning and Technology, 24(2), 8-22.  http://hdl.handle.net/10125/44719
Fryer, L. K., Ainley, M., Thompson, A., Gibson, A., & Sherlock, Z. (2017). Stimulating and sustaining interest in a language course: An experimental comparison of chatbot and human task partners. Computers in Human Behavior 75, 461–468. https://doi.org/10.1016/j.chb.2017.05.045
Følstad, A., & Brandtzæg, P. B. (2017). Chatbots and the new world of HCI. Interactions, 24(4), 38–42. https://www.doi.org/10.1145/3085558
García Laborda, J., & Amengual Pizarro, M. (2022). Revisando el mundo de la evaluación lingüística. Análisis DAFO de 2020. In Romera Ciria, M., & Camino Bueno Alastuey, M. (Coord.) Didáctica de la lengua, multimodalidad y nuevos entornos de aprendizaje (pp. 265-278). Graó Editors, Barcelona, Spain.
García Laborda, J., & Fernández Álvarez, M. (2021). Multilevel language tests: Walking into the land of the unexplored. Language Learning & Technology, 25(2), 1–25. http://hdl.handle.net/10125/73428
García Laborda, J., & Martín-Monje, E. (2013). Item and test construct definition for the new Spanish baccalaureate final evaluation: A proposal. International Journal of English Studies, 13(2), 69-88. https://doi.org/10.6018/ijes.13.2.185921
Goertler, S., & Gacs, A. (2018). Assessment in online German: Assessment methods and results. Die Unterrichtspraxis. Teaching German, 51(2), 156–174. https://www.jstor.org/stable/90026423
Guapacha Chamorro, M. E.  (2022). Cognitive validity evidence of computer-and paper-based writing tests and differences in the impact on EFL test-takers in classroom assessment. Assessing Writing, 51. https://www.doi.org/10.1016/j.asw.2021.100594  
Haristiani, N. (2019). Artificial intelligence (AI) chatbot as language learning medium: An inquiry. Journal of Physics: Conference Series, 1387, International Conference on Education, Science and Technology, 13–16 March 2019, Padang, Indonesia.  https://www.doi.org/10.1088/1742-6596/1387/1/012020
Harley, J. M., Mantou Lou, N., Liu, Y., Cutumisu, M., Daniels, L. M, Leighton, J. P. & Nadon, L. (2021). University students’ negative emotions in a computer-based examination: the roles of trait test-emotion, prior test-taking methods and gender. Assessment & Evaluation in Higher Education, 46(6), 956-972. https://doi.org/10.1080/02602938.2020.1836123
Huang W, Hew K.F. & Fryer L.K. (2021). Chatbots for language learning. Are they really useful? A systematic review of chatbot-supported language learning. Journal of Computer Assisted Learning, 38(1), 237-257. https://doi.org/10.1111/jcal.12610
In’nami, Y. (2006). The effects of test anxiety on listening test performance. System, 34(3), 317-340. https://doi.org/10.1016/j.system.2006.04.005
Jia, J., Chen, Y., Ding, Z., & Ruan, M. (2012). Effects of a vocabulary acquisition and assessment system on students’ performance in a blended learning class for English subject. Computers & education, 58(1), 63-76. https://doi.org/10.1016/j.compedu.2011.08.002
Kim, H. S., Kim, N. Y., & Cha, Y. (2021). Is it beneficial to use AI chatbots to improve learners' speaking performance? The Journal of Asia TEFL, 18(1), 161-178.  https://www.doi.org/10.18823/asiatefl.2021.18.1.10.161 
Kumar, J. A. (2021). Educational chatbots for project-based learning: investigating learning outcomes for a team-based design course. International Journal of Education Technology High Education, 18(65). https://www.doi.org/10.1186/s41239-021-00302-w
Lorenzo Moledo, M., Argos González, J., Hernández García, J., & Vera Vila, J. (2014). El acceso y la entrada del estudiante a la universidad: situación y propuestas de mejora facilitadoras del tránsito = Access and student entrance to the University: status and improvement proposals facilitating transit. Educación XXI: Revista de la Facultad de Educación, 17(1), 15-38. https://doi.org/10.5944/educxx1.17.1.9951
Magal Royo, T., & García Laborda, J. (2018). Standardization of design interfaces applied to language test on-line through ubiquitous devices. International Journal of Interactive Mobile Technologies (iJIM), 12(4), 21-31. https://doi.org/10.3991/ijim.v12i4.9197
Magal Royo, T., & García Laborda, J. (2022). Communicative competence of mediation assessment language learning through the use of chatbots. EDULEARN22 Proceedings. 14th International Conference on Education and New Learning Technologies, 4-6 July 2022 Palma, Spain, 3463-3469. https://www.doi.org/10.21125/edulearn.2022.0846
Martín Mazón, R. (2021). A chatbot on syntactic issues: A proposal for innovative help in the first year of baccalaureate classroom. Alcalibe: Revista Centro Asociado a la UNED Ciudad de la Cerámica, 21, 83-110. http://www.alcalibe.es/images/Alcalibe_21/un_chabot_sobre_cuestiones_sintacticas.pdf 
Nguyen, Q., Sidorova, A., & Torres, R. (2021). User interactions with chatbot interfaces vs. Menu-based interfaces. Computers in Human Behavior, 128. https://www.doi.org/10.1016/j.chb.2021.107093
Nordberg, O. E., & Guribye, F. (2023). Interacting with the news through voice user interfaces. In Følstad, A., et al. (Eds.), Chatbot Research and Design. CONVERSATIONS 2022. Lecture Notes in Computer Science, 13815. Springer, Cham. https://doi.org/10.1007/978-3-031-25581-6_2
Okonkwo, C. W., & Ibijola, A. A (2021). Chatbots applications in education: A systematic review. Computers and Education: Artificial Intelligence, 2. https://doi.org/10.1016/j.caeai.2021.100033
Olani, A. (2009). Predicting first year university students' academic success. Electronic Journal of Research in Educational Psychology, 7(3), 1053-1072. https://doi.org/10.25115/ejrep.v7i19.1351
Pahissa, I., & Tragant, E. (2009). Grammar and the non-native secondary school teacher in Catalonia. Language Awareness, 18(1), 47-60. https://doi.org/10.1080/09658410802307931
Peñate Cabrera, M. (2014). Choosing a speaking test in English as a foreign language for the university entrance exam. Didáctica (lengua y literatura), 26, 377-400.  http://hdl.handle.net/11162/179007
Poehner, M. E., & Lantolf, J. P. (2023). Advancing L2 dynamic assessment: Innovations in Chinese contexts. Language Assessment Quarterly, 20(1), 1-19. https://doi.org/10.1080/15434303.2022.2158465
Poehner, M. E., & Leontjev, D. (2020). To correct or to cooperate: Mediational processes and L2 development. Language Teaching Research, 24(3), 295-316. https://doi.org/10.1177/1362168818783212
Ross, S. J., & Okabe, J. (2006). The subjective and objective interface of bias detection on language tests. International Journal of Testing, 6(3), 229-253. https://doi.org/10.1207/s15327574ijt0603_2
Stenlund, T., Lyrén, P. E., & Eklöf, H. (2018). The successful test taker: exploring test-taking behavior profiles through cluster analysis. European journal of psychology of education, 33(2), 403-417. https://doi.org/10.1007/s10212-017-0332-2
Soodmand Afshar, H. (2020). Test-takers’ perceptions of paired speaking tests and the role of interlocutor variables in pairing. Journal of Research in Applied Linguistics, 11(1), 89-123. doi: 10.22055/rals.2020.15418
Vuorikari, R., Punie, Y., Carretero Gomez, S., & Van Den Brande, G. (2016). DigComp 2.0: The digital competence framework for citizens. Update phase 1: The conceptual reference model. EUR 27948 EN. Luxembourg, Publications Office of the European Union. https://www.doi.org/10.2791/11517
Vuorikari, R., Kluzer, S. & Punie, Y., (2022). DigComp 2.2: The digital competence framework for citizens with new examples of knowledge, skills and attitudes, EUR 31006 EN. Luxembourg, Publications Office of the European Union. https:///www.doi.org/10.2760/490274
Weir, C. J. (2005). Language testing and validation. An evidence-based approach. Research and practice in applied linguistics (RPAL), Palgrave Macmillan, London. https://www.doi.org/10.1057/9780230514577
Winkler, R.., & Soellner, M. (2018). Unleashing the potential of chatbots in education: A state-of-the-art analysis. In Academy of Management Annual Meeting Proceedings (Vol. 1, 15903). https://doi.org/10.5465/AMBPP.2018.15903abstract
Xu, Y., Wang, D., Collins, P., Lee, H., & Warschauer, M. (2021). Same benefits, different communication patterns: Comparing Children's reading with a conversational agent vs. a human partner. Computers & Education, 161. https://doi.org/10.1016/j.compedu.2020.104059
Yin, J., Goh, T. T., Yang, B., & Xiaobin, Y. (2021). Conversation technology with micro-learning: The impact of chatbot-based learning on students’ learning motivation and performance. Journal of Educational Computing Research, 59(1), 154–177. https://doi.org/10.1177/0735633120952067
Journal of Research in Applied Linguistics
Volume 15, Issue 2
October 2024
Pages 43-54

Files

Share

How to cite

Statistics