Tutor vs. Automatic Focused Feedback and Grading of Student ESP Compositions in an Online Learning Environment

Document Type : Research Article

Authors

Foreign Philologies and their Linguistics, Philology, Universidad Nacional de Educación a Distancia (UNED), Madrid, Spain

Abstract

This article discusses the affordances and limitations of an automatic text evaluator in the context of the online teaching/learning of composition writing skills within a specialised linguistic domain, namely, English for Tourism. The system, named G-Rubric, was designed and built by an interdisciplinary team of linguists, psychologists, educationalists, and computer engineers to explore the applicability of data-driven language learning in education, for which it subsequently obtained several awards and distinctions. This article describes the adaptation process of G-Rubric to English for Tourism, contextualized in a distance learning university degree, and analyses its potential to substitute or complement frontline tutors in the task of revising and assessing student compositions. Two types of textual evaluation are provided by G-Rubric: numerical grading and focused feedback on form (writing) and function (content). Content evaluation is based on pattern-matching and machine reasoning against a specialised corpus and associated knowledge previously inserted in the tool as appropriate. The article compares the performance of both tutors and system and proposes specific lines of research to gain insights into their optimal integration.

Keywords

References

Allen, L. K., Jacovina, M. E., & McNamara, D. S. (2016). Computer-based writing instruction. In C. A. MacArthur, S. Graham, & J. Fitzgerald (Eds.), Handbook of writing research (pp. 316-329). The Guilford Press.
Bandura, A. (1997). Self-efficacy: The exercise of control. W.H. Freeman/Times Books/Henry Holt & Co.
Bárcena Madera, E., Martín-Monje, E., & Jordano de la Torre, M. (2016). Methodological and technological innovation in distance teaching of English for tourism. Iberica, 2016(31), 39-61.
Barrantes Montero, L. (2009). A brief view of the ESP approach. LETRAS, 46, 125-143. https://doi.org/10.15359/rl.2-46.7
Bird, B., Downs, D., McCracken, I. M., & Rieman, J. (Eds.). (2019). Next steps: New directions for/in writing about writing. University Press of Colorado.
Bitchener, J., & Knoch, U. (2008). The value of written corrective feedback for migrant and international students. Language Teaching Research, 12(3), 409-431. https://doi.org/10.1177/1362168808089924
Bruton, A. (2009). The vocabulary knowledge scale: A critical analysis. Language Assessment Quarterly, 6, 288-297. http://dx.doi.org/10.1080/15434300902801909
Boulton, A. (2017). Research timeline: Corpora in language teaching and learning. Language Teaching, 50(4), 483-506. http://dx.doi.org/10.1017/S0261444817000167
Boulton, A., & Cobb, T. (2017). Corpus use in language learning: A meta-analysis. Language Learning, 67(2), 348–393. http://dx.doi.org/10.1111/lang.12224
Council of Europe. (2018). Common European framework of reference for languages: Learning, teaching, assessment. Companion volume with new descriptors.
Crossley, S. A. (2020). Linguistic Features in writing quality and development: An overview. Journal of Writing Research, 11, 415-443. Https://doi.org/10.17239/jowr-2020.11.03.01
Curado, a. (2017). Form-focused data-driven learning for grammar development in ESP contexts. Revista de Lenguas para Fines Específicos, 23(1), 13-30. https://doi.org/10.20420/rlfe.2016.327
Dağ Akbaş, R. (2021). A systematic review of the ESP (English for specific purposes)-based post-graduate research in Turkey. Sosyal Bilimler Dergisi, 22, 12-31. https://www.researchgate.net/publication/357735869_A_systematic_review_of_the _ESP_English_for_specific_purposes-based_post-graduate_research_in_Turkey
Díez-Arcón, P., & Martín-Monje, E. (2021). G-Rubric: The use of open technologies to provide personalized feedback in languages for specific purposes, in Edulearn21 Proceedings, 2635-2643. http://dx.doi.org/10.21125/edulearn.2021.0574
Dudley-Evans, T., & St John, M. J. (1998). Developments in English for specific purposes. Cambridge University Press.
Ellis, R. (2009). Task-based language teaching: Sorting out the misunderstandings. International Journal of Applied Linguistics, 19(3), 221-246. http://dx.doi.org/10.1111/j.1473-4192.2009.00231.x
Fleckenstein, J., Leucht, M., & Köller, O. (2018). Teachers’ judgement accuracy concerning CEFR levels of prospective university students. Language Assessment Quarterly, 15(1), 90-101.
Fletcher-Wood, H. (2021). Habits of success: Getting every student learning. Routledge. https://doi.org/10.4324/9781003010074
Flowerdew, J. (2016). English for specific academic purposes (ESAP) writing: Making the case. Writing & Pedagogy, 8(1), 5-32.
Fox, J. & Arteneva, N. (2017). From diagnosis toward academic support: Developing a disciplinary, ESP-based writing task and rubric to identify the needs of entering undergraduate engineering students. ESP Today, 57, 148-171.
Ghanbar, H., & Rezvani, R. (2024). Four decades of publications in English for specific purposes: Mapping the trajectory of empirical research. Journal of Research in Applied Linguistics, 15(1), 32-49. http://dx.doi.org/10.22055/rals.2023.44222.3102
Harris, Z. (1968). Mathematical structures of language. New York: John Wiley & Sons.
Hernández Benítez, M. H., & Santamaría Lancho, M. (2016). G-Rubric: Una aplicación para corrección automática de preguntas abiertas. Primer balance de su utilización. In Nuevas perspectivas en la investigación docente de la historia económica (pp. 473-494). Editorial de la Universidad de Cantabria.
Hutchinson, T. & Waters, A. (1987). English for specific purposes. Cambridge University Press.
Hyland, K. (2022). English for specific purposes: What is it and where is it taking us? ESP Today, 10(2), 202-220. https://doi.org/10.18485/esptoday.2022.10.2.1
Hyland, K., & Hyland, F. (2006). Feedback in second language writing: Contexts and issues. Cambridge University Press. https://doi:10.1017/CBO9781139524742
Jorge-Botana, G., Olmos, R. & Luzón, J.M. (2019). Bridging the theoretical gap between semantic representation models without the pressure of a ranking: some lessons learnt from LSA. Cognitive Processing, 21, 1-21. https://doi.org/10.1007/s10339-019-00934-x
Kao, C.-W., & Reynolds, B. L. (2016). Analyses of EFL teacher feedback on word choice errors from an on-line writing platform. The Academic Conference of Foreign Language Teaching and Intercultural Studies, May, 35-51.
Karpenko-Seccombe, T. (2020). Academic writing with corpora: A resource book for data-driven learning. London: Routledge. https://doi.org/10.4324/9780429059926
Keuning, H., Jeuring, J., & Heeren, B. (2018). A systematic literature review of automated feedback generation for programming exercises. ACM Transactions on Computing Education (TOCE), 19(1), 1-43.
Kittredge, R. (1982). Variation and homogeneity of sublanguages, In R. Kittredge & J. Lehrberger (Eds.), Sublanguage: Studies of Language in Restricted Semantic Domains, Berlin: Walter de Gruyter, 107-137.
Knight, S., Martinez-Maldonado, R., Gibson, A., Buckingham Shum, S. (2017). Towards mining sequences and dispersion of rhetorical moves in student written texts. In Proceedings of the Seventh International Learning Analytics & Knowledge Conference (LAK '17). Association for Computing Machinery, New York, NY, USA, 228-232. https://doi.org/10.1145/3027385.3027433
Knight, S., Shibani, A., Abel, S., Gibson, A., Ryan, P., Sutton, N., Wight, R., Lucas, C., Sándor, Á., Kitto, K., Liu, M., Vijay Mogarkar, R., & Buckingham Shum, S. (2020). Acawriter: A learning analytics tool for formative feedback on academic writing. Journal of Writing Research, 12(1), 141-186. https://doi.org/10.17239/jowr-2020.12.01.06Kurdi
Limpo, T., Nunes, A., & Coelho, A. (2020). Introduction to the special issue on technology-based writing instruction: A collection of effective tools. Journal of Writing Research, 12(1), 1-7. https://doi.org/10.17239/jowr-2020.12.01.01
Lumley, T. (2002). Assessment criteria in a large-scale writing test: What do they really mean to the raters? Language Testing, 19(3), 246-276. https://doi.org/10.1191/0265532202lt230oa
Martín-Monje, E. & Castrillo, M.D. (2021, May 12). Data driven learning en entornos de aprendizaje formales e informales. XI Jornadas de Investigación en Innovación Docente de la UNED. (online conference). https://www.youtube.com/watch?v=mAajXIK6vrA
Martín-Monje, E. & Castrillo, M.D. (2022). Data-driven learning to improve writing skills in foreign languages. In K.D. Rossade, J. Janssen, C. Wood, & G. Ubachs, Designing Online Assessment — Solutions that are Rigorous, Trusted, Flexible and Scalable. Maastricht: EADTU.
Miao, Y., Badger, R., & Zhen, Y. (2006). A comparative study of peer and teacher feedback in a Chinese EFL writing Class. Journal of Second Language Writing, 15, 179-200. https://doi.org/10.1016/j.jslw.2006.09.004
Mitchell, R., Myles, F., & Marsden, E. (2019). Second language learning theories. (4th ed.). London: Routledge. https://doi.org/10.4324/9781315617046
Roscoe, R.D., Allen, L.K., Weston, J.L., Crossley, S.A. & McNamara, D.S. (2014). The writing pal intelligent tutoring system: Usability testing and development. Computers and Composition, 34(1), 39-59.
Sadler, R. (2010) Beyond feedback: Developing student capability in complex appraisal. Assessment & Evaluation in Higher Education, 35(5), 535-550. http://dx.doi.org/10.1080/02602930903541015
Smelstor, M. (1978). A guide to teaching the importance of audience and subject. https://eric.ed.gov/?id=ED176274
Stevenson, M. & Phakiti, A. (2019). The effects of computer-generated feedback on the quality of writing. Assessing Writing, 19, 51-65.
Swales, J. M. (2000). Languages for specific purposes. Annual Review of Applied Linguistics, 20(1), 59-76. https://doi.org/10.1017/S0267190500200044
Trace, J., Hudson, T., & Brown, J. D. (Eds.). (2015). Developing courses in languages for specific purposes. National Foreign Language Resource Center.
Van Beuningen, C. (2021). Focused versus unfocused corrective feedback: from part IV-feedback provider, feedback intensity, and feedback timing. In The Cambridge handbook of corrective feedback in second language learning and teaching (pp. 300-321). Cambridge University Press.
Wang Z, Han F. (2022). The effects of teacher feedback and automated feedback on cognitive and psychological aspects of foreign language writing: A mixed-methods research. Frontiers in Psychology, 13, 909802. https://doi.org/10.3389/fpsyg.2022.909802
Warren, M. J. (2015). Introduction to data-driven learning. In F. Farr & L. Murray (Eds.), The Routledge handbook of language learning and technology. Routledge.
Wilson, J., Ahrendt, C., Fudge, E.A., Raiche, A., Beard, G., & MacArthur, C. (2021). Elementary teachers' perceptions of automated feedback and automated scoring: Transforming the teaching and learning of writing using automated writing evaluation. Computers & Education, 168, 104208.
Journal of Research in Applied Linguistics
Volume 15, Issue 2
October 2024
Pages 22-42

Files

Cited by

Share

How to cite

Statistics