The Robots are taking over STAAR!
Recently, my colleague attending a Curriculum Director’s meeting at the Region 10, Educational Service Center reported back to our team that the Texas Education Agency intends to use artificial intelligence to assess student writing submitted for the state summative assessment- STAAR. There have not been many more details released as to how this will be done/regulated other than that there will be a sample of human graders to provide or ensure calibration against computer scores. This, of course, got my team's Slack channels lit with comments, opinions, and questions about how this would work and what it means for writing instruction. It is inevitable that AI resources and tools will make their way into data analysis in every industry, including education. I don’t disagree that Natural Language Processing could be an appropriate tool in state assessment; however, I would advocate that the contents of Chapter 5 be well understood and adhered to by all decision-making stakeholders.
Most Commonly Used Dimensions of NLP: Descriptive, Lexical, Syntax, Cohesion, Semantic Content
Descriptive indices provide a very basic accounting of what is in a text, like word count, number of paragraphs, etc. With this information, correlations can be made to predictive outcomes for a student. For instance, most students who submit > 500-word discussion posts are more likely to complete a course as compared to other metrics. This dimension is not providing any feedback on the quality of writing. Similarly, lexical indices provide insights into the lexical richness and diversity of a text. This allows for the analysis of the language input, meaning at what level of complexity the text is written. The syntax dimensions evaluate the complexity of the syntactic constructions in writing. This means at what complexity are the sentence structures written, such as the “mean length of clauses, mean length of t-units, or the number of words before the main verb.” Cohesion indices can measure how well the ideas in the text are connected to each other. This metric can evaluate comprehension as a student’s ability to make connections is an indicator of comprehension. Finally, semantic content can be evaluated by “reveal(ing) the main emphasis of a text” or semantic comparison between texts.
Three Primary Stages of NLP Analysis: Input, Process, and Output
The ability to align student resources and texts to their individual skill level is how NLP analysis for input can benefit students. By using NLP to adapt materials to appropriate levels, we can assess for content comprehension rather than reading skills and enhance the learning process by keeping reading levels in their zone of proximal development. The process stage takes into account the processing that is accountable by students to arrive at comprehension. Understanding the processes utilized by students can enhance a program’s ability to be adaptive to student learning needs. Finally, the output from students can be analyzed by AES (essay scoring engine) using NLP. While the focus on AES is often summative in nature, using the dimensions of NLP to help students improve their writing can offer formative feedback in order to attain incremental improvements.
Where are we headed?
The text emphasizes the need to focus on students’ online language production. In other words, spending more time in the process stage of NLP analysis. They suggest research techniques like keystroke analysis to understand the process better while developing the content output. Being able to see what a student writes, deletes, rewrites, and overall how their writing piece takes shape can offer indicators of where best to intervene. So, rather than only being reactive to the final written product, but to really understand how the students arrive at that final product and what redirections could have a beneficial impact on student learning and demonstration of learning.
Back to my connection to Texas state assessments…
Throughout this text and many others referencing the same content, it is said: “to help educators UNDERSTAND.” What does this mean to me? It means that the utilization of AI in the form of AES on STAAR writing could be beneficial IF the metrics are clearly defined and shared with educators to help with the formative assessment and improvement for students. If it is used as a final stamp of approval or denial of student proficiency and its impact ends there, we stand to further reinforce formulaic writing for the sake of meeting a standard rather than a true demonstration of knowledge.
Reference
Gašević, D., Merceron, A., Allen, L., Creer, S., & Öncel, P. (2022). The Handbook of Learning Analytics. In G. Siemens, A. Wise, & C. Lang (Eds.), Solar eBooks (pp. 38–45). https://doi.org/10.18608/hla22
