Inconsistent Scoring Profoundly Weakens the Validity of the ACT Essay

In January we reported on the seemingly inconsistent score results we had observed after the first several administrations of the new ACT essay. In recent weeks, the extent of the inconsistencies has become apparent with the return of hand-scored essays. When our students’ updated essay scores came back, we were taken aback by the magnitude of the net scoring gains. Every one of our students who asked for a hand re-score saw a meaningful increase. One of our tutors, a former professor at Emory, requested a hand scoring and saw his 24 composite essay score rise to a 33. The Washington Post similarly reported that a student in Rhode Island saw a score of 19 rise to a 31. What is going on here?

It is apparent that there are serious problems with consistency of scoring between different graders. Our tutor, who is a highly skilled writer with years of experience grading undergraduate papers, received the following scores on the December ACT essay:

‍

Stunned by his 24 on the essay (compared to his 34 ACT composite score), our professor requested a hand-scored essay, risking the $50 fee that would only be refunded in the event his essay was returned with a net score increase. After six weeks of waiting, his essay was hand graded and returned. Under this higher level of scrutiny, the following scores were returned:

‍

Our tutor was finally recognized by the ACT as the highly skilled writer that he is! But his experience is concerning. When a student has to lay down $50 and wait over a month to attain proper grading, this creates a serious issue of access.

Though these individual cases may be outliers, the fact that they exist threatens the validity of the exercise. The huge score jumps we are seeing with the most recent round of hand-grading far exceed anything we’ve ever seen before. In years past, we’d typically see an ACT essay score of 9 round up to a 10 after hand-grading, and an 11 round up to a 12, whereby the lower of the two graders (scoring on a scale of 1-6) would simply round up to meet the higher grader. When a 24 is effectively equal to a 33, and a 19 is equivalent to a 31, what are college admissions officers to do with these essay scores, besides take them with a grain of salt? This degree of inconsistency renders the scores all but meaningless.

To our knowledge the ACT Inc. has made no effort to explain the significant score changes that have taken place. The score reports simply indicate the new subscores and composite score. Neither the old score (for purposes of comparison) nor the original essay are returned to the student. Without further information, we can only assume that the graders are having trouble consistently applying the new, more complicated scoring rubric.

We anticipate that the new SAT essay will also introduce some initial scoring inconsistency, although perhaps less than what we’ve witnessed with the ACT. Graders on the new SAT will now have to assign up to 12 points to each essay (compared to 6 points on the "old" SAT essays). However, the SAT essay grading may benefit from a narrower focus.  On every essay, students will be asked to analyze a provided example of persuasive writing; by contrast, the revised ACT essay invites students to engage in open-ended exploration across a wide variety of contemporary issues. Importantly, the new SAT essay is highly similar to the College Board’s Document Based Question, a staple of AP tests for decades, so veteran College Board graders may already feel quite comfortable with the new SAT essay format. Still, the AP score range of 1 to 5 is much narrower than the range of 6 to 24 on the new SAT essay. We’ll know the early outcomes later this spring when the first round of new SAT scores are released.

Both the ACT and SAT essays, as measures of student readiness for college-level writing, are superior to the essays they replaced. They are more rigorous, involve greater degrees of critical thinking and evaluation of evidence. The assessments are moving in the right direction; we simply need consistent scoring to realize the full benefit of these improved writing exercises.