Artificial Intelligence will Transform the ACT and Other Tests

Artificial Intelligence (AI) will transform the ACT and assessments as we know them, sooner than you think. The revolution is already underway. I came to understand how much closer we are to computer-generated tests when I had a friendly conversation with the CEO of ACT, Inc., Marten Roorda at the annual conference for NACAC, the National Association for College Admission Counseling. We spoke on Thursday, September 27, a day after the ACT, Inc. announced it was completing the purchase of MGHL Consulting’s technology for Automatic Item Generation (AIG). ACT, Inc. had been working with AIG since 2014 and working in partnership with MGHL for a year before moving to acquire its tech. Having completed the acquisition, ACT, Inc. is closer than ever to using test items and entire tests generated entirely by artificial intelligence.

What surprised me is not the premise that computers could generate items from scratch, but the rate of development of this technology. Roorda mentioned that some 70% of math items on the ACT can now be generated at will using this new AIG technology. Some 40% of Reading and English items could also be created using AIG. Within 3 years, this technology will allow the ACT to generate a completely original ACT test at the touch of a button. This is powerful and disruptive technology, and its implementation is far closer than many in the industry may have imagined.

It’s important to clarify that just because the ACT, Inc. may have the capacity to generate items and entire tests using AIG does not mean the test writers will be ready to introduce these products to the marketplace on a similar timeline. This will take time, testing, verification of the validity of the items and their reliability compared to conventionally generated items. Still, the underlying technology is here and it’s impact on college admissions testing and assessments in general is something we should all be prepared for.

I was not that surprised that the ACT could craft math items from scratch. Math, compared to other subjects, is relatively simple to mimic and the thinking is algorithmic. You look at a previously administered question, study the form of the question and the language and the pattern in the answer choices. Once you understand the structure and foundation of the question, you can iterate from that, developing many other questions, varying the language, the context, the magnitude of the numbers. There are patterns which underpin the questions and answers.

The low-hanging fruit for the ACT is to generate math items using artificial intelligence. The technology the ACT, Inc. acquired from MGHL consists primarily of an item generator, IGOR (short for “Item GeneratOR”). IGOR uses “item models,” structures or frames of a problem that will allow for the generation of thousands of other items using that frame. The AI can modulate the level of language up or down, for 7th graders, or for 11th graders, across languages and cultures. The AI can shift the specific values within a problem to create, in certain cases, a near infinite number of potential test items.

The greater challenge, but one that the ACT appears prepared to tackle, is shifting from math to produce items in science, english, and reading. The Senior VP of Test Development at the ACT, Inc. Donna Matovinovic, wrote a blog about the cognitive modeling the AI uses to generate items for a wide variety of subject areas. In terms of the face validity of the automated items, she writes, “I have placed questions generated by AIG next to human-generated questions and asked people to tell me which were which. They couldn’t do it. It’s impossible to tell the difference. The items are indistinguishable.”

This new technology will allow initial incremental improvements in test security before leading to sea-changes in the space. In the beginning, AIG will allow the ACT, Inc. to create test items and test forms faster than ever, allowing them to generate more test forms, making cheating more challenging. The coup de grace will come when the ACT, Inc. shifts from fixed test forms to Computer Adaptive Tests, whereby every test that a student sees will be unique. Students will be unable to memorize their way through a testing path within a problem bank containing a near infinite number of potential items. The human memory will be no match for these AIs. I’m personally eager to see an ACT that is wholesale generated from a computer-based intelligence, passages and all. We will get there - it’s simply a matter of time. And the fraud and cheating that have plagued paper-based fixed test forms will become a distant memory.