First Impressions of the New Digital SAT

With the release of the first full-length digital adaptive SAT on College Board’s new testing app, Bluebook, students and educators can now get a clearer sense of the SAT of the future. While the app is still in beta, and updates are anticipated to arrive October 18th, it is evident why 80% of students in this summer’s pilot study prefer the digital adaptive SAT to the current paper test. The testing app is generally well-designed, the content is familiar, and the shorter testing duration a significant improvement.

First Impressions

User Experience

The Bluebook app is streamlined and largely intuitive. The digital tools are easy to find, use, and then hide once you are done with them.

• The annotation feature allows students to highlight text and add their own notes. Given the short-form passages, we don’t foresee most students relying too heavily upon this feature.
• The answer-elimination feature is helpful. When activated, the answer choices are stricken through with a bold line and the text grayed out just enough to remain legible without being a distraction.
• The countdown clock is centrally placed and easy to hide or show per student preference.
• The Mark for Review feature is particularly useful: the readily accessible visual display of the entirety of a student’s answer choices will help students navigate each section and quickly toggle between flagged or omitted items. The visual presentation will help eliminate carelessness and unintentionally omitted items.

Testing Content: grouping question types, getting into a rhythm

Within a given section, different question types appear in blocks, rather than at random, allowing students to get into a rhythm. On the Reading and Writing section, a student will see all of the “words in context” vocabulary questions at once, followed by a block of questions focusing on craft, structure, and purpose, later followed by a block of grouped grammar questions. Similarly, on the math section, a student could see a grouping of six consecutive graphing questions. This format is akin to having subsections within a section, and it allows students to learn the question types and move through the test with a coherent, problem-solving mindset.

Adaptive Testing

This practice test gives us our first glimpse of the adaptive algorithm the College Board will use for scoring the new tests. As outlined in our July post, student performance on the baseline sections for math and reading/writing will determine whether one advances to the easier or harder adaptive sections. With 22 math questions (20 operational, 2 experimental) and 27 Reading/Writing questions (25 operational, 2 experimental) in each baseline section, where is the line to advance to the easier or harder adaptive section? We found that for this particular test the line to advance to the harder math section was 15 correct answers out of 22 items on the baseline, and our colleagues over at Summit Education found the line for the verbal module to be 18 of the 27 items. That would indicate a student will need to answer roughly 2/3 of the items correctly to advance to the more rigorous sections. This is a general estimate and does not factor in the 2 experimental questions per module, which will not impact the algorithm.

Scoring Thresholds and Caps

We have more work to do to fully understand the nature of the scoring algorithm once more tests are released, but our initial analysis begins to reveal some of the “edges” of the scoring.

The Easy Upper Limit

If a student misses more than 7 questions on the baseline math module and is therefore routed to the easier adaptive module, that student’s final score appears to be capped at a 650 or so. Even if a student aces the remaining 22 questions in the adaptive module (20 of which are operational, 2 experimental), the missed questions from the baseline will lead to the 650 cap.

The Hard Lower Limit

If a student correctly answers the minimum threshold of 15 questions on the baseline to advance to the harder math module, and then proceeds to miss every single question, that student will attain a score of 460.

So it seems the scoring minimum for a student advancing to the hard math module is roughly a 460 (15 of 44 total items answered correctly), and the maximum score for a student advancing to the easier math module is roughly a 650 (36 of 44 items answered correctly).

It is important to mention that this scale will likely be distinct for this particular practice test, and not generalize to subsequent tests, which will have their own scales. The threshold to advance on math may be 14 or 16 questions or so, but this early analysis gives students a ballpark estimate of how they will have to perform to cross the threshold to the more difficult adaptive sections. More details to follow.

A Deeper Dive into the Reading Section

This practice test, comprised entirely of short, single-question passages, confirmed that short passages are not necessarily easier passages. The reading level on this SAT practice test was as high, if not higher, than other SATs in recent years, including excerpts from Jane Austen’s Sense and Sensibility, a Shakespearean sonnet, and numerous challenging scientific passages. Switching to a new topic, voice, and style every question requires a bit more mental agility. Counterbalancing this, the predictable grouping of questions by type (i.e., all vocab at once, followed by all craft and structure questions at once), does allow a student to get into a particular problem-solving mindset for a stretch of time.

That the passages are so short takes much of the load off of working memory. Students may choose to use the annotation tools, but many will be able to hold all the relevant content in memory while they answer a single question. Annotating digitally requires more effort than annotating on paper. To use the digital annotation feature, Students need to move the cursor to the relevant text, highlight the text, move the cursor to the top right, click the annotate icon, move the cursor to the bottom of the screen, type in the annotation, and then click save. That’s a 7-step process, replacing a much simpler and more automatic written process. Consequently, some students may choose to use the available scratch paper to record their own annotations, instead of using the digital feature, if they chose to annotate at all.

Reading Content

The single-question-per-passage format allows for an exposure to a wider range of content. Here is a breakdown of the passage types on this first practice test.

‍

‍Literature took a back seat on this practice test. The easier adaptive module had more literature, while the harder module focused more on humanities (e.g., the analysis of art). There were no examples of first-person non-fiction writing or persuasive essays. All writing that isn't narrative is informational, which marks a significant content shift for the SAT.

In general, grammar played a much smaller role on this practice test, but we noted an increased focus on logical connectors (words such as “However”, “Therefore”, and “Thus”). Interestingly, focusing on these key words is very helpful on the reading comprehension questions as well, as they often signaled the key hypothesis or detail that the question relied upon.

Vocabulary was noticeably back in focus on this practice test, but did not reflect a return to the “SAT vocabulary” of old. Students will have no use for flash cards of low-frequency words, but they’ll have to attend carefully to how words are being used in the specific context of passages.

One interesting addition was the presentation of reading content in bulleted lists framed by this new question type: “While researching a topic, a student has taken the following notes:” These questions test a student’s ability to summarize a key point, draw appropriate inferences, or effectively use the notes to accomplish a given task; we found them to be a welcome addition that reflects a key skill students must hone throughout their academic careers.

A Deeper Dive into the Math Section

On the new practice test, there were significantly fewer math problems to solve (44 down from 58), and the problems were, on average, less reading intensive. Decreasing the reading burden on the math section will help many students who struggled with the wordier SAT math that emerged in 2016.

The reference sheet of math formulas is still available with the click of the mouse. Students can easily open and close this feature and move the reference section anywhere on the screen; this was much more convenient than flipping back and forth to the reference section on a paper test.

Desmos

The powerful Desmos graphing calculator, baked into the app, plays a major role on this new test, and for many students will fundamentally change the experience of taking the test. The calculator opens and closes with a single click and can remain open during the entirety of the test. While some students feel more comfortable with their own familiar handheld calculator, the Desmos option is right there on the screen, immediately adjacent to the content, which is extremely convenient for problem solving.

We found that students who know how to use a graphing calculator could use it to answer the majority of questions, often by plugging in equations and identifying where functions intersect on the graph. While this may not be the most efficient way to solve every problem, it affords students another pathway towards the correct answer, and can significantly help low and mid-scoring students. In particular, the Desmos calculator made the “easier” adaptive module that much easier for students: the calculator was a passport to solving a surprising number of problems.

This shift towards calculator-involved math feels like a move in the right direction, as we live in an age where we generally have powerful tools at our disposal, including a supercomputer in our pocket, throughout the day. Being able to skillfully use those tools is more relevant to success in the world, than to have memorized a great deal of content or to be facile with hand-calculations.

Content Distribution

The College Board has naturally made some tweaks in the distribution of content assessed on the SAT.

Though the category names have changed slightly, the content assessed on the new SAT is quite similar with that found on the current test:

‍

The most interesting thing is that Problem Solving and Data Analysis (which covers things like percentages, probability, organizing data, and basic statistics) is cut the most. Our current sample size is very small, but right now it looks like the full range of Problem Solving and Data Analysis topics that were tested previously will remain fair game on the new SAT.

Here is the problem distribution by module:

Easier and Harder Adaptive Modules

There is a pronounced difference in average difficulty level between the easy and hard adaptive module. Students who find themselves on easy street for the adaptive section have most likely landed in the lower-level module. By our estimate, 16 out of 22 questions on the easier module were Easy/Medium difficulty compared to only 5 out of 22 questions on the harder module. Here is the breakdown of the levels of difficulty for the baseline, the easy, and the hard adaptive modules.

‍

The weighted average for the difficulty level of the baseline module is a 2.5, compared to 2.0 for the easy module and a 2.95 for the hard module. Using this method of comparison, the easy module is 20% easier than the baseline, and the hard module is 48% more difficult than the easy module.

Reviewing Your Answers

Once a student has completed the testing process on the BlueBook app, they will be redirected to the College Board site to view their score and review their test. The interface is user-friendly, and a student can quickly review each problem they missed.

Looking Ahead

The College Board has so far delivered on its promised deadlines and appears to be poised to roll out the new digital adaptive test to all international students in March 2023. Domestic students will see the digital adaptive PSAT in October 2023, and then the digital adaptive SAT goes live October 2024. Current Sophomores will be the first class to transition from paper to digital testing in the spring of next year.

We are looking forward to the October 18th updates and will keep you abreast of developments as they arise, as we continue to build out our practice materials for this newest iteration of the SAT.