[Originally published February 2, 2021]
Every year, millions of high school students around the world take college entrance exams. Here in the United States, preparing for the SAT or ACT has become an industry of its own, with a myriad of test prep books and private tutors all selling their secret to a perfect score.
Despite this, the emphasis on tests such as the SAT and ACT in determining college admission is diminishing. In recent years, thousands of schools nationwide have become test optional, including top schools such as Harvard, Stanford, and Yale. The reasons behind this shift are numerous, but a large factor is that performance on standardized tests is not a perfect indicator of a student’s intelligence. When it comes to performance on one single assessment, a lot can come down to factors outside of one’s control – the difficulty of the test, the student’s mood and stress levels, even the testing room conditions can all have an impact on the final score.
One way that some tests have attempted to increase the validity of their scores is to institute a penalty for wrong answers. Though neither the SAT or ACT tests have a “guessing penalty”, others, such as the recently discontinued SAT Subject Tests, and the SSAT, used for admission into private high schools, subtract a fraction of a point for every wrong answer. The reason for having the penalty is to discourage guessing and, in theory, allow for luck to play less of a role in the results of standardized tests, making them more accurate and fair.
This is how a penalty induces test takers to behave differently. It is quite obvious that the best strategy when taking SAT is never to skip a question. When you don’t know the correct answer, you have nothing to lose by taking a guess. For tests like the SSAT, it depends. If you are unsure about the right answer, you may do better by skipping the question. Suppose there are four choices and the punishment for a wrong answer is losing 0.25 points. In this case, you would want to answer the question only if you are more than 25% sure that your choice is correct. However, whether you are 25% or 45% sure is a matter of personal judgment. This implies that an overly confident person takes a guess more often than an overly cautious person. There has been evidence in psychology that boys tend to be more confident than girls, and this has implications for test scores when there are penalties for wrong answers.
One test that included a guessing penalty was the PSU, an entrance exam for top colleges in Chile. On the PSU, each multiple choice question consisted of five options, with 0.25 points deducted for every wrong answer, and no points deducted for a question that was left blank. With less guessing involved, it would be expected that the PSU would more accurately represent a student’s academic proficiency. However, when looking at the scores of various Chilean test takers, an interesting pattern occurred. As it turns out, female test-takers took fewer risks and skipped more questions on average than male test-takers. Though the policy succeeded in decreasing the number of questions guessed on (between 2004 and 2014 an average of 29 out of 80 questions were skipped each year), the added risk-taking dynamic created a gender gap. In the years the policy was in place, men outscored women by 0.27 standard deviations, even when controlling for grade point average and other external factors. The PSU’s negative incentive succeeded in discouraging guessing, but failed to make the test more fair because it was less meaningful to male test takers when compared to females. When the policy was removed in 2015, the gap in questions skipped between genders dropped by 70%, and decreased the gender gap in scores by 9%, a small but significant change.