Monday, September 21, 2009

S.A.T. a better predictor than you may think 

There is a common argument that SAT scores don't tell you much about whether students will succeed in college. Catherine Rampell recently showed the correlation between income and SAT score, which caused a bit of a stir over whether it told us anything about the quality of the student.

Now we have a new piece of evidence (h/t: reader jw):
It�s commonly said that the SAT, taken in a senior year of high school, has only about a 40% correlation with a student�s freshman year college GPA.

...I�ve always had a skeptical feeling about the 40% correlation statistic, and so I�ve never relied on it or used it in print. There are two self-selection problems that make it really hard to control the data. First, high schoolers of diverging abilities apply to different schools�the strongest students apply to one tier of colleges, and the average students apply to a less ambitious tier, with some overlap. Second, once students get to a college, they enroll in classes they believe they can do well in. Many of the strongest students try their hand at Organic Chemistry, while more of the less-confident students take Marketing 101. At each of these colleges and courses, students might average a B grade, but the degree of difficulty in achieving that B is not comparable.

Many scholars have attempted to control for these issues, looking at data from a single college or a single required course that all freshman have to take, and their work has suggested the 40% correlation is a significant underestimate. I�ve long wondered what would happen if an economist really took on this massive mathematical mess, on a large scale, harvesting data from a wide selection of universities.

Finally this has been done, by Christopher Berry of Wayne State University and Paul Sackett of the University of Minnesota. They pulled 5.1 million grades, from 167,000 students, spread out over 41 colleges. They also got the students� SAT scores from the College Board, as well as the list of schools each student asked the College Board to send their SAT scores to, an indicator of which colleges they applied to. By isolating the overlaps�where students had applied to the same colleges, and taken the same courses at the same time with the same instructor�they extracted a genuine apples-to-apples subset of data.

It turns out that an SAT score is a far better predictor than everyone has said. When properly accounting for the self-selection bias, SAT scores correlate with college GPA around 67%. In the social sciences, that�s considered a great predictor.
Here's a (gated) link to the paper. It appears that both Berry and Sackett are psychologists, not economists. Does income and selectivity of universities have a significant correlation? Casually I would think yes, but what I've read of this literature tells me not to be casual.

Labels: ,