SAT Scores Are Bad Education Evaluation Data
I am happy to see the public school system coming in for much-deserved criticism. I don't have anything to add to this article that I have not already said about schools many times. But I want to make one complaint about a chart used in the blog post:
SAT scores are a terrible metric for measuring academic performance over time.
First, I am not at all convinced that the test scoring does not shift over time (no WAY my son had a higher score than me, LOL).
But perhaps the most important problem is that all students don't take the SAT -- it is a choice. Shifts in the mix of kids taking the test -- for example, if over time more kids get interested in college so that more marginal academic kids take the test -- then the scores are going to move solely based on mix shifts. Making this more complicated, there is at least one competitive test (the ACT) which enjoys more popularity in some states than others, so the SAT will represent an incomplete and shifting geographic mix of the US. Finally, as students have gotten smarter about this whole process**, they gravitate to the ACT or the SAT based on differing capabilities, since they test in different ways.
To me, all this makes SAT scores barely more scientific than an Internet poll.
** If you have not had a college-bound student recently, you will have to trust me on this, but parents can spend an astounding amount of time trying to out-think this stuff. And that is here in flyover country. Apparently private school parents on the East Coast can be absurd (up to and including hiring consultants for 6 figures). A few years ago it was in vogue to try to find your kid a unique avocation. Violin was passe -- I knew kids playing xylophone and the bagpipes. A friend of mine at a high profile DC private school used to have fun with other parents telling them his son was a national champion at falconry, the craziest thing he could make up on the spur of the moment at a cocktail party. Other parents would sigh enviously, wishing they had thought of that one for their kid.
Joel Grus:
The SAT was explicitly rescaled in 1995 or so.
April 4, 2013, 9:49 amTed Rado:
Grade inflation has been going on for decades. Back in the 80's, I noticed that almost all the young engineers I interviewed had GPA's of well over 3.0. When I was in engineering school in the 40's, the GPA for engineering students at my university was 2.1. There was a piece in the WSJ in the 70's or 80's stating that university GPA's had gone up by 0.7 during the period of their study. There are two problems. Schools and departments compete for students and their funding depends on enrollments. If grading is tough, the students go elsewhere. Also, students feel that getting good grades is the most important thing. Thus they tend to take snap courses for electives.
April 4, 2013, 10:09 amAt the K-12 level, teachers are urged to pass on poor students so as not to hurt their self esteem or ruffle the feathers of the parents.
The result of all this is a mess where 284,000 college grads are working for minimum wage as a result of taking snap courses, a trillion dollars in student loans, and a semi-illiiterate citizenry. We seem to have our fingers firmly pressed on our self-destruct button.
We need to flunk kids that aren't learning, quit teaching worthless courses, and making it clear that one MUST study and work hard. The current entitlement mentality is killing us.
marque2:
Yes I was going to post this myself. They made it much easier to get to 1600 - who knows how that chart is scaled.
Another test the ACT might be better because it actually evaluate knowledge in English, basic Science, and basic history. SAT is mostly about logic puzzles, even in the English language sections, and math tricks in the math section which can easily be taught.
April 4, 2013, 10:15 amMingoV:
"... I am not at all convinced that the test scoring does not shift over time (no WAY my son had a higher score than me..."
The SAT was "renormalized" back in the 1990s. That shifted the scores up by about 100 points.
The SAT can be used to assess student performance over time, but it takes a bit more work. Let's say you want to compare 1973 scores to 2013 scores. You look at the data and note that 25% of high school students took the SAT in 1973 and 65% of students in 2013 (those numbers are guestimates). Because of the huge difference, the scores can't be compared directly. Instead, you compare all the SAT scores from 1973 to the scores of the top 25% of high school students (or the 38% of the SAT-takers) in 2013. After "denormalizing" the 2013 scores, you can compare 1973 to 2013.
April 4, 2013, 2:45 pmRich R:
With regard to your comment about activities (violin, bagpipes, falconry) I think its important to note that this is probably only a factor for students going into the Liberal Arts. My youngest starts college this fall - he has absolutely no outside activities other than a couple of school clubs and playing guitar (what teenage boy can't list that nowadays). He was accepted everywhere he applied; including a large scholarship offer to Baylor. The difference being that he is going into Mechanical Engineering. The key factors were SAT score (Math Level II to be precise) and what classes he took in high school. He could have had his pick of Engineering schools due to these high scores and academic background.
April 4, 2013, 7:00 pmJKB:
I was reading Seth Godin's manifesto on education the other day, 'Stop Stealing Dreams: What are schools for?' Not a lot new material but some good observations.
He did have this one interesting tidbit. The multiple choice test will be 100 years old next year, first used in 1914 to process for WWI.
The multiple choice test was later disowned by its instigator, Dr. Kelly, he was promptly fired from his university presidency.
Godin offered these observations:
"In the words of Professor Kelly, “This is a test of lower order thinking for the
lower orders.”
"The SAT, the single most important filtering device used to measure the effect of
school on each individual, is based (almost without change) on Kelly’s lower-
order thinking test. Still.
The reason is simple. Not because it works. No, we do it because it’s the easy and
efficient way to keep the mass production of students moving forward."
So for the centennial of mass testing "of lower order thinking" in 2014, should we,
April 4, 2013, 7:38 pma. celebrate next year by filling in the oval with a number 2 pencil, being careful to stay within the lines?
b. throw up a little bit because we have test anxiety?
c. weep for humanity?
d. none of the above, because no one cares?
norse:
Some observations from an outsider to the US education system:
Multiple choice tests feel really dumbed down compared to the free essay forms I am used to. There is no development of complex thoughts, it's more of a pattern recognition exercise.
Students I interact with here are surprisingly often uninterested in the subjects they are exposed to and unable to hold an argument (I've seen people lose track of what they were trying to argue in the middle of their two or three sentences)
The scariest difference I can see is that students tend to almost not at all reflect or question. It's regurgitate some barely understood facts, let them stand there, done.
Alltogether, these are tendencies that have me real worried.
April 5, 2013, 9:15 amVargo:
If you think that's fly-over country...
Barry had on all of his college applications that he held 7 national high school records in The Highland Games (six individual events and the overall point total). In his essay, he spoofed himself, pointing out that he was the winner of all six events and the overall title at the first (and, at the time of application, only) high school version of the games. It was good enough to get to Lafayette, where he is exceptionally happy.
April 5, 2013, 1:17 pmDave:
"No way my son had a higher score than me"
That sounds like an SAT grammar question!
April 6, 2013, 4:33 amDaublin:
The mix problem is serious. Some school districts require all students to take the SAT; some stear you away from them unless they think you will do well on them.
As you say, if you want to go to a decent college, then you have to take SATs. So the more people who try to go to college, the more will actually take the test.
Good catch. It's fun to peer at a graph and try to pick something out of it, but if your data is just bad, you shouldn't even try.
April 7, 2013, 10:10 am