Editore"s Note
Tilting at Windmills

Email Newsletter icon, E-mail Newsletter icon, Email List icon, E-mail List icon Sign up for Free News & Updates

June 7, 2007
By: Kevin Drum

MORE NCLB MADNESS....Guess what? In today's education news, we get yet another report that compares the results from state tests to the results from the "gold standard" NAEP test. This one is from the National Center for Education Statistics, and here's the nickel summary: the standards used by states to measure compliance with NCLB are all over the map.

A picture is worth a thousand words, so take a look at the chart on the right. It shows how state standards compare to the NAEP "basic achievement level" for 32 states. This one is for fourth grade reading, and the state standards vary from 161 to 234.

Does this seem like a lot? Well, hold on: the rule of thumb for NAEP is that ten points is about equal to one grade level, which means that Mississippi at the bottom has a passing standard seven grade levels lower than the passing standard for Massachusetts at the top. Overall, more than half the states had passing standards a full grade level lower than the NAEP "basic" level.

So improved scores on state tests probably need to be taken with a grain of salt. Or a grain of statistics. Like this one:

There is also a negative correlation of -0.88 (with a standard error of 0.094) between the estimated NAEP score equivalents and the statewide percents proficient; that is, the larger the NAEP score equivalent, the lower the percent of students in a state deemed proficient.

Now, unlike yesterday's poor correlation between states that showed improvement on their own test vs. states that showed improvement on the NAEP test (suggesting that the improvements we're seeing in state test results might not be very meaningful), 0.88 is a very strong correlation. You won't see one much better in the social sciences. And what that paragraph means in plain English is that the easier the state test, the more students who pass it. No surprise.

None of this means that testing is useless or that NCLB has nothing to recommend it. But it does mean that glowing reports about soaring test scores should be greeted very cautiously. Caveat emptor.

Kevin Drum 12:34 PM Permalink | Trackbacks | Comments (24)

Bookmark and Share

Something does not add up in the 10 points = one grade level bit.

The chart is for fourth grade reading, and the Mississippi is seven grade levels lower, or negative third grade? The rule of thumb is wrong, unless the Mississippi test only looks for letter recognition.

Posted by: MobiusKlein on June 7, 2007 at 12:55 PM | PERMALINK

Ah, I understand now - This are testing systems for students, not CongressCritters - However, Mass versus Mississippi would appear to be correct in either case.

Posted by: thethirdPaul on June 7, 2007 at 1:02 PM | PERMALINK

Pretty much explains why Mississippi votes consistently Republican and Massachusetts goes consistently Democratic.

Posted by: The Conservative Deflator on June 7, 2007 at 1:02 PM | PERMALINK

Caveat emptor.
What was it Inigo said? "I don't think that phrase means what you think it means." Or something like that. . .

Posted by: decanus on June 7, 2007 at 1:13 PM | PERMALINK

yes it does. those of us in Massachusetts appreciate the value of a good education, while it appears those in "other" places do not value reality for their children.

I wonder how many kids who don't pass those ed tests and don't get diplomas will be left behind? Something stinks here, and it seems to be emanating from D.C.

Posted by: dejah on June 7, 2007 at 1:13 PM | PERMALINK

I believe that I know how the right kind of testing can greatly improvement the level of education, but the kind of testing mandated by NCLB doesn't do it.

The right kind of testing is mastery testing. Rather than having ambiguous testing on a broad subject area (such as mathematics as a whole, or language skills as a whole), use unambiguous testing on much narrower subject areas. It's easier to illustrate using mathematics, but I think something similar can be applied to other subject matters.

We can divide the subject of pre-college mathematics into a number of distinct skills (some of which depend on others). Starting with preschoolers, there is the skill of recognizing numbers, counting to 10, counting to 100, writing numbers, doing addition with 1-digit numbers, doing addition with multi-digit numbers, doing subtraction with 1-digit numbers, doing subtraction with multi-digit numbers, multiplication, division, fractions, negative numbers, etc. The topic areas should be organized by a pre-requisite relationship: it doesn't make any sense to try to teach division of multi-digit numbers if the child hasn't grasped multiplication of single-digit numbers.

The main point is that the topic areas should be narrow enough so that it is possible to completely master a topic area. I might not know everything there is to know about mathematics, but I believe that I have completely mastered the topic of adding, subtracting, multiplying and dividing decimal numbers.

The benefits of looking for mastery of narrow topics are that, first of all, it is much more unambiguous that a child has mastered a topic than it is that he shows "skills at the 4th grade level". Second, and more importantly, if you know what a child has mastered, then you know what he is ready to tackle next. Conversely, if you know what a child has not mastered, then you know exactly what he or she needs to work on.

With my own son, I've had the experience of trying to help him with a high-school algebra or trigonometry problem, only to find out that his difficulty had nothing to do with anything being taught in his current course---he was having problems because he never really learned how to work with fractions. Of course, he supposedly learned about fractions way back in the 5th grade (or whenever), but he never really mastered it. He did well enough to pass, but there were gaps in his understanding, and those gaps came back to haunt him in high school, when it was basically too late.

I think that if we insist on mastery of tiny little topic areas, rather than being "good enough" with much broader topic areas, a child's misunderstandings and education gaps will be surfaced much earlier, and they can be addressed immediately, rather than waiting 5 years when it is too late and the child has already concluded that he's "bad at math".

Insisting on mastery does not mean that every child must have exactly the same education. There can be plenty of flexibility in what a child masters at what age, and there can be complete flexibility in how a child is taught a topic. But (at least in "objective" subjects such as math) I don't think there needs to be much flexibility in what it means to have mastered a topic.

Posted by: Daryl McCullough on June 7, 2007 at 1:17 PM | PERMALINK

I remember a few months ago (or sometime last year) education officials in South Carolina were worried that their school system would be labeled as "failing" under NCLB because it had set its standards too high. Mississippi figured out how to game the system: write a state test that sets the bar so low, even the Ralph Wiggums pass with flying colors. Then nearly all the students appear on paper to be improving. The poor South Carolinians actually thought that NCLB was designed to encourage actual learning and push achievement higher. Idiots.

Posted by: jonas on June 7, 2007 at 1:20 PM | PERMALINK

And the states that aren't on there? How do they compare? (I assume as my own state isn't on there, that it's states that do not take part in NCLB).

Posted by: MNPundit on June 7, 2007 at 1:54 PM | PERMALINK

Americans have a dilemma. They value education and want their children to earn a quality education, but they have been flimflammed by W. Mugabe Bush into implementing means testing that does not contribute to their children's education.

If anyone was interested in improving their state's primary education, all they would have to do is imitate Massachusetts' public school system.

Posted by: Brojo on June 7, 2007 at 1:59 PM | PERMALINK
Well, hold on: the rule of thumb for NAEP is that ten points is about equal to one grade level, which means that Mississippi at the bottom has a passing standard seven grade levels lower than the passing standard for Massachusetts at the top.

"Rules of thumb" like that tend to be reasonable very close to the center of the distribution, and fall apart badly farther out. That seems incredibly likely in this case, especially if the "basic cut" score is taken to be a reasonable standard for a "4th grade level".

Posted by: cmdicely on June 7, 2007 at 2:19 PM | PERMALINK

Kevin, if you would have read the methodology then you would have been surprised that the correlation was as low as -.88. The methodology assumes a correlation of -1 for a subset of students. Thus the -.88 is just a measure of how much variation there is between that subset and the state as a whole. Moreover, I think you cherry licked the graphs. The vast majority of the states have pass rate cut-points above the NAEP basic level for math (4th and 8th grades) and reading (8th grade). Your points yesterday were spot on. Today's post is just misleading.

Posted by: rana on June 7, 2007 at 2:26 PM | PERMALINK

All states take part in NCLB. Many states were not evaluated in this study because their data was not as easily available or they don't require schools to give the NAEP.

Posted by: reino on June 7, 2007 at 2:36 PM | PERMALINK

This is just another example of the Republicans - conservatives trying to convince us that we would all be better off if we were more like Mississippi then Massachusetts. It does not compute.

Posted by: spencer on June 7, 2007 at 2:46 PM | PERMALINK

I'm sorry, did you say NCLB does have something to recommend it?

What would that be, precisely? Some sort of testing requirement? Was there a state that DIDN'T perform assessment tests before NCLB came along? Maybe you're referring to 'accountability.' NCLB's version of accountability is laughable: if you fail I'll take away your resources and then maybe you'll succeed; oh, and by the way, you're responsible for your students' performance, even though you only control one facet of your students' lives - their classroom time.

Puh-leeze. Somebody needs to stab this bill to death, and quick.

Posted by: cmac on June 7, 2007 at 2:51 PM | PERMALINK

Right, I covered this back in February:

"In the current NCLB, which was largely the result of an alliance between President Bush and Senator Kennedy (who are also the two leading advocates for "comprehensive immigration reform"—hmmm!). Each state is allowed to concoct its own test to determine whether its own students have reached "proficiency," which the state can define however it pleases.

"Not surprisingly, practically every single state cheats in order to meet the law. For example, Mississippi, that intellectual powerhouse, recently declared that 89 percent of its 4th graders were at least "proficient" in reading.

"Unfortunately, however, on the federal government's impartial National Assessment of Educational Progress test, only 18 percent of Mississippi students were "proficient" or "advanced."

"(The most honest state, surprisingly enough: Louisiana—with Missouri, Massachusetts, and South Carolina deserving honorable mentions.)

"Overall, the typical state claimed that 68 percent of its 4th graders were proficient readers, compared to the 30 percent found by the honest NAEP.


Bush and Kennedy built this massive conflict of interest into the NCLB: states get federal money based on their test score improvements, but they grade themselves!

Posted by: Steve Sailer on June 7, 2007 at 3:20 PM | PERMALINK

For what it's worth, some experts think the NAEP's standards for proficiency are irrationally high. (Gerald Bracey did an op-ed on this subject in the Post a few weeks back.) Inevitably, deciding how well a fourth or sixth grader "should" read involves a subjective judgment.

For this reason, it shouldn't necessarily be shocking if a given state's tests are "easier" than the NAEP. The problem comes when a state makes its tests easier from one year to the next (perhaps intentionally) without informing the public. Passing rates rise, giving the illusion of progress. But the passing rates may be going up just because the test has been "dumbed down."

Many states have shown progress on their own tests in the past decade--progress that isn't reflected when that same state's kids take the NAEP.

Posted by: bob somerby on June 7, 2007 at 4:31 PM | PERMALINK

So let me get this straight...MS dumbed down their tests so they could make it look as if their students were somehow better on tests? People are suprised by this why?

C'mon, you give a moron the ability to determine how smart he and his friends are they'll claim to be einstein.

I like Daryl McCullough's idea. That's exactly how things used to be done, at least when I was in school. Each year you focused on primarily one area of science, math, and English/language arts. That was to prepare you for junior high and high school.

In Junior High and High school it was again one year devoted to areas such as geometry, algebra etc. in math and working ones way up in English to a higher reading level, ie from "Hop on Pop" in kindergarten to Shakespeare by 12th grade and all tests were testing your knowledge of that, and only that subject.

Further HS courses prepared you for the SAT's and Regents exams (NY had regents curriculum and testing) that helped you get into college. You actually felt like you achieved something vs. feeling overwhelmed trying to master several concepts at once.

Posted by: Dreggas on June 7, 2007 at 4:36 PM | PERMALINK

Schools require three things to guarantee success: good administrators, good teachers and a critical mass of well-parented children. Schools lacking in any of the three, especially the last, are doomed to failure no matter the legislation.

Posted by: thinkonit on June 7, 2007 at 5:15 PM | PERMALINK

I only see about 30 states. Kevin, what are you hiding??

Posted by: Rula Lenska on June 7, 2007 at 6:30 PM | PERMALINK

"If anyone was interested in improving their state's primary education, all they would have to do is imitate Massachusetts' public school system."

It would be so much more effective to just imitate Massachusetts' demographics.

Posted by: Rory @ parentalcation on June 7, 2007 at 8:53 PM | PERMALINK

Actually, NCLB does have plenty to recommend it. Namely, it has forced the issue of failing schools into the harsh light of day, compelled the states and school districts to start doing something about it, and required teachers to be qualified and credentialed. The 50 state standard is a problem that DØE is trying to address by forcing the low performers to come up with a better test and standards. It has lots of flaws, but guess what, the status quo that existed before was so god awful in so many schools it couldn't possibly be worse. Instead of trying to kill it, it should be improved based on what we've learned, which it easiy can in the reauthorization.

Posted by: Hebisner on June 7, 2007 at 9:49 PM | PERMALINK

I am amazed at how no one thinks of the one thing that will save our children's educations! The Federal Government needs to get its nose out of the education field and leave that to the schools themselves. No one but the teacher and parents will/should know if that child is completely understanding the topic at hand. The teacher is the one that needs to make up the tests, grade them and then have the power to pass or fail that child. I am of the opinion that public education doesn't work, period - due to the simple fact that there are too many students enrolled. It is impossible, and always will be, to adequately teach a child when the teacher's attention must be distributed among 20-30 other students. Impossible. The school system has become a business machine. It stopped caring about the children decades ago. The gaps in children's education today is incredible! If there were to be one teacher per 5 students who truly cared deeply about each student; if that teacher taught all of the subjects (staying with the student over time allows the teacher to get to know that student deeply); if the curriculum involved community service and manual labor; if the testing and pass/fail were left up to the teacher (without fear of ever being sued); and if each and every family had a mother who stayed home with her children (at least till they graduate from high school,) and they had a loving, moral, upstanding home where there was no TV, nintendo, PS, GameBoy, etc., and where a love of learning was instilled, then I can practically guarantee you a near-perfect education system. Problem is - that doesn't exist, nor will it. So the closest thing we can do is to lower the student/teacher ratio by 300%; leave the testing/pass/fail in the teacher's hands without fear of being sued; have the teacher teach all the classes for those children, have the entire material from the curriculum actually taught (which it is not today,) and ensure that the teachers teaching truly care about the children, and not themselves or "the system" (and please, stop threatening them with state and federal testing!) - then we will begin to see progress. The way things stand right now - there are huge "Black Holes" in our children's education, and they aren't going away anytime soon. That is why we homeschool. We got sick and tired of the "school-business-machine" ruining our children. If my child does not completely "get" fractions, there is no way I am passing him to the next higher grade without going over fractions time and again, until we both are satisfied that he "GETS" fractions. No black holes in this homeschool! (Yes, sadly enough, there are people within homeschool as well as public school and private school communities who do not do their job adequately - but that's a story for another day...) For those of us who do it in a "bonafide manner", homeschool is THE way to educate our children today. (Not to mention the lack of negative socialization, and the huge amount of positive socialization they receive!)

P.S. there are so many things that go into reforming our educational system, that it's impossible to discuss them all in one "sitting." However, I'd like to add one more comment: REVERSE the paychecks between professional sports team members and teachers and that will help to create better teachers as well! All you have to do is take one look at our nation and see who we pay the most to... that will tell you what's important to us. Education is not important to the Nation as a whole; rather football, basketball, baseball, and the likes is what's important. I think sports are an important part of life; even necessary. But not to the tune of billions of dollars, while teachers are being paid (comparatively) pennies and our children are being "brainwashed" in the public school "machine" to turn out dumb voters who have no use of logic, who know no history, science, higher math or language skills and who are falling behind all other industrialized (and soon, third world too) countries! How sad is that!?

Posted by: Nine on June 9, 2007 at 3:06 AM | PERMALINK

The No Child Left Behind law measures schools based on how many students meet the minimum standards. Schools are so focused on decreasing the "achievement gap" between low and high performing students and reducing the drop out rate, a worthy goal, but this should not be at the expense of our nation's motivated, talented and high-achieving students. Many talented and gifted students meet or exceed the grade level standards on tests at the beginning of the school year and yet they do not receive any advanced classroom instruction. There is no incentive for schools to do anything for these students. Our local high schools have even eliminated some honors classes for high-achieving and motivated students. This has resulted in a diluted or watered down curriculum for our high-ability students. Bringing the achievement down at the top may make the schools look like they are reducing the achievement gap, but it ultimately hurts our children and our society. The NCLB law is leaving our nation's brightest students behind.

Posted by: Oregon parent on June 11, 2007 at 11:12 AM | PERMALINK

It's helpful to remember that there is no clear or consistent relationship between the rigor of the "passing score" on a state test and overall achievement in the state. See, for as an example


Posted by: Bert Stoneberg on August 19, 2007 at 7:55 PM | PERMALINK
Post a comment

Remember personal info?



Read Jonathan Rowe remembrance and articles
Email Newsletter icon, E-mail Newsletter icon, Email List icon, E-mail List icon Sign up for Free News & Updates

Advertise in WM

buy from Amazon and
support the Monthly