Editore"s Note
Tilting at Windmills

Email Newsletter icon, E-mail Newsletter icon, Email List icon, E-mail List icon Sign up for Free News & Updates

June 6, 2007
By: Kevin Drum

SOARING TEST SCORES....Here's the headline in the Washington Post today:

Test Scores Soar After 'No Child'

Now, this is a peculiar headline since the second paragraph of the accompanying story admits, "The study's authors warned that it is difficult to say whether or how much the No Child Left Behind law is driving the achievement gains." And indeed, the study from the Center on Education Policy (available here) goes to considerable pains to emphasize that the trend they're reporting started before NCLB was enacted. This, along with other factors, makes it very difficult to say whether, or how much, NCLB is responsible for the gains since 2002.

But put that aside for a moment. An even better question is: even if state test scores are rising, does that indicate that student achievement is also increasing? Bob Somerby suspects that rising scores might actually be due to dumbed down tests, and unfortunately, the study itself suggests he's right. Here's the paragraph that jumped out at me:

When the percentage of students scoring at the proficient level on state tests is compared with the percentage scoring at the basic level on the National Assessment of Educational Progress (NAEP), states show more positive results on their own tests than on NAEP. Moreover, the states with the greatest gains on their own tests were usually not the same states that had the greatest gains on NAEP.

Chapter 6 of the report goes into this in detail (see pp. 68-70), but the bottom line is that there is virtually no correlation at all between rising state test scores and rising NAEP scores — and like it or not, NAEP has long been considered the gold standard for consistent and reliable measurement of student achievement. In fact, the distribution of the scores is rather curious. As you can see in the chart on the right (for middle school reading results), there are some states where scores rose on both tests (top right) and some where they fell on both tests (bottom left). No problem there. It's what you'd expect if both tests were doing a decent job of measuring performance.

However, there are virtually no states that improved on NAEP but fell on their own tests. A rising NAEP score really does seem to indicate better performance that shows up no matter what test you take. Conversely, there are loads of states that showed improvement on their own test even though results fell on NAEP. Peculiar, no? It's almost as though the states tests aren't really testing actual performance very well.

The report suggests several reasons why the results of state tests might not align with NAEP, and score inflation is one of them. More important, I suspect, is the first reason they list: alignment with state curriculum standards. State tests are designed to be tightly constrained to state curriculum standards, and teachers are tightly constrained to teach precisely to those standards. From the report:

For example, Jacob (2007) found that for Texas 4th graders, the difference in scores between NAEP and the state exam was largely explained by differences in the set of math skills (multiplication, division, fractions, decimals, etc.) covered by the two tests, and the format in which test items were presented (word problems, calculations without words, inclusion of a picture, etc.).

Obviously kids are going to do better on a test that perfectly replicates what they're familiar with from class. Frankly, though, fourth graders are all taught basic arithmetic, and if merely making small changes in the format of the problems causes the NCLB gains to disappear, then NCLB isn't doing much to genuinely improve basic skills. More data, please.

Kevin Drum 1:54 PM Permalink | Trackbacks | Comments (63)

Bookmark and Share
 
Comments

I think a big part of any improvement is that the schools are aggressively 'targeting' the tests in their curricula.

This is to say, the proverbial tail is wagging the dog.

I guess that's all fine and well if the tests were an accurate indicator of overall educational quality... but right now the educational programs are being aimed at the test itself so it's pretty hard to tell the difference.

This is an interesting and probably predictable consequence of the mandatory testing and financial consequences attached to student scores.

Posted by: Buford on June 6, 2007 at 2:11 PM | PERMALINK

More data, please.

Spoken like a true liberal. Sometimes in life we don't have all the information we'd like to have, and have to make decisions based on what we know. Because liberals normally don't hold important decision-making positions, they don't understand this.

Posted by: Al on June 6, 2007 at 2:13 PM | PERMALINK

Al, wrong as usual.

Buford, as a Texas community newspaper editor, I have heard the phrase "teaching to the test," or "teaching to the TAKS" (the Texas state standardized battery) for years. Despite superintendents' disclaimers, it's real and it happens regularly.

That said, Texas is getting rid of the TAKS system and going toward a set of exit-level examinations. But, you may have the same problems with that, at least to a degree.

Posted by: SocraticGadfly on June 6, 2007 at 2:18 PM | PERMALINK

I keep telling people, but nobody is listening: when performance measurements are implemented, everyone suddenly spends all their time trying to make their performance look good on paper, which rarely corresponds to good performance in the real world.

Posted by: charlie don't surf on June 6, 2007 at 2:19 PM | PERMALINK

I have a lot of confidence in Bob Somerby's opinion in this area. He was a former school teacher in Baltimore. He knows first hand the kind of games schools play (not to mention outright cheating) to boost their students' test scores. He's been looking a this sort of test for a long time.

IMHO the fact that we can't get reliable test scores from schools is a scandal. To me, it's parallel to a corporation that fails to accurately report its financial results. A mis-reporting corporation is cheating its kholders. A mis-reporting school is cheating its students.

Posted by: ex-liberal on June 6, 2007 at 2:23 PM | PERMALINK

When you pay states extra money to have kids pass a test then you would expect they would do everything they can to get the kids to pass the test.

These things probably include:
1) teaching better (not easy to do)
2) teach to the test
3) help kids cheat on the test
4) make the test easier
5) try to eliminate the dumber kids from taking the test

I could go on but it seems that the most difficult thing to do, teach better, is not the only thing the schools are considering.

So why would you expect the laws of math to trump the law of self preservation?

Posted by: neil wilson on June 6, 2007 at 2:26 PM | PERMALINK
IMHO the fact that we can't get reliable test scores from schools is a scandal. To me, it's parallel to a corporation that fails to accurately report its financial results.

Its not really analogous. There are clear, objective, accepted standards for financial reporting that can be applied.

There are not clear, objective, accepted standards for what is being measured in, e.g., reading proficiency tests.

Posted by: cmdicely on June 6, 2007 at 2:34 PM | PERMALINK

"More data, please.

Spoken like a true liberal."

Well, yes.

Posted by: Matt on June 6, 2007 at 2:35 PM | PERMALINK

cmdicely: There are not clear, objective, accepted standards for what is being measured in, e.g., reading proficiency tests.

True, but there are some standards for what not to do. E.g., Somerby has described schools or classes where the teacher gave the studentsthe answers the day before the test.

Posted by: ex-liberal on June 6, 2007 at 2:40 PM | PERMALINK

With three kids in elementary and middle school, I can tell you that "teaching to the test" is rampant -- to the point where the bright and even not-so-bright kids are simply bored to tears going over and over the same basic literacy and math crap again and again, at the expense of history, science, music, art, etc. etc.

Posted by: Steve on June 6, 2007 at 2:44 PM | PERMALINK

Rarely is the question asked, "Is our students learning?"

Posted by: Your CommanderGuy-in-Cheef on June 6, 2007 at 2:44 PM | PERMALINK

Al on June 6, 2007 at 2:13 PM

F*cking brilliant!

Posted by: thersites on June 6, 2007 at 2:50 PM | PERMALINK

Thanks, Kevin; one of Political Animal's virtues is that you actually present quantitative information and not just the usual rhetorical derring-do. Not common outside of econ and other specialist blogs.

Posted by: idlemind on June 6, 2007 at 2:56 PM | PERMALINK

With respect to Texas, there is the additional problem of a large, raucous block of social conservatives who in Houston forced out nationally-normed testing for years because some of the reading comprehension questions presented information or situations (i.e., biology, evolution, behavior deemed offensive by Religious Right wackjob wingnuts, etc.)that offended their sensibilites. Ditto on school textbooks: the textbook committee in this state has been dominated for a long time by yahoo bible fundamentalists. Since publishers cannot economically prepare regionally-bowlderized editions of textbooks, guess what large, populous state is the tail wagging the dog nationwide to enforce the extreme right's agenda in the schools?

Posted by: biosparite on June 6, 2007 at 3:05 PM | PERMALINK

With respect to Texas, there is the additional problem of a large, raucous block of social conservatives who in Houston forced out nationally-normed testing for years because some of the reading comprehension questions presented information or situations (i.e., biology, evolution, behavior deemed offensive by Religious Right wackjob wingnuts, etc.)that offended their sensibilites. Ditto on school textbooks: the textbook committee in this state has been dominated for a long time by yahoo bible fundamentalists. Since publishers cannot economically prepare regionally-bowlderized editions of textbooks, guess what large, populous state is the tail wagging the dog nationwide to enforce the extreme right's agenda in the schools?

Posted by: biosparite on June 6, 2007 at 3:06 PM | PERMALINK

And what's going on with AL? Their results seem least reliable by a good margin.

Posted by: Jim Lund on June 6, 2007 at 3:16 PM | PERMALINK

How easy do you think it is to memorize a set of "answers" given to you the day before a test? If middle school kids could do that readily, they could learn the course material in the first place. To some extent, learning consists of memorizing information to be used later. How motivated are kids going to be to learn a set of disembodied answers, given that there is no connection between the answers and anything meaningful to the students?

I can believe a teacher might try to do this, but I can't believe it would work any better than trying to teach kids what they need to know to do well on the test (teaching to the test), or teaching the curriculum (teaching to a criterion).

If teachers teach to a test while at the same time the test becomes tailored to what teachers are teaching, isn't there an interactive narrowing spiral of ever-decreasing breadth of content? That would explain why NAEP predicts state tests but not vice versa. Over time, there will be less and less overlap between the two, with the state tests eventually becoming a small subset of the curriculum of the NAEP test. That means even the best-prepared top students in a state are not taught broadly enough to do well on the national test. It also means the national test will not have enough depth in the specific curriculum of the states to distinguish among the state's students. You are criticizing the state exam, but the national exam is also flawed if it goes for curriculum breadth instead of assessing depth (providing a high enough ceiling for state students with higher accomplishments in that state's curriculum). Why the bias in favor of the NAEP? Why do we assume that breadth of knowledge should trump depth?

No one seems to worry about such curriculum narrowing when it happens at the college level and professional schools prepare students for state-specific licensing exams.

Posted by: Perry on June 6, 2007 at 3:25 PM | PERMALINK

Well, see, the dumbing-down of the tests and he rise in teaching to the test, which are responsible for the higher scores, DID come in response to NCLB, so the headline is technically correct! So what that it's correct in a way that's thoroughly misleading and useless. Picky, picky, picky! ;)

Posted by: Steve LaBonne on June 6, 2007 at 3:37 PM | PERMALINK

Why do we assume that breadth of knowledge should trump depth?

No one seems to worry about such curriculum narrowing when it happens at the college level and professional schools prepare students for state-specific licensing exams.

Because some breadth of general knowledge is essential in a democracy. You're assuming that the sole purpose of education is job training, and that's a growing tendency that I find unhealthy.
For example, I write software. At at some point in my later education there needed to be an emphasis on computer crap, at the expense of other subjects.
But without a broader foundation in my earlier education, with a fair amount of history literature etc., I'd be a less-well-informed citizen.

Posted by: thersites on June 6, 2007 at 3:46 PM | PERMALINK

shocking. I am shocked. turns out when you take an AP class, you do better on the AP test. When you take a Kaplan class (or hire me to tutor you) you do better on the SAT. and when you spend two hours a day for five months practicing to take a test, your score go up. It's what my school made me do as a fifth grade teacher.

northzax, who has never taken a test prep class in his life (although he now teaches them), has never scored below the 99th percentile on any standardized test, and kinda wishes that real life was based on testing ability, AND STILL KNOWS TESTS ARE BULLSHIT.

Posted by: northzax on June 6, 2007 at 4:02 PM | PERMALINK

"Test Scores Soar After 'No Child'"


Ad hoc, ergo propeter hoc.

Posted by: Rula Lenska on June 6, 2007 at 4:08 PM | PERMALINK
No one seems to worry about such curriculum narrowing when it happens at the college level and professional schools prepare students for state-specific licensing exams.

Probably because the number of people aware of and following issues in professional education and licensing is much smaller than the number of people aware of and following issues in K-12 education, and (consequently) the forum in which most debates about the former occur is not the mainstream media and major political campaigns, but comparatively obscure professional fora.

Posted by: cmdicely on June 6, 2007 at 4:12 PM | PERMALINK

No one seems to worry about such curriculum narrowing when it happens at the college level and professional schools prepare students for state-specific licensing exams.

That's because it is a fundamentally different sort of narrowing.

Narrowing the curriculum by teaching fractions more, and Roman numerals less, is narrowing in a basically-plausible fashion. "Narrowing" the curriculum by teaching students to solve fraction problems with a calculator, rather than by hand, is more accurately called "gaming the test system" (since ability to work with fractions manually is critical for algebra).

Posted by: SamChevre on June 6, 2007 at 4:28 PM | PERMALINK

The differnce between results on state tests and the NAEP can be attributed to three factors:

1) More teaching to the test.
2) More cheating on state tests.
3) More dumbing down of state tests (more examples than I even want to think about)
4) More statistical exclusion of students who are likely to do poorly on state tests (by labelling them special ed. or ESL, for example).

All this just points out that measuring student achievement solely by standradized testing is pretty silly....it's far too easy to massage the numbers to have them say anything you want by fiddling with the pool of students and teaching to the test.

Posted by: mfw13 on June 6, 2007 at 4:53 PM | PERMALINK

It's very simple: a gigantic conflict of interest was built into NCLB by Bush and Kennedy. The NCLB demands that all states make all students "proficient" (i.e., above average on a scale of far below basic, below basic, basic, proficient, advanced) by 2014 or lose lots of federal aid. Obviously, this is impossible without massive fraud. However, fraud is built into the NCLB because the states are put in charge of making up their own tests and grading them themselves!

Posted by: Steve Sailer on June 6, 2007 at 4:57 PM | PERMALINK

If your final in an AP Calculus test is the same final that the students at the local Big Ten university take, chances are that the grade you earn is real. Sure, the teacher may have known what the students need to expect on the test, but they also had to give the students an idea what is on the AP test and cover the curriculum.

Tests are a great way to provide feedback to students and teachers, but having a testing system like the one used in NCLB is fundamentally flawed, moreso than the inherent flaws in all testing that northzax pointed out. The biggest flaw in the NCLB system is that no one directly involved in giving the tests knows that they have an incentive to get honest results. The students might want honest results, because, ideally, it would be to their benefit, but they most likely do not know that.

Teachers and schools, even good teachers and schools, are given perverse incentives: "If your students do badly on this test, we'll punish you for failing," almost guarantees that the integrity of the testing system will fail.

Posted by: freelunch on June 6, 2007 at 5:08 PM | PERMALINK

How about a scientific survey of college professors to see whether or not their students can write well?

Posted by: Jalmari on June 6, 2007 at 5:09 PM | PERMALINK

5) try to eliminate the dumber kids from taking the test

Does that explain all the extra field trips I went on back in my school days?

Posted by: ThresherK on June 6, 2007 at 5:20 PM | PERMALINK

First of all, there is no need for more data. The NAEP shows clearly that NCLB is a complete failure. If Al wants to argue with one of Kevin's sentences, that's fine, but he can't argue that NCLB has made schools better.

Second of all, the explanation is much simpler and more sinister than the one Kevin presents. The states are making the tests easier and changing the standards for which scores should be counted. In Illinois this year, students who had switched schools in the past three months did not have their scores counted. Students in unstable situations generally test worse than students in stable situations, so our state test scores went up. We still have the same teachers doing the same things with the same students in the same schools as last year, so our NAEP scores didn't go up.

State standards don't amount to much. Good schools ignore them, and bad schools go crazy over them while remaining bad.

Posted by: reino on June 6, 2007 at 5:26 PM | PERMALINK

I don't understand the logic to the position that it's bad to target the test to the curriculum. The curriculum is what we've decided the kids should learn. It would be perverse to test them on anything other than that.

And by the same logic, if the NAEP test is not geared to the curriculum the states have decided is best, why should it be the "gold standard" (leaving aside all of the problems there are with trying to compare different states with different demographics and test-taking patterns).

Posted by: RM on June 6, 2007 at 5:27 PM | PERMALINK

RM--

As a general principle, you are correct that the test should be geared to the curriculum.

However, why should the physics curriculum in Wisconsin be different than the physics curriculum in Minnesota? Also, the curriculum tested on the NAEP was honed by many educators over many years and is generally stronger than the state curricula hammered out quickly by political appointees.

Posted by: reino on June 6, 2007 at 5:39 PM | PERMALINK
Frankly, though, fourth graders are all taught basic arithmetic, and if merely making small changes in the format of the problems causes the NCLB gains to disappear, then NCLB isn't doing much to genuinely improve basic skills. More data, please.

When you work with young kids you discover that, in most cases, small changes in the format of the problems actually almost do make them into different problems. Kids that young don't have much ability to extend their skills / knowledge to new areas, to generalize it, in other words.

Also, the true basic skills are never isolated and taught. Arithmetic, from a kid's point of view, is not a basic skill, even though an adult might see it as a basic skill. Kids think differently from adults.

Posted by: pmacfar on June 6, 2007 at 5:54 PM | PERMALINK

Freelunch,

Thank you for your post. It answered one of my questions.

But, I have another questions for you (and everyone else).

1. Why is it that people (in this case, educators, people who write about education, scholars, politicians, and parents) like certain programs where there is a test at the end (such as Advanced Placement and International Baccalaureat) but disapprove of other tests (such as NCLB)? Is standardized testing (I don't know if IB or AP comes under "standardized testing") inherently wrong, or is it just that the NCLB program is so bad?

I took AP classes, which "taught to the test" to a certain extent. And we did learn a lot. But what is the problem with NCLB. Is it lack of funding for the goals? The punishment? If they changed the testing structure and requirements, would it be acceptable (In other words, is the principle of the tests okay, but the execution is poor?)?

Thank you. And I apologize if these were addressed in a previous post.

Posted by: adlsad on June 6, 2007 at 5:57 PM | PERMALINK

Are state tests being "dumbed down"--i.e., being made easier from one year to the next? Good education journalism should be able to tell. The states should be creating technical manuals explaining how they know that the tests are "equivalent" (equally hard) from year to year. And journos should be examining them.

Of course, this rarely happens, if ever. This question has arisen in New York state in recent years, and the NYT has made no serious attempt to find out what is happening. Simply put, they don't seem to care if the tests are equivalent from one year to the next. They do seem to love those feel-good testing stories.

RE the question of teaching the particular state curriculum: This would likely affect math testing more than reading. Different states actually could teach math in different sequences. But reading is pretty much reading. I don't see how different states would have different skill sequences for reading instruction in a way that would affect test scores much.

Posted by: bob somerby on June 6, 2007 at 6:01 PM | PERMALINK

By the way, I'm thrilled that Kevin followed up on this. More examination of data, please!

To what extent have tests been surreptitiously "dumbed down?" The question is very important.

Posted by: bob somerby on June 6, 2007 at 6:03 PM | PERMALINK

Test scores are bound to improve when teachers start teaching to test material, just as anyone who takes SAT, GMAT, LSAT or GRE prep courses is sure to improve his score on those exams. It doesn't mean, however, that anyone's is actually learning anything other than getting better at test taking.

Unless you have nationally standardized funding for schools, a nationally standardized curriculum, and nationally standardized tests, it is impossible to determine whether an initiative like No Child's Behind is having a positive effect on primary and secondary public education in the U.S.

Posted by: JeffII on June 6, 2007 at 6:13 PM | PERMALINK

NCLB demands that all states make all students "proficient" (i.e., above average...

Is this true? When the day comes that everyone is above average we'll know it's all over for math literacy, and maybe language skills too.

Posted by: thersites on June 6, 2007 at 6:21 PM | PERMALINK

adlsad--

Good question. Different tests are developed for different reasons and have different purposes. The AP Calculus Test, for example, is a good test to determine whether an incoming college student should start with introductory calculus or with advanced calculus, but it is not a good measure of a school's overall math program or even a school's calculus program (because poor scores could be due to poor precalculus courses and high scores could be due to overzealously weeding out average students from taking the test).

I teach at a suburban public high school with very high test scores. Our juniors take the NCLB tests and have great scores. However, if we gave the tests to our incoming freshmen the first week of school, they would do pretty well on the tests--not as good as our juniors but better than the juniors at a lot of other places. Therefore, the tests are not telling us whether or not we are doing a good job--they are just telling us that almost all of our students are above average, which everybody already knows.

Similarly, there are lots of schools in troubled areas where test scores are below average and everybody knows it. NCLB does not determine which schools are doing the best they can under difficult circumstances and which schools are dysfunctional, yet it metes out harsh penalties for schools that don't measure up to it.

In Illinois, half of the NCLB test given to high school students is the ACT. The company that produces the ACT does not believe that there test is a good way to determine which schools are successful and which schools are not--they designed their test to predict college performance for individuals. Therefore, the test is being used for a purpose it is not designed for, and it does not serve this new purpose well.

Posted by: reino on June 6, 2007 at 6:30 PM | PERMALINK

adlsad,

I cannot speak to the other AP exams, but as an AP Calculus teacher who also teaches Algebra II (which ran trial runs of and End of Instruction Exam this year), I can say that the tests are pretty wildly different. This occurs for several reasons. I will discuss what I consider the two main ones below.

1. The College Board, who administers the AP exams, is very meticulous about the integrity of their product. They are a for profit company whose livelihood depends on their test results being accurate indicators of whether or not students who have completed a certain course meet college standards of education. They spend a lot of resources both on content development and on test security.

2. The AP exam tests to mastery, not to proficiency. The expectation is that the students not only have the process skills and a grasp of the basic principles in a topic, but that they have also developed a deep understanding of the concepts involved and can apply them in many varied contexts. I often tell my students that if "teaching to the test" just means showing them how different problems will be presented and what processes to use on each type, then there is no such thing as teaching to the AP Calculus test.

We do not teach our standard Algebra II students to the mastery level. It is unlikely that all or even most students are capable of that level of understanding in several topics. Moreover, even if it were possible, it would be a large misallocation of resources. We don't NEED students who are gurus in every topic. Nor do students need that themselves. What we need, at a baseline, is students who are "proficient" in many topics and can be taught to specialize at a later date.

However, "proficiency" is very hard to test accurately, and is much more likely to be thrown off by outside factors, such as test wording. As such, testing for proficiency tends to narrow the pool of concepts and the way they are taught instead of adding depth to the level of instruction, excepting of course those courses where instruction was already more rigorous in preparation for AP courses to be taken at a later date.

Posted by: socratic_me on June 6, 2007 at 6:37 PM | PERMALINK

NCLB scores prove nothing, since the states individually determine how difficult or easy the test will be, knowing that if they make it too difficult they will be slapped with costly mandates for which the feds will contribute nothing. In addition, the states are free to change their tests anytime to ensure that a higher percentage of students meet the annual improvement requirement.

Until a national standard is established and there is some flexibility to recognize improvement, the NCLB will continue to be counterproductive. I know many schools that pass their annual review but have shown no improvement. Their scores were high to begin with, so as long as they don't fall below the arbitrarily established cutoff, they don't need to improve.

On the other hand, I know of many schools serving economically disadvantaged students who have seen dramatic improvements year over year in scores, yet they continue to be labled "failing" and have been hit with penalties -- sometimes after improving their scores by double digits, yet falling one point short of the bar.

Posted by: DevilDog on June 6, 2007 at 6:42 PM | PERMALINK

I am concerned that when a national test is established, then a national curriculum will also be established. That will even out the quality of education, benefitting states with worse systems. However, given the way every area of government has been corrupted for political purposes by this administration, a national curriculum could be turned into a national system for indoctrination of children. I doubt any of us want that. So, this first step of establishing national testing seems like a slippery slope to me.

Posted by: Perry on June 6, 2007 at 6:54 PM | PERMALINK

Some good points so far, I'd like to reiterate the boredom factor for above average kids. The way we have seen "teaching to the test" is to gear the class to the 3rd quartile student.

Overall the tests are not that difficult, so on average half the class is going to pass no matter what (scale up or down for particular schools). The 4th quartile student is always going to have problems passing the test. But the biggest bang for the buck is getting those somewhat below average students to pass the test. Pretty boring for the above average students though...

The other way we see narrowing, that is specific to NCLB is what happens to schools that are marked as "failing". In that case, the schools are required to put students though an intensive remedial program that emphasizes the basics (reading, math). This means schools have to cutout those worthless elective subjects, like science for example!

Our school district had some major problems when students coming from a "failing middle school" were not able to take (normally required) courses in science. Now the high schools had to add remedial science courses to try and catch-up those poor students who had been caught in the bureaucracy. There were other subjects that had been cut-out but science was the one that really stuck in my head (perhaps because of our proximity to Silicon Valley).

Not only can this narrowing directly hurt the future educational prospects of the student, but it also removes an alternative path of motivation. We all can probably think of examples where so-called electives motivated otherwise poor students (music, sports, art, etc.). Personally I had a hard time writing (additionally handicapped by poor penmanship and spelling) until I took a journalism class. Writing became a lot more fun after that (although my history teachers often commented that my essays had sentences that were short and full of action :-)

Posted by: seaan on June 6, 2007 at 7:00 PM | PERMALINK

Reino and Socratic_Me,

Thank you for your responses to my question(s). I guess that I have a few follow up question. I would also like to take this time to apologize for my lack of knowledge on NCLB.

Reino,

Are there any incentives for students at your school to test well on the NCLB? In other words, is there any reason to work for a higher than score than just a passing mark? Or is that what the ACT portion is designed to do?

Socratic_Me,

Is there any way to improve, or to raise the standards of, "proficiency?" For example, would it be possible, in the future, for all high school students to take Calculus (I don't know what the Mathematics requirement for High School gradutates is now)? Are standardized tests for "proficiency" completely useless?

Again. Thanks for the responses.

Posted by: adlsad on June 6, 2007 at 7:09 PM | PERMALINK

adlsad--

One of the advantages of making half the test an ACT is that students can use their score on that half of the test as their ACT score to apply to college. Because colleges in the Midwest like the ACT and our students are almost all college bound, they have a big incentive to do well on that part of the test. (I went through the downsides above, the biggest of which is that the test is used to judge our school even though it does not provide a meaningful measurement of our school.)

The other part of the test is so easy for most of them that they do well even when they don't try very hard. Also, failing scores get mentioned on transcripts, and students want to make sure that they don't fail.

As a high school math teacher, I'll try to answer your question to Socratic_Me as well...

There will be no time in the future when all students can do well on the AP Calculus Test. For one thing, it is too difficult a test. For another thing, it is better to have students who struggle with Algebra to spend more time on Algebra than to have them move on to Calculus. Students should be comfortable with a variety of different functions (linear, quadratic, polynomial, exponential, logarithmic, trigonometric) in a variety of formats (algebraic, graphical, and in a table), and they should be able to connect those formats, before they study Calculus. It's analagous to asking whether we'll need to raise basketball hoops once everybody learns how to dunk.

Posted by: reino on June 6, 2007 at 7:36 PM | PERMALINK

I am a middle-school Language Arts teacher. I imagine that there exist certain principles, and even entire school districts, where cheating happens. But the truth is, it is very difficult to "cheat" on these high-stakes, standardized tests. Encourage "slow" kids to "not show up" for the tests? It doesn't work. If an insufficient percentage of our enrolled students take the test, we will FAIL AYPs, no matter how high our scores are.


"Give kids the answers"? No chance, in my school district. If anyone--including the test administrator--opens a test booklet early, her group will have to start over with a new test, and the administrator (who is required to be a certified teacher) will be fired. How can it be proved? Again, simple: I don't administer the test alone. I have a proctor, who--as often as possible--is also a certified teacher, and part of his job is to make certain that I don't cheat. I have never once opened a test booklet before my students did--and my students have never opened a test booklet prior to the scripted moment during the actual administration of the test.

I believe that most of the people who talk about "teachers who give the answers in advance," or find other ways to cheat, are simply propagandists--or worse, liars. The system is simply juiced against cheating. People who don't understand this, don't know anything about NCLB.

Nevertheless, single-administration, high-stakes tests do not measure anything worthwhile. They certainly don't measure actual achievement. Let me put it this way: I have never, and neither has any teacher I have ever known, ONCE said, "Well, I thought Kimmy was a good reader, because she summarizes what she reads so eloquently, reads for pleasure, and gets excited about books and stories. But since she didn't get a
Level 3" on the state tests, it's clear that Kimmy is a failing student."

Do I teach kids strategies for taking multiple-choice tests? You bet! When I was preparing to take the GRE, I paid a lot of money to people who taught me the same principles. Do I "teach to the test?" Not if I can help it. My goal is to help students to generate their own questions, and then pursue their own answers. I don't want to raise a bunch of test-bots. If I only cared about scores, I wouldn't be earning $30,000 per year and working 60-hour weeks. I didn't go into teaching to get rich, or to win awards, or to sit in the balcony at the president's next State of the Union address. I went into teaching because the rich, detached guys who think up these idiotic educational obstacle courses WON'T. If any of these policy-makers would spend a year in a public middle-school, I would take their proposals seriously.

But they won't. And so, neither will I.

Posted by: Lisa Rathert on June 6, 2007 at 7:41 PM | PERMALINK

I am a middle-school Language Arts teacher. I imagine that there exist certain principles, and even entire school districts, where cheating happens. But the truth is, it is very difficult to "cheat" on these high-stakes, standardized tests. Encourage "slow" kids to "not show up" for the tests? It doesn't work. If an insufficient percentage of our enrolled students take the test, we will FAIL AYPs, no matter how high our scores are.


"Give kids the answers"? No chance, in my school district. If anyone--including the test administrator--opens a test booklet early, her group will have to start over with a new test, and the administrator (who is required to be a certified teacher) will be fired. How can it be proved? Again, simple: I don't administer the test alone. I have a proctor, who--as often as possible--is also a certified teacher, and part of his job is to make certain that I don't cheat. I have never once opened a test booklet before my students did--and my students have never opened a test booklet prior to the scripted moment during the actual administration of the test.

I believe that most of the people who talk about "teachers who give the answers in advance," or find other ways to cheat, are simply propagandists--or worse, liars. The system is simply juiced against cheating. People who don't understand this, don't know anything about NCLB.

Nevertheless, single-administration, high-stakes tests do not measure anything worthwhile. They certainly don't measure actual achievement. Let me put it this way: I have never, and neither has any teacher I have ever known, ONCE said, "Well, I thought Kimmy was a good reader, because she summarizes what she reads so eloquently, reads for pleasure, and gets excited about books and stories. But since she didn't get a
Level 3" on the state tests, it's clear that Kimmy is a failing student."

Do I teach kids strategies for taking multiple-choice tests? You bet! When I was preparing to take the GRE, I paid a lot of money to people who taught me the same principles. Do I "teach to the test?" Not if I can help it. My goal is to help students to generate their own questions, and then pursue their own answers. I don't want to raise a bunch of test-bots. If I only cared about scores, I wouldn't be earning $30,000 per year and working 60-hour weeks. I didn't go into teaching to get rich, or to win awards, or to sit in the balcony at the president's next State of the Union address. I went into teaching because the rich, detached guys who think up these idiotic educational obstacle courses WON'T. If any of these policy-makers would spend a year in a public middle-school, I would take their proposals seriously.

But they won't. And so, neither will I.

Posted by: Lisa Rathert on June 6, 2007 at 7:42 PM | PERMALINK

Yikes, I noticed that I had written "principles" when I meant "principals" at the very instant when I hit send. Just to let you know, I really AM a middle school teacher, and I do know the difference between these two words!

Stupid homophones!

Posted by: Lisa Rathert on June 6, 2007 at 7:51 PM | PERMALINK

How refreshing to read a post that reflects statistical literacy!

Posted by: Vaughan on June 6, 2007 at 7:53 PM | PERMALINK

Lisa Rathert: I believe that most of the people who talk about "teachers who give the answers in advance," or find other ways to cheat, are simply propagandists--or worse, liars. The system is simply juiced against cheating. People who don't understand this, don't know anything about NCLB.

I'm glad to hear this. I seem to recall an old column where Bob Somerby mentioned a case he knew of where the teacher had access to the test and went over specific test questions with her class the day before. I'm relieved to know that this sort of thing is not possible under NCLB.

Posted by: ex-liberal on June 6, 2007 at 8:14 PM | PERMALINK

Sorry, ex-liberal.

I have not proctored the test for a few years because of a scheduling issue, but it would have been very easy for me to cheat. There sometimes is only one teacher in the room, and that teacher could point out mistakes that students are making or wait a few extra minutes before calling Time. Additionally, the teacher sometimes walks the tests back to the testing center, and some of the answers could change during that time.

I have never cheated, and I don't think cheating goes on at my school (we don't proctor our own students and do fine without cheating), but there are opportunities to cheat, and I would guess that some people take advantage of those opportunities. The penalties are harsh when people get caught, but I think that a lot of people cheat without getting caught.

Posted by: reino on June 6, 2007 at 9:01 PM | PERMALINK

Thanks, Ex-Lib, and Reino. I didn't express myself well. Widespread cheating? If most teachers are cheating, then I am teaching in an unusual school. It's just that my experience is nothing like Somerby's or even yours, Reino. Even if I wanted to cheat, I have never had the opportunity.

Ex-Lib: how clever of you, to discount my comments with a bit of sarcasm. I bet you believe that Al Gore actually claimed to have invented the Internet.

But what I'm really trying to say is this. What kind of teachers care so little about their profession, and their charges, that they would gladly cheat, if given the chance? Ex-Lib, is that what you think of ministers and rabbis? CEOs? Senators? Presidents? In your world, will everyone cheat, given the chance, if it's at all possible?

Reino, I can only say that in my school, the situation you describe simply doesn't happen. I have never given one of the standardized tests by myself. I am not even allowed to have the test booklets in my possession until the morning of the test, and even so, only with another teacher monitoring what I do.

Some other schools may promote cheating; I don't know. I've worked in three school districts in my state, and nothing like you describe has ever been the case.

Posted by: Lisa Rathert on June 6, 2007 at 9:52 PM | PERMALINK

Since the passage of NCLB, schools have become standardized test preparation factories instead of places of learning, so it's not surprising that test scores would be going up. However, ask virtually any teacher whether NCLB has made kids smarter or better prepared for life, and their answer will be almost unanimously - NO!!!

If we paid teachers half of what we pay the 100,000 mercenaries we now have in Iraq, we might be making some real progress in educating our children.

Posted by: The Conservative Deflator on June 6, 2007 at 10:08 PM | PERMALINK

Curriculum narrowing is such a nice little buzz word. Of course no one talks about the curriculum broadening that happened over the last couple of decades.

Our schools teach the same things, to the same students, year after year, hoping that if it's repeated enough, some of it will sink in. Of course this is a very inefficient manner of teaching. It's half ass year after year.

K-3 should concentrate on basic reading and math skills.

4-8 grades should be used to gradually increase students background knowledge.

9-12 grades should then begin to merge the background knowledge, put it in to context, and teach critical thinking.

Instead our schools try and teach critical thinking to 8 year olds, basic reading skills to 8th graders, and background knowledge to Juniors in High School.

The discrepancy between State scores and NAEP performance has been well covered by the education blogosphere, including take-downs of several NYC articles.

For example.

http://d-edreckoning.blogspot.com/2007/03/madison-cooks-books.html

http://d-edreckoning.blogspot.com/2007/03/schemo-responds.html

http://rightwingnation.com/index.php/2007/03/21/3109/

http://rightwingnation.com/index.php/2007/03/17/3098/

Its amazing to me that the political blogosphere only cares about education when it can be used to score points against which ever political party they are against. The rest of the time, all that interests anybody is whether some guy named Scooter is going to serve any prison time.

Meanwhile there are thousands of parents out there railing against the system trying to encourage the smallest amount of reform.

Peace, I'm outie.

Posted by: Rory Hester on June 6, 2007 at 10:49 PM | PERMALINK

Lisa Rathert, please don't blame me for those comments about cheating. I would love to believe trhat George Bush's NCLB is working.

I was quoting Somerby, just as Kevin was. Somerby is a liberal, and, by the way, was Al Gore's roommate at Harvard.

Posted by: ex-liberal on June 6, 2007 at 11:01 PM | PERMALINK

Lisa Rather, it's obvious from your comments you are a good teacher who invests a lot of time and energy in your work. Your students are lucky.

Ex-liberal, you don't know what the fuck you're talking about. It would be obvious if a school or class cheated because it would create a huge anomaly in the test results, way too conspicuous to hide. You can quote all the right wing blogs, or any other sources, you care to, but the people who work in the profession are the experts. They are on site and administer the tests.

Your lack of knowledge makes your commentary on this subject laughable. And the transparency of your agenda only further undermines your credibility.

Posted by: DevilDog on June 7, 2007 at 12:12 AM | PERMALINK

DevilDog, please pay attention. I quoted a LEFT WING blogger, the same one Kevin did in his original post.

I believe Bob Somerby knows what he's talking about. He was an inner city school teacher after graduating from Harvard. He has thought about and written about these tests for many years. I consider Bob Somerby to be one the most knowledgable people in the matter of test accuracy.

I assume Kevin Drum agrees that Somerby is a reliable source, since he based his post on what Somerby wrote.

Posted by: ex-liberal on June 7, 2007 at 12:25 AM | PERMALINK

adlsad,

reino said it much more eloquently than I would have.

As for cheating, I have heard of it done to various degrees. My mom worked at a school with very high scores that got extra money for perfect scores and one teacher (Superintendant's daughter, wouldn't you know) was known to make a key that fit over their tests leaving little holes where the right answer would be. She would check when they turned them in and then tell them "You missed X questions. Go back and find them and correct your work." However, I have found this sort of thing to be very rare.

What I do not doubt is that for struggling schools, the incentive to cheat goes through the roof. Given her descriptors both of her teaching as well as the test administration, I suspect that Lisa Rather works at a school that would love to see better scores for prestige reasons, but is no where close to being in trouble from NCLB.

Once you hit that list, I am betting that you start thinking that the best thing you can do for your profession and especially for your students, is to game the test a little so that their already ridiculously underfunded and overmandated school doesn't come completely unglued. Thankfully, I have never had to worry about such a situation. I would like to think I would stick with the honest approach in those dire straights, but I can certainly understand why pragmatism should rule the day.

Posted by: socratic_me on June 7, 2007 at 2:09 AM | PERMALINK

DevilDog, beware of responding to the author and not the content of the post.

This is one of the most intelligent and useful threads I've seen in years.

Posted by: MFB on June 7, 2007 at 4:33 AM | PERMALINK

DevilDog said "It would be obvious if a school or class cheated because it would create a huge anomaly in the test results, way too conspicuous to hide."

My response is that it depends how well they cheated. One of our sender schools got caught cheating when a teacher, who got to keep the tests overnight in between test days (which is not supposed to happen) fixed a ton of scantron answers. The Principal turned him in after a quick look at the sheets made it obvious that a lot more erasing had taken place than normal. If the teacher had cheated less--by, for example, limiting himself to three corrections per student--he could have raised his class's scores and never gotten caught.

I have never heard of teachers seeing the tests beforehand--that sort of thing would require school-wide corruption. Additionally, schools whose cheating gets out of control get caught, and people get fired. However, some schools cheat.

Posted by: reino on June 7, 2007 at 7:17 AM | PERMALINK

This chart compares gains on both tests. If a state has done a great job over the years and their students have historically scored near the top on nationally standardized tests, then we should expect only modest gains. States like Iowa, Wisconsin, and Oregon fit this category. Unfortunately, this chart would indicate that they haven't done a good job, when in fact, they have been doing a great job for many years. So, we need to be careful interpreting these comparisons.

Posted by: heyitsmom on June 7, 2007 at 8:30 AM | PERMALINK

I grew up in New York, which has had its Regents Exams for a very long time.
I also remember the bright red Barron's Regents Exams and Answers books in big racks in every bookstore and newsstand. Everybody (who wanted to get a Regeents Diploma and not the basic diploma, which meant everybody who didn't expect to be working the line at the GM plant) carried them around, read them in study halls. It was quite a business.
I was also thrown out of my Physics class and given a library pass because I was asking questioms that weren't in the Regents curriculum.

And I agree, this is a great post and a great thread.

Posted by: pbg on June 7, 2007 at 11:23 AM | PERMALINK

There are all kinds of ways to cheat, and (some) teachers and principals are determined to use them. In the past decade or so, various authorities have made it harder to cheat in certain ways. (More attention is being paid to the need to test all kids, not just the bright ones, for example.) But this has led, in at least one case, to a new type of abuse.

When I taught in the 70s and 80s, school systems used the big nationally standardized tests (the Iowa Tests, for example); typically, those tests would be re-developed every seven years. Therefore, test items would stay the same for at least seven years. In one way, this is very good; it makes test scores directly comparable from one year to the next. But it also facilitates cheating. Any teacher who wanted to know could know all the items on next year's test.

Today, states tend to change test items from year to year. I assume they are doing this in part to defeat this kind of cheating. But this introduces a new problem. Is this year's test as hard as last year's test? Inevitably, there's a margin of error, even if expert test developers try very hard to develop "equivalent" tests. And what if a state actively wants to "dumb down" a test to improve the state's passing rate? Under this new approach, they can do that every year. This approach has made it harder for teachers to cheat--and easier for states to do so.

Posted by: bob somerby on June 7, 2007 at 12:06 PM | PERMALINK

Dead thread, I know, but I've been offline for a few days. And after this--

Test Scores Soar After 'No Child'

Now, this is a peculiar headline since the second paragraph of the accompanying story admits, "The study's authors warned that it is difficult to say whether or how much the No Child Left Behind law is driving the achievement gains."

--I just had to weigh in. How often do you get to say "post hoc ergo propter hoc" after all? And it's such a nice example of the fallacy ain't it.

Posted by: DrBB on June 7, 2007 at 11:12 PM | PERMALINK




 

 

Read Jonathan Rowe remembrance and articles
Email Newsletter icon, E-mail Newsletter icon, Email List icon, E-mail List icon Sign up for Free News & Updates

Advertise in WM



buy from Amazon and
support the Monthly