Editore"s Note
Tilting at Windmills

Email Newsletter icon, E-mail Newsletter icon, Email List icon, E-mail List icon Sign up for Free News & Updates

April 1, 2007
By: Kevin Drum

FLAT MAXIMA.... SOLVED!.... Yesterday I asked whether NFL scouts were really able to choose the best players out of a college draft populated exclusively by the finest athletes in the country. Or were the differences between the top players so small that they were essentially picking randomly?

The chart on the right shows the answer. It's from a paper by Cade Massey and Richard Thaler, who mapped the performance of all NFL players drafted from 1991-2002. They chose five measures of performance, and by all of those measures it turned out that players chosen in the first round of the draft did better than those chosen in the second, second round picks outperformed third round picks, etc. The measures aren't perfect, but taken together they seem to make it clear that even when confronted by a small group of the most elite performers on the planet, scouts can indeed predict future performance reasonably well.

What this appears to show is that, at least in a case where scouts have truckloads of information on each prospect, the principle of the flat maximum doesn't hold up. Yes, everyone chosen in the first few rounds of the NFL draft is a phenomenal athlete, but there are still differences large enough to predict future performance with at least moderate accuracy.

On the other hand, the operative word is still "moderate." If you're interested in more (and who wouldn't be?), scroll down to Figure 7 in the paper, which compares players by position and draft order. (That is, it compares them not merely by draft round, but by the actual order they're picked.) Players picked earlier do get more starts than players picked later, but their Pro Bowl performance is nearly identical. Within the first draft round, the first player chosen at a given position is only barely more likely to make the Pro Bowl than a player chosen four spots later. Within all rounds, the difference is a little greater, but still only about 55%. The difference in pro performance between the third linebacker chosen and the seventh linebacker chosen is very, very small.

Of course, in most situations decisionmakers don't have nearly as much information about their prospects as NFL scouts. If this is as good as they can do, how likely is it that a college admissions committee can choose between a group of students who are all like this? Probably not very.

Kevin Drum 1:32 AM Permalink | Trackbacks | Comments (47)

Bookmark and Share

The only thing that I'm suspicious about is the fact that teams feel they have more of an investment in players drafted in the higher rounds. They get more attention, they get more playing time, they get more opportunities. Obviously, scouts seem to have some clue what they're doing, but I wonder if lower round players really get the opportunities to show just how well they can perform.

Posted by: Royko on April 1, 2007 at 3:59 AM | PERMALINK

Are Al and Hawk taking the day off for April Fools' Day?

One thing we didn't talk about in the earlier discussion is that Los Angeles is an NFL-Free Zone, and better off for it. I'm thinking of having a bumper sticker made saying that. I wonder why any rational being with less than a seven figure income would support bringing a team here. (Particularly when we already have a superb professional team already playing in the coliseum).

My prediction about how draft order correlates with being a starter would have been different (and I will take credit for pointing out that this would be one testable measure), and this is why doing the experiment or collecting data are important. If only the supply side ideologues would take the same hint.

Posted by: Bob G on April 1, 2007 at 3:59 AM | PERMALINK

Moral of the story cited seems to be that if you want to get admission to an elite private school, don't mention "theology" as a career objective.

It's all about the benjamins.

Posted by: Michael Robinson on April 1, 2007 at 5:01 AM | PERMALINK

Or, to put it another way, it's all about the self-perpetuation of the upper class.

Posted by: Michael Robinson on April 1, 2007 at 5:03 AM | PERMALINK

I'm not sure that the example is relevant to the op-ed. The principle, as Schwartz articulates it, isn't that you can't ever differentiate within the top echelon, but rather that:

a) there are some cases where the level of uncertainty in differentiating within the top echelon is larger than the degree of differentiation*, and

b) college admissions is probably one of those cases.

I don't know that I agree--particularly if you allow that there may be dozens of useful ways to reform admissions criteria--but I don't think that a counterexample would refute his argument.

*Which has eery echoes to me of how superfluids come about**

**Obviously it's way too late and I've had a bit too much wine to be commenting on the internets.

Posted by: Adam on April 1, 2007 at 5:41 AM | PERMALINK

Has anybody read Moneyball by Michael Lewis? It is a "baseball" book that focuses on the approach to performance taken by Billy Beane, Oakland's General Manager, and others. It is not just a book for the uber fan or the uber geek, but it is a book that people in the recruitment business should read.

One of it's basic messages is that we tend to judge potential on traditional metrics that may or may not be relevant in the real world. We are really reluctant to reexamine those metrics. We are often unwilling to retest. We do so at our peril. Talent scouting is always a crap shoot. There are a lot of variables. Maybe the day the scout saw the prospect play, the prospect had a cold? Maybe he had just broken up with his girlfriend?

Maybe the prospect has other interests? Kansas City had a player some years back who was more interested in molecular biology than baseball. He was a great prospect, he is a better biologist.

Anyway, most (nearly all) prospects fail for some reason or another.

One of the biggest is the "he looks like a baseball player" prejudice. We want our athletes to have tall well defined "Greek god" type bodies. Maybe the prospects body type doesn't fit the "ideal?" If it doesn't scouts tend to ignore him.

The subject of the book, Billy Beane was one of the 5 tool can't miss kids who just looked like a player. Beane failed as a player. His heart just wasn't in playing the game. Ambition, drive, desire, or whatever you call it is very important. Often overlooked.

Beane solution was relatively simple, in an age when the NBA is drafting younger and younger kids are being drafted, he elected to focus on college players and on screening kids already in baseball's system.

At the level of the first and second round NFL draft pick the physical and mental differences are minute. Ambition plays a much bigger role in success. You have to really be hungry to be successful.

Anyway if you want to understand this subject you need to read Moneyball.

Posted by: Ron Byers on April 1, 2007 at 6:41 AM | PERMALINK

Aw, Kevin!

Ur such a geek.

Posted by: Tilli (Mojave Desert) on April 1, 2007 at 7:02 AM | PERMALINK

Recruiting students may be a crap shoot when you're talking about the top few percentage points of students, but picking research and grad school staff isn't. It's those that make and keep the school attractive. The way undergrad classes are taught these days is a good argument for extending public education another two to four years.

Posted by: joe on April 1, 2007 at 7:13 AM | PERMALINK

When I saw the first post, I wondered if anyone had done a similar study on baseball, especially since the baseball draft involves a good number of players who are just getting out of high school (kind of like the college recruiters, eh?), and there are SO many more rounds in that draft, and so many more players taken.

Posted by: Marc in Denver on April 1, 2007 at 8:09 AM | PERMALINK

Ron Byers--

It's worth noting, though, that of those players Beane drafted in that season, the one who turned out to be a star was Swisher, the guy who looked like a baseball player, and not, say Brown.

The subtler message of Billyball is to draft people with characteristics that are undervalued. That's harder to do now, because the ideas Bill James wrote about in the 80s are now pretty widely adopted. While nobody talks much about secondary average, OBP is now at least as important as BA, and you don't hear much about steals.

Posted by: jay ackroyd on April 1, 2007 at 8:20 AM | PERMALINK


This idea applied to college admissions is based on a false premise. Elite colleges don't try to admit the best students. Harvard claims to reject a classful of 1600 SAT scores and a classful of valedictorians every year. Elite colleges try to construct a class with goals that change over time. When standardized testing became common, Harvard found that it would let in too many Jews if it relied too heavily on testing, so it set quotas. Likewise, today schools seek ethnic diversity. Princeton looks for squash players, Johns Hopkins for lacrosse players. Kansas used to be (may still be) a school that had a strong debate team.

IMO, what these classes are constructed to do is maximize the future income stream of the endowment. Legacy applicants get a huge break. And I believe that the rest of the class is constructed to give that group training in how to get ahead in the world of high-salaried work and to have a memorable and pleasurable college experience. Today that means dealing with a diverse, fairly meritocratic environment. So that's what the classes look like. Hence the focus on athletics, which is a way you can participate in the school's community for the rest of your life.

I went to one of those eastern elite institutions. At the time, they let you look at your own academic file. The file included a projection of where you would finish in your class. My file projected me, accurately, as finishing in the middle of the class. Yet they took me over students that were projected higher. Why? I came from a rural state, a catholic high school and had been successful in both academic and athletic extra-curricular activities. There were easily a dozen kids from Bronx Science or Newton North who would have had a more successful academic career than I did. But I got in and they didn't, even though the school could (and did) accurately project our relative performances.

Posted by: jay ackroyd on April 1, 2007 at 8:35 AM | PERMALINK


In 2 1/2 threads on this topic your comment is the best. Congratulations.

Posted by: Ron Byers on April 1, 2007 at 9:19 AM | PERMALINK

Typically, NFL teams seem to use high first round draft picks to get players in positions where the team is currently lacking a solid player. So, it would make sense that those early picked players would play more games (which would then color the other categories used by the study) regardless of their relative skill level to the other players picked.

This is true even for the part of the study that looked at the relative merit of successive picks at each position.

Posted by: Luke on April 1, 2007 at 9:55 AM | PERMALINK


This is why the Bakke case was so stupid. Even if they had thrown out the least "qualified" student, there's no reason to think that Bakke would have been the next one in. There were certainly many applicants more "qualified" than Bakke who also didn't get itn.

Posted by: jay ackroyd on April 1, 2007 at 9:57 AM | PERMALINK

I've been doing a lot of thinking recently and I realize that I'bve been wrong on many, many issues. I realize now that we should get out of Iraq, we should have universal health care, we should have a decent minimum wage. And most of all, I realize that our president is an incompetent idiot.

In all humility, I am asking you liberals to forgive me and accept me. But if your liberal arrogance and self-righteousness prevents it, I understand.

Posted by: Al on April 1, 2007 at 10:23 AM | PERMALINK

As I recall Allan Bakke was admitted to the UC Davis Medical School. Did he remain? Is he practicing medicine today?

Posted by: Ron Byers on April 1, 2007 at 10:31 AM | PERMALINK

Al, you April Fool's joke is far too obvious. As has been said many, many times, conservatives just can't do comedy.

Posted by: Ron Byers on April 1, 2007 at 10:40 AM | PERMALINK

Isn't it also the case that teams tend to select those players who play positions for which the team has an actual need? I know this isn't always true, but it does seem that a player is more likely to play simply because the team has a serious need for someone in his position. And the lower you are picked the less likely it is that you might play simply because the need for your services is less acute, and that's why you were picked lower? Did the paper try to account for this potential variable?

Posted by: Barbara on April 1, 2007 at 10:47 AM | PERMALINK

I do want to point out that there is talent and there is talent. The flat maximum theory holds better for some kinds of talent than others. Think back to your high school days. Remember the top mathnerd, the best jock, and the prettiest girl. They were all likely to have remained tops (for the high school class) five years later. And the mathnerd is likely to remain tops for the rest of her life. Now remember the best writer, the best politician, and the most sensible person in high school. Exactly! The standings changed quite a bit, even within five years.

Some kinds of talent are easier to sort than others, and sorting has more predictive value. This is especially true for math talent. CalTech can reliably get the most mathematically-gifted freshman class in the country, year after year. Okay, the flat max theory applies to CalTech as well, because they are really looking for research talent, which is not quite the same thing. But this notwithstanding, math talent is fairly easy to sort out. Almost as easy as talent in sprinting.

Other kinds of talent are a lot harder to figure out, and the flat max theory applies much better.

Posted by: Joe S on April 1, 2007 at 10:48 AM | PERMALINK

Ron Byers,

An aside about peers not always recognizing what a seasoned scout sees - Had a friend who grew up on the West side of LA - He played many a game against the Brett brothers - He said that everyone thought Ken was a sure thing, but that George didn't have the makings of a ball player.

And now for the Coopertowns envelope, please.

Posted by: thethirdPaul on April 1, 2007 at 10:53 AM | PERMALINK

I do think that information is vital to make accurate predictions. And to take up for college admissions folks, they have loads of information: high school grades, standardized tests (now with writing samples and some content-based tests), AP test scores (in some cases - certainly for top 40 schools), analytical work by the students, letters of recommendation, the actual student application, data from student interviews.

Colleges are just looking to recruit potential talent and at the high end, I think that they are able to do that. Which doesn't mean students at other schools aren't capable, because college is the business of talent development and 18 year olds have A LOT of room to grow. But I don't think that there are no difference between these applicants. Some have been exceedingly well-prepared to compete at the top levels.

Disclosure: I teach at a college prep school. Obviously, there is a whole host of questions to be asked about whether every high school graduate in America has been given the education and skills they need to compete. They haven't.......but that's an issue for another day.

Posted by: Stacy on April 1, 2007 at 11:10 AM | PERMALINK

One problem you have when it comes to seeing how people perform after college is that in many professions where you go to college is a signficant sifter. That's why you don't see any Supreme Court judges who went to an oustanding state law school like the University of Michigan much less lower-tiered law schools like say the University of Georgia. Brilliant, late-blooming lawyers who attended say UCLA law school or the University of Illinois law school don't make the Supreme Court cut. Also, some firms like McKinsey only hire from schools like Harvard or Stanford.

If you want to see some areas that seem to fit your thesis, entertainment and politics both seem to fit. There are many, many people in the entertainment industry (and I am not just talking about performers but executives as well) who either did not go to college, did not finish or went to schools far removed from elite institutions. The same is true for politics where many, many politicians did not go to elite schools, and yet have managed to carve out signficant careers. Finally, many CEOs came from much more humble backgrounds.

Posted by: Guscat on April 1, 2007 at 11:41 AM | PERMALINK


Excellent points - Some of the finest, sharpest gunslinger trial lawyers, I have known, graduated from George Washington, Texas, Loyola and Southwestern Schools of Law - They have all defeated many of the finest from far more prestigious schools and have all been included in the top 100 trial lawyers in the land. One of them had been an undergrad at a Cal-State.

And then there is Alberto, from HAHvard.

Posted by: thethirdPaul on April 1, 2007 at 11:55 AM | PERMALINK

I think we should scout prospective lawmakers right out of college. Maybe they stand a chance at outperforming the "old linebackers"? How would "W" have done in the draft? Cheney? Obama? McCaine? Clinton? We need DNC & RNC scouts!

Posted by: coldlouie on April 1, 2007 at 12:06 PM | PERMALINK

I guess to a significant degree, I disagree with the entire premise.

College admission opens up a world of opportunity, prestige, and intellectual engagement. Colleges are not seeking to maximize the number of academics in their pools. Colleges seeks to perform a social function - and it's not clear that the social function is maximized by putting the best performers all in one school. A public university may do more good by admitting a class that includes people who are likely to work within particular communities, create businesses, go to professional school, or become academics. So, can you choose the best students to do dozens or hundreds of different things best?

Posted by: MDtoMN on April 1, 2007 at 12:44 PM | PERMALINK

It struck me as interesting that the NYTimes article focused so much on a girl who most likely did NOT really do so spectacularly well on all measures relevant to college admissions. Evidence of this is that she was not admitted either to Amherst or Middlebury. Given her obviously impressive extracurriculars, and her pretty remarkably good essay, I've got to conclude that something else was below par -- most likely, her SAT scores.

On the other hand, the writer mentions another student who got 2400 on her SATs -- which puts her in the company of just about 200 odd high school students in her year.

Why not feature this other student? Too geeky? Too Asian? Uncool parents? Uncool attitudes ("hot" is important for her)?

While articles like this pretend to be breaking out against stereotypes, typically they simply reinforce them.

Posted by: frankly0 on April 1, 2007 at 1:19 PM | PERMALINK

These are team sports, Keven, the player in question is only one part of the equation when it comes to what that player's pro stats will be. You could put 2 exactly equal players on two different teams and you would expect them to generate different stats. The differences could be quite large depending on the teams.

Posted by: QrazyQat on April 1, 2007 at 2:15 PM | PERMALINK

The difference between football players and Ivy League students is that the football players are playing a winner-take-all game. So when fielding a team it's extremely important to have the Best, not just the Very Very Good.

Posted by: Allen K. on April 1, 2007 at 2:16 PM | PERMALINK

I went the Newton North. Class of '96

Three big things they don't mention.

1) NN has tracking. AP > honors > I > II > IIb. The best teachers are given the AP and honors classes.

2) Very few students are taking more than two AP classes. I would estimate less than 50 of 500 fit in the profile presented in the article. They all know each other, hang out together, generally form a school-within-a-school.

3) Ethnicity and class is hardly mentioned in the article. The school and Newton in general is very Jewish. The primary ethnic division (and it's pretty big) is between the Jews (almost entirely Reform) and Protestants (almosted entirely mainline) on one side and the Italian Catholics that live in the next town over, which also provides students for NN, on the other. There is very limited association between the two groups (and lots of nasty name calling). The view of NN in the article is entirely from the perspective of the Jew/Protestant/Secular folks with 3+ APs. They probably represent less than 10% of the student body.

Posted by: Adam on April 1, 2007 at 2:51 PM | PERMALINK

That graph you cite has huge problems. It is a graph of means over all players in the given category.

The real statistic that is meaningful in this case is the probability that player X > player Y given that player X is draft pick 1 and player Y is draft pick 2 (for example). It seems (from the closeness of the means) that with any reasonable real world distribution the above probability would be

I think the whole point of the paper was that the first round draft picks were overvalued.

Posted by: Adam on April 1, 2007 at 3:07 PM | PERMALINK

I believe that over a large number of such individuals grades, SAT/ACT, and extracurricular activities will produce a similar result as the NFL draft study. That Kevin doesn't want it to and would rather draw straws is irrelevant. Even at the highest level. While the differences would definitely narrow, it seems this is much preferable to going to some random system with the elite students.

Posted by: Chad on April 1, 2007 at 3:28 PM | PERMALINK

Um, the metrics presented here completely circularize the argument. If I picked somebody in the first draft then the principle of sunk costs will encourage me to start him more, hang on to him longer and to generally give him a greater chance to prove himself. Late picks, with no sunk costs, can be kept or discarded at will.

Maybe the study also measured receiver ratings, tackles, touchdowns or interceptions per quarter played or some other metric that objectively determines whether early picks actually do better than others who spend the same amount of time on the field. If it does, great. Otherwise I would not put very much weight on the figure that you presented.

Posted by: Tim F on April 1, 2007 at 4:13 PM | PERMALINK

OK, the pro bowl metric is useful but also depends somewhat on a player getting enough field time to qualify, so it is at least partially derivative to the same circularity that I mentioned above.

Posted by: Tim F on April 1, 2007 at 4:15 PM | PERMALINK

Football is probably not a great or supporting example for flat maximum. The game is so studied, so coached, so game planned, and played at such an elite level that really the reverse of the flat maximum is in effect - all of these guys are elite but the smallest differences in ability in some areas, a little extra burst or better sprinter speed, may make the difference between getting through the hole or going the distance after catching a simple in route.

Even though Mike Shanahan, for example, has great success cycling in running back after running back into his downhill running scheme, there's little doubt that a guy with that extra quickness and burst, or with the adaptable vision to detect the hole, is going to be more effective. That said, Mike's cost-benefit is usually to downplay getting a special running back in favor of other positions higher in the draft, though he is also gifted at finding diamonds in the rough, which plays into his successful strategy.

Posted by: Jimm on April 1, 2007 at 5:01 PM | PERMALINK

Students probably have problems figuring out which colleges are suitable to apply to. In Florida, UF gets top students, regardless. I suspect the place has always had better students than it deserves, just because it's the "flagship" state university.

In the biological sciences, run-of-the-mill colleges seem to serve as feeders for top graduate programs, so surprising numbers of elite scientists have undergrad degrees from something like a California State University campus. A friend (a supermarket manager) started at Keene State College in New Hampshire, MS at North Carolina, PhD in a distinguished program, and has made an extremely productive career in a staff position at Harvard.

Posted by: Dave on April 1, 2007 at 6:06 PM | PERMALINK

Also, in football, there is a very limited "set" of elite "applicants", at least in comparison to college admissions.

Posted by: Jimm on April 1, 2007 at 6:40 PM | PERMALINK

I think reading the whole pdf is worthwhile-I skimmed some of the methods and focused on the discussion. But what the researchers are really getting at is Rational Expectations Theory (RET) in economics. The idea that markets are always rational given reasonably good information. What you see in the research is a psychological trait of overvaluing CHOICE. That is a cornerstone of conservative economics in many areas. What they do here is blast this concept very effectively. Here's the first paragraph of the study with a few snippets and then the concluding paragraph:

"Two of the building blocks of modern neo-classical economics are rational expectations and
market efficiency. Agents are assumed to make unbiased predictions about the future and markets are assumed to aggregate individual expectations into unbiased estimates of fundamental value. Tests of either of these concepts are often hindered by the lack of data. Although there are countless laboratory demonstrations of biased judgment and decision making (for recent compendiums see Kahneman & Tversky , 2000 and Gilovich, Griffin, & Kahneman, 2002; ) there are fewer studies of predictions by market participants with substantial amounts of money at stake. Similarly, tests of market efficiency are plagued by the inability to measure fundamental value. (Even now, in 2005, there is debate among financial economists as to whether prices on Nasdaq were too high in 2000.)"

"...we compare the market value of draft picks with the historical value of drafted players. We find that top draft picks are overvalued in a manner that is inconsistent with rational expectations and efficient markets and consistent with psychological research."

"...A combination of well-documented behavioral phenomena, all working in the same
direction, creates a systematic bias: teams overestimate their ability to discriminate between stars and flops...."

"...While this means we will not be able to pin the blame on any one underlying cause, it
strengthens the case for our overarching hypothesis: teams overvalue the right to choose..."

"...Overconfidence is a closely related concept in the psychological literature. Simply put, people
believe their knowledge is more precise than it is in fact..."

"...When people have more information on which to base their judgments their confidence can
rationally be greater, but often information increases confidence more than it increases the actual ability to forecast the future..."

"...False consensus suggests that teams will overestimate the extent to which other teams covet the same player, and therefore overestimate the importance of trading-up to acquire a particular player. Such a bias will increase the value placed on the right to choose..."

"...In this section we estimate the *surplus value* of drafted players, that is the value they provide to the teams less the compensation they are paid..."

"The implications of this study extend beyond the gridiron. Football players are surely not the
only employees whose future performance is difficult to predict. In fact, football teams almost certainly are in a better position to predict performance than most employers choosing workers. Teams get to watch their job candidates perform a very similar task at the college level and then get to administer additional tests on highly diagnostic traits such as strength and speed. Finally, once hired, performance can and is graded, with every action visible on film from multiple angles! Compare that to a company looking to hire a new CEO (or an investment bank hiring an analyst, a law firm hiring an associate, etc.). Candidates from outside the firm will have been performing much of their job out of view. Outside observers see only a portion of the choices made, and options not taken are rarely visible externally. And, even once a CEO is hired, the company’s board of directors is unlikely to be able measure his or her performance nearly as accurately as a team can evaluate its quarterback. In our judgment, there is little reason to think that the market for CEOs is more efficient than the market for football players. Perhaps
innovative boards of directors should start looking for the next Tom Brady as CEO rather than Eli Manning."

Posted by: Doc at the Radar Station on April 1, 2007 at 7:23 PM | PERMALINK

The key point to determine whether the principle of flat maximum should be applied is whether the situation involves a capped maximum or whether it's essentially unbounded. In an unbounded situation, just looking at a bell curve indicates that there will be more separation between individuals at the extrema, not less. A good hypothetical example is test taking. If a test is fairly easy such that the median grade is over 90%, then you have a flat maximum -- it's really fairly random whether the individuals scoring 100% are that much better than those scoring 95% because the later might have just made some fluke errors. If the test was very difficult, with a median score of 50%, then there would be wider separation of scores at the top.

Looking at the two concrete examples we have here:
In football, there can always be someone better and faster. Hall-of-fame caliber players routinely make athletes that are in the top fraction of the top percentile of human capability (ie, league average players) look terrible. The best players in the draft group really do separate themselves from the rest of the pack enough on average to allow NFL teams to make genreally good choices.

However, the argument for why college admissions might be a flat maximum situation is that a lot of the criteria used for admissions are bounded pretty low, especially for the top high school students. There are many more high school students with perfect grade records and maximum SAT scores than there is room in the top universities. While some high school students do find out-of-the-box ways of distinguishing themselves, for the most part even the best and brightest confine their pre-college activities to whatever is provided by their high school. And so many of them are able to max out on not only grades but sports, student government, other extra-curriculars, etc. In the end, decisions probably come down largely to highly subjective evaluations like application essays and interviews.

Posted by: Will Hutchinson on April 1, 2007 at 7:30 PM | PERMALINK


You flunked out of Cal Tech, right? (No shame, I thought hard about applying to Cal Tech but didn't in the end precisely because I figured I'd flunk out.) Surely, you noticed while at Cal Tech that there were lots of people smarter than you (and a lot smarter than me)?

What you observed was that there is no principle of a flat maximum (the maximum at Cal Tech in your day went all the way up, at least, to Richard Feynman). Instead, college admissions suffer from a problem of a flat maximum. In our day in the 1970s, lots of guys at a place like Cal Tech (and quite a few guys at a place like Rice, where I went) scored 800 on the Math SAT test. The local culture distinguished between students with a run-of-the-mill 800 and students who could have scored a lot higher if the top score went up higher. The just-barely-800 guys needed good work ethic, good study habits, immunity to homesickness, and other traits to succeed at Cal Tech, and they are less likely to go into the history of science books. The few guys who could have scored a 1000 (two standard deviations higher than 800), well, they are different. They could still screw up, but they are something special.

Back then, however, the SAT Verbal test was extremely hard to score 800 on. A few dozen kids per year nationwide did it. So, there was less of a problem of a flat maximum.

In 1995, ETS and the College Board revamped scoring on the SAT to make it much easier to score 800 on the Verbal, intentionally creating a problem of a flat maximum where it previously hadn't existed.

The flat maximum: it's not a principle, it's a problem!

Posted by: Steve Sailer on April 1, 2007 at 8:01 PM | PERMALINK

Will H:

Bill James wrote about this phenomenon.

He pointed out that the distribution of talent in professional baseball is not a bell curve. It's the extreme right hand tale of the bell curve that characterizes the general population. He drew the picture, which I can't do here very easily. It's very fat on the left and tapers to the right. He drew a line on the fat part and labeled that "the replacement level." At the extreme left hand tail there are one or two players--Barry Bonds--and at the replacement level is about half the majors, one short step away from AAA. It's like Feynmann vs the rest of the CalTech physics faculty.

Part of the point of that article was that arbitrators do a poor job of using statistics (in the players' favor). A guy who hits .300 doesn't deserve 80 percent of what a guy who hits .360.

Posted by: jay ackroyd on April 1, 2007 at 9:46 PM | PERMALINK

Paul, I have actually heard George Brett say that his brother Ken was a far more natural ball player. He is still in awe of his older brother. Of course, George is in the hall of fame and Ken is just another retired journeyman ball player.

When George came to the big leagues he competed head to head with one of his best friends, a guy named Jamie Quirk. Quirk was heavily recruited. Brett wasn't. George paid close attention to a hitting coach named Charlie Lau, and won the competition with Quirk, although it was close. The Royals refused to trade Quirk. Quirk was forced to change positions. He became a back up third baseman and a backup catcher. Stuck, he worked hard trying to learn how to be a catcher. It was his only chance to become an everyday player. Once he spent a season or two trying to learn how to be a cathcer he never really returned to his previous level of performance as a third baseman.

It is important to remember that Quirk was a better prospect. Almost by accident Brett met a hitting coach who taught a hitting style that perfectly matched his abilities. Brett was smart enough to pay attention to Lau. Quirk wasn't helped as much by Lau. The competition was a near run thing, but Brett won. Quirk was stuck in the Royals organization. He wasn't allowed to move at a time when he could still maximize his potential. Had Quirk been drafted by another organization, his might be a well known name today. Instead he is just known as the guy who was stuck behind George Brett because his organization decided to waste his talents as Brett's backup. Quirk is now a coach. His chances for greatness are gone.

Chance played an enormous role in the George Brett story. It played an equal role in the Jamie Quirk story. Ying and Yang. Had it not been for the divine intervention of Charlie Lau, the fates of both might have been reversed.

Posted by: Ron Byers on April 1, 2007 at 11:15 PM | PERMALINK

Ken Brett was an amazing all-around athlete, who spent 14 years in the majors and was one of the best-hitting pitchers ever, with a .262 batting average and a .406 slugging average. His career OPS was above that of the average middle infielder or catcher of his time.

To the extent that he had a problem, it was that majro league baseball, unlike lower level baseball where he exceled, couldn't accomodate a generalist who was both a pitcher and a hitter. Even Babe Ruth had to choose between those jobs.

Posted by: Steve Sailer on April 1, 2007 at 11:51 PM | PERMALINK

I just remembered something. Sadly Ken Brett died at age 55 a little more than 3 years ago. He was a victim of cancer.

Posted by: Ron Byers on April 2, 2007 at 12:56 AM | PERMALINK

Kevin Drum, your last paragraph is not convincing, there are plenty of athletes with impressive accomplishments in high school but they are not equally good college prospects and the recruiters can tell the difference. Similarly for the girls in the NYT story you linked, a 2400 SAT is more impressive than >2000.

Now of course if Harvard admits say 1000 out of 10000 applicants there won't be much difference between number 1000 who gets in and number 1001 who doesn't. That doesn't mean there is no difference between number 100 and number 1000 or number 1000 and number 10000.

As others have pointed out the SAT is not hard enough to optimally discriminate among top students. If the elite colleges like Harvard cared as much about getting the very best students (in academic terms) as the top college sports teams care about getting the best athletes they would demand a harder test. Also note academic peformance at Harvard does not really separate out the top students either because Harvard is notoriously undemanding once you are admitted.

Posted by: James B. Shearer on April 2, 2007 at 4:33 AM | PERMALINK

One small practical note that I don't think has been mentioned. When you're looking across rounds, you're comparing the guy a team picked at position X with one picked at X+(# of teams). Comparing within rounds gets you significantly finer granularity. It may be completely consistent with the notion of flat maxima that (relatively) gross differences among the set would be observable. What you're seeing in this case, I believe, is the effect that most of the other GMs of NFL teams are at least "pretty good" at picking from the pool remaing after your pick X, making it pretty likely that your pick X+1 will be worse.

Posted by: Mike Jones on April 3, 2007 at 2:50 AM | PERMALINK

Give please. Never despair; but if you do, work on in despair. Help me! I can not find sites on the: Teaennitngng bnnuiedn mannnnanr cufactuurncueatrus. I found only this - old tanning beds. The cutwe are wearing this to fashion week. We also sell sun tanning lotions and sun tanning equipment such as sun tanning goggles and eyewear. Thanks :eek:. Jacy from Finland.

Posted by: Jacy on August 18, 2009 at 5:51 AM | PERMALINK

Good evening. Smell is a potent wizard that transports you across thousand of miles and all the years you have lived. Help me! It has to find sites on the: Domain name website hosting. I found only this - host unlimited domain. Your tag has to be european of clients at the many query, yet if it's an advertising college, domain hosting. Throughout the infection of 2007 and 2008, myspace advanced well-known of the clients of its example in both eligibility and in meaning, domain hosting. Thanks for the help ;-), Floria from Fiji.

Posted by: Floria on March 4, 2010 at 4:12 PM | PERMALINK
Post a comment

Remember personal info?



Read Jonathan Rowe remembrance and articles
Email Newsletter icon, E-mail Newsletter icon, Email List icon, E-mail List icon Sign up for Free News & Updates

Advertise in WM

buy from Amazon and
support the Monthly