Editore"s Note
Tilting at Windmills

Email Newsletter icon, E-mail Newsletter icon, Email List icon, E-mail List icon Sign up for Free News & Updates

July 12, 2007
By: Kevin Drum

SURFING THE NET....The Britannica Blog sure is weird. For example, a couple of weeks ago contributing editor Gregory McNamee decided to write ten rules for separating the wheat from the chaff during online searches. This is an extremely worthy topic, and McNamee started out with a fine suggestion: don't always trust the first result from Google. Check out a few other results too. But then comes #2:

Interrogate your sources as Detective Sergeant Joe Friday would interrogate a hippie....And interrogate the facts themselves, relentlessly. Spend a portion of each day asking, Which came first, the chicken or the egg? I don't know, but I do know this: In 1960 humans consumed 6 billion chickens. This year the number will be around 45 billion. And since the 1930s chickens have doubled in weight while eating half as much feed. This has implications. Think chemicals.

Whoa. Dude. You're, like, blowing my mind. Later (suggestion #8) McNamee advises "condensing facts to their most essential form" and offers this:

Here, by way of example, is the shortest fact I know: fish fart.

I can do better than that. How about: "I am." Maybe you don't consider it a fact, but I sure do. Certainly a factoid at the very least.

But this did get me thinking: McNamee's list is a little too Delphic for my taste, but it really would be a service to humanity to provide some sound advice about the pitfalls of doing research online and how to avoid them. The best answer, of course, is to go to college and then read widely for a couple of decades so that you have loads of context and background you can use to evaluate what Google turns up. Lacking that, though, a Top Ten list of pithy prescriptions about Internet research would be good too. Maybe we should come up with one.

Kevin Drum 8:37 PM Permalink | Trackbacks | Comments (41)

Bookmark and Share

Descartes published a "Discourse on the Method for properly employing one's reason and discovering the truths in the sciences."

Leibniz read it, and joked that Descartes vaunted method amounted to "take what you need, and do what you should, and you'll get what you want."

This was an early employment of the "Shorter X" trope.

McNamee gives something far less useful than Descartes did.

Shorter McNamee: I got nothing.

Posted by: Count Cant on July 12, 2007 at 8:50 PM | PERMALINK

It's very difficult to teach people how to evaluate information. The simple screening questions such as "Does that sound reasonable?" or "Does that fit with my experience of how the world works?" don't do a good job for people who simply want to believe what they want to believe. Look at Michelle Malkin from six months ago, or see what right-wing talk radio was overheated about last year. You'll find it laughable, but people believed it, at least until they found another shiny button to distract them.

Posted by: jimbo on July 12, 2007 at 8:57 PM | PERMALINK

"I am."

If you think it, it's a fact.

Posted by: low-tech cyclist on July 12, 2007 at 9:03 PM | PERMALINK

Go back and read Toffler's Future Shock. A major flaw in discussions like this is the peculiarly western assumption that all problems and all issues are essentially simple and can be distilled down to a few easily understood components. E.g., John Edwards spends a lot of money on haircuts. Who needs to no more?

If the information universe is complex and confusing it's because the information universe is complex and confusing.

Posted by: fugitive on July 12, 2007 at 9:03 PM | PERMALINK

... know more ... I no, I no!

Posted by: fugitive on July 12, 2007 at 9:05 PM | PERMALINK

Wittgenstein stated that the universe is not the aggregate of all things, but the aggregate of all facts.

Did you know that Wittgenstein's training was as a (pre-Wright Brothers) aeronautical engineer?

Posted by: pbg on July 12, 2007 at 9:09 PM | PERMALINK

In my favorite Dragnet episode, Friday and Gannon raid a house full of hippies tripping on acid. To get their attention, Friday blurts out: “Now sober up and try to listen to this!”

Posted by: Warren Terraplane on July 12, 2007 at 9:14 PM | PERMALINK

Existence is not a predicate.

Posted by: larry birnbaum on July 12, 2007 at 9:24 PM | PERMALINK

The other day I read an article on theHill.com. I couldn't make heads or tails of it. It had lots of factoids, but it seemed to me that something was missing. The Defense Department wanted to insert language in the defense appropriate bill repealing something called the Smith Amendment. The Smith Amendment limits who dealing with the DoD can be issued security clearances. A bunch of Republicans joined with a handful of Democrats in the Senate intelligence committee to reject the repeal. Something didn't feel right. It wasn't until I read the original reports and comments that I understood what nobody else seemed to want to admit--Kit Bond, 4 other Republicans and a handful of Democrats protected Congressional turf. They actually told the Administration that the Congress will legislate rules for who gets security clearances thank you very much. I don't know what deals were made to reach that result, but I think theHill.com article got all the facts right, but missed the story.

I never would have understood what was going on without reading the original documents. Sometimes if a story doesn't pass the smell test, you have to look further. I don't care how many factoids the authors include.

Posted by: corpus juris on July 12, 2007 at 9:25 PM | PERMALINK

*cough* kitties?

Posted by: Angela on July 12, 2007 at 9:27 PM | PERMALINK

My first 2 rules of critical thinking/reading seem to work surprisingly well for this:

1) Consider the source (though this one's already inherent in Kevin's question).

2) Ask: (Molly Ivins' -- may she rest in peace -- formulation) "Who's got a dog in this fight?" That is, who has a vested stake in the outcome?

For websites, this means either they've proven reliability through long experience -- i.e., they'
ve never lied to you (Kevin, TPM, Glenn Greenwald and TheDailyHowler are on my list); or (especially if linked by someone with an apparent agenda), the very first thing you do is read "About Us" or whatever the site offers regarding who writes it, what their mission is, crucially, who funds it, etc. In my experience, it's surprising how revealing this info can be; i.e., surprising that more don't just lie about what they're up to and who's paying for it (though that part's likely to be kept best hidden).

To me, this seems as basic as it's possible to get, but it's astonishing how many people don't seem to know to do even this.

Posted by: oaguabonita on July 12, 2007 at 9:28 PM | PERMALINK

Senate Intelligence Committee, not House Intelligence Committee. Preview is my friend.

Posted by: corpus juris on July 12, 2007 at 9:29 PM | PERMALINK

Hey, Kev. Please drop the "Susie for President" ad. I'm really starting to get a bit torqued about it. At least cover up her pallid face. Photoshop a veil or burkha over it or a big handlebar moustache and muttonchops. And get her out of that damn denim jacket that betrays her wispy, sickly, vegetarian frame. Give her a breast augmentation while you're at it. Have fun with it. Just get her out of my sight, PLEASE!

Posted by: forsythe on July 12, 2007 at 9:34 PM | PERMALINK

"Fish fart" is no shorter a fact than "bush lies."

Posted by: Paul on July 12, 2007 at 10:25 PM | PERMALINK

I used The Google today to find out how to remove a temporary tattoo from a 5-year-old.

Baby oil.

Posted by: Noam Sane on July 12, 2007 at 10:46 PM | PERMALINK

Of course, generation '. . . n' can get away with a short, text-messagy 'I m."

Posted by: ferd on July 12, 2007 at 10:49 PM | PERMALINK

Assessing the reliability of web sites just seems to be a special case of assessing the reliability of sources generally. I won't be telling anybody anything they don't already know if I list some well-known rules of thumb: look for a source with a good track record; look for things they say about subjects you know about (if they get that wrong, then don't rely on their pronouncements about stuff you don't know about); look for a source with good credentials; look for one respected by other reputable sources...lower on the list: check to see whether the source has a vested interest in the matter at issue.

None of these are magic bullets...none give automatic, easy, fool-proof evaluations.

Posted by: Winston Smith on July 12, 2007 at 11:25 PM | PERMALINK

Nevermind the earlier prod for kitties. I IM'd my boyfriend to say there had not yet been Friday catblogging here, and he said, "On Thursday?"

Sigh. It's been such a long week.

Posted by: Angela on July 12, 2007 at 11:34 PM | PERMALINK

Bookmark the Skeptic's Dictionary http://skepdic.com/contents.html
and Snopes' Urban Legends reference page http://www.snopes.com/ Others have already done a lot of your skeptical thinking for you.

Posted by: fyreflye on July 12, 2007 at 11:43 PM | PERMALINK

When it's important and academic-ish, dont google X. Google 'X bibliography'.

That way, you only get sites that have a bibliography.

Perfect ? Nope. But it does filter a lot of crap.

Ian Whitchurch

Posted by: Ian Whitchurch on July 12, 2007 at 11:57 PM | PERMALINK

My wife is am English teacher, and I've helped her grade suburban honors high school juniors' research papers. Talk about your indiscriminate use of google as a research tool. I'd flip to their works cited page and it's nothing but one long url string after another. This is helpful in one sense, because plagiarism can be caught through turnitin.com, but at the same time it's rather frightening because so many of their sources are sketchy.

The first thing these kids need to be taught is the value of paper research, for my money. Yes, there are tons of shitty, poorly researched or outright fraudulent books published every day in this country, but when you go to, say, Encyclopedia Britannica, each entry tends to have its own bibliography, and each of those sources I'd tend to rank in the reliable category.

So the number one rule I'd make is initially consult either a.) a respectable paper source such as the EB or to visit the website of same. I know they have subscriptions at her school to online encyclopedias, but I'm uncertain as to whether or not they include the bibliographical material.

The second rule would be to then branch out your initial research based on that first step.

Okay, so you have the bibliographical material on Saturn, now you google the history of the Saturn satellite launches, etc.

We graded one paper of a student who was writing about a novel and it was clear that she'd only typed a search string like "huck finn theme psychology" because her primary source was the number one hit I got when I googled a string of words that I thought was the essence of her argument. I went and read that first hit material and the hash she had made of its ideas demonstrated that she didn't fully read it or understand what little she did read.

So my third rule would be to use very specific search strings when researching a topic. The more specific you are the fewer hits you tend to get, but oftentimes you find that in those fewer hits it is easier to find material suited to your topic. So while "huck finn theme psychology" might turn up hits about psychoanalyzing Huck and Jim's relationship as stand-in father etc. etc. typing "huck finn mark twain biography psychology" might be more suited to what you wanted to write, which was a psychoanalysis of the author. (And yes, just as likely to turn up shit sources, but it's the general point here.)

Fourth rule, sticking with google as our example, is how to refine a search or use the advanced search function. Selecting what terms to limit your search can also be of assistance as can limiting the domain you're filing through. knowing we want to reject any "huck finn theme psychology" hits that mention "toilet training" is helpful. if those keep turning up but aren't germane, maybe that perfect source hit was on the 20th page of your search and now it moves itself up to page 3 and you actually find it.

Fifthly, I'd make teachers teach how to online search effectively. Sit down, put together a rubric, go online and find good examples of crap leads from vague search terms and bad sources. Demonstrate why those are bad searches and sources. Part of the teaching of research papers is the teaching of how to make note cards and how to use MLA style citations. Why not teach how to use google? It's as important a skill if not more important by several orders of magnitude.

The only issue here is that biased sources, one way or the other, are appealing, especially when you're writing under deadline. The process of teaching how to see past one's biases is a lifelong task and beyond the scope of this comment, I'm sure.

Posted by: The Critic on July 13, 2007 at 12:04 AM | PERMALINK

The most important thing is to have a comprehensive, consistent, unshakeable world view. This will enable you to immediately differentiate between truth and falsehood. Often, just knowing the source will allow you to determine if something is true.

Posted by: Al on July 13, 2007 at 12:44 AM | PERMALINK

OMG! If you have a vinyl version of ZZ Top's "Eliminator", put it on side "X2", at the ending of the last song and let the groove carry it the end of the record. It will repeat "Oh Mercy!" over and over. I just now noticed that after 25 years of owning this record!

Posted by: Doc at the Radar Station on July 13, 2007 at 1:23 AM | PERMALINK

It will repeat "Oh Mercy!" over and over.

You know, I would "peer review" that, except 24 years ago I bought a linear tracking turntable and the cartridge picks up and returns to home when it runs out of grooves.

Now I'm thinking furiously - who do I know with a direct-drive turntable???

Posted by: Blue Girl, Red State (aka G.C.) on July 13, 2007 at 1:41 AM | PERMALINK

BGRS, I've had the same JVC *semi*-automatic turntable the whole time. But the gear busted inside recently that normally always made it return at the end of the record. So now when records end if I'm not right there, they sit and "click" over and over on the inside groove by the label. I heard a funky squiggly noise and checked it out and it was "Oh Mercy!" over and over. Really blew my mind. I *mysteriously* noticed this just as I was reading Warren Terraplane's post about the hippies on Dragnet. Queue the "Twilight Zone" theme I guess.

Posted by: Doc at the Radar Station on July 13, 2007 at 1:54 AM | PERMALINK

since the 1930s chickens have doubled in weight while eating half as much feed. This has implications. Think chemicals.

Tsk, tsk. Another lesson that needs to be learned is not to go too far with your assumptions before thinking about and researching the problem. Chemicals might be reason #3, with breeding #2 and caging instead of free-range in the barnyard being #1 [or vice versa].

Posted by: natural cynic on July 13, 2007 at 3:04 AM | PERMALINK

"I am" is not a fact if "modal realism" is true, because then every "possible world" is just as real as any other, and there are no distinction between descriptions that "exist" and those that do not (all exist in the platonic sense...)

Posted by: Neil B. on July 13, 2007 at 8:38 AM | PERMALINK

Learn how to be skeptical. "I can't believe that" isn't skepticism. "Are all the steps needed to support the claim properly documented or supported by evidence" is. Religious and political zealots and others like them have a preconceived notion about what is true and what is not. They are incapable of verifying reality.

Posted by: freelunch on July 13, 2007 at 9:08 AM | PERMALINK

I'm a librarian and my advice. If you spend hours searching for information and can't find it by doing Google searches alone, there is probablly a reason for that. Ask For Help. Even better, if you don't know enough on the topic you are researching to even begin to know how to start - ask for help. Talking to someone (in the library world we call that a reference interview) will help you figure out what you really want (as opposed to what you think you want) and may reveal sources and strategies that cut down your time. It's OK and even expected that you don't know everything, asking for help from people who research for a living or who are experts on a topic is not a sign of weakness it actually shows smart/strategic thinking.

Posted by: ET on July 13, 2007 at 9:18 AM | PERMALINK

I tell my students that a quick way to filter through to sources of information that tend to be more authoritative in Google searches is to go to "Advanced Search". After typing in your search term parameters at the top ("Find results), down on the left side of the "Advanced Search" page look for "Domain", across to the right you see a field where you can type (below it, it says "e.g. google.com, org", type in "edu" (without the quotation marks) and hit return or the "Google Search" button. This brings up a list that is only from educational institutions. You do have to scan the articles to make sure they aren't written by some undergraduate student who isn't particularly authoritative, but often you can fairly quickly find articles that are written by professors with expertise in the question you are researching. Those you can generally be confident are reasonably authoritative - though, even there, there needs to be some healthy skepticism.

Sometimes a similar search limited to the "org" domain yields worthwhile stuff.

Posted by: TK on July 13, 2007 at 9:41 AM | PERMALINK

I just couldn’t resist. Here are a few additions to my earlier suggestion. They are a bit more practical (and less well written) than those given in that link.

1. Ask an expert. Experts in the field will know the lingo, they will know the big sources. They will help you understand what to look for.

2. Ask a librarian, especially one in the filed. In addition to what you would get from an expert you can also get a good handle on the best sources, specific strategies, as well as have expectations that are realistic. While I can’ speak for all librarians, I am sure I speak for many, I get enormous satisfaction from helping people with their research needs. I am a bit thrilled (yes I am a bit of a geek) when after explaining Library of Congress Subject Headings the patron gets it.

3. Set realistic expectations. First, there is not one big ginormous database with all the information you need in exactly they way you need it. Second, everything is not necessarily on the Internet and it is not always free. Third, when you are talking about information you may use for research there is more information behind firewalls in databases and in print than there is electronically. I am sure they are more but I am sure the point has been made.

4. Don’t for pity’s sake, assume everything is free on the Internet. Some of the best data is on any topic is behind the firewalls of paid vendors. As for historical data, it is there on the Internet but likely what you are looking for may not be.

5. And just in case the Internet and the Internet alone is your idea of research, learn to use the advanced functions. Just typing in keywords randomly in the basic/simple search is not going to be good enough. In Google using quotes, searching with in certain domains can reduce 100,000 hits to a more manageable level. Also, experiment with other search engines you may find one that are designed specially for certain subjects or have different ideas of what constitutes a different search.

6. Just because it is in print and not on the Net does not make it suspect. Actually, the reverse is more the truth. Anyone can post anything on the Internet. Would you trust the KKK to give a fair and unbiased account of the Civil Rights Movement? No. Question the source. Do critical thinking.

Posted by: ET on July 13, 2007 at 9:59 AM | PERMALINK

Middlebury College recently banned its students from citing Wikipedia as a source for anything in a college term paper submitted by a student for credit.

That should tell you something.

I find the internet useful especially for retrieving primary source documents, like laws, government documents, census data, original sourcings on articles. Can't beat the original source.

Posted by: pj in jesusland on July 13, 2007 at 10:52 AM | PERMALINK

I would simply suggest not stating with certainty anything you learned quickly.

And remember, even Harvard has some blinkin' idiots in tenured positions. The path to enlightenment is not an easy one.

Posted by: serial catowner on July 13, 2007 at 11:21 AM | PERMALINK

1) Think critically about what you read.
2) Get more sources to answer your questions.
3) Engage in a dialogue.

Posted by: calling all toasters on July 13, 2007 at 11:46 AM | PERMALINK

How about just "am"?

Posted by: Billy Bob Shranzburg on July 13, 2007 at 11:59 AM | PERMALINK
Middlebury College recently banned its students from citing Wikipedia as a source for anything in a college term paper submitted by a student for credit.

That should tell you something.

What it tells me is this: "Middlebury College doesn't teach the skills to use sources appropriately, and instead just bans sources which its untaught 'students' are prone to misuse from citation entirely."

Posted by: cmdicely on July 13, 2007 at 12:43 PM | PERMALINK

1,490,000,000 Google hits for i am. Not really a creative idea.

1.37 million for fish fart.

1 for erotolingus.

Go figure.

Posted by: slanted tom on July 13, 2007 at 1:53 PM | PERMALINK

In addition to the various suggestions above (and asking your friendly reference librarian is always a good idea when doing research), it is also important to realize that different search engines return different--and unique--results. James Fallows writes about this on his Atlantic Monthly blog. "Of first page results, 69.6% of Google’s items were unique to Google, 79.4% of Yahoo’s were unique to Yahoo, 80.1% of Live’s were unique to Live, and 75.0% Ask’s were unique to Ask. All in all, according to the survey, only 1% of results appeared on the front page of all four search engines."

The original research report is available in PDF.

Posted by: RefManTim on July 13, 2007 at 3:03 PM | PERMALINK


As I understand it, the reason Middlebury banned citing Wikipedia is because Wikipedia sourcing is notoriously unreliable, facts are unverified, assertions are unsupported and because so much of the material in Wikipedia is itself copied from other sources, often without proper or correct attribution.

Colleges want to encourage their students to submit orginal work in order to advance the body of knowledge in their students' chosen fields. Teachers don't need to read the same Wiki-bullshit over and over again.

Posted by: pj in jesusland on July 13, 2007 at 3:59 PM | PERMALINK

First, it should be clear that different types of research require different techniques.

Historical research, for instance, is often jumpstarted by beginning with a wikipedia article. This provides the researcher with additional keywords to doublecheck wikipedia's data, if the article seems otherwise to provide the info needed, or to further narrow the search, in an attempt to generate results from older histories that have been scanned.

On historical research, it is useful to know that all numbers need to be double and triple checked, because there are some amazing errors out there. Particularly in less recent history.

For more recent history, errors tend to occur because accounts are partisan and local, reflecting only a partial awareness of the broader landscape.

But all research isn't historical.

Let's say, you or a loved one has come down with something, and you'd like to know what illness you've got and how to treat it. A simple keyword search on google is a reasonable place to start, but less than an hour into it, you will realize that those tens of thousands of medical sites are all just providing you with the info they learned at medical school in Diseases 101. The info is often outdated, severely limited, or even false. After doing this preliminary medical research, it is essential to move on to keyword searching within Groups, where the actual experiences of people with these diseases generate a much fuller range of information about symptoms, their potential remedies, and the pitfalls. This info also needs triple checking and a search for actual research data backing up anecdotal claims, which may or may not exist, but this research greatly broadens an individual's knowledge base before a doctor's visit, surgery, etc.

Posted by: catherineD on July 14, 2007 at 3:39 PM | PERMALINK

Finally, an issue that I am passionate about. I have looked for info of this caliber for the last a number of hours. Your site is greatly appreciated.

Posted by: web watcher on January 30, 2011 at 12:28 AM | PERMALINK
Post a comment

Remember personal info?



Read Jonathan Rowe remembrance and articles
Email Newsletter icon, E-mail Newsletter icon, Email List icon, E-mail List icon Sign up for Free News & Updates

Advertise in WM

buy from Amazon and
support the Monthly