College Guide

Blog

February 19, 2013 5:17 PM No One Really Reads Academic Papers

By Daniel Luzer

Academics do a lot of research. The pressure to perform research in order to earn tenure generates, by some estimates, about 1.5 million new articles a year.

Some scholars have critiqued the quality of this research, pointing out that only 45 percent of the articles published in top journals are cited within the first five years after publication, but scholars are supposed to build on existing knowledge and use that to develop their own thinking.

Except it turns out that most of the cited research probably isn’t read. In fact, most of the research academics actually cite in their own papers they likely haven’t looked at.

That’s according to a paper by scholars at University of California, Los Angeles indicating that some 80 percent of authors include citations to articles they probably haven’t read. As the paper explains:

We report a method for estimating what percentage of people who cited a paper had actually read it. The method is based on a stochastic modeling of the citation process that explains empirical studies of misprint distributions in citations. Our estimate is that only about 20% of citers read the original.

The researchers apparently estimate this based on repeated misprints. In other words, if my references repeat misprints used in a study published earlier, it’s likely I haven’t actually read the study I’m misprinting but, rather, just copied the earlier study’s references.

All of this is not to say that researchers are straight-up dishonest (nowhere in an academic paper does it actually say, after all, “I certify and affirm that I have read and verified all information presented in paper and guarantee that the information is true and correct and that any documents I/we have provided with this study are genuine and that the information contained therein is also true and accurate”) but it is a little, well, disconcerting to think that there’s a possibility that most of the researchers out there are really just cutting and pasting other people’s papers, unread and unimportant, before they introduce their own material.

Disclaimer: I read the paper in question, “Read Before You Cite!” by M. V. Simkin and V. P. Roychowdhury published in Complex Systems, 14 (2003) 269-274, but it was only five pages long, and I basically skipped the equations on pgs. 272-73.

Daniel Luzer is the news editor at Governing Magazine and former web editor of the Washington Monthly. Find him on Twitter: @Daniel_Luzer

Comments

  • RSA on February 19, 2013 7:41 PM:

    Without an analysis of errors in widely used bibliographic databases (which are much more convenient to use than typing out citations from scratch--think Google Scholar, in earlier incarnations), Simkin and Roychowdhury's conclusions are still just speculation.

  • troglodyte on February 19, 2013 9:34 PM:

    Different issue. Even if you read a paper, it is much easier to cut and paste the citation for it from someone else's PDF into your own reference list. That would propagate a reference typo as well.

  • peggy on February 20, 2013 3:08 AM:

    The paper with the mistaken citations is in a physics journal from 1973! Obviously it opened up a new field and has been given credit ever since then. The authors learned the material in their undergraduate or graduate courses and picked up the damaged citation at that point. Since science pedagogy is not based on the U. of Chicago "Great Books" model, why would anyone ever read the original paper? One learns a condensed form of the material in a lecture or review and never looks backwards. It's probably difficult to find 1973 journals in the average university library.

    Try for a better story on academic frailty next time.

  • nobody on February 20, 2013 6:08 AM:

    What troglodyte said. That's exactly how I enter the references in my own articles. Still, looking at citations to my work, somewhere around 1/3 of citations seem to have missed the point of the paper they're citing, so maybe we're not reading carefully enough.

  • Ron Mexico on February 20, 2013 7:41 AM:

    Daniel--the conclusion of the paper in question is: "misprints in scientific citations should not be discarded as mere happenstance, but, similar to Freudian slips, analyzed." The authors also consider the explanations offered in comments above, but claim without evidence that "a modest reflection would convince one that this is relatively rare." I think a modest reflection would convince one otherwise, but I have no evidence either, except, you know, the Internet and stuff.

    As for your post....all academics must "cut and paste" previous work in the field. It's called a literature review. You have to show that a) you have expertise in the field that you are writing on, and b) that the questions you are asking are important to that field, before you can describe your research.

    The article is just a high-class trolling.

  • matt w on February 21, 2013 10:49 AM:

    Dittos to everyone -- it's perfectly possible to copy-paste a reference to an article you've read. There's often a big delay between when you read the article and when you write the bibliographic entry.

    The authors try to account for this by saying "Surely, in the pre-internet era it took almost equal effort to copy a reference as to type in one’s own based on the original, thus providing little incentive to copy if someone has indeed read, or at the very least has procured access to the original."

    But this is hogwash; in the pre-internet era, it would still have been easier to copy en masse bibliographical entries from another article that cited many of the same papers than to go through and find your own copies of every paper and extract the bibliographic information from them. In fact, this would be especially true in the pre-internet era, when the original articles would be sitting in the bound journals off in the campus library -- it'd be a lot less effort to find a citation in another article lying around in your office than to truck off to the library every time you discovered you didn't have a complete citation for something you've read.

    And the fact that they've restricted themselves to "celebrated papers" is really going to confound the data. Those are the papers that someone's most likely to have read five years before, even if they don't have a copy of it lying around.

  • JoanneinDenver on February 22, 2013 8:11 AM:

    Has anyone ever tried to get mistakes corrected, not in academic papers, but in books that are then cited in academic papers? I have. There is no real platform.
    In one widely used book in public administration/political science classes, there were two historical errors about the Peace Corps. One statement was that only male trainees were send to Outward Bound training in Puerto Rico. The other
    statement said that Peace Corps staff occupied Peace Corps Headquarters in Washington during the Nixon administration to protest the Vietnam War.

    I corresponded with the author and presented documentation as to the error of those statements. He reluctantly agreed that if there were a second edition, he would consider making corrections. Meantime, the book circulates. Students are tested on its contents and the falsehoods are perpetrated.

  • N.Wells on March 14, 2013 8:33 PM:

    I just want to second (or fourth or something) the opinion that the analysis is misguided. In the pre-internet, pre-copy-and-paste era, it was often necessary or best to copy a citation from trusted source like a major article in the field rather than to figure out citation details missing at the time of typing up the bibliography. This is not a question of not reading the sources, but a consequence of such things as a) older journal articles not always carrying volume and issue numbers, last page numbers, or even journal titles on an article's front page (thus leading to having photocopies without needed bibliographic information, if you neglected to write it all down at the time because of the queue at the photocopier), b) missing page numbers, etc., when trying to fit two journal or book pages onto a single photocopy page, and c) not being able to read one's own crabbed handwriting about page numbers and the like on a page of notes scrawled about an article read (but not photocopied) toward the end of a day in the library. In those sorts of cases, it's much easier to copy multiple citations from an earlier trustworthy paper, rather than to head back to the library to recollect all the details.