Editore"s Note
Tilting at Windmills

Email Newsletter icon, E-mail Newsletter icon, Email List icon, E-mail List icon Sign up for Free News & Updates

March 31, 2004
By: Kevin Drum

CORPORATE RESEARCH....Max is annoyed that anyone would take seriously a research report from, say, Global Insight or McKinsey. I'll tentatively agree.

I spent the last 20 years of my life in the software industry reading reports like these, which are generally written for corporate clients (who either subscribe to a service or buy them one off) and are often commissioned by trade associations. I wouldn't go as far as to say they're completely useless, but I would sure recommend taking them with a big grain of salt unless you have a very good reason to think the methodology is unusually accurate. When you read them carefully, it usually turns out they've spun a very small amount of actual data into a very impressive looking report.

It's not that these firms don't do genuine research. They do, and their market research arms in particular spend a lot of money on large phone banks that do a lot of box counting. The problem is that raw numbers can usually be sliced and diced to get any result you want based on which assumptions you use, and my experience is that these firms are not very sophisticated about picking assumptions. Sometimes this is deliberate, especially with commissioned reports, and sometimes it isn't, but it's reason for skepticism in either case.

Also (again, from personal experience only), the lead analysts at these places are incredibly harried trying to crank out reports while still taking calls from clients and touring the country with the sales guys trying to drum up new business. Turnover is also high. And the "analysts" who do a lot of the quantitative work are very often low-paid 22-year-olds fresh out of college with tools no more sophisticated than an Excel spreadsheet.

I was constantly under pressure to subscribe to some of these services, and I did on occasion if only to stay on their good side so they'd write favorably about our company. But I usually dropped the subscriptions pretty quickly because the data simply wasn't very good. Everyone loves having precise numbers in black and white from an authoritative source (it's especially good for reports to the board of directors), but the fact is that most of the numbers aren't much better than stuff you can get with a few minutes of Googling.

Max says, "You might as well put credence in the email you got this morning on how to make your balls bigger." I wouldn't go that far, but it's a fair point. You probably shouldn't take these kinds of reports very seriously unless someone you trust gives you a good reason, and you should never take them seriously based only on their press release. Caveat emptor.

Kevin Drum 2:46 PM Permalink | Trackbacks

Bookmark and Share
 
Comments




 

 

Read Jonathan Rowe remembrance and articles
Email Newsletter icon, E-mail Newsletter icon, Email List icon, E-mail List icon Sign up for Free News & Updates

Advertise in WM



buy from Amazon and
support the Monthly