68% of all statistics are misleading

May 10, 2002

Jupiter Analyst, RIAA Trade Barbs Over P2P Findings

To counter Jupiter, the RIAA presented its own findings...RIAA's survey asked listeners, "What were some reasons for not buying more music in 2001?" Among its findings:

- Twenty-three percent of music fans said they did not buy more music because they "download or copy most music for free."

- Thirty-eight percent of "heavy music buyers" under age 30 said they "did not buy more music because they could download or copy most music for free."

This is a good example of designing a survey to prove a predetermined conclusion by using leading questions. Asking about causes for an event that did not happen is generally bad. "Why didn't you buy more music?" is a very leading question. This is a question not about actual buying habits-- "why did you buy less music?" or "why did your music purchases stay the same?" would have been two questions that would have led to more accurate results. Instead, respondants were asked what kept them from buying even more music than they actually did. Even with this leading question designed to demonstrate the largest impact of downloading, only 23% of music fans cited downloading as a cause.

"People can come up with statistics to prove anything. 14% of people know that." -Homer Simpson

In addition to the RIAA's survey design being constructed with an inherent bias, this press quote is misleading, because it doesn't show what other causes were cited by survey respondants, how the differing answers compare in popularity and how the questions were posed: as a multiple-choice or freeform. Where do "music is priced too high" and "most music is crap" rank in comparison with "I can download."