Forget thinking we don’t know what we don’t know. We don’t even know what we do know.
| Jun 1, 2012
What is the value to a drug manufacturer or policy maker of having high-quality estimates of the size of the population affected by a particular medical condition? Living in the Information Age, we’re easily overwhelmed with more data than we could ever process. I’ve heard people say we don’t know what we don’t know, but in fact, most of the time we don’t even know what we do know. How often is data placed on the shelf, left to grow dusty and isolated from other data? Conversely, how often do we hold on to a piece of data that’s become outdated? So while we conduct our ultimate data-mining efforts searching for the “diamond in the rough” insight, separating good information from bad is not as easy as it may seem.
The ubiquitous nature of internet-based search engines may give some a false sense of security with respect to the completeness and quality of information for making high-stakes decisions. For example, submitting the search string “multiple sclerosis prevalence united states” to Google returned about 1,140,000 results in 0.18 seconds. The search also included what Google considers to be “scholarly articles.”
This is quite impressive and almost magical for those of us who remember having to search through the stacks at our university library for a single article. But search engines cannot generate a metric regarding the appropriateness, meaningfulness and usefulness of the information with respect to its intended use; we humans still have to make that evaluation. (Although Google is doing its best to eliminate this step; it’s pledging to make its searches more “human-like.”)
Two factors often overlooked when evaluating information, particularly when the ever-present specter of time is ticking away, are the interaction of the age of the information and the trajectory of the prevalence of the disease in the population. For example, the article with the greatest number of citations related to MS prevalence in the U.S. (691 citations) was published in 1997 and reported a relative value of 58.3/100,000 persons. If one were to assume that value is constant (a big assumption), we would expect the current U.S. MS prevalence to be about 132,000 given a non-institutionalized adult civilian population of approximately 226 million people. A 2009 memo from the National Multiple Sclerosis Society puts the estimate at 400,000; however, no references could be found on the Society’s website to support that value. The prevalence of multiple sclerosis derived from Kantar Health’s National Health and Wellness Survey is 1,394,012.
How can these estimates be so different? And how do you reconcile these estimates when your decisions will affect portfolio management, supply chain management and budgetary and resource requirements? Timothy Victor, Ph.D., Vice President of Science and Strategy at Kantar Health, wrote an excellent article, “The Value of Information,” that explores how to critically evaluate data sources to find the most accurate data to drive your decision-making process.
Overestimating prevalence may lead to wasted resources, and underestimating prevalence may result in missed opportunity. Taking the time to reevaluate what you think you know may make the difference.