3 Ways to Recognize Bad Stats
We need to be shrewd when it comes to statistics and reality: things are not always as they seem.
Often times, a statistic is like a piece of candy thrown at a
parade—you really don't know if you should bite into it or not. We've
all heard Mark Twain's famous quote about lies and statistics. There is a
reason so many people have had skepticism toward stats. Too frequently,
people repeat inaccurate, bad, or explicitly made-up numbers.
I've written about the issue before—on many occasions. Here at the
blog, you can read about a lot about stats, including specifics about bad marriage stats and why we like bad stats in general.
There is a reason so many people have had skepticism toward stats.
Still, I keep hearing statistics quoted at conferences and through
blogs and social media that make me scratch my head in amazement. I'm
not sure where some of these stats originate, and I'm the president of LifeWay Research.
So how can you really discern good stats from bad?
1. Be Wary of Statistics in Promotions
Statistics are effective at moving people toward action, but it can
border on manipulation, depending on how you use or change the numbers
and information. Using unverified statistics can be an example. Even if
the action is good, it is not acceptable to use a statistic that is not
valid.
For example, maybe you've heard of books that say the church is dying—and this book can fix it (Yet the numbers show that the church is not dying). Or, that 80% of church plants fail—and this conference can stop that (yet, about two-thirds of church plants are around in four years after starting).
You've probably heard someone say the stats show that pastors are
miserable and just want to quit—and our ministry saves them (when the
vast majority are happy and consider it a privilege to serve).
Or that the Super Bowl is the biggest sex trafficking event in
history—and you should support our ministry to stop it (when there is no law enforcement evidence that this is true).
All of these things are important: we need the church to thrive, church
plants to be successful, pastors to be happy, and sex trafficking to
end. But we don't need made up facts to evidence the situation and the
need.
2. Be Wary of Stats that Cannot be Verified Scientifically
Secondly, is there really a way to know the fact claimed? In other
words, you should always ask "How do you know that?" when someone quotes
a statistic.
For example, we have heard that 94% of evangelical youth drop out of
church after high school never to return. The three words "never to
return" should raise red flags. How could anyone know that?
You should always ask "How do you know that?" when someone quotes a statistic.
Realistically, how could they keep track of those 94%? Did a researcher
track them their whole lives or implant them with transponders and see
if they come back?
We have also heard the statistic that 80% of pastors don't make it to
retirement. How would someone know that? Some of this is just common
sense.
If you can't think of a way to verify a statistic, it is probably not
true. Unless someone spent millions of dollars on research conducted
over decades, some of these stats could not be verified.
If it is not verifiable, you should not be quoting it.
3. Be Wary of Stats that do not Line up with Reality
Thirdly, be skeptical about statistics that do not line up with your
reality. For example, one statistic seems to indicate that Christians
and non-Christians divorce at the same rate. That hasn't been my
experience. The truth is that those who practice their faith have a much
lower rate of divorce.
I wrote a post called That Divorce Stat You Quoted Was Probably Wrong
due to the confusion over this issue. The oft-quoted statistic comes
from a Barna study. However, if you read what Barna reported, the
research found those that say they're Christians divorce at the same rate as those that say they're not Christians. But the reality is that a much larger number of people claim to be Christians than really are.
So, stop getting up before your church and leading them to believe that
they have a 50/50 chance of divorce. That just demoralizes them.
Sociologist Bradley Wright, author of Christians are Hate-filled Hypocrites… and Other Lies You've Been Told,
indicated that 60% of people who identify as Christians but rarely
attend church have been divorced. But of those who attend church
regularly, only 38% have been divorced. That is a huge difference.
Not to mention you can use your own common sense. I recently read the
50% of pastors will get divorced. Because this stat is all over the
place, I'm going to write a full-length post on it in the future.
We need to be more skeptical when it comes to statistics.
But just use some common sense for the moment and ask the question. Do
we REALLY believe that half of all pastors get divorced? Does that
REALLY make any sense? If the 50% divorce rate is a myth for all
Americans (and it is), do we REALLY think that would be the case for
pastors? Of course not.
We need to be more skeptical when it comes to statistics. If a
researcher cannot tell you their research methodology and the questions
they asked, you should immediately disregard the statistic. Researchers
have an ethical code that we supposedly follow—if they don't, stop
listening to them, and never quote them.
Not all Research is Created Equal
Occasionally, researchers will come up with different conclusions to the same concept. It depends on the question posed.
I recently read some comments from David Kinnaman of Barna about Millennials and evangelism.
He says that while most born-again Christians say they should share
their faith, few do. At Lifeway Research, we've found the same thing.
However, David's report found that Millennials who are born again are
the exception, and share their faith at a higher rate. Our data shows
something different, something I hope to discuss with David soon. My
guess is that different samples and wording made the difference between
the samples—but you should be able to tell by looking at the
methodology!
But, in both cases, there is a reasonable methodology that led us to
these different conclusions. As a reader, you should look at the
methodology of study in order to know where the researcher is coming
from, and to know the context for the statistics.
Every well-framed question tells us something, as long as it is done correctly.
To discern if research is of good quality, it may help to understand
some of the process. In any study, a researcher needs to ask, "Does this
question ask what it needs to ask?" Every question is imperfect. There
is only so much money available to research a topic therefore the number
of questions is limited.
So a researcher is only going to be able to ask a limited number of
questions (sometimes only one or two) about a topic. From that research,
we're not going to be able to draw every conclusion—although some try
to extrapolate it to ridiculous conclusions. But, every well-framed
question tells us something, as long as it is done correctly.
So, How Do We Know?
What are some ways to tell if the stat you've heard is trustworthy? You can tell the quality of the research if it is:
- done on a good sample (a random set with a large enough sample, not a group at a conference, a website where people click to vote, etc)
- a good question (is it fair and not leading), and
- a good methodology (is it transparent, honest, and credible).
You can guard against believing and spreading bad statistics by
determining if its source is a promotional piece, if it cannot possibly
be verified scientifically, and if it does not line up with reality.
Then, if it is quality research, quote away.
Anything else is just spreading misinformation. We can and should do better than that.
No comments:
Post a Comment