Join Our Newsletter

Events Calendar

« < March 2017 > »
26 27 28 1 2 3 4
5 6 7 8 9 10 11
12 13 14 15 16 17 18
19 20 21 22 23 24 25
26 27 28 29 30 31 1
The Dangers of Poorly Done Research PDF Print E-mail
Written by Dr. Van Bennekom   
13 Jul 2005

The Dangers of Poorly Done Research, Written by Dr. Frederick C. Van Bennekom, Great Brook Consulting.

Imagine you're planning to conduct a survey to support some key business decision-making process.? But you have to get results fast.? So, you throw together a quick survey because something is better than nothing.? Right?? Wrong.? Why?

Because the survey process may contain biases and errors that lead to incorrect and misleading data.? Thus, the decisions will be made based on delusions of knowledge.?

Consider the following matrix.? (As a quasi-academic, it's a requirement to present ideas in 2x2 matrices).

??????????????????????????????????????????????????????????????????????????????Our True Knowledge of

????????????????????????????????????????????????????????????????????????????????????Objective Reality
Our Perception of Our Knowledge of Reality

We Do Understand
We Do Not Understand

We Think We Understand

Delusions of Knowledge
We Think We Don't Understand

Area of Opportunity
No Loss

In this matrix, our true understanding of objective reality - we know the truth or in fact we don't - is contrasted to our perception of our understanding of reality - we think we know the truth or we think we don't.? Let's look at each quadrant.

Northwest Quadrant: We think we know reality and in fact we do.? This is our goal.? We want to live in this cell.? Nothing but blue skies do we see…

Southwest Quadrant: We think we don't know reality but in fact we do.? This is a good zone because of the opportunity it presents.? We're not going to run off half-cocked and do something stupid.? We're cautious in this position awaiting someone - a manager, a mentor, our mate - to tell us that we know more than we thought and to give ourselves more credit.

Southeast Quadrant: We think we don't know reality and in fact we don't.? This is another good zone since we're unlikely to make wrong decisions.? However, that opportunity in the previous quadrant is lacking since we lack real knowledge.? Our mentor first has to help us learn, and then realize what we have learned.? We want to move to the Southwest quadrant and not to the Northeast quadrant.

Northeast Quadrant: We think we know reality but in fact we don't.? This is our black hole of decision making where we're operating under delusions of knowledge.? Many versions of this exist.? Maybe one customer or employee complains and the assumption is made that this is a real problem that everyone confronts.? Maybe we've done some “research” but the research plan or execution is flawed.? A whole host of reasons can create delusions of knowledge.? You probably know many people who fit this quadrant: “He doesn't know what he's talking about.”

But how could a survey research project create such delusions?? “A survey is so simple,” you say.? “I get surveys in the mail all the time, and I've thrown together my own surveys.? All research is beneficial.? How could any survey effort not help?? This isn't brain surgery, after all.”? But it is (sort of).? We are trying to get into the brain of some group of interest -- our customers, employees, stockholders, suppliers, etc. - and measure their perception of us and perhaps collect some other hard data.?

A survey project presents many, many opportunities to screw up. The list below shows the areas where biases or errors can affect our survey results - and thus our decisions.

Questionnaire Design
Instrumentation bias (bias from question wording, etc.)
Questions that do not support desired analysis

Survey Administration
Incorrect sampling procedure
Improperly defining the relevant population
Administration technique bias
Non-response bias???
Response bias??

Data Analysis
Bad model for analysis
Weak presentation of the findings?
Error in analysis?

In any research program, including surveying, the researcher faces a constant struggle to avoid introducing biases into the data and errors into the findings and conclusions.? For a novice surveyor real danger exists.? Why?? Because the novice isn't even aware that these biases could exist, they don't know enough to avoid them, they won't know they've created these biases, and they won't know to take caution when interpreting the collected data set.? The novice will execute the flawed project, analyze the data and draw conclusions using data that doesn't represent the true feelings of our survey population, our group of interest.?

Future articles will address each of those three major areas in depth.? I'll add links when they're written.? The bottom line message is that rigor is needed in any research project to be sure that you're truly learning something about the area of interest.?

In summary, would you rather make decisions based on:

1) Sound research data, or
2) Data you thought was right but was wrong or
3) Gut feel and business intuition??

While I would always prefer the first choice, I'd rather operate from intuition than delusions of knowledge.? As some wise folks over the millennia have stated in various forms:

''The trouble with most folks isn't so much their ignorance. It's know'n so many things that ain't so.'' - 19th century humorist Josh Billings

“To know that you do not know is the best.? To pretend to know when you do not know is a disease.”? Lao-Tzu

"It ain't what you know that hurts you. It's what you do know that ain't so."???? Will Rogers

I'm sure Mark Twain, Winston Churchill, and Disraeli had a version of the same quote.?

Fred Van Bennekom, Dr.B.A.
Principal Great Brook Consulting
Author, "Customer Surveying: A Guidebook for Service Managers."
Bolton, MA 01740 USA

Last Updated ( 02 Aug 2005 )
< Prev   Next >


How important is market research to start-ups in the current economic climate?

RSS Feeds

Subscribe Now