The Dangers of Poorly Done Research

The Dangers of Poorly Done Research, Written by Dr. Frederick C. Van Bennekom, Great Brook Consulting.

Imagine you're planning to conduct a survey to support some key business decision-making process.? But you have to get results fast.? So, you throw together a quick survey because something is better than nothing.? Right?? Wrong.? Why?

Because the survey process may contain biases and errors that lead to incorrect and misleading data.? Thus, the decisions will be made based on delusions of knowledge.?

Consider the following matrix.? (As a quasi-academic, it's a requirement to present ideas in 2x2 matrices).

??????????????????????????????????????????????????????????????????????????????Our True Knowledge of ????????????????????????????????????????????????????????????????????????????????????Objective Reality
Our Perception of Our Knowledge of Reality	We Do Understand	We Do Not Understand
We Think We Understand	Nirvana	Delusions of Knowledge
We Think We Don't Understand	Area of Opportunity	No Loss

In this matrix, our true understanding of objective reality - we know the truth or in fact we don't - is contrasted to our perception of our understanding of reality - we think we know the truth or we think we don't.? Let's look at each quadrant.

Northwest Quadrant: We think we know reality and in fact we do.? This is our goal.? We want to live in this cell.? Nothing but blue skies do we see…

Southwest Quadrant: We think we don't know reality but in fact we do.? This is a good zone because of the opportunity it presents.? We're not going to run off half-cocked and do something stupid.? We're cautious in this position awaiting someone - a manager, a mentor, our mate - to tell us that we know more than we thought and to give ourselves more credit.

Southeast Quadrant: We think we don't know reality and in fact we don't.? This is another good zone since we're unlikely to make wrong decisions.? However, that opportunity in the previous quadrant is lacking since we lack real knowledge.? Our mentor first has to help us learn, and then realize what we have learned.? We want to move to the Southwest quadrant and not to the Northeast quadrant.

Northeast Quadrant: We think we know reality but in fact we don't.? This is our black hole of decision making where we're operating under delusions of knowledge.? Many versions of this exist.? Maybe one customer or employee complains and the assumption is made that this is a real problem that everyone confronts.? Maybe we've done some “research” but the research plan or execution is flawed.? A whole host of reasons can create delusions of knowledge.? You probably know many people who fit this quadrant: “He doesn't know what he's talking about.”

But how could a survey research project create such delusions?? “A survey is so simple,” you say.? “I get surveys in the mail all the time, and I've thrown together my own surveys.? All research is beneficial.? How could any survey effort not help?? This isn't brain surgery, after all.”? But it is (sort of).? We are trying to get into the brain of some group of interest -- our customers, employees, stockholders, suppliers, etc. - and measure their perception of us and perhaps collect some other hard data.?

A survey project presents many, many opportunities to screw up. The list below shows the areas where biases or errors can affect our survey results - and thus our decisions.

Questionnaire Design

Instrumentation bias (bias from question wording, etc.)

Questions that do not support desired analysis

Survey Administration

Incorrect sampling procedure

Improperly defining the relevant population

Administration technique bias

Non-response bias???
Response bias??

Data Analysis

Bad model for analysis

Weak presentation of the findings?

Error in analysis?

In any research program, including surveying, the researcher faces a constant struggle to avoid introducing biases into the data and errors into the findings and conclusions.? For a novice surveyor real danger exists.? Why?? Because the novice isn't even aware that these biases could exist, they don't know enough to avoid them, they won't know they've created these biases, and they won't know to take caution when interpreting the collected data set.? The novice will execute the flawed project, analyze the data and draw conclusions using data that doesn't represent the true feelings of our survey population, our group of interest.?

Future articles will address each of those three major areas in depth.? I'll add links when they're written.? The bottom line message is that rigor is needed in any research project to be sure that you're truly learning something about the area of interest.?

In summary, would you rather make decisions based on:

1) Sound research data, or
2) Data you thought was right but was wrong or
3) Gut feel and business intuition??

While I would always prefer the first choice, I'd rather operate from intuition than delusions of knowledge.? As some wise folks over the millennia have stated in various forms:

''The trouble with most folks isn't so much their ignorance. It's know'n so many things that ain't so.'' - 19th century humorist Josh Billings

“To know that you do not know is the best.? To pretend to know when you do not know is a disease.”? Lao-Tzu

"It ain't what you know that hurts you. It's what you do know that ain't so."???? Will Rogers

I'm sure Mark Twain, Winston Churchill, and Disraeli had a version of the same quote.?