Most of this book describes formal survey methods; a technique sometimes known as quantitative research. Surveys work like this:
- The population is defined, and a representative sample is selected.
- A questionnaire is prepared, in which everybody is asked exactly the same questions, with exactly the same wording.
- The results of the survey come from counting the number of people who gave each answer to each question.
Qualitative research is quite different. Though the process is very different from surveys surveys, it can be used to reach similar conclusions. Qualitative research works like this:
- The population is defined, in the same way as a survey.
- Respondents are selected, using a sampling method.
- Respondents are often interviewed in groups.
- Instead of a questionnaire, a list of topics is used.
- The results are not expressed in numerical form, and formal counts are seldom made.
But both types of research share the same context: typically this:
- The managers of an organization have a problem. To solve it, they feel they need data about their audience, users, or customers.
- A research study is carried out.
- The results of this study are used to solve the organization’s problem, by producing audience data.
The organization’s managers may not care exactly how the research is done; they simply want an answer to their questions, or a solution to their problems. From this point of view, it may not matter what form the research takes.
Qualitative research produces a wealth of information, not in the form of numbers, but in the form of words. People whose inclinations are verbal rather than mathematical (like most media workers I know) often have trouble interpreting the results of surveys, but they find the results of qualitative research easier to understand and use. However, qualitative research has been regarded as too difficult for untrained people to do successfully. The prerequisite has normally been an advanced degree in psychology.
The most common form of qualitative research is a focus group.
The most common form of qualitative research is the focus group. These are widely used for assessing the viability of proposed new services or products. In each group, about 8 people meet to discuss a particular issue. The group is led by a highly trained moderator, who begins the discussion at a very general level, then gradually focuses in on the specific topic. Respondents are not told this topic in advance, only the broad area of interest.
For example, if a TV station wants to assess a new type of current affairs program, the people chosen for the group could be those interested in watching information programs on television. The discussion might begin with the types of information program participants like and dislike, and the reasons for those feelings. The discussion might then move onto current affairs programs in general, then some specific current affairs programs, then onto current affairs programs on that channel. At this point the participants might be shown a short pilot of the proposed new program, and asked to discuss it. Such a group typically lasts from 1 to 2 hours.
So the focusing begins with the general, and moves towards the particular. Focusing is also called funnelling - but a focus group is never called a funnel group.
Everything the participants say is recorded, on either audio or videotape. For the moderator, the hard work now begins. The actual moderating can be learned quickly - it’s mainly a matter of ensuring that everybody has a chance to speak, that some don’t dominate others, and so on. However the analysis is much more difficult and time-consuming. The moderator often watches the video or listens to the tape several times, noting participants’ expressions and gestures as much as the content of their speech. Advanced training in psychology is almost necessary, as is long experience at running focus groups. With untrained moderators, interpretation of focus groups is highly subjective: two different moderators may reach quite different conclusions about the participants’ reaction to whatever was being studied.
When the moderator has studied the tapes or transcripts, he or she writes a report. There is no simple method of converting what participants say and do into conclusions.
Focus groups are usually done in sets of 3 to 6 groups, each group with a different type of person. For example, the assessment of a pilot current affairs TV program might require 4 groups: perhaps men under 35, men over 35, women under 35, and women over 35.
To analyse focus group proceedings thoroughly usually takes a full day’s work for each group, then another day or two to write the report. For this reason, commissioning focus groups from market research companies is expensive. Though few people take part in the groups, far more time is spent on each person than on interviewing respondents in a survey - and a much higher level of skill is needed than for normal interviewing.
Because of the high cost of professionally organized focus groups, some organizations are now running their own focus groups, and even gaining some useful insights. However, their lack of experience often leads them to misleading conclusions. It may seem to them that their customers are highly satisfied, or would watch a proposed new program in large numbers. Later, they often discover that their conclusions were wrong: that the innovation that the participants seemed to welcome is far from popular among their whole audience.
In the mid-1980s, I produced the first edition of my Radio Survey Cookbook, so that radio station staff could do their own surveys. Some of them managed to do successful surveys, but others found that surveys required too much work, too much organization, and too much skill at manipulating numbers. So I set out to develop a simpler research method.
Bearing in mind the skills of radio staff, and the great interest they took in reports from focus group studies, I gradually developed a qualitative research method that could be done easily and accurately by people with little training. This is the consensus group, which is sort of halfway between a focus group and a public meeting. It also includes elements of other research techniques and negotiation techniques.
Principles of consensus groups
In every survey, the questionnaire ensures that everybody is asked the same questions. The only variation can be in the number of people giving each answer to each question. So surveys begin with words (questionnaires), but the results are always expressed in numbers.
Consensus groups work in the opposite way: the numbers remain constant (more or less), but the wording of each statement is adjusted until the great majority of participants agree.
It’s important to realize that a consensus group does not try to create a consensus among participants: that’s peace-making. This is research: it simply tries to find and define any consensus that already exists. Unlike a focus group (which narrows in on a single topic) a consensus group normally covers a broad range of topics.
The technique has two main stages: recruiting participants, and holding discussions. Like focus groups, consensus groups are never done singly, because with a small number of participants, any one group may be atypical. Consensus groups are normally done in sets of 3. There can be more than 3, but every extra group adds less and less information.
Before participants are recruited, the planning process is much the same as a survey: the organizers must decide what is to be covered, among what population. When the subject and scope of the study have been decided, nine steps follow:
1. Within the area to be studied, three (or more) sampling points are chosen, contrasting as much as possible.
2. At each of the sampling points, a venue is arranged. All you need is a suitable room.
3. A short screening questionnaire is prepared.
4. At each sampling point, people are interviewed using the screening questionnaire. The purpose of these interviews is to find people who are both eligible and willing to attend a consensus group.
5. The group meets, either immediately, or up to several weeks later. Each meeting lasts for two to three hours. Approximately 12 participants are present, as well as 2 organizers: a moderator and a secretary.
6. The first stage of the meeting is introductory. The participants briefly introduce themselves, giving some relevant background information.
7. In the second stage of the meeting, the topics are discussed by all participants. The moderator manages the discussion, ensuring that everybody speaks freely, while the secretary takes notes.
8. In the final stage of the meeting, consensus is sought. The secretary puts up statements which most participants are expected to agree with. Statements are modified, depending on what participants say. When each statement is ready, participants vote on it. On average, about 20 statements are agreed on. This list of statements is the main outcome of the meeting.
9. When three meetings have been held, the three lists of statements are compared. Any statements shared by at least two groups are the outcome of the study.
The rest of this chapter considers the nine steps in detail.
1. Choose sampling points
A sampling point is the centre of a geographical area, where a small survey can be carried out. It can be either a group of homes, or a public place where passers-by are recruited for the study. For consensus group sampling to work effectively, at least three sampling points are needed. They are chosen to be as different as possible. For example, if your study is being done for a radio station, the population will probably be all people who live in that radio station’s coverage area. Within this area, you should identify three contrasting localities, where the people are as different as possible from each other. For example, if the station broadcasts to a city and outlying rural areas, you might choose:
• An inner-city area
• An area near the outer edge of the city
• A rural area.
Another way in which localities often vary is in wealth. Therefore it would be useful to choose one wealthy area, one poor area, and one middle-income area. If there are ethnic, racial, or tribal divisions among your audience, it may be essential to have a separate consensus group for each ethnic group.
In some countries, women won’t give their true opinions in the presence of men, so you need to have separate consensus groups for men and women.
Whatever the basis for selecting the sampled areas, the main goal is to make them as different as possible. This is known as maximum-diversity sampling. It is explained in detail in the chapter on sampling.
There is no reason why you should not have more than three sampling points. We have found that each additional sampling point adds less and less to the value of the study. However, in some situations, more than three sampling points are needed to adequately cover the variations in a population. A technique we have often used compares a station’s potential listeners with its current listeners. This requires choosing two groups at each sampling point, or 6 groups in total. If both the sampling points and the type of person invited to a group are different, you cannot make clear conclusions about the causes of any differences in the results.
If you hold separate groups for men and women, you will probably need four groups: two or men and two of women. If you want to be able to compare the opinions of men and women, you need to use the same sampling points for each sex.
2. Organize a venue
A venue for a consensus group is a space that will hold about 15 people and is free from interruptions. It need not be a room; in Papua New Guinea we held some group discussions outdoors, with no problems.
In some countries, hotels and restaurants often have rooms available for hire for meetings. We have also used clubrooms, and borrowed office areas outside working hours. Another possibility is to use a private house with a good-sized room, paying the owner a small fee for the use of their space.
It is often better not to use the premises of the organization sponsoring the research, as people may be reluctant to criticize an organization when they are on its premises: for example, if your research is for a radio station, avoid using its office as a venue. But this depends on the type of people attending, and on the organization. In Hanoi, Vietnam, we held some consensus groups in the head office of the national radio network, the Voice of Vietnam. Despite the imposing building, and the presence of numerous Voice of Vietnam staff, the participants weren’t inhibited in giving their opinions. But in nearby Laos and Cambodia, where people are less assertive, such a venue would not have been successful.
Here are some factors to take into account when choosing a venue:
• We usually provide something to eat and drink for the participants. We find it helps them to relax. One advantage of hotels and restaurants is that catering is included.
• A venue should be easy to find, specially for people who have never been there before. Large public buildings are usually well known to people living in the area.
• A venue should not be a place that some people would not want to visit. In some cultures, for example, women do not like to go to hotels.
• A venue should be quiet, particularly if the meeting is to be recorded on audio tape. Noisy air-conditioning can be an unexpected problem, making voices much more difficult to understand, even though participants may be hardly aware of the background noise. Noise is often a problem in hotels and restaurants.
• The venue must be close to the sampling point. We have found that some people are reluctant to travel for more than about 15 minutes to a group discussion. Sometimes you may have to change a sampling point, because no suitable venue can be found there.
3. Prepare a screening questionnaire
The purpose of the screening survey is to find participants for the consensus groups: people who are both eligible and willing to participate. If, for example, you are assessing the programs on a radio station, there is little point in inviting people who don’t listen to that station. The key question on the screening questionnaire could be
"Do you listen to FM99 at least once a week?"
(This is better wording than asking simply "Do you listen to FM99?" If no time limit is included, people who listen only very rarely to the station would answer Yes, and would not be able to discuss the programs in detail.)
If you are interested in increasing the station’s audience, you will need to speak to potential listeners to the station. There are several ways to define potential listeners on a questionnaire. When a station increases its audience, this is usually because people who did formerly listen to the station (but infrequently) began to listen more often. So most of your potential listeners are probably occasional listeners already. In a screening survey, you could ask
"How often do you listen to FM99: at least once a week, occasionally, or never?"
All people answering "occasionally" would be considered potential listeners.
If you are trying to assess the audience to something that does not yet exist (such as a planned new radio station), you will need to define the potential listeners in another way. This can be done from two starting points:
(a) A demographic group, such as "rich people living near the city." If these were defined as the target audience, the purpose of the study would be to find the kind of radio program that most appealed to these people.
(b) A program idea, for example "a radio station specializing in jazz, blues, and reggae." In this case, the purpose of the study would be to estimate how many people are interested, what type of people they are, and exactly what program content would most interest them. You may need to do a small survey to find the participants for consensus groups.
When you have found a respondent who is eligible to take part in a consensus group, the next step is to find out if they will attend. We normally use wording like this.
"With the answers you’ve given, you’re eligible to take part in a group discussion, to talk about news and current affairs in more detail. We’d like to invite you to come to this discussion next Tuesday night, the 15th of October. We’re holding it at the Flinders Lodge, in Dequetteville Terrace, Kent Town, starting at 7pm, and finishing no later than 10pm. People usually find these meetings very interesting, and if you attend we’ll pay you thirty dollars to cover your expenses. Would you like to come along?"
1 Not interested
2 Interested, but can’t come
3 Agreed to come -> get name and address, and say we’ll send a letter
Name . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Address . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
The essential points included in the above script are:
• Invitation to attend
• Approximate description of the subject
• Date, time, and place of meeting
• Maximum duration of meeting
• Incentives, such as payment to respondents, offer to feed them, meet interesting people, and so on.
The third type of question that can be included in a screening questionnaire is the demographic question: their sex, their age group, and perhaps their occupation and other similar data. There are two reasons for obtaining this demographic information:
• To help ensure a proper balance of sexes, age groups, etc. in the groups.
• To help find out the differences between the type of people who attend the groups and those who do not.
4. Find participants for the discussions
Interviewing for a screening survey is done in the way described in the chapter on interviewing. However when you do a screening survey for consensus groups, it is not essential to interview people in their homes, following a prescribed route. When you are looking for characteristics that everybody shares, sampling makes much less difference: for example, if you wanted to find out how many legs humans have, almost any sampling method would do.
Unless the group discussions are to be held immediately after the recruitment, it is best to use only one or two interviewers at each sampling point. Because you will be aiming for 12 participants in each group, not many interviews will be required — unless the people eligible to take part are only a small percentage of the population.
In order for 12 people to turn up, you will probably need more than 12 to agree. In Australia, even when we send a letter to confirm the details of the discussion, and then telephone each participant the day before the discussion, up to 20% of those who accept will fail to turn up. We have also found that people who say they "might" come usually don’t. Therefore, we usually get acceptances from two more people than we really want: if we want 12, we obtain 14 acceptances.
Attendance rates are higher when:
- Participants are paid well for attending.
- They are allowed to attend with a friend or relative.
- The meeting is at a convenient time of day.
- Participants are mostly over 25.
- Participants are regular users of your service, and they know the group is about that service.
- The lead time between the invitation and the group is short.
- Participants are reminded of the meeting the day before it takes place.
- Participants are sent letters confirming the arrangements.
- These letters have practical details of how to get to the venue - e.g. a map
The worst way to organize a consensus group is to extend a weak invitation to a lot of people to come. As this is very easy, it may seem tempting. If you are running a radio station, you may think "Why not advertise on air that we are doing a research study, and invite listeners to come along?"
The problem here is that you have no control over the number of people who turn up. It could be nobody, and it could be hundreds. We have found that 12 people is about the ideal number for a consensus group. With fewer than about 8 participants, there is too much danger of the responses being atypical. With more than about 15, many participants are unable to give their complete opinions.
In Australia, usually less than 50% of people eligible to attend a group will agree to do so - but in developing countries, usually 90% will agree. Considering all the people who are not eligible, and the eligible people who do not want to come to the group, and those who say they will come but do not, sometimes it takes a lot of interviews to fill one group of 12.
For example, if one person in 10 is eligible, and a third of those attend a group, that’s 30 interviews for each person who attends, or 360 interviews to fill a group. It is therefore not a good idea to make the eligibility criterion too restrictive.
Another step you can take to reduce the number of interviews is to offer an incentive to attend. If you can persuade two thirds of the eligible people to attend instead of one third, only half as many interviews will be needed. Therefore, if you pay the people for attending, this can greatly reduce the total cost. (The participants get more money, but the interviewers get less.)
In Australia, we normally do the screening surveys between one week and two weeks before the discussion. If given more than two weeks’ notice, people tend to forget. With less than a week’s notice, many people can’t attend, because they have already made plans to do other things at the time.
But it is not essential to wait that long. Another possibility is to hold consensus groups on the spot. We did this in Hanoi, Vietnam, where a number of interviewers were sent out to interview people in the street outside the meeting venue. All people who met the criteria, and had an hour to spare, were invited to a group discussion then and there. It took only ten to fifteen minutes to find enough participants. Of course, this would not work unless a large number of eligible people were nearby.
The above description of screening questionnaires involves a separate interview for each person. In the resulting groups, each participant will usually not know any of the other participants — except in small towns, where many people already know each other. In some ways this can be an advantage, because participants will not offend their friends by giving opinions they feel the friends might disagree with. But in other ways, it can be a disadvantage to hold a discussion among strangers: participants may feel unwilling to reveal their opinions to people they do not know and cannot trust.
Which of these two disadvantages is the stronger will vary from country to country. In Australia, my experience is that when the participants in a group already know each other, they tend to express their feelings more freely, when the topic is one of broad interest - such as broadcasting. But if the topic is something that may embarrass people, such as sexual behaviour, participants will be more honest in the presence of people they have never met before and probably will never meet again.
Sometimes it is better to have separate groups of younger and older people. In some countries, it may be best not to mix supporters of different political parties in the same group. This separation can be done partly by careful selection of sampling points, and partly through screening questionnaires. Remember that the purpose of restricting a group to a particular type of person is to enable the participants to speak more freely to each other.
Another approach is to organize a discussion among a group of people who already know each other, such as members of a sporting club, a group of people who work together, or a group of students. These groups, formed by people who already know one other, are called affinity groups.
The groups need not have a special purpose: they can simply be groups of friends or neighbours. However, you should not have a group made up only of people from a single family. There is too strong a chance that they will not be typical of the population, because the entire study would then be limited to three families.
If the purpose of the group is to collect opinions, it’s usually best to discourage husbands and wives from coming together: they tend to inhibit each other. Often only one of them will join the discussion. As each group is quite small, it would be better to invite two people, who would give separate opinions. But if discovering behaviour is your main interest, husbands and wives can correct each other.
When affinity groups are used for a study, each group needs to be as different as possible from each other (replacing the three sampling points). For example, don’t choose three sporting clubs, or three groups of neighbours. This type of sampling is most effective when there is the largest possible contrast between the types of person in each consensus group.
One problem that restricts the use of affinity groups is that not everybody in an affinity group may be eligible. If a radio station is studying its listeners, it does not matter if a few people in an affinity group are not listeners, but if most people are non-listeners, the group will not provide useful information. Also, if people are not interested in the topic being studied, they may disrupt the discussion by talking among themselves. Therefore affinity groups are best when all (or almost all) the population are eligible to be participate.
Updated 26 May 2002
Article contributed by Audience Dialogue 19 October 2011