Trochim’s Survey Research Methods – Knowledge Base [Evelyn]
Trochim presents thorough and easily understandable instruction on how to develop an effective survey instrument. Provided below is a summary; however, his work is so fundamentally rich in explaining the basic how-to’s and rationale for designing the survey as instructed, that I have placed a fuller treatment in the Dropbox. The document is useful as a survey design handbook and could be a beneficial tool in our research toolbox.
Survey research is one of the most important areas of measurement in applied social research. It includes any measurement procedures that involve asking questions of respondents, ranging from a simple, short paper-and-pencil feedback form to an intensive one-on-one in-depth interview. The two broad types of surveys are questionnaires and interviews. The distinction between the two is sometimes blurred, as they share some characteristics.
1) Know the two types of surveys and the pros and cons of each.
2) Know how to select the appropriate survey method using key area guiding questions.
3) Know how to construct a survey; specifically, how write survey questions:
- Types of questions
- Question content
- Response format
- Question wording
- Question placement
4) Know how to prepare for and conduct interviews.
The path of survey research can be guided by the following key steps and overarching questions, with more focused questions asked as the survey development progresses:
- Set the goals – What do you want to capture?
- Decide on the target population and sample size – Who will you ask?
- Determine the questions- What will you ask?
- Pre-test the survey – Test the questions
- Conduct the survey – Ask the questions
- Analyze the data collected – Produce the report
TYPES OF SURVEYS
Questionnaires -There are three types, each with pros and cons: mail questionnaires, group-administered questionnaires, and household drop-off surveys. Questionnaires are typically written; completed by the respondent; and can include both closed- and open-ended questions.
Mail questionnaires are low-cost; easy to administer; convenient for the respondent; and the same instrument can be used for a broad audience. They tend to have a low response rate and are not the best instrument to ask detailed questions.
With a group-administered questionnaire, a sample of respondents is gathered together to respond to structured questions. Each respondent is handed an instrument to complete while in the room. Respondents work individually. Pros are the convenience of the group setting, high response rate, ease of assembling groups in an organizational setting, and opportunity to clarify ambiguities on-the-spot.
With a household drop-off survey, the interviewer drops off the instrument to respondent’s home. The respondent mails it back or interviewer picks it up. It blends the advantages of the privacy and convenience of the mail questionnaire with the personal interaction and ability of the respondent to ask questions afforded by the group method There is an expected increased response rate.
Interviews are more personal than questionnaires. The interviewer can ask follow-up, probing questions. They are easier for the respondent, especially if impressions or opinions are requested. They can be time-consuming and are resource-intensive. The interviewer is considered part of the instrument and has to be trained how to respond to contingencies. In a group interview, the group works together, listens to each other’s comments, and answers the questions. The interview is not completed individually. Someone takes notes for the group. Group interviews have evolved into focus group methodology.
Telephone interviews facilitate rapid information-gathering. Most public opinion polls are based on telephone interviews. They allow personal contact between interviewer and respondent. The interviewer can ask probing and follow-up questions. Cons are that many people don’t have telephones or publicly-listed phone numbers, can consider telephone calls intrusive, and can consider the telephone interview an imposition, so it needs to be kept relatively short.
Trochim’s Quick-and-Dirty Pros and Cons of Survey Methods
|Are Visual Presentations Possible?||Yes||Yes||Yes||Yes||No|
|Are Long Response Categories Possible?||Yes||Yes||Yes||???||No|
|Is Privacy A Feature?||No||Yes||No||Yes||???|
|Is the Method Flexible?||No||No||No||Yes||Yes|
|Are Open-ended Questions Feasible?||No||No||No||Yes||Yes|
|Is Reading & Writing Needed?||???||Yes||Yes||No||No|
|Can You Judge Quality of Response?||Yes||No||???||Yes||???|
|Are High Response Rates Likely?||Yes||No||Yes||Yes||No|
|Can You Explain Study in Person?||Yes||No||Yes||Yes||???|
|Is It Low Cost?||Yes||Yes||No||No||No|
|Are Staff & Facilities Needs Low?||Yes||Yes||No||No||No|
|Does It Give Access to Dispersed Samples?||No||Yes||No||No||No|
|Does Respondent Have Time to Formulate Answers?||No||Yes||Yes||No||No|
|Is There Personal Contact?||Yes||No||Yes||Yes||No|
|Is A Long Survey Feasible?||No||No||No||Yes||No|
|Is There Quick Turnaround?||No||Yes||No||No||Yes|
SELECTING THE SURVEY METHOD
Method selection is a critical decision but, in many contexts, there is no clear and easy way to make this decision and no singularly clear best approach. It is more art than science, a judgment call that weighs and trades off the pros and cons of the survey types. Important: If you select an inappropriate method or one that doesn’t fit the context, you can doom a study before you even begin designing the instruments or questions themselves. (Think back to Immekus’s Research Methods and Measures course re how to select a research method for your research question.) Use guiding questions about: the population; sampling; types of questions needed; content; bias; and administrative issues to help you decide on the method. Questions for each area are:
- Can the population be enumerated? Is it accessible?
- Is the population literate?
- Are there language issues?
- Will the population cooperate?
- What are the geographic restrictions
- What data is available?
- Can respondents be found?
- Who is the respondent?
- Can all members of population be sampled?
- Are response rates likely to be a problem?
Sometimes the nature of your questions will determine the type of survey you select.
- What types of questions can be asked?
- How complex will the questions be?
- Will screening questions be needed?
- Can question sequence be controlled?
- Will lengthy questions be asked?
- Will long response scales be used?
The content of your study can also pose challenges for different survey types.
- Can the respondents be expected to know about the issue?
- Will respondent need to consult records?
People come to the research endeavor with their own sets of biases and prejudices. Sometimes, these biases will be less of a problem with certain types of survey approaches.
- Can social desirability be avoided?
- Can interviewer distortion and subversion be controlled?
- Can false respondents be avoided
Consider the feasibility of the survey method for your study.
- Cost is often the major determining factor in selecting survey type. Can you afford the postage to send out an extensive mailing?
- Facilities – Do you have the facilities (or access to them) to process and manage your study?
- Time –Do you need responses immediately (as in an overnight public opinion poll)? Have you budgeted enough time to mail the surveys and follow-up reminders, and get the responses back by mail?
- Personnel – Interviews require motivated and well-trained interviewers. Group-administered surveys require trained group facilitators. Technical studies might require the interviewer to have some degree of expertise.
CONSTRUCTING THE SURVEY
Survey construction is an art. Numerous small decisions — about content, wording, format, placement — can have important consequences for your entire study. While there’s no one perfect way to accomplish this job, there are ways to increase your chances of developing a better final product. Some aspects of survey construction are just common sense, but if you are not careful, you can make critical errors that have dramatic effects on your results. Once you have your questions written, there is the issue of how best to place them in your survey.
Thee three areas involved in writing a survey question are:
- determining the question content, scope and purpose
- choosing the response format that you use for collecting information from the respondent
- figuring out how to word the question to get at the issue of interest
Types of Questions
Survey questions can be divided into two broad types: structured and unstructured.
- Dichotomous Questions – only two possible responses: e.g., Yes/No, True/False, M/F.
- Questions Based on Level of Measurement – Remember “noir,” the French ford for “black” for the four levels of measurement– nominal, ordinal, interval, and ratio.
These are open-ended questions such as, “Tell me three things that you like about the program and three areas you would like to see improved. Open-ended questions call for an unstructured response format.
For each question in your survey, ask yourself how well it addresses the content you are trying to derive.
- Is the question necessary/useful?
- Are several questions needed?
- Do respondents have the needed information?
- Does the question need to be more specific?
- Is the question sufficiently general?
- Is the question biased or loaded?
- Will the respondent answer truthfully?
The response format is how you collect the answer from the respondent. Structured and unstructured formats are distinct and have their pros and cons.
Top of Form
Structured Response Formats – help the respondent respond more easily and help the researcher to accumulate and summarize responses more efficiently. But, they can also constrain the respondent and limit the researcher’s ability to understand what the respondent really means. These include: fill-in-the-blank; check the answer, and circle the answer. Whenever you use a checklist item, be sure that you ask whether:
- All of the alternatives are covered
- The list is of reasonable length
- The wording is impartial
- The form of the response is easy and uniform
If you are not sure all options are there, allow the respondent to write in another option.
Unstructured Response Formats – There are relatively few unstructured ones. Generally, it’s written text. If the respondent (or interviewer) writes down text as the response, that’s an unstructured response format. These can vary from short comment boxes to the transcript of an interview. Open-ended questions call for an unstructured response format.
Slight wording differences can confuse the respondent or lead to incorrect interpretations of the question. Here are questions you can ask about how you worded each of your survey questions.
- Can the question be misunderstood?
- Is the time frame specified?
- How personal is the wording?
- Is the wording too direct?
- Other wording issues
One of the most difficult tasks facing the survey designer involves the ordering of questions. If you leave your most important questions until the end, your respondents might be too tired to give them the attention you would like. If you introduce them too early, they may not yet be ready to address the topic. Use your judgment. Consider the following questions about placement:
- Is the answer influenced by prior questions?
- Does the question come too early or too late to arouse interest?
- Does the question receive sufficient attention?
The Opening Questions – First impressions are important. The first few questions determine the tone for the survey and can help put your respondent at ease. Thus, the opening few questions should, in general, be easy to answer. Start with some simple descriptive questions to get the respondent going. Never begin with sensitive or threatening questions.
Sensitive Questions – Before asking difficult, sensitive, or uncomfortable questions, attempt to develop some trust or rapport with the respondent by starting with easier warm-up ones. Use a transition sentence between sections of your instrument to give the respondent some idea of the kinds of questions that are coming.
A Checklist of Considerations – There are lots of conventions or rules-of-thumb in the survey design business. Use this checklist of important items to review your survey instrument:
- Top of Formstart with easy, nonthreatening questions
- put more difficult, threatening questions near end
- never start a mail survey with an open-ended question
- for historical demographics, follow chronological order
- ask about one topic at a time
- when switching topics, use a transition
- reduce response set (the tendency of respondent to just keep checking the same response)
- for filter or contingency questions, make a flowchart
Bottom of Form
The Golden Rule – You are imposing in the life of your respondent and are asking for their time, attention, trust, and often, for personal information. Do unto your respondents as you would have them do unto you! In practical terms:
- Thank the respondent at the beginning for allowing you to conduct your study
- Keep your survey as short as possible — only include what is absolutely necessary
- Be sensitive to the needs of the respondent
- Be alert for any sign that the respondent is uncomfortable
- Thank the respondent at the end for participating
- Assure the respondent that you will send a copy of the final results
Interviews require personal sensitivity, adaptability, and the ability to stay within the bounds of the designed protocol. Following is a description of the required preparation and process of conducting the interview.
Preparation – The interviewer’s role is complex, multifaceted, and includes the following tasks:
- Locate and enlist cooperation of respondents.
- Motivate respondents to do good job.
- Clarify any confusion/concerns.
- Observe quality of responses.
- Conduct a good interview.
Training the Interviewers – In many ways the interviewers are your measures and the quality of the results is totally in their hands. Organize in detail and rehearse the interviewing process before beginning the formal study. Major topics that should be included in interviewer training:
- Describe the entire study.
- State who is sponsor of research.
- Teach enough about survey research so that the interviewers respect the survey method and are motivated.
- Explain the sampling logic and process.
- Explain interviewer bias.
- “Walk through” the interview.
- Explain the respondent selection procedures.
The Interviewer’s Kit – should include all of the materials they need to do a professional job and be easily carried. Important materials include a “professional-looking” 3-ring notebook; maps; sufficient copies of the survey instrument; official identification (preferable a picture ID); a cover letter from the principal investigator or sponsor; a phone number the respondent can call to verify the interviewer’s authenticity
Whether it is a two-minute phone interview or a personal interview that spans hours, every interview includes some common components: the opening; the middle game; and the endgame.
Opening Remarks – get the respondent’s attention to persuade participation.
- Gaining entry – dress professionally and cultivate a manner of professional confidence to put the respondent at ease. Smile. Be brief. State why you are there and suggest – don’t ask — what you would like the respondent to do.
- Introduction. Without waiting for the respondent to ask questions, introduce yourself. Memorize this part of the process so you can deliver the essential information in 20-30 seconds at most.
- Explaining the study. There are three rules to this critical explanation: 1) Keep it short; 2) Keep it short; and 3) Keep it short! You should have a one or two sentence description of the study memorized. No big words. No jargon. No detail. Bring materials to leave. This is the “25 words or less” description. You should spend some time assuring respondent of the confidentially of the interview and that their participation is voluntary.
The Middle Game – Asking the Questions
- Use questionnaire carefully, but informally.
- Ask questions exactly as written. Don’t try to improve on it by altering a few words. to make it simpler or more “friendly.” Ask the questions as they are on the instrument so that the interview is standardized.
- Follow the order given.
- Ask every question. Do not be tempted to omit a question because you thought you already heard what the respondent will say. Do not assume anything.
- Don’t finish sentences. If you finish their sentence for them, you imply that what they had to say is transparent or obvious, or that you don’t want to give them the time to express themselves in their own language.
Obtaining Adequate Responses – The Probe – To elicit a more thoughtful response to a brief, cursory answer, you probe.
- Silent probe. The most effective way to encourage someone to elaborate is to do nothing at all – just pause and wait. It works because people are uncomfortable with pauses or silence. It suggests that you are waiting, listening for what they will say next.
- Overt encouragement. At times, you can encourage the respondent directly. Try to do so in a way that does not imply approval or disapproval of what they said (that could bias their subsequent results). It could be as simple as saying “Uh-huh” or “OK” after the respondent completes a thought.
- Elaboration. You can encourage more information by asking for elaboration. “Would you like to elaborate on that?” or “Is there anything else you would like to add?”
- Ask for clarification. Sometimes, you can elicit greater detail by asking the respondent to clarify something that was said earlier. “A minute ago you were talking about the experience you had in high school. Could you tell me more about that?”
- Repetition. This is the old psychotherapist trick. You say something without really saying anything new. “What I’m hearing you say is that….” Then, pause. The respondent is likely to say something like “Well, yes… In fact, …”
Recording the Response – Most interview methodologists don’t think it’s a good idea to record a respondent. Respondents are often uncomfortable when they know their remarks will be recorded word-for-word and may strain to only say things in a socially acceptable way. Assuming the use of paper-and-pencil approach:
- Record responses immediately.
- Include all probes.
- Use abbreviations where possible to help you capture more of the discussion.
Concluding the Interview
- Thank the respondent. Even if he or she was troublesome or uninformative, it is important for you to be polite and thank them for their time.
- Tell them when you expect to send results. It’s common practice to prepare a short, readable, jargon-free summary of interviews that you can send to the respondents.
- Don’t be brusque or hasty. Allow for a few minutes of winding down conversation; however, use your judgment so that you do not overstay the visit.
- Immediately after the interview, go over your notes and make any other comments and observations — but be sure to distinguish these from the notes made during the interview (you might use a different color pen, for instance).
1. Which survey method best suits your current doctoral research question, and why?
2. With the hindsight of a year’s doctoral coursework, discuss a previous survey you conducted that you might have conducted differently had you known then what you know now about survey design and qualitative research.
3. Trochim asserts that we sometimes don’t stop to consider how a question will appear from the respondent’s point-of-view or the assumptions behind our questions. Discuss some common survey assumptions you have encountered or are aware of that have had survey questions that were inappropriate or not easily understood by respondents.
4. For any variety of reasons, people who consent to an interview might not provide accurate responses or might refuse to respond, resulting in refusal or response bias. Discuss strategies that could be employed to elicit forthright responses from an evasive or reluctant respondent or how you could account for the bias if you were unable to overcome it.
5a. Design a brief customer satisfaction survey (or other survey instrument) of the CSUF/CSUB doctoral program targeted at Cohorts 1 and 2 or CSUF / CSUB faculty. What are your guiding questions?
5b. Design a brief survey instrument to gauge the interest in and need for of a doctoral program in Kern County that could be used to inform DPELFS.