(Trochim) Research Methods Knowledge Base (The Qualitative Debate,…Qualitative Validity) [Mike]

Trochim, Research Methods Knowledge Base.

Trochim (2006) champions qualitative research as a method to develop new ideas and study phenomena in great detail and understanding.  As a tradeoff, qualitative research is not generalizable as opposed to experimental studies, and qualitative research finds much greater difficulty receiving funding.  This is due to the tendancy for that time frames in qualitative research to be much more nebulous, and research questions more likely to change with time.

However, Trochim (2006) also notes that in many cases, there is little real difference between qualitative and quantitative research.  Qualitative data can usually be coded quantitatively, and many supposed purely quantitative surveys use Lichert-scale measures that involve participants making qualitative judgments.   Trochim claims a main difference between the two methods is that qualitative researchers are far more concerned about the context of the phenomena being studied, while quantitative researchers look for the ability to generalize.

One other main difference Trochim (2006) claims between quantitative and qualitative research is that the traditional concept of validity and reliability is in some ways irrelevant to qualitative research.  Trochim instead lists a number of similar concepts: credibility, transferability, dependability, and reliability.  Interestingly, credibility is a measurement from the subjects’ perspective on whether research outcomes are believable. Transferability is related to quantitative generalizability:  can the conclusions be used in other contexts, or only in this specific context.  Dependability, a cousin to quantitative reliability, looks at whether outcomes are likely repeatable.  Confirmability speaks “to the degree to which the results could be confirmed or corroborated by others” (Trochim, 2006).

Four main approaches exist to conduct qualitative research: ethnography, phenomenology, field research, and grounded theory (Trochim, 2006).  Ethnography involves researchers immersing themselves in the culture or phenomenon to be studied and gather data from direct observation and interviews. In the Phenomenology approach, researchers investigate subjects’ subjective experiences.  Field research comprises researchers examining phenomena firsthand.  Grounded theory appears the most complex; after generating research questions, preliminary data is gathered and analyzed.   This data is used to generate new questions to focus further data; this eventually leads to the development of theories regarding many integrated sets of data and viewpoints.

Qualitative data exists in three main forms: transcripts or recordings of interviews, direct observations, and existing documents.  The methods used to collect this data are also varied.  Participant observation involves the researcher becoming involved and accepted in the native culture; this method is tied closely to the ethnographic approach.  Direct observation is distinct from participant observation in that the researcher is solely an observer.  Unstructured interviews in another method used; these interviews involve the researcher modifying questions based off of subjects’ answers.  Finally, case studies are often used to look at phenomena in a specific context.

In Trochim’s (2006) description of qualitative research, one yet unmentioned theme runs throughout.  Trochim claims that many qualitative researches are dubious that there is any one reality that exists; instead each person’s perceptions color their own reality, and no reality is the true reality.

Questions for discussion:

  1. Do you think that each person’s perceptions define their own reality or context?  Is there one true reality?
  2. How important is context in educational settings?
    1.  K-12: Assume well-planned intervention program for students at risk for the CAHSEE was implemented with pure fidelity at two schools, would it have similar results at Clovis West High School and McLane High School?
    2. 13+: Assume a new research-based basic skills program was mandated for all community colleges.  If implemented with fidelity, would it have similar results at COS and Reedley College?
  3. What is the difference in my thought pattern if I choose an 8 or a 9 on a 10 point scale?  Is the difference truly quantitative?
  4. Why do you think most educational research funds go to experimental research?
  5. Have you conducted interviews that were unstructured in any setting?  If so, was it more comfortable or less so?

24 thoughts on “(Trochim) Research Methods Knowledge Base (The Qualitative Debate,…Qualitative Validity) [Mike]

  1. 1. I think that it is nearly impossible to remove your experiences, values, and perceptions in order to be a purely objective observer. The best you can do is know your tendencies, and work to give equal weight to observations both in favor, and opposed to your beliefs.
    3. Without good descriptors for the various values they are pretty much the same. More steps give the impression of a more detailed response, but will people actually use the steps, or group things towards the ends and middle?
    4. In experimental quantitative studies the methods are clearly defined in advance. You know how log it will take, the resources needed, have a clear definition of how the results will be measured, and how they will be analyzed. It is also much easier to determine a final outcome based on the hypothesis testing. Qualitative research is much more up in the air.

  2. 1. Ultimately each person’s reality is going to shape and define their reality, even if their perceptions of reality is incorrect. Having observed and taught children for as long as I have, this concept has become more clear each day. Whatever the “true” reality is, the student’s perceptions become the only important reality to themselves. Students who suffer from paranoia and Schizophrenia perceive reality in different ways and to them reality is what they may or may not be perceiving.
    2. To say with certainty that the same program implemented in different high school settings would have similar results, would be a difficult assumption to make. Successful programs require extensive research and committed implementation on all levels. In a perfect world I believe any researched-based “best” practice, would have beneficial results on at-risk students.
    3. It is difficult to use quantitative measures to describe elements that are purely qualitative in nature. There seems to be too many variables that could determine the reasoning for the reasons why the respondent chose an 8 or 9. While this kind of Likard scale does have its place and is important, without qualitative input to go along with this information, the information is inconclusive.
    4. Experimental research is the final say-all end-all of the arguments and often why it gets most of the federal funding. Data in many eye’s proves with concrete examples. It is difficult to say something does not have an effect or vise-versa when data is backing the hypothesis.
    5. I often use unstructured types of interviews during an IEP meeting when I am looking for important input from the parent. Without knowing where the parents concerns may lie, I allow the parents responses to guide where the interview is headed.

    • Follow-up to #1:

      Granting that a person’s perceptions of reality are only what’s important to them, do you think there is a “true” reality that research can distinguish? Or should we concentrate on improving student’s experiences from their own frame?

      • I believe we need to concentrate on improving a student’s experience from their own reality. There is no one “reality.” What works in one setting will not work in another setting. I find this true even in schools that are considered similar. One experience does not equate another. At each school the student makeup and size may be similar but differences in attitudes, teachers, administrators, and culture can change the reality of that school.

      • I agree with Cari’s comments for the most part. I do agree that we should concentrate on improving student’s experiences from their own frame. However, the trick with this is that you also have the frame of the educator, administrator, parents, community, etc. to work with. I think we have to realize that some commonalities must exist for all concerned. We have to help students continue to develop their view of reality or “re-frame” that view to help them be the most successful. A difficult task indeed.

      • I agree, Donna.
        I think there are many commonalities that exist with which we can make some basic assumptions. The trick is knowing your population (administrators, teachers, parents, and students) and tailoring interventions to the individual frames that exist in your reality.

  3. 1. I agree with both Tim and James, as I suspect everyone will, with regards to an inability to have “one true reality.” We may be able to come to some form of consensus, but that consensus would likely look different to different people. We all have a lens that we look through based upon our own experiences. However, we can reframe our reality by acknowledging that everything we think we know is actually colored by our lens of life experiences. As James said, the best that we can do is make the necessary adjustments to filter out our own biases.

    2. Context is everything. Unless both schools were equally ranked at the beginning of the intervention with student to like student, which would include student attitude and internalized expectation, it will be impossible to have similar outcomes. It is comparing apples to oranges. However, one could measure the percentage of individual growth. If students from CWHS went up 20% in their scores but did not pass the CASHEE, while students from MHS went up 20% in their scores but did pass the CASHEE, one could say that both schools went up 20% – although outcomes would be different.

    3. We come back to perception. One person’s 8 could be another person’s 10 dependent on individual experience. As Tim said, “too many variables.” However, these variables could be managed somewhat by explicit descriptions for each place holder – but, even then, the quantitative outcomes may be unreliable.

    4. Funding institutions are likely looking for black and white, yes or no, type of research answers that are often associated with quantitative studies because they need to be able to go back to their own stakeholders and show that they have made wise investments. Qualitative research projects may not be able to provide this type of absolute outcome.

    5. All conversations / interviews that I have with parents/teachers/students are parent/teacher/student driven. I may have specific information that I need to gather, but the path I take to get this information is largely dependent upon how they answer the initial questions. I always consider my audience and the emotional needs that they bring to the table when seeking answers to my questions.

    • As a follow-up to #5:

      The IEP process in general seems to be a very qualitative process. While certain ways to qualify are supposed to be concrete; I know that in practice they are not. Do you think that being an integral part of that process prepares you well for qualitative research?

      • I think that I understand that a person’s perception is always part of the equation and that concrete decisions are often made by a combination of both quantitative and qualitative information. Does this make me better prepared? Maybe.

  4. A highly trained qualitative researcher who set parameters for collecting data in a novel environment can collect empirical data that informs about that setting, particularly data based on the number of times something occurs. That type of data can be empirical and not trigger an emotion based response from a researcher. Collecting data about human responses to a particular stimuli that evokes a strong emotional response would require more indepth analysis which in turn would likely cause the researcher to draw more from their emotional experiences.If there is one single reality, we do not currently have evidence of that phenomenon.
    Research of school culture indicates context in educational settings is very important. However, it is difficult for me to discern the distinction between compliance to a norm of what school culture is and an actual measurement of learning environment for each individual student.
    A measurement of a difference in a thought pattern on a 10 point scale is quantitative when there is sufficient data to fit the setting. The numeric value of the response is not the most significant part of the measure. The measurement of change in a significant number of responses is what gives that measurement value and is quantitative in nature.
    From the readings in our other two classes this semester, funding is awarded when there is a clear indication that the results can be measured. Experimental designs that appear to have an acceptable metric are given priority.
    Our student outcomes assessment exit interviews use very open ended questions that prompt responses that reflect student perceptions. Looking over a pattern of student perceptions can guide our thinking in making programmatic decisions. In some cases student perception is more viable data than numeric data.

    • Nan,

      Do you think looking at patterns of student perceptions via interviews is more useful than a survey? If you wanted to use quantitative data, do you think you could effectively convert interview responses into quantitative form to perform the analysis you wanted?

  5. 1.There is not one true reality as your experiences and biases continue to change as life experiences are gained
    2.Context in educational settings are like code switching and langauge. Such as proper standardized speech and inflections may not get you very good directions if you are lost in the deep south. With that said, you have to know your clientele and culture that you are dealing with.

    3.No- because of bias and beliefs. Your opinion and criteria for a 7, 8, 9 may be different from someone else’s.
    4.Empirical reseach that is measurable may have student impact but I beleive that it also has political and monetary influences. Instead of funding for new research, how about focusing on best practices and making them better.
    5.I have conducted interviews in an unstructured setting, I found it to be more comfortable but the the length of time goes past a certain point and the number of participants grows, I tend to lose focus of what I am looking for.

      • It is possible to develop programs that are beneficial for differentiated school settings. Calibration starting at the top is crucial. Also matching your resources to the data you are attempting to gather is one method to drive the program in the right direction

      • I really think we need to stop looking for that “magic bullet” program and start analyzing our instructional practices within each classroom. Instruction and intervention must be tailored to meet individual needs. A program may be great for some of our students but we will never reach all our students.

  6. 1.Do you think that each person’s perceptions define their own reality or context? Is there one true reality?
    Of course each person’s perceptions define their own reality or context as well as how the respond and react in every situation every day. This is simply how the brain works and is a survival mechanism. Our brain will automatically interpret situations based on memories that have been stored. For example one who has been mugged in a dark parking lot is likely to percieve dark parking lots as more dangerous then one who has never been a victim of a mugging or has never heard of someone being mugged.
    2.How important is context in educational settings? A. K-12: Assume well-planned intervention program for students at risk for the CAHSEE was implemented with pure fidelity at two schools, would it have similar results at Clovis West High School and McLane High School?

    I think the key word is similar. I believe the intervention is likely to have similar results but one can never expect identical results in differing context just as one has to examine the generalizability of any study. This is true for quatitative research in that we know that no sample perfectly mirrors the population.

    3.What is the difference in my thought pattern if I choose an 8 or a 9 on a 10 point scale? Is the difference truly quantitative?
    I think there is truely a difference in the thought pattern between choosing an 8 or a 9 but it is not always a valid measure of what one is intending to measure. The thought process may infact be a function of what the respondent believes the perception of someone who marks an 8 is versus someone who marks a 9.

    4.Why do you think most educational research funds go to experimental research?
    Most educational research funds likely go to quantitative experimental research because we are a society built on the quick fix and are far more comfortable with the black and white than shades of grade particularly when public funds are involved. So politicians would rather invest taxpayer money in something they can say is supported by research so they are able to blame later failure on the implementation of whatever was funded.

    5.Have you conducted interviews that were unstructured in any setting? If so, was it more comfortable or less so?
    Yes but I have also guided Master’s level students through the process and they found it very unsettling. I think you must be comfortable and at ease with the process and aware that the process is the goal rather than anxious to find “the answer.”

  7. My background in counseling has allowed me to explore various perceptions and how people define their own reality. During client sessions, I was able to serve more of a “mirror” role with clients and was able to dismiss my biases because of the training I had completed and practiced regularly. Working as an administrator has taken me away from this way of working with students. I serve more of a disciplinarian role and use my own sense of reality daily. I think people are able to serve as “mirrors” after acquiring the right tools, such as in qualitative research.

    It’s easier to generalize and try to place things together, but we know that is not always the case. People’s Mental Models to the Clovis and McLane question might tell them that one school will do better than the other, regardless of the program being implemented with pure fidelity at the two schools. Context is important, but other components should accompany the context. We can assume a lot in this situation. The results may have nothing to do with how the program is implemented.

    Most of the funded programs for research purposes that I have worked with have had a mandatory data collection component. Funding has been provided based on the trends of improvement that are highly valued by many people/researchers. Qualitative research may not always produce a visible trend. Structured formats become favorable, because people are able to see the trends. Working as a counselor demonstrated to me that structured was more favorable. Personal counseling sessions were not seen as important because I didn’t have a way of showing that they were effective. Personal counseling sessions were also confidential most of the time, so personal student improvements were not something I proved to my boss. Academic counseling was more favorable at times, because I had data that my boss could review. The personal well-being of students was essential and time consuming, but others felt it was not the best use of my time.

  8. I agree with your statement regarding counseling training. I often try to exercise the skills learned as a psychologist so they remain sharp and in my toolbox. A certain level of comfort comes from not always having to have the answer (as many administrators often feel they must) but allowing others to arrive at their own answer.

  9. 1. Do you think that each person’s perceptions define their own reality or context? Is there one true reality? — Yes, I happen to think that the limitations of human perception and the variations between individual perceptual faculties make reality appear very different to different people/groups. I also think that there exists a strong argument to be made for a stable, objective reality outside of human perception, and that even though cultural analysis yields tremendous variation in apprehending it, this is the only outcome when humans create arbitrary signifiers, network them, and superimpose them on the world.

    2 How important is context in educational settings? (13+: Assume a new research-based basic skills program was mandated for all community colleges. If implemented with fidelity, would it have similar results at COS and Reedley College?) — Context is perhaps the most important element to consider with respect to providing the most pertinent services to specific populations. In order for this basic skills program, implemented with fidelity, to find even basic threshold levels of success would need to first precisely identify the student population that are identified as needing basic skills intervention.

    3 What is the difference in my thought pattern if I choose an 8 or a 9 on a 10 point scale? Is the difference truly quantitative? — This is a topic of actual ongoing debate among quantitative methodologists. Theoretically, a factor analysis or cronbach’s alpha (or any other test of internal validity/reliability) would achieve a greater degree of accuracy when providing the subject with a greater range of response. I wouldn’t say the difference matters on the part of the respondent, but is more an issue of instrument validity/reliability.

    4 Why do you think most educational research funds go to experimental research?
    Have you conducted interviews that were unstructured in any setting? If so, was it more comfortable or less so? — Most educational research funds are directed into experimental research because it produces new insight or knowledge that has been empirically validated. This extends beyond a specific program or certain implementation of practices, but rather extends to all instances of things like it, by challenging, reevaluating, or outright dismissing certain assumptions. I have never conducted an unstructured interview – nor would I.

  10. Is there one true reality? If I say that there is not, then is that a true reality? If we say that the only true reality is that which we believe to be true, then there is no truth aside of our own understanding. If that is the case, then how could any person make a claim that what they believe to be true actually is?

    This question compelled me to reflect on conversations I participated in as a student of philosophy. The Socratic conversations about absolute truth v. relativism, which we beat to a pulp, revealed to me that every person at some point in their lives will have to face this question. I also thought about philosophers René Descartes and Friedrich Nietzsche, who articulated their thoughts about realities and truths. I will say that before my studies in those courses, everything that I firmly believed to be unequivocally true became questionable truths.

    How important is context in educational settings?
    K-12: Assume well-planned intervention program for students at risk for the CAHSEE was implemented with pure fidelity at two schools, would it have similar results at Clovis West High School and McLane High School?

    Context is very important, but equally important are the experiences (culturally, educationally, emotionally, etc.) that students and staff bring with them. Not to mention, the culture of the organization, which may or may not be part of the formal context?

    That is not to say that effectively implemented intervention programs will not have a positive impact at either school. I believe that the impact that it will have will vary, not so much by school, but by the individual teacher(s) and cohort(s) of students.

  11. 1. Do you think that each person’s perceptions define their own reality or context? Is there one true reality? I do believe that each person’s perceptions define their own reality. Take two siblings who grew up in the same house. Each will have different memories of shared family events, different experiences and perceptions of their parents, and different feelings about their family in general. Who’s reality is the true reality?

    2. How important is context in educational settings? 13+: Assume a new research-based basic skills program was mandated for all community colleges. If implemented with fidelity, would it have similar results at COS and Reedley College? It would have similar results IF the program was implemented in a similar manner – with a similar infrastructure, similar funding levels, and similar supportive resources. However, because community colleges are run with different organizational structures and priorities, the outcomes for the basic skills program would be different.

    3. What is the difference in my thought pattern if I choose an 8 or a 9 on a 10 point scale? Is the difference truly quantitative? It is different, because the an 8 point scale puts the extreme values closer to the middle value than the 10 point scale.

  12. I have conducted semi-structured interviews and have found them to be very useful. You can gain much insight by branching your questions off of answers that are given. This allows you to delve deeper into topics. One caution is that you will not always cover all of your critical questions if you are not careful to bring the interview back to the focus. I have been side tracked by intriguing answers and then found when analyzing my responses that I forgot to ask some important questions.

    I agree with everyone else that the difference in point value on likert scales is very subjective. Even my own responses are not always consistent depending on the length of the survey.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s