Step 7: Consider Reliability and Validity

To sharpen code definitions and their reliable and appropriate application to data, two people code the same data set, and review and compare the codes applied. Questions that guide this process are:

1. Do two coders working separately agree on how big a “codable” block of data is?
2. Do they use the same code for the same blocks of data?

Guba and Lincoln proposed four criteria for judging the soundness of qualitative research and explicitly offered these as an alternative to more traditional quantitatively-oriented criteria. They felt that their four criteria better reflected the underlying assumptions involved in qualitative research. Their proposed criteria and the “analogous” quantitative criteria are listed in the table.

Credibility

The credibility criteria involves establishing that the results of qualitative research are credible or believable from the perspective of the participant in the research. From this perspective, since the purpose of qualitative research is to describe or understand the phenomena of interest from the participant’s eyes, the participants are the only ones who can legitimately judge the credibility of the results.

Transferability

Transferability refers to the degree to which the results of qualitative research can be generalized or transferred to other contexts or settings. From a qualitative perspective, transferability is primarily the responsibility of the one doing the generalizing. The qualitative researcher can enhance transferability by doing a thorough job of describing the research context and the assumptions that were central to the research. The person who wishes to “transfer” the results to a different context is then responsible for making the judgment of how sensible the transfer is.

Dependability

The idea of dependability emphasizes the need for the researcher to account for the ever-changing context within which research occurs. The researcher is responsible for describing the changes that occur in the setting and how these changes affected the way the researcher approached the study.

The traditional quantitative view of reliability is based on the assumption of replicability or repeatability. Essentially it is concerned with whether we would obtain the same results if we could observe the same thing twice. But we can’t actually measure the same thing twice—by definition if we are measuring twice, we are measuring two different things. In order to estimate reliability, quantitative researchers construct various hypothetical notions (e.g., true score theory) to try to get around this fact.

Confirmability

Qualitative research tends to assume that each researcher brings a unique perspective to the study. Confirmability refers to the degree to which the results could be confirmed or corroborated by others. There are a number of strategies for enhancing confirmability. The researcher can document the procedures for checking and rechecking the data throughout the study. Another researcher can take a “devil’s advocate” role with respect to the results, and this process can be documented. The researcher can actively search for and describe negative instances that contradict prior observations. And, after the study, one can conduct a data audit that examines the data collection and analysis procedures and makes judgments about the potential for bias or distortion.