8 Checking the Analysis
Where this chapter fits. Step 7 of the research process ends with a check on the analysis. This chapter covers how to test whether the findings can be trusted.
8.1 A Second Coder That Works in Seconds
A researcher has 40 short complaints that consumers sent to a federal agency. She has a codebook with six codes. She gives the complaints and the codebook to a language model and asks it to assign one code to each complaint. The model returns 40 codes in a few seconds.
The complaints are real. Consumers sent them to the Consumer Financial Protection Bureau and agreed to have them published. The codes are also real. A language model assigned them for this book in November 2026, in one pass, and no one corrected them.
Here is one of the 40. The model assigned it the code Money withheld or not returned.
“My IRS tax return was sent to a closed account and the bank refuses to return the check back to the IRS.” (Complaint 10916232)
Most readers would agree with that code. Here is another. The model assigned it Dispute or claim denied or stalled.
“I was scammed on the phone into buying $1400.00 in XXXX XXXX XXXX cards using my [company] debit card in exchange for a $5000.00 loan, which the scammers agreed to provide but did not. When reported to the [company], [company] posted a $1400.00 provisional credit and thirty days later removed the credit without completing a full investigation.” (Complaint 15766556)
This one is less clear. The complaint begins with a scam, which fits the code Unauthorized activity or fraud. It ends with a bank that withdrew a credit without finishing its investigation, which fits the code the model chose. A careful human coder could choose either. The codebook needs a rule for a complaint that contains both.
The model’s 40 codes look like a finished piece of work. Are they right? The question has no direct answer, because no answer key exists for what a passage means. The useful questions are narrower. Does a second coder, working alone, assign the same codes? Where the two differ, what does the difference show about the codebook? This chapter covers those questions and the other checks that make a qualitative analysis worth trusting.
8.2 Trustworthiness
Quantitative research judges a study by its validity and reliability. Lincoln and Guba (1985) proposed a parallel set of standards for qualitative research. They used trustworthiness as the general term and divided it into four parts. Table 8.1 lists them.
| Standard | The question it asks | Nearest term in quantitative research | Main checks |
|---|---|---|---|
| Credibility | Do the findings give an accurate account of what participants said and meant? | Internal validity | Triangulation, member checking, negative case analysis |
| Transferability | Can a reader judge whether the findings apply in another setting? | External validity | Detailed description of the setting and participants |
| Dependability | Would the analysis hold up if someone examined how it was done? | Reliability | An audit trail, a second coder |
| Confirmability | Do the findings come from the data and not from the researcher’s wishes? | Objectivity | Reflexive memos, an audit trail, quotations as evidence |
No single check establishes any of the four. A researcher chooses several checks that suit the study and reports which ones she used.
8.3 Six Checks
Triangulation. The researcher compares findings from more than one source, method, or analyst. In the housing study of Chapters 6 and 7, tenants, a landlord, and a caseworker all described help that came late. Agreement across three kinds of participants is stronger evidence than agreement among tenants alone.
Member checking. The researcher takes the findings back to participants and asks whether the account fits their experience. She can send a one-page summary or meet with a few of them. Participants may correct a detail, object to a label, or add what was missed. They do not have a veto. A participant may dislike a finding that is accurate. The researcher records each response and what she did with it.
Negative case analysis. The researcher searches for cases that contradict her themes and revises the themes to account for them. Chapter 7 showed this with one interview.
Thick description. The researcher describes the setting, the participants, and the context in enough detail for a reader to judge whether the findings may transfer. The term comes from the anthropologist Clifford Geertz (1973).
Peer debriefing. The researcher presents the analysis to a colleague who was not part of the study. The colleague questions the themes and asks for the evidence behind each one.
Audit trail. The researcher keeps a record of every step. The record includes the interview guide and its revisions, the codebook and each change to it, the memos, and the reasons for each decision. An audit trail allows another person to follow the path from the transcripts to the findings.
8.4 A Second Coder
A code is a judgment. One way to test a judgment is to ask a second person to make it independently. Intercoder agreement is the extent to which two coders assign the same codes to the same passages.
The procedure has five steps.
- The first coder writes the codebook and trains the second coder on it.
- Both coders code the same sample of the data, working separately. A sample of 10 to 25 percent of the material is common.
- The two sets of codes are compared, and agreement is calculated.
- The coders meet and discuss each disagreement. Most disagreements point to a definition that is unclear.
- The codebook is revised, and the process is repeated until agreement is acceptable. The remaining data are then coded.
The fourth step is the most valuable. A disagreement between two careful coders means that the codebook allows two readings. The scam complaint at the start of this chapter is an example. The remedy is a new rule in the codebook, such as “When a complaint describes a fraud and a failed dispute, code the failed dispute.”
8.5 Measuring Agreement
The simplest measure is percent agreement. It is the share of passages to which both coders gave the same code.
Percent agreement has a weakness. Two coders will agree some of the time by chance, even if they assign codes at random. With only two codes, random coding produces agreement about half the time.
Cohen’s kappa corrects for this (Cohen, 1960). It compares the agreement that was observed with the agreement that chance would produce.
\[\kappa = \frac{p_o - p_e}{1 - p_e}\]
In the formula, \(p_o\) is the observed agreement and \(p_e\) is the agreement expected by chance. A kappa of 1 means perfect agreement. A kappa of 0 means agreement no better than chance.
Here is a worked example with invented numbers. Two coders read 40 passages. For each one, they decide whether the code Fraud applies. Table 8.2 shows their decisions.
| Coder B: yes | Coder B: no | Total | |
|---|---|---|---|
| Coder A: yes | 14 | 4 | 18 |
| Coder A: no | 2 | 20 | 22 |
| Total | 16 | 24 | 40 |
- Calculate observed agreement. The coders agreed on \(14 + 20 = 34\) passages. So \(p_o = 34 / 40 = 0.85\).
- Calculate each coder’s rate of saying yes. Coder A said yes to 18 of 40, or 0.45. Coder B said yes to 16 of 40, or 0.40.
- Calculate the chance that both say yes. Multiply the two rates. The result is \(0.45 \times 0.40 = 0.18\).
- Calculate the chance that both say no. The rates of saying no are 0.55 and 0.60. Their product is 0.33.
- Add the two chances. So \(p_e = 0.51\).
- Apply the formula. Kappa is \((0.85 - 0.51) / (1 - 0.51) = 0.69\).
The coders agreed on 85 percent of the passages. After the correction for chance, kappa is 0.69.
Landis and Koch (1977) proposed labels for ranges of kappa that are still widely used. Values from 0.41 to 0.60 are “moderate,” values from 0.61 to 0.80 are “substantial,” and values above 0.80 are “almost perfect.” These labels are conventions. Many researchers treat 0.70 or higher as acceptable for a codebook that is ready to use.
Two coders can agree and both be wrong. If the codebook defines a code badly, two well-trained coders will apply the bad definition in the same way. High agreement shows that the codebook can be applied consistently. It does not show that the codes capture what matters. That question belongs to the other checks, such as member checking and negative case analysis.
Some qualitative researchers do not calculate agreement at all. Braun and Clarke (2022) describe a reflexive form of thematic analysis in which coding is one researcher’s interpretation. They argue that a second coder’s agreement is the wrong test for it. They ask for careful reflexive memos in its place. A researcher should choose the checks that match her approach and say why. For applied studies with a team and a codebook, measuring agreement is standard practice.
8.6 The Model as a Second Coder
Return to the 40 complaints. The codebook that the model was given is in Table 8.3.
| Code | Use when the complaint is mainly about |
|---|---|
| A. Unauthorized activity or fraud | An account or a transaction the consumer did not authorize, a scam, or identity theft |
| B. Dispute or claim denied or stalled | A dispute or claim that the company refused, reversed, or did not finish |
| C. Money withheld or not returned | Funds, a refund, or a check that the consumer did not receive |
| D. Fees and interest | An overdraft fee, an annual fee, or interest that the consumer contests |
| E. Credit reporting and records | Late-payment marks, wrong information on a credit report, or the sharing of personal information |
| F. Service and access | Trouble reaching the company, moving money, or getting an explanation |
Figure 8.1 shows how the model distributed its 40 codes.
The chart describes what the model did. It does not show whether the codes are sound. That requires a human coder who codes the same 40 complaints without seeing the model’s codes. The tool for this chapter lets you be that coder.
Readers who try it should expect three things.
First, agreement will be high on complaints with one clear topic. A complaint about an overdraft fee is coded Fees and interest by nearly everyone.
Secondly, disagreement will be most common on complaints with two topics. The scam complaint is one. Here is another, which the model coded as Unauthorized activity or fraud.
“There is a transfer of $11000.00 and I have no clue how it got on my account nor any clue on who did but I am aware I do not have access to the funds” (Complaint 3943369)
The consumer reports a transfer that they do not recognize. The consumer also cannot reach the funds, which could fit Service and access. A disagreement here is a finding about the codebook, which gives no rule for choosing a main topic.
Thirdly, a model’s errors are not random. A human coder who is tired makes scattered mistakes. A model tends to make the same kind of mistake each time. It often follows the first topic in a passage, or the words that match a code’s name. An error of that kind shifts every count in one direction.
A sound procedure for using a model as a coder has five steps.
- Write the codebook yourself, with definitions and rules for unclear cases.
- Code a sample yourself before you see the model’s codes.
- Give the model the same sample and the codebook. Calculate agreement.
- Read every disagreement. Revise the codebook, and repeat until agreement is acceptable.
- Let the model code the remaining material. Check a random sample of its work and report the agreement you found.
This is the same procedure as for a human second coder. The model is faster and never tires. It also has no knowledge of the setting and no stake in the result. Its account of its own reasoning cannot be relied on. The human coder remains responsible for every code in the final dataset.
8.7 Work With the Data: Coder Agreement
The section above compared a human coder with a language model. The tool puts you in the place of the human coder. It shows 40 real consumer complaints and the six codes used in this chapter.
Open Coder Agreement in its own window
- Read each complaint and choose the one code that fits best. The model’s codes stay hidden while you work.
- When you finish, read your percent agreement with the model and Cohen’s kappa.
- Compare kappa with the agreement expected by chance, which the tool also reports.
- Read every complaint on which you and the model differ. For each one, decide whether the codebook needs a new rule.
Expect kappa to be lower than percent agreement. Kappa removes the agreement that two coders would reach by chance. Some disagreements may come from complaints with two topics, for which this codebook has no rule. Agreement shows that two coders applied the codebook in the same way. It does not show that either coder is right.
8.8 Reporting the Checks
A methods section should state which checks were used and what they showed. Here is an example for the housing study.
One researcher coded all 12 transcripts. A second coder coded three transcripts independently with the same codebook. Agreement on those three was 82 percent, with a kappa of 0.74. The coders discussed each disagreement and revised two code definitions. The first researcher then recoded all transcripts. A summary of the themes was sent to the 12 participants, and four replied. One reply led to a change in the name of a theme. The codebook, its revisions, and the memos are available on request.
The numbers in this example are invented for illustration. The form is the point. A reader learns who coded what, how agreement was measured, what changed as a result, and where the records are.
A researcher sends her themes to participants. Three of them say that one theme misreads their experience. She is confident in the theme and has a deadline. She could report that “member checking was conducted” and say nothing more. That sentence would be true and would mislead. A check has value only if its result is reported. State what participants said and how you responded, including the points on which you kept your own reading and why.
It is tempting to ask a model whether its own codes are correct. The model will answer, and the answer will usually be yes. A second run of the same model is not an independent coder either. The two runs share the same training and the same blind spots, and their agreement with each other will overstate how reliable the coding is. An independent check needs a different kind of reader. The strongest one is a person who knows the setting. Report agreement between the model and a human coder, and say how many passages the human coded.
8.9 Putting It All Together
Trustworthiness has four parts. They are credibility, transferability, dependability, and confirmability. Researchers support them with triangulation, member checking, negative case analysis, thick description, peer debriefing, and an audit trail.
A second coder tests whether a codebook can be applied consistently. Percent agreement is the share of passages coded the same way, and Cohen’s kappa corrects that share for chance. Disagreements show where the codebook is unclear. Agreement shows consistency and does not show that the codes capture what matters.
A language model can serve as a second coder under the same procedure as a person. The researcher writes the codebook, codes a sample first, measures agreement, reads every disagreement, and checks a sample of the model’s later work. A model’s agreement with itself is not an independent check.
8.10 Looking Ahead
So far, the text in this book has come from interviews. A great deal of useful text was never collected by a researcher at all. People write complaints, public comments, reviews, and answers to open-ended survey questions. Chapter 9 covers how to analyze such text, using 1,000 real consumer complaints. It also covers how to write a report in which each claim can be traced to its evidence.
8.11 Further Reading and References
- Lincoln, Y. S., and Guba, E. G. (1985). Naturalistic Inquiry. Sage. The source of the four standards of trustworthiness.
- Cohen, J. (1960). A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20(1), 37 to 46.
- Landis, J. R., and Koch, G. G. (1977). The measurement of observer agreement for categorical data. Biometrics, 33(1), 159 to 174.
- O’Connor, C., and Joffe, H. (2020). Intercoder reliability in qualitative research: Debates and practical guidelines. International Journal of Qualitative Methods, 19, 1 to 13.
- Birt, L., Scott, S., Cavers, D., Campbell, C., and Walter, F. (2016). Member checking: A tool to enhance trustworthiness or merely a nod to validation? Qualitative Health Research, 26(13), 1802 to 1811.
- Geertz, C. (1973). The Interpretation of Cultures. Basic Books. The source of the term thick description.
- Braun, V., and Clarke, V. (2022). Thematic Analysis: A Practical Guide. Sage.
- Consumer Financial Protection Bureau. Consumer Complaint Database. The source of the 40 complaints.
8.12 Key Terms
- Audit trail. A record of every step and decision in a study.
- Cohen’s kappa. A measure of agreement between two coders that corrects for chance.
- Confirmability. The degree to which findings come from the data and not from the researcher’s wishes.
- Credibility. The degree to which findings give an accurate account of what participants said and meant.
- Dependability. The degree to which the analysis would hold up under examination of how it was done.
- Intercoder agreement. The extent to which two coders assign the same codes to the same passages.
- Member checking. Taking findings back to participants and asking whether the account fits.
- Peer debriefing. Presenting the analysis to a colleague outside the study for questioning.
- Percent agreement. The share of passages to which two coders gave the same code.
- Thick description. A detailed account of the setting, participants, and context.
- Trustworthiness. The general standard for the quality of a qualitative study.
8.13 Exercises
Check Your Understanding
- Name the four parts of trustworthiness and one check for each.
- Why is percent agreement alone a weak measure?
- Two coders reach a kappa of 0.90. Does this show that their codes are correct? Explain.
- Why is a second run of the same language model not an independent check?
Apply It
- Two coders judge 50 passages for one code. Both say yes on 12. Both say no on 30. Coder A alone says yes on 5, and Coder B alone says yes on 3. Calculate percent agreement and Cohen’s kappa. Show each step.
- Use the file
cfpb-coding-set-40.csv. Cover the model’s codes. Code the first 15 complaints yourself with Table 8.3. Calculate your percent agreement with the model. - Write a codebook rule that settles how to code a complaint that describes both a fraud and a failed dispute.
Think Deeper
- After member checking, three of twelve participants reject one of your themes. List the possible reasons. How would you decide what to do, and what would you report?
- A team plans to have a language model code 5,000 open-ended survey answers. A human will check 50 of them. Is that enough? What would you want to know before deciding?
- Braun and Clarke argue that a second coder is the wrong test for some kinds of thematic analysis. State their argument in your own words. For which kinds of study would you still want a second coder?