9 Text Beyond Interviews
Where this chapter fits. Step 8 of the research process is to write the report. This chapter also returns to step 2, because text that already exists can sometimes answer the question.
9.1 A Thousand People Who Were Not Asked
No interviewer asked the question. A person sat down, opened a form on a government website, and wrote this.
“The said bank charged me XXXX dollars in overdraft fees – – -the account was in the positive – – [company] does not respect veterans at all – – – – – – – – —-” (Complaint 6922392)
The writer was not answering a research question. The writer wanted a bank to return some fees. The dashes and the final sentence are in the original. They are part of what was written, and a researcher should not tidy them.
The Consumer Financial Protection Bureau has received complaints from the public since 2011. When a consumer agrees, the bureau publishes the consumer’s own account after removing personal details. The public database now holds several million of these accounts.
This chapter uses 1,000 of them. They concern checking and savings accounts, credit cards, and mortgages, and they were received between May 2017 and May 2026. They are real. This book has masked most company names. Each complaint keeps its number, and a reader can look it up in the bureau’s database.
Text of this kind is everywhere. People write complaints, reviews, public comments on proposed rules, posts in online forums, and answers to the last question on a survey. Much of it is public, and nearly all of it was written without a researcher in the room. This chapter covers how to analyze such text and how to report findings so that each claim can be traced to its evidence.
9.2 Text That Already Exists
Researchers sometimes call this material found text. It differs from interview data in ways that help and in ways that hurt.
Found text has three advantages. It is plentiful, so a study can read a thousand accounts where it could afford twenty interviews. It was written without a researcher’s questions, so the interviewer’s wording could not have shaped it. It also often comes with useful facts attached, such as a date, a place, or a product.
It has three limits.
No follow-up. The researcher cannot ask the writer what a phrase meant. A complaint that says the bank “refuses to return the check” cannot be probed.
A purpose of its own. Every document was written to do something. A complaint is written to obtain a remedy. The writer has a reason to stress some facts and omit others. A researcher should ask who wrote the text, for whom, and to what end.
Self-selection. The people who write are not a sample of everyone affected. People who complain to a federal agency knew that the agency existed, had access to the internet, and believed that writing could help. Many people with the same problem never write.
The last limit matters most. The complaints show what kinds of problems consumers describe and how they describe them. They cannot show how common each problem is among all bank customers.
9.3 Know Where the Text Came From
Before reading a word, a researcher should establish how the documents were selected. For this chapter’s file, the answer has four parts.
- The bureau publishes a complaint’s text only if the consumer agrees. Complaints without that agreement are not in the public text.
- The bureau removes personal details and replaces them with the letters “XXXX.”
- A university research project drew these 1,000 complaints from the public database for a study of three products. The project kept only complaints that a company had closed with an explanation or with some form of relief.
- The file is therefore not a random sample of all complaints. It contains more recent complaints than older ones, because complaint volume has grown.
A report has to state these facts. A reader who knows them can judge what the file can support. Figure 9.1 shows what it contains.
9.4 Counting What Is There
Content analysis is a method that sorts text into categories and counts them. It treats text as data to be measured. Krippendorff (2018) gives the standard account.
Some counts need no reading at all. The complaints range from 5 words to 3,943 words. The median is 176 words. Companies closed 771 of the 1,000 with an explanation only, 149 with monetary relief, and 80 with relief of another kind.
Other counts come from searching the text for words. Figure 9.2 shows how many of the 1,000 complaints contain each of ten terms.
The most common term in the figure concerns contact with the company and has no direct link to money. In 384 of the 1,000 complaints, the writer mentions calling or speaking with the company. That is more than the number that mention fraud (243) or a dispute (209). The count suggests something to look for in a closer reading. Many of these accounts may be about a failed attempt to get an answer as much as about the original problem.
A word count is a lead and not yet a finding. It has three weaknesses.
A word is not a meaning. “Interest” can be a charge on a credit card or a consumer’s lack of interest in an offer. A search finds both.
Writers use different words for one thing. One consumer writes “unauthorized.” Another writes “I never opened this account.” A search for the first word misses the second complaint.
A word can be denied. “This was not fraud” contains the word “fraud.”
For these reasons, a keyword count should be checked by reading. Draw a random sample of the complaints that contain the term and a sample of those that do not. Read both, and estimate how often the search was right.
9.5 Counting and Reading Together
The stronger design combines the two methods.
- Read a sample closely. Draw a random sample of 40 to 100 documents. Code them as Chapter 6 describes, and build a codebook.
- Check the codebook. Have a second coder code part of the sample, as Chapter 8 describes.
- Apply the codebook to the rest. Human coders can do this for a few thousand documents. Beyond that, researchers use keyword rules or a language model, and they check a random sample of the results by hand.
- Count the codes. Report how many documents received each code.
- Return to the text. For each code, read a sample of the documents and choose quotations that show what the code looks like in the writers’ own words.
Chapter 8 carried out the first step for 40 of these complaints with a six-code codebook. The codebook would need more work before it could be applied to all 1,000. In particular, it has no rule for a complaint with two topics.
The counts from step 4 describe the documents in hand. With these complaints, a sentence such as “a quarter of the complaints in the file mention fraud” is accurate. “A quarter of bank customers experience fraud” is a different claim, and the file cannot support it.
9.6 Work With the Data: Text Explorer
The five steps above combine counting with reading. The tool supports both for the 1,000 real complaints. Part 1 searches and counts. Part 2 ties each claim to the complaints behind it.
Open Text Explorer in its own window
- In Part 1, search for “fraud.” The count matches Figure 9.2.
- Read the five complaints shown. Press “Show five other matches” and read those. Look for a complaint in which the word is denied or means something else.
- Search for a second word for the same idea, such as “scam” or “unauthorized.” Compare the counts.
- In Part 2, write one claim that your reading supports. Press “Attach” on each complaint that supports it.
- Download the table. It lists each claim with the complaint numbers behind it.
A claim with no complaint attached is not yet a finding. The table from step 5 has the same form as Table 9.1 later in this chapter. That table matches the wording of a report to its evidence.
9.7 Other Kinds of Found Text
Each kind of text has its own hazards.
Open-ended survey answers. Most surveys end with a box for comments. Many respondents leave the box empty, and those who write tend to hold stronger views than those who do not. The answers are short. Code them with a small codebook and report how many respondents wrote something.
Public comments on proposed rules. Federal agencies publish the comments they receive. Many are copies of a form letter that an advocacy group wrote and its members submitted. A count of comments then measures how well a group can organize. Identify the duplicates first and report them as a separate category.
Reviews and posts online. These are plentiful and easy to collect. Some are written by people who were paid, and some by programs. A site’s terms of use may also restrict collection. Check both before starting.
Organizational documents. Minutes, case notes, and reports record what an organization chose to write down. What they omit can be as informative as what they include.
9.8 Writing the Report
A qualitative report has to do two things at once. It tells the reader what was found, and it shows enough of the evidence for the reader to judge the finding. Most reports use the following order.
- The question and why it matters.
- Methods. Who took part or which documents were used, how they were chosen, how the data were collected and analyzed, and which checks were applied.
- Findings. One section for each theme. Each section states the claim, explains it, and supports it with quotations.
- Discussion. How the themes connect, what they mean for the decision at hand, and what the study cannot show.
Two published checklists list what a report should contain. The first is the Consolidated Criteria for Reporting Qualitative Research, known as COREQ. It has 32 items for studies based on interviews and focus groups (Tong, Sainsbury, and Craig, 2007). The second is the Standards for Reporting Qualitative Research, known as SRQR. It has 21 items for qualitative studies in general (O’Brien and colleagues, 2014). Many journals require one of the two.
9.9 Quotations as Evidence
A quotation in a report is evidence for a claim. Four rules govern its use.
Choose the typical quotation over the most dramatic one. The most striking quotation in a dataset is often unusual. Use a quotation that represents the segments under the theme. If you use an extreme one, say that it is extreme.
Quote exactly. Do not correct grammar or spelling. Mark any cut with an ellipsis in brackets. Do not join two passages into one quotation.
Identify the source. Give a pseudonym or a document number with each quotation, so that a reader can see that the evidence comes from more than one person. In this chapter, each complaint carries its number.
Explain the quotation. A quotation does not interpret itself. Say what it shows and how it supports the claim.
A reader should be able to trace every claim to its evidence. A simple table makes the link visible. Table 9.1 shows one for the housing study of Chapters 6 and 7.
| Claim | How far it extends | Evidence |
|---|---|---|
| Help reaches people only after a crisis. | 7 of 12 interviews, including tenants and a caseworker | “You have to be all the way at the bottom before they can help you.” (Avery, H01). “I’m telling them to call back when it’s worse.” (Akira, H08) |
| One informed person often decides the outcome. | 8 of 12 interviews | “Once I had a person next to me, the company had a person too.” (Jordan, H04). “one person who checked a list for me” (Rene, H12) |
| A move by choice was described as a gain. | 1 of 12 interviews, a negative case | “Something we did. That’s the difference, right?” (Quinn, H06) |
A researcher who cannot fill in the third column for a claim does not have a finding. Build the table before writing the findings section and use it as the outline.
9.10 Matching Words to Evidence
The wording of a claim should match the strength of the evidence behind it. Table 9.2 gives a guide.
| The evidence | Wording that fits | Wording to avoid |
|---|---|---|
| Present in nearly all cases, across different kinds of participants | “Participants consistently described” | “Everyone feels” |
| Present in more than half | “Most participants described” | “The majority of renters” |
| Present in several | “Several participants described” | “Many people” |
| Present in one or two | “One participant described” | “Some participants,” when it was one |
| A relationship the researcher inferred | “These accounts suggest” | “This shows that” or “This proves” |
The right-hand column has one fault in common. Each phrase speaks about a population or claims a certainty that a small, purposive sample cannot support.
State who was not included, what the documents could not show, and which checks you did not carry out. For the complaint file, the limits are plain. The writers chose to complain. The file holds three products. The bureau does not verify what consumers report. A reader who knows the limits can use the findings with more confidence.
The complaints in this chapter are public, and the writers agreed to their publication. That settles the legal question. It does not settle every ethical one. A person who posts in an online forum about a health problem has made the post public. That person did not agree to be quoted in a research report. A quotation can be traced back to the person with one search. For public text that was not written for publication, paraphrase where you can and remove identifying details. Ask whether the writer would be surprised to find the words in your report. For the complaints, this book masks most company names, because a complaint is one side’s account and has not been verified.
A language model can read 1,000 complaints and return a one-page summary. The summary is useful as a first look. It has four known weaknesses. A model gives most of its attention to common content and tends to leave out the unusual complaint, which may be the one that matters. It may state how many complaints mention a topic, and that number is often wrong. Count with a search or a program. It may also present a “quotation” that no one wrote. Search the file for every quotation before you use it. Finally, a summary carries no record of which documents support which sentence. Ask the model to list the complaint numbers behind each point, and then read those complaints.
9.11 Putting It All Together
Found text is plentiful and was written without a researcher’s influence. It also allows no follow-up, was written for a purpose of its own, and comes from people who chose to write. A researcher first establishes how the documents were selected and reports it.
Content analysis sorts text into categories and counts them. A keyword count is a lead that has to be checked by reading, because a word is not a meaning. The stronger design codes a sample closely, checks the codebook, applies it to the rest, and returns to the text for quotations.
A report states each theme as a claim and supports it with quotations that are typical, exact, identified, and explained. A table that traces each claim to its evidence keeps the report honest. The wording of each claim matches the strength of the evidence.
9.12 Looking Back
This book has followed one path. A question that numbers could not answer led to interviews. The interviews were planned, conducted with consent, and transcribed. The transcripts were coded, the codes were sorted into themes, and the themes were tested against the cases that did not fit. A second coder and the participants checked the analysis. The findings were written so that each claim could be traced to what someone said.
Before the Numbers, the companion to this book, follows the parallel path for research that produces numbers. Many questions need both. A count shows how large a problem is, and people’s own accounts show what the problem is like and why it happens.
9.13 Further Reading and References
- Krippendorff, K. (2018). Content Analysis: An Introduction to Its Methodology (4th ed.). Sage.
- Tong, A., Sainsbury, P., and Craig, J. (2007). Consolidated criteria for reporting qualitative research (COREQ): A 32-item checklist for interviews and focus groups. International Journal for Quality in Health Care, 19(6), 349 to 357.
- O’Brien, B. C., Harris, I. B., Beckman, T. J., Reed, D. A., and Cook, D. A. (2014). Standards for reporting qualitative research: A synthesis of recommendations. Academic Medicine, 89(9), 1245 to 1251.
- Hsieh, H.-F., and Shannon, S. E. (2005). Three approaches to qualitative content analysis. Qualitative Health Research, 15(9), 1277 to 1288.
- Consumer Financial Protection Bureau. Consumer Complaint Database. consumerfinance.gov/data-research/consumer-complaints. The source of the 1,000 complaints.
- Franzke, A. S., Bechmann, A., Zimmer, M., Ess, C., and the Association of Internet Researchers (2020). Internet Research: Ethical Guidelines 3.0. Guidance on the use of public online text.
9.14 Key Terms
- Content analysis. A method that sorts text into categories and counts them.
- COREQ. A 32-item checklist for reporting studies based on interviews and focus groups.
- Found text. Text that was written without a researcher’s involvement.
- Self-selection. The process by which people choose for themselves whether to write or take part.
- SRQR. A 21-item standard for reporting qualitative research.
9.15 Exercises
Check Your Understanding
- Give two advantages and two limits of found text compared with interviews.
- Why can the complaint file not show how common a problem is among all bank customers?
- Name three reasons a keyword count can mislead.
- State the four rules for using a quotation as evidence.
Apply It
- Use the file
cfpb-complaints-1000.csv. Count the complaints that contain the word “refund.” Read ten of them. In how many is the consumer asking for a refund that was not given? - Rewrite each sentence so that its wording matches the evidence. (a) “Renters feel ignored by landlords,” from 3 of 12 interviews. (b) “This proves that early help prevents eviction,” from one caseworker’s account.
- Build a table like Table 9.1 for the theme Instability that is not counted. Use the file
housing-coded-segments.csvfor the evidence.
Think Deeper
- An agency receives 20,000 public comments on a proposed rule. Eighteen thousand are copies of one form letter. How should a report describe public opinion on the rule?
- You find a public forum where people describe being evicted. The posts would make strong quotations. List the ethical questions you would settle before quoting any of them.
- Ask a language model to summarize 50 complaints from the file and to give three quotations. Search the file for each quotation. Report how many you found word for word.