Research Methods in Mass Communication

Story

2.4 million answers, the wrong winner

October 1936. The US presidential election is weeks away, and the magazine Literary Digest has mailed about 10 million people a ballot to find out who will win. About 2.4 million came back: a number no poll today could dream of. The magazine announced that the Republican candidate, Alfred Landon, would win with 54 percent of the vote.

Franklin Roosevelt won: more than 60 percent of the popular vote and 523 electoral votes.

What had happened? The mailing list had been compiled mainly from telephone directories and automobile registrations; Gallup’s diagnosis was that this list over-represented the well-to-do. On top of that, only about 24 percent of the ballots came back. Later studies showed the other side of the story too: according to Squire, both the sample and the question of who responded were skewed, and together they sent the forecast off course. But even if everyone had answered, the magazine would at least have named the right winner.

We start the lesson with this story because the whole methods course boils down to one question: where did this number come from? Every percentage, every “effect” and every headline you read in communication research is the product of a method. Reading a result without understanding the method is like rating a dish without seeing the recipe.

Card

The calendar of methods: a handful of turning points

Methods did not fall from the sky; each is an answer to the blind spot of the one before. The skeleton of this lesson:

• 1933: The first volumes of the Payne Fund studies were published; the effects of films on children were examined with three different designs (next card).
• 1936: The Literary Digest poll became the emblem of a large but skewed sample.
• 1940: The Erie County voter panel: monthly interviews with 600 people from May to November. Directed by Paul Lazarsfeld; the book appeared in 1944.
• 1946: Robert K. Merton and Patricia Kendall’s article on the “focused interview”.
• 1952: Bernard Berelson’s book Content Analysis in Communication Research.
• 1969: George Gerbner laid out the “Cultural Indicators” programme and the analysis of message systems.
• 1976: Gene Glass introduced the term “meta-analysis”. 1979: Robert Rosenthal proposed ways of calculating how many null results it would take to overturn a finding, against the “file-drawer problem”.
• 1982: Iyengar, Peters and Kinder’s television-news experiments argued for a revival of experimentation in political communication.
• 2011–2015: Simmons and colleagues’ false-positive warning (2011) and the Open Science Collaboration’s repetition of 100 studies (2015).

Instead of memorising dates, remember the order: first “whom did you pick?” (sampling), then “what did you compare it with?” (design), and last “can we trust it?” (replication, meta-analysis).

Story

1933: Three designs, one question, an honest confession

In 1928 William H. Short, executive director of the Motion Picture Research Council, gathered psychologists, sociologists and educators from several universities and put the big question of the day: what are the movies doing to children? According to the preface by the chairman, W. W. Charters, nobody in the country yet knew the answer in a general and impersonal way; everyone told their own anecdote. With the Payne Fund’s money the work ran for four years, 1929–1932, and the result was twelve studies published by Macmillan in volumes, the first of them in 1933.

Three of those twelve read like three separate methods lessons:

• Experiment (Peterson and Thurstone). Pupils get an attitude scale, then watch a film, and the next morning the scale is repeated. The question: can a single film move attitudes?
• Comparative survey (Shuttleworth and May). Children who go to the movies often are compared with those who go rarely; the groups are equated for age, grade, sex and family background. The question: which behaviours go together with the movie habit?
• Life histories (Blumer). Young people and workers write “motion-picture autobiographies”: 634 university students, 481 college students, 583 high-school students, 67 office workers and 58 factory workers. The question: how do movies enter their lives?

The honest confession is inside the texts. Shuttleworth and May write that the comparative design leaves them without precise experimental controls and that it is hard to say how much of the observed difference can be attributed to the movies. Blumer, for his part, notes that the frequencies he tabulated should not be taken as conclusive. A researcher who knows the limits of the method makes the result speak within those limits.

Card

The core claim: a finding is only as strong as its method

The claim of this lesson is: a research finding is only as strong as the method that produced it; so when you read a study, before asking “what did they find?” you have to ask “from whom, measured how, compared with what?”

Three questions form the backbone of the whole lesson:

1. From whom? Do the respondents, or the messages examined, represent the whole being talked about? (sampling; Literary Digest.)
2. Measured how? Does the measurement measure what it claims to, and does it give the same result every time? (validity, reliability.)
3. Compared with what? Does the difference really come from the cause under study, or from something else? (correlation and causation.)

In the previous lesson (What Is a Theory? Paradigms and Classifications) we talked about theories needing to be testable. The testing itself starts here: a hypothesis that is not turned into data by some method stays a wish. Most later lessons describe a study, for example The Payne Fund Studies and the Invasion from Mars, The Voting Studies and Limited Effects and Agenda-Setting. You will be able to ask these three questions of every one of them.

A warning: knowing methods means being sceptical, not cynical. No method is flawless; a good researcher knows its flaw and states the result as boldly as that flaw allows. This lesson does not say any method is “bad”; it shows what each one can see and what it cannot.

Card

The language of methods: a glossary in ten entries

Methods debates revolve around a few words. Your glossary:

• Population and sample (sampling): the population is the whole you want to talk about; the sample is the part you actually measure. Representativeness is determined not by size but by how the sample was selected.
• Variable: something that takes different values (minutes watched, type of news, a fear score).
• Operationalisation: turning an abstract concept into something measurable. “News exposure” = the number of evenings in the last seven days on which you watched the evening news, say.
• Reliability (reliability): the consistency of a measurement; Turkish sources describe it as an indicator of the stability of measured values.
• Validity (validity): the degree to which you really measure what you meant to measure. Internal validity: does the difference really come from the cause you studied? External validity: does the finding extend to other people, settings and times?
• Confounding (third) variable: a third factor that makes two things vary together (correlation and causation).
• Effect size (effect size): how large a difference or relationship is. Statistical significance only answers “could this be chance?”.
• Meta-analysis (meta-analysis): combining the results of many studies.
• Replication (replication crisis): repeating the same study to see whether the same result appears.
• Experiment, survey, content analysis, qualitative methods: the four main families of methods; you will meet them as we go (experiment (research method), survey research, content analysis, qualitative research).

Lab

Mini lab: pick the method for the question

You have four tools and four questions. Get to know the tools:

• Survey (survey research): puts the same questions to a mass of people. The question it answers: how much, how many, in which groups?
• Experiment (experiment (research method)): you change a condition, split people into groups by lottery, and watch the difference. The question it answers: why, does this cause that?
• Content analysis (content analysis): asks messages, not people. The question it answers: how often, how, involving whom?
• Qualitative methods (qualitative research): interview, observation, reading texts. The question they answer: how, in what sense?

Now the questions (all imaginary):

1. Has the word “shock” become more common in a news site’s headlines over the last five years?
2. What percentage of young people follow the news mainly on their phones?
3. Does putting a correction note under a false story reduce belief in it?
4. What does the evening news mean in the daily life of an older viewer?

The rule: each question has one “best fit” tool, but another tool may add side information. The real habit is this: when you read a study, first ask what type of question it asks, then whether the method fits it.

Myth check

Is “qualitative research” just arbitrary interpretation?

What people assume. No numbers, no science; interviews, observation and reading texts are “just interpretation”, and the researcher finds whatever they want.

What the evidence says. The qualitative tradition took this objection seriously and wrote its own criteria of trustworthiness. Yvonna Lincoln and Egon Guba proposed credibility, transferability, dependability and confirmability for qualitative (naturalistic) inquiry; these were conceived as counterparts of the conventional criteria of internal validity, external validity, reliability and objectivity. So the question is not “is there a criterion?” but “which criterion suits which question?”.

An example of a qualitative researcher writing down his own limits: in his 29 group interviews with viewers of the television programme Nationwide, David Morley chose group interviews over individual ones because much individual interview research treats people as social atoms cut off from their context. Later he also noted his own shortcoming: he had talked with the groups in shared places such as colleges, not at home, where television is normally watched.

Our interpretation: numbers carry assumptions too. “I watch two hours of television a day” is a measurement, and its accuracy has to be tested separately (see the measurement card). The qualitative–quantitative divide is not a “scientific–unscientific” divide but a divide between different questions; the “interpretive paradigm” in the What Is a Theory? Paradigms and Classifications lesson was exactly the tradition that asks these questions.

Step by step

A survey, step by step: where do you become “Literary Digest”?

1. Define the population. Whom will you talk about: “university students”, or “everyone aged 18–29”? If the population is fuzzy, the sample will be fuzzy too.
2. Find a sampling frame. You need an accessible list of the population. In 1936 this list was mainly telephone directories and automobile registrations; when the frame falls short of the population, the sample starts out skewed.
3. Select the sample. Random selection, where everyone has a calculable chance of being chosen, is not the same as asking whoever you can reach first (convenience sampling). Sheer size does not make up for how people were selected.
4. Write and try out the question. Wording moves the answer (see the deep card). Try the question on a few people beforehand.
5. Go into the field and watch the response rate. Who is not answering? The Digest’s response rate was about 24 percent.
6. Correct for nonresponse, but do not oversell it. Lohr and Brick showed that if the Digest had weighted with the information it collected on 1932 votes, it would have seen Roosevelt winning, but the bias would still have remained very large.
7. State the limits. “This result holds for this population, this period, this response rate.”

There is one more design choice. A cross-sectional study asks everyone once and gives a snapshot. A panel study returns to the same people and answers whose mind changed, and how (deep card: the Erie County panel). survey research

Story

A free newspaper arrives at the door

Autumn 2005, Virginia. Researchers picked 3,347 households in Prince William County, about 25 miles from Washington, who said they took neither the Washington Post nor the Washington Times. Then a lottery was held: some households would be sent the Post free for ten weeks, others the Times; a third group would get nothing. One paper was considered fairly liberal, the other conservative. The question: does reading a newspaper change people?

The interview and the state records after the election showed this: neither paper measurably changed political knowledge, stated opinions or turnout. But whichever paper they got, households that received a paper gave slightly more support to the Democratic candidate. The authors put this down more to encountering a newspaper at all than to the paper’s slant.

This is the logic of the experiment: the lottery (random assignment) makes the households similar to begin with; from then on only the newspaper can explain a difference. We call this assurance internal validity. It has a price: the study was run only in households without either paper, in one county, and of the 3,347 households 1,081 answered the final interview. Whether the finding extends to other people, other places (external validity) is a separate question. A field experiment (in real life, by lottery) is valuable for this reason; a laboratory experiment increases control but makes the setting artificial. experiment (research method)

Step by step

Content analysis: ask the messages, not the people

Surveys and experiments ask people; content analysis asks the message itself. Berelson’s classic definition of 1952 treats it as a technique for describing the manifest (surface) content of communication objectively, systematically and numerically. So not “what is the series about?” but “what is the share of female characters in these series, and in how many scenes does someone harm someone else?”.

1. Narrow the question. Not “is the news full of violence?” but “how has the share of crime stories in evening bulletins changed over the years?”.
2. Choose the sample. Which channel, which period? In Cultural Indicators a chosen week of broadcasts is recorded every year. message system analysis
3. Define the unit of counting. Will you count stories, paragraphs, scenes or characters?
4. Write a codebook. Each category must be so clear that two people reach the same decision.
5. Train the coders and run a pilot first.
6. Measure intercoder reliability. Two people code the same content independently and their agreement is computed. Percent agreement is easy but also counts agreement that happens by chance; Hayes and Krippendorff proposed Krippendorff’s alpha as the standard measure.
7. Count and interpret. content analysis

Lombard and colleagues examined 200 content analyses published in 1994–1998 and found that intercoder reliability was often not assessed or not reported and that percent agreement was frequently used; the authors call that measure overly liberal.

Card

Qualitative methods: four doors into the same room

The question numbers cannot answer is this: what does this mean to these people? Qualitative methods exist for that (qualitative research). Four doors:

• In-depth interview: talking at length with one person, with prepared but flexible questions. An answer nobody thought of may turn up; in return you work with few people.
• Focus group discussion: getting several people to talk together. Merton and Kendall’s 1946 article on the “focused interview” describes an interview designed to establish the reactions of people who have been exposed to a situation (for example a radio programme) that the researcher has analysed in advance. Morley’s 29 group interviews with Nationwide viewers follow a similar logic (our interpretation).
• Ethnography: placing yourself in a community’s everyday life for a long time and observing. In media studies, audience ethnography grew from the 1980s into an important branch of reception research.
• Textual and discourse analysis: reading a text (a news story, a letter, a series) closely to see how it builds meaning. Ien Ang’s Watching Dallas is an analysis made by reading forty-two viewers’ letters written in reply to her own advertisement.

What they share: small, deliberately chosen groups, open-ended data, interpretation. What sets them apart is what you examine: interviews examine what is said, ethnography what is done, textual analysis the trace left behind. The price is the same for all: you cannot answer “what percentage?”. Letter writers are self-selected people; the study shows not “how many” but “what kind of meaning” (our interpretation).

Myth check

“If you ask, they will tell you”? The two tests of a measurement

What people assume. The most accurate way to find out how much news a person watches is to ask them.

What the evidence says. Markus Prior compared news-viewing claims in surveys with Nielsen measurements; Nielsen is a measurement that does not rely on a person’s own report. Survey estimates inflated viewing of network news by a factor of three on average and by as much as eightfold for some groups. Prior’s conclusion is unsettling: apparent “media effects” may arise not from real differences in exposure but from unknown differences in how accurately people report their exposure.

A measurement has two tests:

• Reliability: if you measure again under the same conditions, does the same result come out? reliability
• Validity: is what I measure what I meant to measure? validity

The measurement ladder goes like this: an abstract concept (news exposure) → an operational definition (the number of evenings on which the evening news was watched in the past seven days) → an item (“how many evenings last week did you watch the news?”) → a score. Something can get lost at every rung. Reliability does not guarantee validity: a survey that gives the same inflated number every time is very reliable and still wrong.

None of this means self-report is useless; it means you must check, for each question, which measurement is sufficient.

Lab

Mini lab: design a one-page study

Imagine a university’s student newspaper wonders: do short video summaries leave more information than a long article? Draw five boxes on paper or in your head:

1. Population and sample. Whom are you curious about: all students, or only the paper’s readers? Whom can you invite?
2. Method. Survey or experiment? Which one if you are asking about causation?
3. Measurement. What does “leaving information” mean? A five-question test? How will you check the test’s reliability and validity?
4. Comparison. Who will see the video and who the text, and how? Who will be assigned to groups, and how?
5. Threats. Write down three things that could spoil the result.

A hint: if you compare the format volunteers chose for themselves, the cause of the difference may not be the format but the people who chose it; assign by lottery and the problem disappears. If you make the test too easy, everyone gets full marks and no difference shows. If you take the groups from a single class, you cannot know whether the result extends to other students.

There is no answer key. A good design has two marks: the method fits the question, and its limits are written down openly.

Story

Does television scare you, or does the neighbourhood?

Imagine a finding: people who watch a lot of television find the world more dangerous. The first explanation that comes to mind is that television scares them. Two researchers, Anthony Doob and Glenn Macdonald, asked a dull but clever question: do these people really live in neighbourhoods with different crime rates?

They ran a door-to-door survey in Toronto. They chose one high-crime and one low-crime area downtown and one of each in the suburbs. Respondents reported their media use and how likely they thought it was that they themselves would become victims of violence. When the four areas were pooled, the earlier finding was replicated: people who watched more television were more afraid. But looked at within areas, the average relationship was insignificant; that is, once the neighbourhood’s real crime level was held constant, no general relationship remained between viewing and fear.

When you see a relationship there are three explanations (correlation and causation):

• X causes Y (television scares people);
• Y causes X (for example, frightened people stay at home and watch more television);
• a third thing causes both (the neighbourhood you live in).

Doob and Macdonald’s data bring out the third explanation, but this is a single study, and the debate over cultivation theory belongs to a separate lesson (Cultivation: Critiques and Later Findings). The real habit is to ask “what else could it be?” before accepting a relationship as “evidence”. The Payne Fund researchers were aware of the same problem in 1933; equating frequent movie-goers on things like family background was for this reason.

Card

“Significant” is not the same as “important”

January 2012, for one week. In an experiment run on Facebook, the news feeds of 689,003 users were altered by lottery: in one group some positive posts were left out, in another some negative ones. Then the users’ own posts were examined. Was emotion “contagious”?

Yes, statistically: when positive posts were reduced, the share of positive words in people’s own posts fell a little, and some comparisons gave p < 0.001. But the effect size was tiny: Cohen’s d lay between 0.001 and 0.02 across four comparisons. The authors themselves argued that the effects were small but could matter at this scale.

Here is the point: statistical significance answers “could this difference be chance?”, and as the sample grows even very small differences turn out “significant”. Effect size answers “how big is the difference?” (effect size). The American Statistical Association’s 2016 statement says the same: a p-value, or statistical significance, does not measure the size of an effect or the importance of a result.

So how small is “small”? For correlation-type effect sizes (r) Funder and Ozer propose this frame: .05 is very small for single events but can matter over time; .10 small; .20 medium; .30 large; .40 and above is often a gross overestimate in psychological research.

When you see “a significant effect was found” in a news story, let your first question be “how big?”.

Step by step

Meta-analysis: the study of studies

A single study is a witness statement; meta-analysis tries to listen to all the witnesses and extract a common voice. Gene Glass coined the term in 1976 for the statistical synthesis of the results of similar studies. The steps (meta-analysis):

1. Write the question and the search rules. Which studies are included, which dates and databases are searched?
2. Collect. Look for unpublished ones too.
3. Extract an effect size from each study. That makes different scales comparable.
4. Combine. Compute the average effect size.
5. Explain the differences. Under which conditions is the effect large, under which small?
6. Test for bias. Are there results missing?

An example: a 2023 meta-analysis combining cultivation studies that link social media use to beliefs collected 460 effect sizes from 66 independent samples and found an overall effect of .123; the effect was higher in studies looking at exposure to specific content, in younger samples and in Asian samples, and lower when active use was measured. A 1997 meta-analysis of cultivation research had found an average effect of .09.

There are two traps. The file drawer: Rosenthal argued that journals may be filled only with studies that turned out significant, while those that did not stay in drawers, and proposed methods for estimating how much damage that does. Bad measurement: according to Scharkow and Bachl, measurement error in content analysis and in self-report pulls effect estimates downward; combining flawed studies combines the flaw too (this last sentence is our interpretation).

Story

Did a song make you younger?

In one study, 20 University of Pennsylvania students listened either to the Beatles’ “When I’m Sixty-Four” or to an instrumental called “Kalimba”. Afterwards they wrote down their birth dates. The result: those who listened to the first song were statistically about a year and a half younger (adjusted means 20.1 and 21.5 years; p = 0.040).

This is not a joke, though the point behind the joke is serious. Joseph Simmons and colleagues deliberately “proved” an impossible result; the article says everything reported really happened, and that the aim was to show how easily flexibility in data collection and analysis produces false positives. Combining four such flexibilities can push the false-positive rate to 61 percent (deep card).

Then the problem was measured. The Open Science Collaboration repeated 100 experimental and correlational studies from three psychology journals: 97 percent of the original studies had significant results; only 36 percent of the replications did, and replication effects were about half the size of the originals. A survey of more than 2,000 psychologists by John and colleagues suggested that “questionable research practices” were surprisingly common and that some might have become the norm. Matthes and colleagues, looking at communication research, found signs in experiments in four leading journals such as small and unjustified samples, unreported effect sizes and a pile-up of p-values just below .05. (replication crisis)

A caution: failing to replicate does not mean fraud; more often flexible analysis, small samples and publication bias combine.

Card

The three special troubles of studying media effects

Problems of sampling, measurement and comparison exist in every field. In media-effects research three are especially sharp:

1. People choose for themselves what to watch. People who watch the news are different kinds of people from those who do not: will you attribute the difference to watching, or to the person who watches? The newspapers Gerber and colleagues distributed by lottery are valuable for exactly this reason: they remove the choice.

2. Measuring exposure is hard. In Prior’s study, claims were inflated threefold. Scharkow found that for internet use, self-reports were only moderately related to log data and that the errors were systematic (especially over-reporting) rather than random. Scharkow and Bachl’s simulation argues that error in measuring content and exposure pulls effect estimates downward and could thereby artificially produce some “minimal effects” findings. So in the “limited effects” debate, method may be part of the result (our inference).

3. Time order. Do people who watch a lot of violent programmes become aggressive, or do aggressive people choose them? The two are called the “effect” and the “selection” hypotheses; longitudinal studies, which follow the same people over time, exist to tell them apart (deep card).

Seeing all three together, you understand both why a media-effects study deserves caution and why it should still not be dismissed: these are recognisable problems, and often solvable ones.

Myth check

Is the experiment the “gold standard”?

What people assume. The experiment is the only way to establish causation; surveys and observations are at best preliminary work.

What the evidence says. The experiment is among the strongest tools for internal validity, but it pays for it in external validity. In 1986 David Sears wrote that since 1960 social psychology had worked mostly with college students in academic laboratories, and that this narrow database might have distorted the discipline’s portrait of human nature. In 2010 Henrich, Heine and Norenzayan argued that claims in the behavioural sciences rest mostly on samples from Western, educated, industrialised, rich, democratic (WEIRD) societies, and that these people are often outliers compared with the rest of the species. Gerber and colleagues’ field experiment was also limited to people who did not already take a newspaper.

Surveys, on the other hand, reach wide and varied groups but cannot settle causes. The story of agenda-setting research shows this well: the Chapel Hill study comparing undecided voters’ important issues with media content (Agenda-Setting) showed a relationship; Iyengar, Peters and Kinder’s experiments showed that television news affects which problems viewers take to be important.

The conclusion: there is no “best method”; there are methods that complement one another. Theories become robust not through a single study but when different methods point in the same direction.

Card

Platforms: new opportunities, old questions

Social media gave researchers two gifts: huge trace data and the ability to run lotteries inside a platform. Kramer and colleagues’ experiment is an example of the second: participants were selected at random by user ID and 689,003 people were reached. But about a month after the article appeared, PNAS’s editor published an “expression of concern”: questions had been raised about informed consent and the opportunity to opt out. The authors had written that the experiment was consistent with Facebook’s data use policy, which they took to be consent; the note said that although Facebook as a private company was under no obligation to follow the relevant rules, it was “a matter of concern” that the data collection may not have been fully consistent with the principles of consent. So scale does not remove ethics.

Trace data (logs) come with a promise to make up for the shortcomings of self-report. The meta-analysis by Parry and colleagues, based on 106 effect sizes, found the relationship between self-reported and logged digital media use to be r = .38; of the 49 comparisons of average usage, self-report was accurate in only three. But a log tells you “how much”; it does not tell you “why” or “in what sense”. And platform users are not the whole population.

The rule: a new type of data does not abolish the old three questions; it only moves “from whom, measured how, compared with what?” to a new place.

Lab

Mini lab: interrogate a headline

Imagine you see this headline on a news site: “Study: 70 percent of young people now get their news only from short videos.” (The headline and number are invented, only for practice.) Ask six questions with what you have learned in this lesson:

1. From whom? Who are “young people”? How were they selected? How many, and how many of those approached answered? (Sampling, response rate; Literary Digest.)
2. How was it asked? How was “only” defined? Wording can move the answer. (Rugg.)
3. Measured how? People’s own reports, or logs? (Prior, Parry.)
4. Compared with what? It says “now”: relative to whom, relative to when? Were the same people followed over time?
5. How big, how precise? 70 percent is an estimate; is there a margin of error?
6. What else could it be? Does the headline imply a cause (“videos killed the news”)?

To do: pick two of the six questions; for each, think about where in the study’s report the answer should be written. If the report does not say, you lack a reason to trust the result; that is not a reason to distrust it. Absence of evidence and refutation are different things.

Card

The methods vocabulary in the Turkish literature, and a classification muddle

The Turkish literature gives the same concepts more than one name; knowing this saves time when reading sources.

• Reliability and validity (güvenirlik, geçerlik). Ercan and Kan (2004) describe the two basic properties of a scale like this: reliability is an indicator of the stability of measured values; validity is an indicator of the degree to which the intended characteristic is measured correctly. Other Turkish texts also show the forms “güvenilirlik” and “geçerlilik”.
• Content analysis (içerik analizi / içerik çözümlemesi). Metin and Ünal (2022) present the method as a technique within qualitative research that analyses documents and texts by rules such as sampling, coding and categories in order to reach objective, measurable, verifiable knowledge. Their article identified 32 doctoral theses using this technique in sociology and communication sciences in 2013–2021 and analysed 16 of them (eight each); it reports that the communication theses did not use the technique as an auxiliary method and took care over categories and codes.

The second item is interesting: Berelson’s classic definition treats content analysis as numerical description (content analysis). The Turkish article places it within qualitative research. It looks like a quarrel, but it is often a matter of names: the same word can cover both a study that counts categories and a study that interprets text (our interpretation). In an exam, answer in the language of whichever framework the question uses; but know what the concept does.

Card

This is how exams ask it

Frequent patterns

• “Does sample size guarantee representativeness?” → No; how the sample was selected and the response rate decide (Literary Digest, 1936).
• “What does random assignment secure?” → That the groups are similar at the start, that is, internal validity; not generalisability (external validity).
• “Reliability versus validity” → being consistent is not the same as being right.
• “Is correlation causation?” → No; third variable, reverse direction, time order.
• “Statistical significance versus effect size” → significant does not mean large.
• “What is meta-analysis, and what is the file-drawer problem?”
• “Panel versus cross-sectional study” → returning to the same people (Erie County, 1940) versus a one-time snapshot.

Tempting wrong answers

• Swapping reliability and validity.
• Confusing random assignment with random sampling: one is about splitting people into groups, the other about selecting from the population.
• Thinking content analysis measures the “effect on the audience”: it only counts the message.
• Immediately calling a study that fails to replicate “fake”.

Name–year pairs

• Berelson (1952): content analysis. Merton and Kendall (1946): the focused interview. Glass (1976): meta-analysis. Lazarsfeld and team: Erie County, 1940. Lincoln and Guba: trustworthiness criteria for qualitative research.

Alternative names

• güvenirlik / güvenilirlik (reliability); geçerlik / geçerlilik (validity); içerik analizi / içerik çözümlemesi (content analysis); the focused interview (1946) and the focus group discussion (the later name) are close but not identical.

Exam-type questions here were written from general patterns; whichever source you read, run the question through the filter “from whom, measured how, compared with what?”.

Summary

Summary card: five questions for reading a study

• Main idea: a finding is only as strong as its method. Ask: from whom, measured how, compared with what?
• Sampling: how people were selected and the response rate decide, not size (Literary Digest, 2.4 million answers, wrong forecast).
• Survey: gives proportions; a panel returns to the same people to catch change (Erie County, 600 people); question wording moves the answer (Rugg).
• Experiment: random assignment gives internal validity; external validity is a separate question (Gerber and colleagues: 3,347 households; Sears, WEIRD).
• Content analysis: counts the message; a codebook and intercoder reliability are essential (alpha; percent agreement can mislead).
• Qualitative methods: look for meaning; they have their own trustworthiness criteria (Lincoln and Guba); they do not answer “what percentage?”.
• Measurement: reliability (consistency) ≠ validity (accuracy); self-report can be inflated (Prior: threefold).
• Correlation ≠ causation: third variable, reverse direction, time order (Toronto; effect/selection hypotheses).
• Significant ≠ important: in a large sample a tiny effect also turns out significant (Kramer: d 0.001–0.02); ask for the effect size.
• Meta-analysis and replication: combining studies helps, but the file drawer and bad measurement are traps; in the Open Science Collaboration 36 percent of replications were significant.
• Exam traps: reliability–validity, random assignment–random sampling, significance–size, panel–cross-section.