Be wary of data writes John Patkin

The temptation to employ data to generalise and subsequently change programming may be costly. Have you ever thought about the source of your information or sample? Is it scientific or is it flawed by personal judgement?

In his book Thinking, Fast and Slow, Nobel Prize winning economist Daniel Kahneman refers to the availability heuristic as “the ease with which instances come to mind.” Kahneman provides examples of how convenience often tempts us to make a decision based on personal taste, a recent experience, or perhaps a poor sample group. The availability heuristic is worse than a lottery. In a lottery of ideas, each one has a more or less equal chance of being pulled out of a hat; however, the availability heuristic can cloud your judgment. If we could see our brain’s behaviour during episodes of poor judgment, we would never make the same mistake again.

But instead of searching for old bulk erasers to build an MRI scanner, head to the library archives for Radio Research 1942-43. Work by Paul Lazarsfeld and Frank Stanton, and Herta Herzog could be considered the foundations of today’s radio research. Antipodal in approaches, their studies would be considered scientifically sound.

Lazarsfeld and Stanton pioneered complex mixed methods radio research in the 1940s with The Program Analyzer. It involved recruiting a group of ten participants, booking a studio, a number of research assistants, a stenographer, and technicians. The participants would be asked to relax and listen to recorded radio programs and rate them with a like or dislike. They would hold down a green button for like and a red one for dislike. Their choice would be printed simultaneously which would allow researchers to cross-reference their approval or disapproval to the audio and transcript. The participants would then fill a questionnaire which would be scanned by the researchers. The researchers would complete the data collection by asking participants to explain their choices.

The research team accepted some shortcomings. For example, they knew it was hard to recreate the same listening environment in their laboratory. The participants were now among strangers and focused on listening instead of having the radio on while doing other things such as household chores. This was seen as a positive because comments were focused. Researchers also learnt that a dislike did not equal tuning out as it meant listeners might have become frustrated by a mystery, a comment, or a news story; similar to having a love-hate relationship with a shock jock or the reaction to news an interest rate rise will affect mortgage payments.

The design of the Program Analyzer is tapered and systematic. It combines quantitative and qualitative methods that lead to an interpretation. Conversely, Herta Herzog’s investigation into the uses of radio serials would seem wild as it involved thousands of questionnaires and hundreds of repeat interviews. Herzog’s researchers went door to door so they were able to capture the interviewees’ opinions in their homes. Curious about the experience of more than 20 million American women, Herzog discovered the power of radio serials and that knowledge of the programmes raised listeners’ social status. One of the outcomes was that Herzog found that the audience used media for their benefit which reversed earlier beliefs that media used the audience.

Variations of the The Program Analyzer and Herzog’s study are employed in research today, but replication is an expensive logistical nightmare. I know you’re wondering, “No one has time for another survey;” “Who’s going to organise it?” and “Who’s going to pay?” You are right. People are worn out from research and it’s time-consuming and expensive. Finding willing and motivated participants on a tight budget remains a challenge, though some ideas can be found in an earlier article Radio research on a tight budget for regional stations (https://www.radioinfo.com.au/news/radio-research-tight-budget-regional-stations). For pure indulgence, the Program Analyzer of the future could be a science fiction-inspired system that would give us real-time listener data, but ethically it’s a can of worms. So for now, we’ll just have to believe what research participants are telling us.
 
Recommended reading:
Dobelli, R. 2013, The art of thinking clearly.
Herzog, H. 1979, “What do we really know about day time serial listeners?” in Radio research, 1942-1943, eds. P.F. Lazarsfeld & F.N. Stanton, Arno Press, New York.
Hollonquist, T. & Suchman, E.A. 1979, “Listening to the listener” in Radio research, 1942-1943, eds. P.F. Lazarsfeld & F.N. Stanton, Arno Press, New York.
Kahneman, D. 2011. Thinking, fast and slow.

John Patkin is a regular contributor to radioinfo. He is a Hong Kong-based Australian media researcher.