"As planning lead for public relations, I constantly deal with the need for better data to define a business problem and set ROI (Return On Investment) measurement beyond free PR coverage," says Marion McDonald managing director, strategy and measurement, Asia Pacific Ogilvy Public Relations. She is not alone. PR agencies and in-house corporate communications professionals are constantly struggling with this problem. The Barcelona Principles unveiled in June 2010 called upon the PR industry to move beyond the Advertising Value Equivalents (AVE) and agree upon a minimum new standard of measurement. Since then industry bodies, PR professionals and experts have attempted to experiment with new models.
Communications industry finds itself groping the dark when it comes to identifying the outcomes of PR and corporate communication and its value to an organization. The problem stems from the fact that traditional media relation has become only one, and an increasingly narrow, aspect of the scope of public relations. The business landscape for the industry has become so vast and diverse that it now covers areas which were previously considered the domain of marketing and creative advertising. So it has become that much more difficult to measure and evaluate its contribution. Prof Jim Macnamara of the University of Technology, Sydney has identified thirty odd metrics are broadly used for measuring PR and corporate communication today (Table 1).
|
Basic output metrics |
Outputs -> Outtakes |
Outtakes ->Outcomes |
Counts of press clippings
Unique visitors
Engagement
Audience
Views
Influence
Reach
Likes
Impact
Target audience reach
Followers
Awareness
Impressions
Fans
Attitudes
Opportunities to see (OTS)
Clickthroughs
Trust
Share of voice
Downloads
Loyalty
Cost per thousand (CPM)
Comments
Reputation
Hits
Tone
Relationships
Visits
Sentiment
Return on investment (ROI)
Table 1: Breaking the PR measurement and evaluation deadlock: A new approach and model; Jim Macnamara, University of Technology, Sydney
According to the history of PR measurement by Tom Watson the search for methods to measure and demonstrate the value of public relations really became the subject of intensive focus only after the 1970s. In 1994 the International Public Relations Association published a so-called Gold Paper that became a rallying call by industry leaders for communications practitioners to conduct valid and rigorous measurement and evaluation of their activities. In 1987 John Pavlik, an American academic and author who wrote on the impact of technology on media said measurement had become the "Holy Grail" of PR. But despite experimenting feverishly with different models the industry, in the words of Prof Jim Macnamara, has not really "cracked" the measurement and evaluation "nut". Renowned academics David Michaelson and Donald Stacks concluded as recently as 2011 that "public relations practitioners have consistently failed to achieve consensus on what the basic evaluative measures are or how to conduct the underlying research for evaluating and measuring public relations performance".
According to Macnamara there are still some disagreements over the definitions of some key terms that get bandied about in the industry. For instance, there are inconsistencies in the understanding of terms such as ‘reach’ and ‘impressions’ he says. In most advertising, PR literature, engagement is poorly defined and described in superficial ways that regard ‘click throughs’, ‘following’ and ‘likes’ as ‘engagement’ when, in fact, as Macnamara puts it, engagement is a "deep psychological concept".
A progress report presented at the fourth European Summit on Measurement in Dublin in 2012 recommended that "ROI should be strictly limited to measurable financial impact". Macnamara has identified 10 different types of ROIs in industry literature. But this is not conducive to standards or to achieving understanding of the value of PR and communication. A number of other tools for measurement and evaluation have not been given significant attention. For instance, the Benefit Cost Ratio (BCR), which has been applied to PR by theorist James Grunig as far back as the 1980s and Cost Effectiveness Analysis (CEA) developed by management consultant Fraser Likely are based on "sound business principles" says Macnamara. Similarly, market mix modeling advocated by Mark Weiner, various logic models to connect PR processes to outcomes, and communication performance management and communication controlling methods advocated by Ansgar Zerfass are rarely applied.
Barriers
Macnamara has identified three barriers that he believes are causing a deadlock in implementing measurement and evaluation of PR. The first is the obsession with numbers. The punch line of the Institute for Public Relations (IPR), for instance, is "the science beneath the art". This clearly indicates a view that PR should be underpinned by scientific knowledge and quantitative methods of research. But such a view may be flawed. The communications industry is trying to quantify outcomes that are difficult to quantify. "Human interactions; relationships, feelings, attitudes, loyalties, perceptions and engagement do not yield easily to numeric quantification," says Macnamara. Applying "interpretive" qualitative research, he says, is a better way to measure PR outcomes. But qualitative research, which requires in-depth interviews, focus groups studies and ethnography, are usually more difficult to do well than quick self-reporting surveys, ratings and scores that "look good" in marketing meetings.
Second, is the conflation of measurement and evaluation. The two are often conducted in a concurrent or linear process and based on a very narrow range of data and often centered exclusively on metrics as determined by the concerned organisation. According to Macnamara measurement and evaluation are two distinct processes. Measurement is the taking of measures such as counting items, collecting ratings on a scale, or recording comments in interviews, and analyzing these.
The third major barrier to demonstrating the value of PR is that measurement and evaluation processes look at what has been done in the past. So measurement and evaluation often fail to give an organisation anything other than a retrospective performance review of work done. Sometimes M&E is seen as little more than an exercise in self-justification.
New Models
Emerging new models use Big Data to identify insights that can inform strategy. According to Macnamara the communications industry has a tendency to jump to conclusions about findings without adequate data or analysis. He says new models should collect qualitative as well as quantitative data. Allowing time to reflect on findings after a ‘cooling off period’ can be very productive. This allows for in-depth analysis, which can draw on contextual information, published research literature, databases and case studies. Most importantly rather than simply reporting past achievements, insights are forward-looking, creating potential for value addition. (e.g., in sales, reputation, costs, risks or employee loyalty). This forward-looking approach helps bridge the gap between PR and outcomes, which has been seen widely as a big hurdle. "Armed with powerful information and technology in the age of big data, that value (of PR) can go even higher," Jim Weiss, CEO, W2O Group wrote in PRWeek. It also addresses a troubling contradiction at the heart of the PR measurement dilemma – that is, despite demands for results and accountability employers often will not pay for and sometimes even do not want rigorous measurement and evaluation. But with new models they are far more likely to pay for what they don’t know and what can change the future. "Coursing into the seemingly expensive 'river of data' scares marketers as much as it does their agencies. But smart investments in sharper insight development upfront, and measuring the critical behaviour change impact throughout a campaign can save many thousands in beautifully executed but irrelevant messaging", says McDonald.
How to improve measurement
Source: Prof Jim Macnamara, Univ of Technology, Sydney
The Jim Macnamara model of measurement and evaluation
Another emerging model that can be used as a guideline for smarter measurement is the Valid Metrics Framework (VMF). It takes the form of a matrix, with the underlying logic that the model can be applied to a number of different types of PR campaigns. It uses outputs, impacts and outcomes as a framework for setting out what needs to be measured and identifying what tools will be required to get this data. The advantage of using this framework is that it can be adapted to most types of campaigns. "It can help agencies and clients show how PR goes beyond ‘how many people are talking about you’ to finding out ‘what behaviour change are we driving, to our advantage," explains Marion.
The Valid Metrics Framework, AMEC
The VMF was developed by a taskforce set up by the International Association for the Measurement and Evaluation of Communications (AMEC) following the call of the Barcelona Principles. It was intended to provide an alternative to AVE. The framework distributes PR output in three distinct phases and splits marketing campaign in five stages on a simple matrix grid (see above). It works by applying five steps to the matrix. First, choose the grid that is relevant to a particular campaign; then determine the activities to be conducted during the course of the campaign; select the appropriate metrics; review and determine the appropriate resources available and finally decide which of the outcomes are relevant to the client. The drawback of this model is that it is simply a guideline that serves as a framework for identifying possible metrics for individual PR programs. It is not a definitive rule of measurement. At the time of introducing the framework AMEC itself said the model was not designed to be a rulebook and advised PR firms to choose metrics that fit their budget and objectives.
Social media
There are five buckets in which most social media measurement results fall: measuring an organisation’s own activities; measuring the attention that social media brings to the brand; measuring the awareness of people who have interacted with online, and the actions people take as a result of that interaction. The key digital metrics that PR professionals traditionally tracked were clicks, comments and shares. Thanks to Big Data digital communication allows us to measure almost anything. The problem is that there is little guidance on what makes sense.
According to Dr Walter Carl, founder and chief research officer of ChatThreads the way to determine ROI for social media is not the same as traditional media like TV and digital. It should be measured independently of other channels. But at the same time is important to apply the same standards. She believes PR firms should focus on business goals, shift social media measurement metrics from ‘activity’ metrics to "value". In the world of social media people share and amplify the impact of other media, and therefore a value needs to be attached to this. "Measurement must be holistic or else you will miss interaction effects," she says. According to a study conducted by ChatThreads 80 percent of the value came from network amplification. It is therefore important to take that into account while calculating the outcome of a PR exercise.
The example above explains how social media campaign can be measured. Source: ChatThreads
Many firms still use traffic and page views, which have value, but keywords, bounce rate, time spent on site, pages per visit and repeat visitors may be better indicators of audience engagement. Keyword Associaton Mapping (KAM) is an analytical model that has been developed by CIC (above), a leading social business intelligence in China. It provides the user a visual dashboard to assess the effectiveness of a social media campaign.In his book Listen to your B2B Market, Generate Major Accounts Leads, and Build Client Relationships, Paul Gillin and Eric Schwartzman say it is better to focus on three or four elements to measure.
"Based on nearly three years of our own data and insights (quantitative) as well as hundreds of conversations with some of the best and brightest PR professionals, CMOs, and communications executives in the world (qualitative), I believe three distinct areas need to be considered in terms of measuring PR outcomes...," says Sharam Fouladger-Mercer, CEO of AirPR."...these are revenue; key message pickup and amplification; and how you fare against your competitor. By assigning comparative value to industry players, PR will be able to better measure the effectiveness of their efforts.
Agencies jump into the ring
Until recently it were the clients that led the development of new models. In 2013 General Electric, McDonald's, General Motors, and Southwest Airlines adopted the first round of measurement standards proposed by the US-based Coalition for Public Relations Research. The standards' first two stages required the Coalition to propose initial ideas, then develop them into proposals. The third part required a customer panel to review and approve the measurements. The idea was that this would ultimately lead to a common standard.
But now PR agencies have also got into the act. Ogilvy PR Australia has launched its own proprietary measurement system -Impact4, which offers its clients over 70 different tools and techniques to measure public relations and communication campaigns.
It combines the latest measurement techniques, analysis software and tools, and a mathematical "thread" that is said to offer a "unique" combination of predictive analytics, statistical outcome modeling, independent analysis and integrated search optimisation methodology. Impact 4 is built on a framework of different measures, and allows clients to build their own unique model. Built by Metrica, a UK-based media evaluation consultancy the interface provides a graphic dashboard interface to analyse, in real time, the results of PR campaign.
Above: The Metrica dashboard
David Rockland, a partner at Ketchum has come up with the marketing mix modeling. It is a technique that draws "market-influencing" data from disparate sources, applies advanced statistical analysis and provides insights into the efficiency and effectiveness of ongoing marketing programs. Using traditional variables including sales, advertising Gross Rating Points (GRP) or the content analysis of news coverage to represent PR in the form of media relations results, researchers examine the data to create more complete views of the marketplace and to estimate the impact on sales.
The marketing mix model seeks to quantify the impact of any individual marketing activity on sales volume. It also accounts for the effects of controllable (e.g advertising, pricing), semi-controllable (e.g public relations, sponsorships) and uncontrollable externalities while calculating the relative and absolute returns of each of the various marketing activities. By taking each communication expense into account, brands can empirically determine the best budget allocation from among their different marketing investment options. The importance of knowing in advance what factors, or combination of factors, drive sales, and their relative efficiency in doing so, this model promises to answer that million dollar question that often eludes CMOs – what part of the marketing spend is being wasted?
|
Steps |
The Marketing Mix Model |
|
I |
Determine how the variables are collected in terms of frequency and reach |
|
II |
Select an approach to collect data representing desired media coverage |
|
III |
Create the marketing mix model |
|
IV |
Refine model |
The drawback of the marketing mix model, as with many others, is that is only as good as the inputs it receives. In other words it is very difficult to take into account every possible factor that drives sales. Most PR professionals are unfamiliar with statistical techniques and most statistical modelers are unfamiliar with the position of PR in the marketing communications mix. " Few PR people even know what it is, and fewer clients use it, despite IBM proving that companies that do can generate upto 12X greater profit, by knowing which elements of marketing spend are working," laments McDonald.
It is no more a matter of debate whether the PR industry needs better models of measurement. Everyone recognizes that need. What is needed is a common minimum standard that everyone can agree upon. As Jaideep Shergill, CEO, MSLGROUP India puts it bluntly - "If PR industry doesn't know how to measure…it is not going to get us anywhere."