Measurement Roundtable: Defining value

Be it tactics, investment, or integration, the measurement conversation continues to fundamentally change.

Be it tactics, investment, or integration, the measurement conversation continues to fundamentally change. Erica Iacono and Tonya Garcia were recently in New York for a BurrellesLuce-sponsored roundtable on the topic.

Saswato Das, director of global communications, SAP AG

Mark Stouse, director and global communications leader, BMC Software

David Kellis, senior group PR manager, The Clorox Company

Tim Marklein, EVP of measurement and strategy, Weber Shandwick

Katie Delahaye Paine, CEO, KDPaine & Partners

Natasha Fogel, EVP and GM of global analytics, StrategyOne

Chris Atkins, VP of communications, Standard & Poor's

Lt. Col. Michael J. Paoli, chief of media and opinion Leader Engagement, U.S. Air Force

Johna Burke, VP, BurrellesLuce

Erica Iacono (PRWeek): What do you see as the biggest changes in traditional PR measurement over the past couple of years?

Mark Stouse (BMC Software): It's less about conventional PR metrics. Not that those are not interesting and important, but it's much broader. We're looking more at substantiating the impact to the business. If you approach it as a business person first and communications executive second in the same way that your CFO or chief counsel does, you're going to be saying ‘How do I influence sales' or whatever it is that's the final reason for being for your organization.

Saswato Das (SAP AG): You're seeing two trends happening at the same time. The conventional print media is shrinking, there are fewer journalists, more papers are going online. At the same time, you're seeing online social media mushrooming. There's just an overwhelming snowball effect when it comes to information from blogs, Twitter, and other social media.

For the PR professional, it's a question of how do you manage the information explosion online yet make sure you get traditional media covered and measure it because they're still very important and will continue to be so.


David Kellis (The Clorox Company): Social media is a real opportunity for PR. It's a somewhat undefined space. We looked at it within Clorox; who's going to own social media? Is this an online marketing thing? Is this a PR thing? [There are] PR professionals who have decided [they're] going to make social media a part of what [they] do and therefore give PR more prominence within the organization.

As PR professionals, if we take the lead on social media, understanding how to do it, and how to measure and tie it back to sales or things important to organization, it's going to give us better seat at the table in terms of overall impact on marketing communications for a company.

Lt Col Michael Paoli (United States Air Force): In the Air Force, our success is still tied to our impact on public sentiment, which we still tie back to traditional media, especially print. But now we've incorporated regular analysis of the blogosphere into our measurements and are looking for direct correlations on how [that's] impacting the traditional press.

Tim Marklein (Weber Shandwick): We're seeing an expansion of the media, but also an expansion of the PR measurement toolbox. It used to be focused on media content analysis. We're seeing much more recognition that there needs to be more than that. We need to be looking at more survey research than I think PR people have traditionally done. We need to take advantage of Web analytics, and search engine optimization data, and the other things that are happening on the online world. We need to be taking advantage of the third-party surveys and scorecards that are being published that are essentially free data that [can be] leveraged within a measurement system.

When you pull all those things together, the big trend that we're seeing with clients is a big focus on integration. It's no longer about what I would call “random acts of measurement.” It's how do you take these different metrics and tie them together to get a holistic sense of what is going on in the business, what are the drivers, what are the meaningful metrics,… and figure out what's driving what.

Chris Atkins (Standard & Poor's): When you start looking at the tactical aspects of measurement, whether it's in the blogosphere, mainstream media, or survey of audiences, [etc], you can get really hung up on the specifics of what you learn and lose sight of the overall picture, which is particularly difficult when you're going through a rough patch as we have been. It's tempting to get hung up on the blogger that won't leave you alone or the reporter who keeps scratching away at the open wound.

It's important to find a way to step back and say “Where is our problem here? Is it really in the blogoshere? Is it something we can counter through other communications channels?” Frankly, that's a difficult thing for people in a resource-constrained environment to do.

Katie Delahaye Paine (KDPaine & Partners): It used to be that PR measurement was about justifying your existence. Finally people are realizing we need the data to figure out where to spend our resources and where to spend our efforts and energies. If something isn't working, take the money and do something else. But we need to know what's working, what's not working, and how to continuously improve our program.

I [have] said, “Anybody who's not measuring today won't be around for very long.” Somebody is going to come along and say, “I only have this many resources.” We're going to use measurement finally as the tool that it was designed to [be], which is to get better.

Changing the definition

Stouse (BMC): It's operational analytics. Measurement, I'd like to propose, is a dated word. Measurement historically has been about what just happened and justification. The way we're using it today is that we want to be able to project forward and [determine what] we believe the coverage will look like and get ahead of that and meet it.

Marklein (WS): Another change that's important to that operational analytics notion is a shift in the time cycle. Measurement traditionally in PR had been a quarterly kind of thing. In this day and age, it's happening in real-time. You're not making decisions in real time unless you're in a crisis. But we at least need to get to a weekly or monthly standard in order to get to that operational analytics [issue] because otherwise the data is not really helping you make decisions.

Kellis (Clorox): We're getting the data in real time on an online basis and being forced to react to it. It's taken the PR industry a number of years to get to the point where we're able to measure PR and its impact on sales. We [Clorox] were able to do that a couple of years ago. It's a marketing mix model and it's impressions that are driving sales. Our group looks at the number of impressions generated, treats it somewhat similar to advertising, and maps that against sales that are going on market by market.

So impressions became the thing and now social media comes along and it's really difficult to put social media into the impressions category because you can tell how many people potentially read a blog or link to a blog, but you don't know the ripple effect.

Paine (KDPaine): You don't know how many people read my blog unless I tell you.

Kellis (Clorox): Right. So therefore it's harder to measure, which gets back to “How serious should I take what's going on in social media?” We just figured out that generating media impressions increases sales. Everyone talks about social media, but because it's so difficult to measure and fit into the model we've all created to measure PR you're stuck

Das (SAP): We have to figure out how people are consuming information today and in the future. We want to provide information there.

Paine (KDPaine): We have to do the research. [We have] 40 years worth of research that said if “I hit this many people with this kind of message, they're going to buy this stuff.” We need to start doing the research that says, “So there's a human being doing Twitter… and six months later we sold this much more.” It's just experimental research that needs to be done.

Paoli (USAF): In the Air Force, we tie research with analysis. You rarely see them pulled apart. We're doing it monthly beginning with communication audits; are our messages resonating out that there. We have monthly focus groups and surveys that then tie back into our strategy and analysis.

One of our greatest challenges has been and I think always will be not how well we measure things. When you get into the weeds of it, you can argue over the percentage points regarding impressions and so forth. I can tell you, big picture, if your story is on page one of the New York Times, that's generally an indicator of how everything else is following suit. Our challenge is once we measure it, what do we do with the information. There are a lot of unknowns out there, especially with social media.

Paine (KDPaine): For years we've gotten caught up in this conversation about we need a standard way to measure. Measurement for years might have meant to one person impressions, to somebody else it might have been sales. Now we're forcing people, because of the nature of social media, to say, “Am I doing this because I need to improve the reputation and the way people perceive me or for sales?” If it's sales, you get your Web analytics tools out and you do that. If it's reputation, then you're going to do reputation studies, relationship studies, [etc]. It's forcing people to make a decision up front about what their measures of success really are.

Stouse (BMC): In the haste to quantify, we also are in danger of losing highly relevant anecdotal information. For example, one of the things that we do with our executives and sales guys, we'll sit after a major sales meeting, and we'll get a complete debrief.

Last week when we had our story in the Wall Street Journal, it came out the morning [of a] meeting with two other large companies. The presence of that article that morning completely changed the dynamics of that meeting and actually accelerated the deal. That's anecdotal but it's highly relevant.

Kellis (Clorox): We're working through the swine flu issue at Clorox. We've been able to establish that a number of our products kill H1N1. Because this is a public health crisis, we didn't want to be out there pushing a story or marketing to fear. What we did was rely on our influencer network of doctors that we've built up relationships with.  Not even pushing the doctors out, but knowing the media would call upon medical experts to talk about this issue.

Over the years as we've been able to measure the effectiveness of impressions, the proportion of our budget has shifted more toward media relations and less toward influencer work. But if we weren't doing that influencer work over the years, we wouldn't in this time of crisis have had these people that we can reach out to. You're talking to a marketing VP and we [say] we want to spend on influencer relations and they say “what's going to be the result of that this year?”, we say “building better relationships.” Now in a time of crisis, the years that we put into it have come back to help us in a way when we couldn't communicate ourselves about something.

Marklein (WS): The anecdotal can become quantitative. It's that anecdote that can spark a firestorm for you or against you.

The other dimension is the qualitative factors that go into the discussions we're engaged in, which don't fit the realm of a lot of marketing communications metrics because there is no qualitative factor. You publish something and people read it. That's what makes PR measurement hard, but makes PR powerful. We live at this intersection of qualitative and quantitative; of anecdotal and these big numbers.

Atkins (S&P): You can't expect your boss to go into a board meeting and say that the rate of decline in our reputation has slowed to 3%. And yet the ability to say our efforts are paying off and pass out a clip of a Wall Street Journal article has much more emotional meaning. I say this almost sadly because it is to some degree a certain amount of show-and-tell that just becomes part of explaining how we do what we do and what the result is. And it would be comforting to be able to go back and say the numbers tells the story that we want them to tell.

Paine (KDPaine): But it is measureable. You can measure trust and commitment. I do it every day.

Marklein (WS): The anecdotal part can be as persuasive as the measurement.

Atkins (S&P): Or more so. Edelman has made a whole business out of the trust barometer. Edelman, in the interest of full disclosure, is our firm. But I know there are examples where people have gotten very happy or upset depending upon what they happen to be reading in the Wall Street Journal in a given moment that went far beyond the reaction they got from a Power Point slide that talked about the numerical calculation either in the quantity or quality of what we were doing.

Das (SAP): The good clips are easy to measure. Their influence is easy to see. But when you keep a bad story out or you tone it down, how do you measure that? That's a more fundamental challenge.

Stouse (BMC): One of the things that we've been working on is putting together a chain of analytics that supports a revenue contribution. So that we can credibly say, for example, that in [any given quarter], [we can establish] a revenue [contribution]. When it comes down to budget time, it's not at that point about defense. It's about additional investment. Let's invest in success. Whatever the equivalent is for revenue in your organization, if you can prove that, you've got it locked.

Johna Burke (BurrellesLuce): How did you come to that agreement with the rest of the teams to be able say that is this is the percentage that's attributed directly to our PR efforts?

Stouse (BMC): In my particular case, it's made somewhat easier by the fact that we essentially don't advertise. BMC puts the vast majority of its market awareness on the back of comms. So from that standpoint, it's a lot easier to lay claim to. But you still have to stitch together the data points logically to say this is how I got here where it's defensible.

Marklein (WS): What we've seen is that what Mark and BMC is similar to what others have done with market mix analysis. But there's a lot you can do even short of market mix analysis to get where Mark is talking about and it does get to revenue contribution. The studies that I've seen with different clients, it ranges from 1% to 20% in a b-to-b enterprise sales environment.

Think about the PR community at large, if everybody invested the time, and work, and energy to get to that place where you know whatever percent it is for your company, we would be having a very different conversation in the PR industry about our business. The majority of people still out there don't believe PR can contribute to sales.

Growing importance


Iacono (PRWeek): There's been a lot of talk that measurement during this recession is even more important. Are clients investing more in measurement now?


Kellis (Clorox): Absolutely it's become more important. Each [marketing communications] program that's put forth is being questioned and is under scrutiny. So [if there's] $20 million to spend what is [a] $2 million program going to drive? If I don't have any historical data on ROI of a spend of that nature then I'm not sure [the company is] going to spend $2 million on PR.

[We] go back to “we've measured PR three times; the ROI is always over $1.” There's a perception now at our company at a relatively modest level of spending, you're always going to get your money back and then some. The question is if we want to take it to the next level and have PR become an even more dominant player in terms of percentage of the overall marketing budget, that's where the data isn't there as much.

The challenge in this economy is that it costs money and resources to measure PR. It's kind of a double-edged sword where it's needed even more but maybe the resources aren't there.

Natasha Fogel (StrategyOne): With some of our even smaller clients, it started with even just measuring impressions or total share of voice. They're moving to [aligning] measurement to the business objectives. Let's start there. So when [they] talk to a businessperson about [the] organization, it's in a language… that they're able to frame the conversation around what they're measuring, their PR activities [and] to that specific business goal that they have.

Paine (KDPaine): Those people that are tying stuff to sales [or] revenue, they are investing in more measurement because they… know this stuff works, [they] want to know what element doesn't work so we can take the money away from there. Measurement gets paid for by the cost savings.

Marklein (WS): If you're doing an ad campaign, measurement is part of it. If you're the CFO, measurement is part of your job. Why does PR ask for separate funds? I always tell people 10% to 15% is probably the best practice. The reason to invest that is it makes the other 85% work harder. You're getting into the right media that will actually drive your business outcome. And the only way you're going to know that is if you do the measurement.

Fogel (StrategyOne): And it increases the probability for success.

Das (SAP): You can't be efficient without measurement. And in hard times, you have to be more efficient.

Atkins (S&P): S&P does no advertising. The interesting aspect of measurement that we've started to move into, which is both interesting and troubling at the same time, we're now working with business process management people, to look at the effectiveness of the organization in responding to events from a media perspective. How long does it take us from the point at which an event is to when we reengage; what is the process, and can that be improved? We as communicators are so dependent on operations people so often and we're not perceived as the most important thing on their plate at a given moment. When you couple actual measurement of results with the operational efficiency and begin to see that there's a connection, it becomes much more powerful to be able to say this needs to be something that's a management function and taken as seriously.

Fogel (StrategyOne): There's an opportunity for increased operational awareness of the PR function. It's not as integrated as marketing and IT.

Iacono (PRWeek): Part of the issue is bringing it into a business language. I know there are still people who consider media monitoring or impressions as their form of measurement. But that seems as though that will deter PR from getting ahead if you're not speaking that business language?

Paine (KDPaine): I think this is enabling PR to challenge the fact that for years advertising claimed responsibility for everything. This is an opportunity for us to change the whole conversation.

Paoli (USAF): Now is the time when we all stand the best chance to get it integrated; when the dollars are tight. In the Air Force, we have a strategic communication working group where we have a representative from every major division sitting on a strategic communication board. We're not doing that simply because we want to do a better job of raising public trust and awareness across the board on a variety of issues. We're doing it in earnest now because even in government dollars are tight. And we have our end-state business objectives and we have to ensure all our communications efforts are tied towards those objectives.

If support for the Air Force has risen, but it hasn't risen in those end-state objectives, we need to go back and look at it. It's the long-term measurement that drives that train.

If you take all those short-term anecdotal pieces you can look back and say we did a good job. But it's that long term measurement, [such as] how are we compared to last year on these end-state business objectives, [that let us know] that simply mitigating and educating in a reactive fashion isn't working.

Atkins (S&P): The challenge is to remove yourself from the immediacy of what's happening right now and ask are we moving the ball down the field. It's only with the perspective of how things have changed over time [that] you sleep at night

Marklein (WS): There's a tradition in PR that it can't be measured. I still hear that a lot. More PR people [should go] back to the CEO after [a] crisis and [ask] “Why did you come into my office when that story hit?” The CEO knows and we all know that that media coverage matters. Start there… and figure out how [to] work [together] to figure out how to quantify that in some meaningful way.

Fogel (StrategyOne): And that's part of the investment in research and measurement – the post-mortem that goes into these projects and the planning that goes into that next event or program.

Stouse (BMC): A lot of times we talk in terms of strategic measurement. But there's strategy, there's the operational level, and the tactical level, and measuring all three of them is very important.

For example, one of the most illuminating measurements that I do is the lost opportunity metric. If there's a roundup that you're not in, why?

We have [also] pulled the sales management approach into my organization. All of my people on both the agency side and internal side have a quota. We take what kind of story it is, the tonality, [etc] and that creates a matrix that delivers a certain number of points. If there are stories that we really want to drive, those stories [earn bonus points]. It also helps you understand who your talented people are. A lot of times, it gets to be such a team situation that you lose that transparency into who's closing the deals. That kind of tactical measurement is also critical to understanding what's going on and how can we improve.

Das (SAP): We take a similar approach. We are trying to run communications similar to a sales organization and the ability to forecast good stories, bad stories, [and] what will appear. We just rolled this out in the past year and we're having very good success with it. What Mark just said about identifying the people who can close the deal - it also allows you to shift resources. There's something going on that you need to pull in people who might otherwise be occupied, it helps you to do that.

Paine (KDPaine): What you're doing is setting up metrics according to what organization wants to become. The organization has never wanted to become more impressions.

Stouse (BMC): When we first started doing it, everybody groaned. Yet fast-forward to today, there's greater esprit de corps, there's greater pride, the morale impact alone is worth it.

Burke (BurrellesLuce): You create your own savings opportunities too by identifying who your key people are and letting them do the media training.

Marklein (WS): This speaks to the issue that communications is not one thing and it shouldn't require just one skill set. Measurement is another one of those disciplines that needs to become core. Not that everyone on the team needs to be a measurement guru, but measurement needs to be somewhere in the organization by somebody who knows enough or gets the right resources to do enough.

Atkins (S&P): The number of tools in the toolbox is vastly greater than most operational functions within an organization. If you try to develop strategy by measuring the success of individual tactics, you'll get very quickly lost in a morass and lose sight of the broader outcome. Having some means of assessing the overall change of perception over time would be of far greater value.

Social Media

Iacono (PRWeek): Going back to social media, what do you see as the biggest challenge in measuring it?

Paine (KDPaine): If nothing else, social media is going to kill forever this notion that PR needs one metric. Social media is eminently more measureable than traditional media. Typically it's online and you can track traffic, it's much easier to count outcomes.

Kellis (Clorox): The power of social media is the immediacy of it. When you do something online and you see the results immediately, people can see that and you can send that out. On the blogosphere, it's there right away and you can forward it. There's a perception that that's doing something. It may not be measureable in the long-term, but in the short-term… to be able [to know how you're] doing within hours of launch. Eventually we'll figure out how to measure it. But it's buzz to get people excited.

Marklein (WS): [Social media is] customers, it's conversations that are happening with our without you. You need to figure out how you're going to engage. Thinking of it more in a customer service or sales mentality helps because you're not thinking of impressions, some sort of big mass media kind of number. You're thinking about having a conversation and [the other person] might become a customer or might tell a friend. If you can use social media to break down the walls and engage in live, real human conversations with people, that starts to change reputation. We can measure it now, but it's going to be a different kind of measurement.

Paine (KDPaine): Don't you think that PR people fundamentally are comfortable dealing with the media than directly with customers? It's all about direct conversations and you're going to have to evaluate that in a different way. It's not a million direct conversations. It's 10 direct conversations that impact your rep in different ways.

Fogel (StrategyOne): One of you said if you're going to measure think Web analytics. If you're going to think about reputation, you should be doing some kind of reputation study. It goes back to what's the goal?

Atkins (S&P): We go out into the field, we do omnibus studies on a regular basis, …we look at a variety of metrics. I think it goes back to what Mike was saying: “There's research and there's analysis.” I would only add, “there's action.” Finding out what happened isn't hard. Finding out why it happened and what you're supposed to do about it is hard. We're moving towards a kind of a why, but [figuring out what to do] is more art than science. An awful lot of what we have to do is tailor what we hear and learn to what works for our individual organizations.

Fogel (StrategyOne): We have years of rich traditional media data and experiences with journalists. Social media, new media, customer engagement, we just don't have that. My discomfort lies there somewhat, which is why we're collecting and starting to look at trends. It's making me more comfortable in working with clients because now you're able to make decisions based on some historical patterns.

Paoli (USAF): If today's media relations expert doesn't also become a community relations expert, they will not survive. Social media, talking to 400 people one-on-one is community relations.

Iacono (PRWeek): What is the general attitude toward ad value equivalencies? Is there is a place for them in measurement?

Atkins (S&P): I have been a vocal critic of the concept of advertising value equivalents for 25 years. It makes no sense as a measurement of your output.

Marklein (WS): In my book, we have to kill the term ad value equivalency. The term is flawed. First, it's talking about ads, which isn't our job. And the equivalency term throws a big monkey wrench in it, which is why we've debated it for 25 years.

What I'd call it is earned media value, and I think there are legitimate ways to calculate [that]. It's fundamentally critical in a lot of organizations where we use it because in most companies, they're choosing [whether to] spend in advertising, PR, and [a number of] disciplines. The lingua franca in that discussion is media value, or cost per thousand, or impressions. It's a sense of how many people are you reaching and what is the value through that medium in which you're reaching. Every time I've seen media value calculated that way, PR ends up looking a lot better than the other disciplines. Not only is it valuable, but it works in our favor.

Stouse (BMC): We're not really talking about the value. We're talking about the cost. What would make it valid is [if] you like this article [and] you bought this as an advertisement, it would've cost you X. We delivered it for X less 90%. All of a sudden, that focuses [on] the fact this it's a good business deal. That's valid. When it really gets ridiculous is when you put the multiplier on it. We've all seen the story that had more impressions than there are people in the world. We have to properly contextualize.

Kellis (Clorox): It's CPM. And when you start to measure PR CPM in terms of efficiency against advertising, [etc], our CPM looks really good. That's a language that executives can understand.

Paine (KDPaine): But that assumes that getting millions of eyeballs is going to sell lots of product.

Marklein (WS): But I would say that that assumption is built into every marketing dollar that companies are spending. So we need to compete on that level. I do think it calls into question whether that's the correct assumption for marketing overall. And that's where I think social media starts to upset some apple carts too. But it is at least the playing field that we're on.


Stouse (BMC): You look at what's happening to the ad agency model, it's so imploding. Do we really want to be correlated with that?

Paine (KDPaine): What we're all looking for to a certain extent is a way to [ask whether a] blog mention, story, [or] Tweet do any good. That's different from earned media value.

Have you registered with us yet?

Register now to enjoy more articles and free email bulletins

Already registered?
Sign in