The firm distributed questionnaires to 1,000 students and 40 teachers before the launch, and follow-up telephone interviews and questionnaires throughout the program. The measurement techniques demonstrated that students answered questions about proper nutrition with greater accuracy after the CD-ROM was released, and that the interest in talking to other family members about the importance of eating fruits and vegetables increased from 45% to 67%. "Not only did we find that the needle moved in the right direction, but it gave the company information for future strategic planning."Such in-depth analysis doesn't come cheap. Lindenmann estimates the Dole analysis cost between $20,000 and $30,000. "The general rule of thumb most researchers recommend is that you devote 10% of your total budget to research,
he explains. "If you don't do it, it's like penny-wise, pound-foolish."
Survey work, the most expensive of all measurement offerings, can cost up to $50,000, depending on the sample size. Media analysis is also variable depending on the number of outlets being covered and analyzed. Steve Lombardo, president and CEO of StrategyOne, a measurement subsidiary of Edelman, says that can cost anywhere from $5,000 to $15,000 a month. Biz360's clients tend to pay anywhere from $3,500 to $12,000 a month, depending on their size.
Measurement experts want to change the perception that analysis is expensive, saying that it is less pricey than a poorly conceived program. "If a company has an agency, and you have 10 people on a daily basis selling your message or telling your story, if they aren't doing it against the clear objective, you've wasted a lot of effort,
says Applied's Marklein. Applied charges $500-$1,000 a month for each performance indicator.
Katie Paine, founder of KDPaine & Partners, targets small and mid-size companies - the ones that don't get much media coverage, and need to know how measurement is valuable for their businesses and budgets. After answering questions on an interactive survey, the client gets a custom-designed program with recommendations for how to assess its existing program.
PRTrak offers another budget approach: a software tool for evaluating PR outputs, costing anywhere from $100 to $245 a month. A typical report will include reports on broadcast, print, and other media reports, with breakdowns of ad equivalency, length of story, and circulation.
In terms of cost of in-depth research, there seems to be no real difference between retaining a stand-alone company or using the PR agency of record to also execute measurement on its programs. But there is some debate about whether a PR agency should be evaluating the success of its own programs.
"I think there is a difference,
says VeriSign's Galvin. He retains Applied Communications as agency of record. And even though Applied has its own research unit, Galvin just hired Biz360 for media analysis. "When you are using your own PR agency, it's close to what you are doing, which can be good and bad. Anyone who plays some role in it obviously has some attachment to the results."
Some PR firms also see an advantage to outside research, even when they have their own proprietary tools. "It's a valid point to make,
says Chris Genasi, who was recently named global director of strategy for Weber Shandwick Worldwide. "If the budget's there, we'll always recommend using an outside company."
But Genasi and others who do the research make a point of being transparent about how they evaluate the media and execute other methods of evaluation.
"At the end of the day, you just have to trust us, because we aren't going to put our reputation on the line and fiddle the results,
says Genasi. "Because it's just kind of pointless."
All things being equal, some clients are just philosophically more oriented toward measurement than others. Companies in the hi-tech and pharmaceutical space - industries with rich R&D traditions of their own - need little in the way of explanation of measurement's benefits.
StrategyOne's Lombardo has done analysis work for the Pharmaceutical Research and Manufacturers of America (PhRMA), as well as Johnson & Johnson and Pfizer. "These companies easily adapted and understood the idea of developing the program on the front end using research, and then evaluating it on the back end,
he says. He adds that the tradition of research in the company also makes it easier for companies to understand when PR programs aren't working. "It's easy for them to understand," Lombardo says, "because these companies spend millions on the front end of research on products that often don't make it past the FDA. A good idea may not be enough."
Applied is retained by SeeCommerce, a technology company steeped in the research tradition. "We are hyperanalytical just by the nature of what we do,
explains Mike Mansbach, the company's VP of marketing and business development. "If I state something and the audience is hearing something else, then I'm failing as a marketer. With Applied's metrics, we're able to understand the impact of any messaging we could use."
Measurement companies say more clients are seeking evaluative solutions than ever before. But it is not clear that companies see its potential the same way the vendors do.
Oracle recently issued an RFP for help in reevaluating its media measurement.
Stacey Torman, director of external communications, says, "It's a great indicator of past trends, and a lot of the systems can give you information about recent trends too. But PR doesn't wait for a media measurement report. It's a handy tool to see if you can focus on trends to target coverage in the future. But you've got to know the media client yourself."
The potential for using measurement in an international context has not even begun to be tapped, says USC's Swerling. "We have a tendency to take a very US-centric view of things,
he says. "The information in this should be an eye-opener vis-a-vis the different mindsets and ways of doing business at work abroad. The information is out there."
One industry debate still involves whether or not there should be consistent standards of measurement, like those in the ad industry. While the Institute for Public Relations and others believe a standard is important, there is no consensus. "I think, to a very great degree, there already are standards,
says Carma's Barr.
Barr would rather see the industry focus on the use of technology and the internet. While often a cost-effective research bank, it does not always reflect the same information as the print or broadcast media.
Others look even further ahead to the future of measurement. David Senay, regional president and senior partner at Fleishman, sees metrics as a way for PR people to increase their relevance in the corporation, so long as the focus is on proving it helps drive the underlying business objectives.
"Communicators across the industry continually encounter people who have the information - people who know what they want to do, but don't know how to get it done,
he says. "There isn't a group better equipped across the enterprise to do that than the communicators. If you buy into that, you have really upped the ante in the endgame for measurement in the communications milieu."
SURVEY OF ATTITUDES TOWARD MEASUREMENT
Delahaye Medialink research on methods of and attitudes toward PR measurement yielded some encouraging results for those who endorse an analytical approach to PR measurement. The online survey yielded 263 responses, 60% representing corporate PR departments, 32% from PR agencies, 6% nonprofits, and 2% government.
"Those who measure changes in awareness, understanding, attitude, and preferences reported a higher level of satisfaction with their PR measurement system than those who spend less,
explains Bruce Aube, Delahaye Medialink's director of survey research. A higher level of satisfaction was also found among those organizations spending more than 5% of their budget on metrics.
Another interesting point the survey reveals is that while measuring press clipping volume is the second most-common method for proving ROI of PR, it is ranked second from the bottom in terms of its importance as a yardstick. Only advertising value was judged less significant, but agencies and companies do not always think alike. "Corporate PR believes that senior management thinks measuring press clipping volume is less important than do PR agencies,
Mark Scott is a great example of how far measurement in PR has come, and how far it has to go. As VP of marketing for HomeBanc Mortgage Corporation, Scott retains Burrelle's Information Services to help him justify his PR budget to the CEO and head of marketing.
His most important metric is advertising equivalency, one of the most controversial tools in the measurement business. Based on advertising rates, it measures how much money a company would have paid to appear in the publication or broadcasting outlet, compared to the "free
inclusion PR efforts garnered.
Scott knows this approach is often criticized. "Most of my friends in PR don't like putting dollar ad equivalent numbers to PR results because they say an editorial story is infinitely more valuable than a similarly sized ad,
he explains. "I understand that, but when working in a corporate structure, I need to justify my expenses by some concrete value. This works very well."
Measuring the effectiveness of PR is a critical issue, as every marketing discipline within a corporation fights for resources. Services are offered from a variety of sources, including research firms like Carma International, Delahaye Medialink, and Biz-360, and the measurement units of PR firms like Fleishman-Hillard, Applied Communications, and Edelman.
That's just to name a few. Katie Paine - who founded Delahaye, and just launched her own research firm, called KDPaine & Partners - has been putting together a website with links to measurement companies around the world.
So far, she has identified 31 companies that just do media analysis, not to mention the many other services out there. "My guess is by the end of the month we'll have over 100,
Some of the debate about measurement revolves around the development of consistent standards, or the differences between stand-alone companies and PR agencies. But the firms themselves are preoccupied with other issues.
Many want to use their expertise to provide in-depth counsel that forms strategy on a regular basis, and not just offer hard data.
Mark Weiner, CEO of Delahaye Medialink, says it's like the difference between a child's report card and a tutor. "If clients can take the report card and do something about it, there is an opportunity,
he says. But too often, he adds, clients need help interpreting the information and recognizing how to use data to enhance their programs - just like a tutor can benchmark a child's progress while guiding the process.
But for Scott and others in corporate PR, playing the numbers game is crucially important. Stand-alone measurement companies and PR agencies with proprietary tools have to accept that fact, and understand the internal pressures clients face. According to the metrics provided by Burrelle's, the ad equivalency for HomeBanc's editorial coverage last year was about $810,000. "When you consider that I make considerably less than that, it's OK,
Scott says. "For what they paid me and the expenses I incurred, I don't think it exceeded $150,000 to $200,000."
It's tough to argue with that logic, particularly in a belt-tightening market. Scott sees the value of competitive analysis, but it's not something he wants to pay for right now. He also understands the potential pitfalls of his system - primarily, that it can thwart a strategic approach. "You can be tempted to sacrifice strategic PR for stuff that's going to bring in the numbers,
Scott once spent five months getting a premium placement on the front page of the South Florida Sun-Sentinel business section. "It was a nice thing to have, but extremely time consuming to make happen,
"So if I were less good at managing my time, I might not have done that, and gone for quick personnel news releases and other simple things that I know will get the ink, but don't have the strategic value. Everyone likes to see the big articles, but at the end of the month, they forget those and look at the numbers."
For those who like the ad equivalency model, it will always be available to them. The beauty of measurement is that companies can pretty much find out anything they want to know about the impact of their PR programs.
Companies and agencies that carry out research are all convinced of one thing: Good measurement engenders better PR programs. Just like PR, there are many ways of accomplishing an objective.
Media analysis is still the most popular metric going, which makes sense, as media relations is the bread and butter of PR. But the basic offering has many applications.
Obviously, the essence of effective media evaluation is looking at the quality of the coverage, not just the quantity. "You can't judge a Business Week cover the same way you do a page-17 story in the business section of the San Francisco Chronicle,
says Tom Galvin, director of corporate communications for VeriSign. Galvin just retained Biz360 to research the company's media performance. "Unfortunately, many of the measurement companies don't make that strong a distinction between the two."
On the strategic side, companies can formulate strategic plans based on the coverage of a certain issue or industry sector. Biz360's product Market360 offers a real-time analysis across print and web media, and now includes the broadcast component through an alliance with Multivision.
When Biz360's client 3Com was going through layoffs last year, the company was able to develop its media relations effort through its review of the issue's coverage. "We tracked which journalists were writing about the layoffs and who their influencers were,
explains Keith Goldberg, VP of marketing. "We got a good idea about who to pre-pitch to, and to see how their competitors were positioned on the same subject."
PR21 has its own research division, called competitive intelligence services, and has also used media analysis to evaluate the efficacy of programs.
PR21 uses information from a variety of vendors, including PRTrak, and conducts analyses based on a combination of that and its own data.
Competitive media analysis is a popular tool. Recently, a client that wanted to assess the effectiveness of its Winter Olympics promotional plan, retained PR21. The client's chief competitor had chosen to become an official sponsor, but this company decided to instead focus on supporting individual athletes.
"The client believed it could attract more favorable coverage and do so on a more cost-effective level than being a traditional sponsor," explains William Black, VP and head of competitive intelligence services. "They asked us to track that, and advise them as to whether their diverse campaign had equal or greater success than their competitors'."
Black says the goal of the research was to inform the company's strategic PR plan for the future, not just serve as an audit of past performance.
"The opportunity in media analysis is to take that information and look ahead to the future, and make forward-looking recommendations and give strategic advice,
Another interesting application that a number of companies offer is the evaluation of people who journalists actually quote in their articles, like market analysts or academics, or industry leaders. "This suggests that the folks quoted are people you should reach out to as third-party influentials,
explains Jerry Swerling, a PR consultant and associate director of the University of Southern California's Annenberg Center for Strategic PR. "It also indicates a company's most aggressive competitors in terms of executive communications."
Looking at a client's competitors can yield unexpected results. Applied's Mediashare Performance Indicators include 13 different products to help evaluate PR programs, including competitive analysis. Tim Marklein, SVP and partner responsible for research and planning, says that companies do not always know who, in terms of media share, their true competitors are.
"If you talk to the CEO of a company, you will get the sales competitors. The chief engineer will look at the three products that are most comparable,
he explains. "But that product and sales comparison is very often out of synch with the media comparison.
Marklein says revealing those gaps between perception and reality creates the opportunity for having an impact on strategic PR planning.
Albert Barr, president and CEO of Carma, believes that media analysis still needs refinement and targeting. His 16-year-old company offers a media analysis that juxtaposes press reports with things like share price, as well as more in-depth survey tools. "The area of media analysis has evolved out of PR; it needs to evolve more into advertising, marketing, and IR,
he says. "What's a bad thing is that we are not considered part of the bag of marketing tools that marketing, advertising, and IR people use."
The evolution of measurement
Media analysis, popular though it is, is only one measurement tool. Weiner, of Delahaye Medialink, says that measurement has been evolving in two critical ways: One is rooted in the advanced technology that enables greater automation and speed, and the other is a greater awareness of the importance of measuring PR's impact on a target audience.
An important feature of that evolution is observing how PR works together with other marketing functions. Together with a company called The Hudson River Group, Delahaye Medialink has been able to put the PR efforts in that total marketing context. One example is the work the company did with AT&T in 1999.
The company had been tracking the connection between AT&T's advertising expenditure and brand loyalty for years, and typically observed that there was a direct correlation between the two. But there were sometimes inconsistencies between the amount spent on PR and brand loyalty. In short, the company was not always getting the brand payoff for its ad investment.
During one such gap, Delahaye juxtaposed the PR effort with the advertising spend. The study revealed that the gap in AT&T's ad spending and public attitudes about the company was connected to a negative piece that ran about the AT&T One Rate plan on 60 Minutes. "The ads that were running at that time were in support of the One Rate plan,
explains Weiner. "So in the minds of consumers, this was a big disconnect, and they attached the negative understanding from 60 Minutes to AT&T every time they saw the ad."
Data like that can really make a company take notice - particularly when millions of dollars are at stake. "What's most meaningful about this research is it doesn't just help to measure a PR program. It puts PR in a much larger context,
says Weiner. "AT&T's PR budget may be a few million, but the ad budget may be hundreds of millions. That ad money can be undermined if the PR is not working.
AT&T's strategy for advertising now always involves a PR component, Weiner says.
Fleishman has had a research department for 26 years, back in the days when there was only one office. While its media analysis tool called ECHO (Every Communication Has Objectives) is its most popular offering, the firm has developed other programs that evaluate perception and consumer action.
One Fleishman service called READ (Reputation Equity Assessment Direction) is a focused-perception audit using survey research to assess awareness and perception among a client's target audience. READ is particularly geared to helping in the strategic planning of PR programs, and is usually conducted at the beginning of a PR relationship.
"Typically, there are two to four groups a client wants to focus on developing new communications tools for, and doesn't really have a hold on what those groups are thinking,
says Lisa Richter, senior partner and director of Fleishman research.
The firm conducts surveys of the company's impact on the target audience, in conjunction with its major competitors, and then follows up on them periodically to assess whether attitudes have shifted.
Until two years ago, Dr. Walter Lindenmann was head of Ketchum's research unit, and is now an independent consultant specializing in measurement.
Ketchum handled the evaluation of a campaign for the Dole Food Company, which had launched a CD-ROM to help educate third-graders on the importance of eating fruits and vegetables.