HOW MUCH DO YOU WANT?: How do PR pros really feel about research and evaluation? In an exclusive Proof survey, PRWeek reports on the techniques, benefits and obtacles

PRWeek's campaign to boost the industry's research and evaluation efforts takes another step forward this week with the launch of an exclusive report on PR professionals' attitudes to the subject. The results are controversial and in some cases alarming.

PRWeek's campaign to boost the industry's research and evaluation efforts takes another step forward this week with the launch of an exclusive report on PR professionals' attitudes to the subject. The results are controversial and in some cases alarming.

In May, the magazine staged a videoconference in New York and London to unveil a Research and Evaluation Toolkit (PRWeek May 17, p20) which had been produced by PRWeek UK in conjunction with the leading trade bodies in the UK: the Public Relations Consultancy Association (PRCA) and the Institute of Public Relations (IPR).

This toolkit aims to provide an industry standard for measuring the effectiveness of PR programs, and following its launch, two leading US-based Fortune 500 companies ? GM and AT&T agreed to test the toolkit.

But regardless of its success, the overall aim of the Proof Campaign has always been to promote greater and more judicious adoption of R&E techniques in the first place. And as with all R&E, a benchmark is required in order to measure changing habits as the industry develops.

With this in mind, PRWeek has endeavored to discover how agencies and corporate communicators really feel about R&E. Are they genuinely committed to evaluating their work? What techniques do they use and what obstacles lie in their way?

These were some of the questions raised by the Proof Campaign US Benchmark Study, conducted by CARMA International in June. This comprehensive insight into the use of R&E was based on extensive phone interviews with 301 US PR professionals working at agencies (54%), in corporations (31%), and at government departments (6%). They were randomly chosen from PRWeek's database of subscribers and readers.

An identical study commissioned from CARMA by our UK sister publication had concluded that agencies are paying lip service to the evaluation issue ? and that very few of them are actively measuring the effectiveness of their campaigns in a sophisticated manner.

In some respects, the US results provide more cause for optimism. It was pleasing to note that many agencies are carrying out some form of evaluation ? 69% carried out R&E into their latest PR campaign. And respondents agreed overwhelmingly (92%) that R&E was important, with 62% agreeing strongly that they were personally committed to evaluation and 30% agreeing somewhat.

Elsewhere, however, the study paints a picture of uncommitted clients, inconsistency of approach, confusion over methodologies and a disturbing lack of confidence in the effectiveness of R&E as a business tool.

These problems exist among not merely junior PR staff, but PR pros at the highest level. Of the respondents to the survey, 23% were presidents or CEOs and 23% were directors. Nearly 30% of the remainder were managers or vice presidents. They also work across disciplines, from the corporate sector (21%), business-to-business (17%), and consumer (12%) right through to specialist areas like cyber-marketing (1%).

Only a paltry 7% said measurement and evaluation was the main challenge facing the industry ? less important than keeping up with new media developments or gaining access to upper management (see chart three). Yet 56% said the most crucial task was to enhance the credibility of PR ? the very issue evaluation addresses. More pros thought evaluation aided strategic planning (36%), than saw it as a tool for showing the value of PR activity (30%) or demonstrating a return on investment (21%) (see chart two). Chart two confirms that only 7% see an increase in professionalism and the reputation of the PR industry as a key benefit of evaluation.

How does this square with the 92% of respondents who felt either strongly (63%) or somewhat (29%) that the industry needed to improve its efforts at evaluation, or, for that matter, the 92% who claimed they were personally committed to evaluating their own PR efforts. Perhaps it is a matter of semantics, since it stands to reason that whether it's as an aid to strategic planning (36%) or to show that objectives have been met (33%) or to help show the 'value' of PR activity (30%), it all amounts to an increase in credibility.

Techniques used
Methods of evaluation vary considerably (see chart one). By far the most popular techniques used in general (see green-colored bar) are consumer/stakeholder surveys, which have been used by almost half of those questioned ? a considerable improvement on the UK, where the figure was closer to 20%. Media content analysis and press clippings were widely employed (39%), followed by focus groups (28%) and interviews with the target audience (18%). Media reach and opportunities to see scored more than 16%.

Despite the potential of PR to influence the bottom line, sales and share price rises did not seem to be regarded as a particularly useful measure (7%). The antiquated method of using advertising value equivalents (AVEs) has become discredited and was cited by just 7% of respondents. More alarmingly, pre-testing was virtually non-existent.

When the focus was narrowed down to their most recent campaign, an impressive 69% of respondents said they had used some kind of evaluation technique.

This, of course, means that 31% did not conduct evaluation at all. But more worrying is the marked contrast between the methodologies respondents claimed to use, and the technologies they actually employed (see red-colored bars in chart two).

In the most recent campaign, media content analysis or press clippings were used by 68% compared with 39% overall. And more than 10 times as many (58% vs. 5%) used anecdotal feedback or gut instinct ? clearly the least reliable form of evaluation. AVEs, which at 7% appeared to be discredited, were in reality adopted by 24% of respondents, more than four times as many. One-to-one interviews (from 18% to 48%) and media reach/opportunities to see (47% up from 17%) also tripled in use. And bizarrely, industry prizes and awards accounted for as much attention as sales and share price increases (18%).

Of course, it's not always appropriate to conduct more sophisticated R&E techniques, but nevertheless, the heavy reliance on gut instinct and AVEs is still disappointing.

Those who reported that they had devoted resources to evaluation, spent, on average, 9.4% of their budgets, close to the Proof Campaign target of 10%. On the other hand, since 31% did not conduct evaluation, the average figure is obviously lower. Also, the most often reported budget size was dollars 10,000 so this amounts to less than dollars 1000.

The most worrying figures to emerge from the survey is the lack of confidence in evaluation methods. Only 18.6% felt the results of consumer surveys were likely to convince clients or budget holders to increase spending.

They had even less confidence that media content analysis would do the trick (14%). In fact, 14.6% were unable to name a single effective method of squeezing additional cash out of budget-holders! No wonder only 8% feel that a key benefit of R&E is to justify bigger budgets.

The blame for this lack of confidence is put fairly and squarely on those who hold the purse strings. More than 40% of pros questioned said it was hard to get budgets for measurement (see chart four) and 26% blamed lack of client/senior management buy-in.

That's obviously because 67% feel that research and evaluation is a difficult concept to sell to clients and senior management. Respondents agreed that a set of common standards would make the idea more acceptable. There was considerable interest in an evaluation toolkit ? even though awareness of PRWeek's product was disappointingly low at 11%. Fewer still mentioned the efforts of the PRCA (5%) leave alone the efforts of the PRSA (4%) and the IPR (1%).

Pros clearly want budget holders to take measurement more seriously.

More than 70% agreed companies should take a holistic view of measuring communication and its contribution to the bottom line. A massive 90% asserted that a company's reputation was a measurable asset. Three quarters thought that PR activity could be measured against other marketing disciplines.

Yet it's no surprise to learn that only 3% listed evaluation as the most recent task they carried out. Traditional PR assignments are at the top of the list, with 27% saying that their last task was writing news releases.

Advertising/awareness scored 15%, product placement 12.6%, and organizing news conferences 11.6%.

Still work to be done
Brian Collins, president of CARMA's Toronto office, who is an expert in the research field, welcomed a number of aspects of the study. 'In particular, the significant number of PR professionals who engage in evaluation and measurement was encouraging. It was also encouraging to see they are following PRWeek's guideline of setting aside 10% of budgets for evaluation.

However, there is still significant work to be done. Quite a few PR people are still using 'gut feel' for measurement when they should be adopting more replicable, valid forms of research such as survey research.'

Collins is impressed by the interest PR practitioners are showing in evaluation. 'Of course, this commitment will repay PR people, and they are aware of it. They know that it will show whether they have done their job and show the value of their profession.'

And he offers a sound reason why clients should regard evaluation as essential. 'In our experience, companies that support the PR function and measurement from the executive levels are some of the most successful companies and typically have a strong position in their industry.'

Reacting to the study, Richard George of the PRSA says many of the results are not surprising. His experience suggests too many agencies base evaluation on instinct, 'just sitting down with the client at the beginning of a campaign and doing nothing at the end of it.' As the survey implies, George believes the first step is to convince clients and decision-makers to take a more scientific approach. 'From the initial contact, it needs to be driven home that research and evaluation is crucial. I don't think there's one other discipline which has not recognized that.'

Jack Bergen, president of the Council of PR Firms, confirms the results have important implications for agencies as well as corporate PR professionals. 'The standing of the industry is clearly related to the extent to which it can demonstrate effectiveness. If we can't do this, other disciplines will come forward and start taking on some of the responsibilities of PR. We don't want to be relegated to a second class role.'

Bergen believes management consultants present the greatest threat. 'It's clear that they intend to use the issue of research against agencies. If that happens, it means that the corporate PR professionals may find that they lose control of strategy.'

However, he adds that many clients find themselves in a Catch-22 situation. 'When I was on the client side, I wanted extra money to spend on research. But I couldn't get that money until I could prove the effectiveness of my programs.'

Outlaw poor methods
While Bergen accepts that relationships are often intangible ? 'I don't want to do away with instinct altogether' ? he wants the industry to effectively outlaw poor evaluation methods like advertising value equivalents. 'It's always going to be difficult to come up with one solution, but I think we should agree that there are certain things we won't present to clients.'

Bob Druckenmiller, CEO of Porter Novelli International, says it is unlikely the industry will come up with 'a magic answer' to the evaluation problem anytime soon. But he comments: 'With accounts becoming bigger and the likelihood of more performance-based contracts coming along, we have to value what we do.'

He points out that the lack of historical data on which to base evaluation is a major stumbling block. 'We can tell a client we achieved a seven million reach and that two-thirds of this carried our key messages, but then the client might ask, "Is seven million good?" Perhaps the next survey should poll agencies for historical data about campaigns, so we'd know what to expect of campaigns with similar variables.'

The Proof Study shows that while the US may be ahead of the UK in its commitment to evaluation, there is no room for complacency. One wonders how significant are the other figures in chart four: the 32% who said there wasn't time to do research; and the 21% who felt they lacked definable measures. Agencies must continue to push clients for additional resources, while simultaneously making greater efforts to prove incontrovertibly the effectiveness of their work.

That's not easy because the budget does not appear to warrant it. But if more attention were paid to sales, share price, web site/call center contacts - all techniques that appear at the bottom of the pile ? perhaps the quality of the work would get better, and, in turn, help to achieve the enhancement of credibility that PR pros clearly hold so dear.


Type %

Hard to get budgets 42

Lack of time 32

Senior management / client buy-in 26

Lack of definable measures 21

Have you registered with us yet?

Register now to enjoy more articles and free email bulletins

Already registered?
Sign in