Can the PR industry be trusted to measure its own work impartially? While the ranks of teachers, doctors and social workers who are beholden to government performance targets might scoff at such a concept, this is exactly what happens when a PR consultancy is tasked with evaluating its own work for its clients. But this may be about to change.PR agencies might soon find their work open to greater scrutiny than ever before.
A recent survey conducted by media evaluation and research company Metrica has highlighted what appears to be encouraging trends in evaluation among senior PR directors and managers at the UK's top 1,000 companies.
Also, long gone are the days of gut feeling. The survey revealed that, over the past two years, the number of large organisations using media analysis has risen by 17 per cent to 90 per cent. PR measurement has also been adopted by 95 per cent of organisations.
Even better news for evaluation evangelists, Advertising Value Equivalents (AVEs) appear to be gradually on the wane with only 34 per cent still clinging to measuring outputs rather than outcomes.
More worryingly, however, the survey also highlights the fact that one in three senior PR managers and directors believe that PR firms are bad at planning and measuring PR programmes, up from 29 per cent two years ago to 34 per cent today - a statistic that does not make for encouraging reading.
Although steps taken to improve the quality and sophistication of evaluation in PR have moved in positive directions as borne out by the results, the issue of how PR firms perform when assessing their own performance remains a sticking point.
'Very few agencies are brave enough to do evaluation of their own work that say: "We did a shocking job on the launch of your new product and don't deserve to be your agency any more",' says Prudential director of PR James Murray.
This lack of impartiality is becoming a growing concern for some clients in tandem with the increasing investment in evaluation.
Pharma firm Wyeth corporate affairs director Gill Markham says evaluation is one of the most important elements considered when selecting an agency, but that standards vary greatly. 'At pitches, we invite three to four agencies, and we always look to see if the agency has its own evaluation tool, and an interesting observation is the wide variety in evaluation tools and methodology,' she says.
Need for standardisation
It was this lack of standardisation that led consultancy Kaizo to develop its own evaluation tool - Value Flow - two years ago, claims agency managing director Paul Smith.
'We are focused on not just the communications but the business objectives within client organisations,' says Smith. '(The lack of focus on business objectives) is not just a weakness among agencies, but clients themselves.
There is a lot more to be done among marketing departments to link communications programmes to business objectives,' Smith argues.
Kaizo is so committed to its evaluation system that Smith says: 'As a client, you cannot work with us unless you interact with Value Flow.'
Kaizo is not alone in offering its own evaluation tool. Firefly offers its clients a four-tier system of evaluation called Fire Proof, each with differing levels of sophistication and cost, from AVEs to reputation analysis and competitor campaign analysis.
Firefly associate director Robin Wilson says 80 per cent of the agency's clients, who include Motorola and Fujitsu Services, have adopted some form of analysis with the agency.
But Wilson adds: 'It tends to be the case that if a client signs up to Fire Proof, they can specify a third-party analyst (as part of the evaluation).
'Ideally we do like to use third parties as this has more value to the marketing department. In fact, it's always best to have third-party analysis to give that objectivity. Interpreting that analysis, however, is best done by the agency and the client together,' he adds.
While a few years ago these initiatives would have been applauded, they now have critics - most notably, and unsurprisingly, the specialist media analysis companies.
Metrica MD Mark Westaby says that despite such strongly held views self-assessment will never work in PR - for either agencies or in-house departments: 'There are a number of problems with agencies doing their own evaluation, the major one being that an agency can never be independent of its work and this inevitably leads to bias - no matter what an agency says, this will always be the case.
'Another is lack of expertise - results from good evaluation should be easy to understand but the system underlying them will be complex. Agencies simply do not have the technical expertise to do the job properly,' he adds.
The growing importance of PR measurement is illustrated by the survey in its finding that 64 per cent of large organisations believe more money should be invested in evaluating PR - on average ten per cent of the PR budget - compared to only 55 per cent who held that view in 2002.
Investment is something with which Smith takes issue: 'Clients say, in surveys, that evaluation is very important to them, but when it comes to talking money they stay away from it,' - (a point borne out by the two in-house PROs who released the percentage of their budget devoted to evaluation to PRWeek - see p27 - and spend less than ten per cent on evaluation).
Evaluation has become a crucial issue to the industry. In the survey, 91 per cent of respondents said better use of planning, research and evaluation is important if PR is to continue to gain boardroom credibility.
Murray agrees, saying Prudential UK's media performance is a standing agenda on the UK board's weekly management meeting. He adds: 'It's up to any client to manage their agency and how the work is evaluated, making sure they are asking the right questions rather than allowing them to set their own agenda for success - which shouldn't be just the volume of coverage.'
Good evaluation, after all, leads to improved performance.
IN-HOUSE PROS: THE IMPORTANCE OF EVALUATION
REBECCA BELL, RAC HEAD OF PR
'We have an evaluation overview on a monthly, quarterly and six-monthly basis, and we have agencies that support different areas of the business, such as legal, roadside and financial services.
'We also have a third-party agency that supports our overall evaluation.
That way we have consistent media evaluation on volume, tone, and key message.
'We were spending more than we are now on evaluation, but getting less value for money. RAC now spends five per cent of the PR budget on evaluation.
'I do think evaluation helps to highlight our work in the PR department and help to move it higher up the corporate agenda.'
ALISON CAMPBELL, SIEBEL SYSTEMS SENIOR DIRECTOR, EMEA PR
'Too often we hear the cry that top management "doesn't understand" PR. So the onus is on us PROs to prove its value and gain their trust.
'A well-presented report based on real, current data and addressing immediate business concerns will usually make the board sit up and take notice.
Evaluation then becomes a valuable tool in the boardroom instead of a budget justification exercise, which is too often the case.
'If the business value of PR is well articulated the budget sign-off - for ongoing campaigns and their evaluation - is much more straightforward.'
PIP WOOD, J SAINSBURY HEAD OF MEDIA
'I'm a firm believer that evaluation has to inform thinking going forward, whether it clarifies that you are doing the right things or provides evidence that change is required. I think it's too easy for evaluation to become "something we do".
'It can make us lazy if we rely on data to do our thinking - but the two work well together.
'You should also challenge its value. We regularly review our requirements to make sure they are relevant and have moved away from trend analysis to more project-based analysis to determine if we have delivered specific business requirements.'
PAUL CHARLES, EUROSTAR COMMS DIRECTOR
'In theory, evaluating PR coverage should be simple. With an agency, you can agree on share of voice, coverage by outlet, key message strike rate and the competitive context. In practice, however, agencies are unable to offer a full evaluation service.
'There is a knowledge gap as so much broadcast PR, a vital part of the campaign mix, is either unmeasurable or not measured properly by research agencies in the UK.
'I invest around two per cent of my budget on evaluation but don't see a need to increase that. I don't think it's acceptable to rely on an agency to evaluate the coverage they are involved in creating. It is not impartial enough. I prefer to use a specialist firm.'