'Interminable meetings talking about measurement just make me want to scream,' admits Pirate Communications managing director David Pincott.
'Agreeing on all sorts of finely honed factors around measurement wastes time that could have been spent doing the job well. I've never left meetings like these where a consensus was reached.'
Pincott is not the only PR practitioner to take a sceptical view of the possibilities of evaluating PR. Indeed, while other marketing communications disciplines have long-established best practice and industry standards, the PR industry has long suffered from lip service to evaluation.
Last month there was a signal that things might finally be changing.
The PRCA joined forces with the IPA, Incorporated Society of British Advertisers and the Marketing Communications Consultants Association to produce Evaluation: a best practice guide to evaluating the effects of your campaigns.
The authors spoke to 65 clients and 109 marketing services agencies. It confirms that 60 per cent of agencies still do not use econometrics, fewer than 20 per cent of agencies consistently evaluate their contribution to the profitability of their client, and only 36 per cent have had training on the subject. It suggests more arguments for more widespread evaluation, but the question is whether the guide will offer anything new to persuade PROs to adopt a more constructive attitude to measuring their craft.
Paul Miller, an independent consultant who led the PRCA's contribution to the guide, says the time has come to force evaluation on to PROs if they are to survive budgetary threats: 'Budgets are being scrutinised.
All too often, where cuts are made they're made to PR spend, and in many cases this is because PR is less able than other forms of marketing comms to evaluate its work and so justify the investment. This can, and must, change.'
If this is not warning enough, a far more threatening shot across the bows is contained within the pages of the guide - which is that clients are getting fed up with PR's apparent lack of commitment to measurement. Eighty-four per cent of clients said evaluation was now as important to them as creative product. In addition - something that should wake PROs up - 90 per cent of clients believe that evaluation data should be used to help determine agencies' remuneration.
But herein lies the problem. Adopt evaluation and it is a further step down the road to adopting what many do not want to accept - payment by results (PBR). Like Pincott, many PR agencies steadfastly resist the notion of PBR. Fuse PR communications managing director Blaise Hammond says: 'A few years ago there were a number of start-up agencies adopting very aggressive PBR models to win new business. They appealed to some clients, but you won't find too many of them around now. Not enough thought was put into the longer-term effects of adopting a near-telesales approach to media relations.'
There are compelling reasons why PR practitioners should evaluate. Some see it as a learning tool. BMW corporate communications manager Mark Harrison says: 'I do it solely for our own benefit and not to justify our budgets to the board. I don't think it's appropriate for every campaign, but with some press launches I will closely measure which messages were picked up by the media and then conveyed by them to consumers. If we weren't getting our messages across, I'd look into why and see what we could improve next time round.'
Mostly, though, evaluation is used to prove the benefits of PR and to strengthen the case for future expenditure. Director of research at Porter Novelli UK Mary Baker says: 'We always insist that we set clear and measurable targets for client programmes, and that we put in place the mechanisms for evaluating activity before starting work. Most clients want to see their campaigns evaluated properly, as they often need to produce return on investment figures that justify their PR spend to internal audiences.'
If this is the case, should evaluation really be such commercial suicide?
And more importantly, could there be a benefit in promoting PBR to evaluation-hungry clients?
Some agencies, it seems, are beginning to introduce an element of PBR into their client contracts, with agency costs covered by a basic fee, and bonuses paid for achieving pre-agreed objectives. Eclat Marketing customer services director Dianne Canham believes PBR is good for relationships with clients: 'Eighty per cent of our hi-tech and telecoms clients operate on a PBR basis. It means that we are sharing the investment with them.'
Given the potential benefits of evaluation and PBR, it seems strange that few agencies are set up like this. Some, such as MD of Elemental PR, Tim Gibbon, believe there is little excuse for this: 'A lot of consultancies and agencies use the measurement and evaluation confusion as a cushion, because it provides a comfort zone if something doesn't go quite to plan. A lot of professionals have been hiding behind the mystique and the "value of PR" for too long.'
Others, though, argue there are genuine and significant obstacles in the path of PR evaluation. Most attempts fall at the first hurdle - deciding what to measure in the first place. For some, PR is only effective if it has increased sales or positively affected some other business measure. But in almost every case it is impossible to isolate the impact of PR from other factors, such as other promotional activity, product quality and competitor activity.
Berkeley PR managing director Melanie Kanarek points out: 'It is rare for an individual or a business to make a purchase based on editorial coverage alone, whether we consider a bottle of shampoo, a digital camera or a software package. The purchaser will be influenced by advertising and discounts, as well as by editorial coverage.'
Paying a fee to an agency based on the amount of coverage it produces looks like an easy solution, but it fails to consider the quality of that coverage. But even solutions that assess the content of media coverage remain open to the accusation that they are measuring output rather than outcome. Some in the industry believe that the only way to find out whether or not PR has been effective is to ask the intended audience whether their attitudes have changed.
Independent PR consultant Michael Bland warns: 'What people say they think is often very different from what they actually think, and in many cases bears no resemblance to what they'll do. If you ask me, the best way to tell if a PR campaign is working is to ask an experienced practitioner for his or her gut feeling. That will be much more use than all the management data in the world.'
Lack of rigour
Gut feeling is hardly the sort of rigorous methodology the PRCA had in mind with its evaluation guide. It is certainly not a method that many clients would be prepared to accept.
But it does raise the question: how do you evaluate meeting a journalist who might not write about you for months? Or one who keeps your company or personal name out of a negative story because you have taken the trouble to meet up and introduce your firm?
Faced with these difficulties, and in the absence of clear industry standards on evaluation, the number of agencies have begun to develop their own evaluation methodology.
Lewis Communications marketing manager David Cunningham describes a new system he is rolling out: 'It combines a measure of the agency's activities with qualitative and quantitative evaluation of the coverage we've achieved. Each clipping will have a score that indicates the value it has from the client's marketing point of view. In brief, this is calculated by multiplying the publication's tier with the type of coverage, and then adding points if key messages were conveyed.'
Whiteoaks, a technology specialist with 70 staff and offices across Europe, calls its evaluation system Wordsworth. Client services director James Kelliher says he starts with agreeing deliverables. 'These may be case studies, interviews, or quality and quantity of coverage. We then evaluate each piece of coverage on size, positioning, photography, readability, quality of publication and whether or not it conveyed our messages. Finally, we operate a message tracker. This can show clients, on a monthly basis, where and when their messages have been conveyed.'
These two examples are fairly typical of the approach PR agencies are developing to evaluation. In fact, some in the industry do not accept the generally held view that PR agencies avoid evaluation. Director and head of technology at GCI UK, Caroline Randle, says: 'Our own research into this area has indicated that almost 90 per cent of companies are evaluating their PR campaigns. Evaluation has become the norm rather than the exception. We evaluate all our client campaigns. It's part of the way we work. We can never expect to get proper credit for what we do in public relations until we can find an effective way to measure it.'
Even clients say one thing and do another, says Christian Kollmann, MD of Austrian agency Communications Matters. He says: 'We go to many conferences where corporate PR people say they're all for evaluation and are just on the lookout for a proven methodology. When we tell them we have a proven methodology they can use for just EUR1,000 a month, they aren't interested. They tell us they'd have to cut a press conference to pay for the evaluation, and they're not prepared to do it.'
The PRCA might have chosen an opportune time to release its guide.
The signs are that the industry is becoming convinced of the potential benefits. The pressing need now is for a universally accepted evaluation methodology.
Counting the cost
The main stumbling block looks likely to be who pays for evaluation, but to an extent that could be mitigated by linking evaluation to PBR.
Few clients would refuse to pay for an industry-standard evaluation if it was tied to reduced payments for under-performance. Of course, those clients might soon discover that their agencies are performing so well that they could end up paying them more.
More importantly, if the PR industry can agree on a best practice evaluation methodology, it could drive up standards and drive out those low-quality practitioners who use the current confusion surrounding evaluation to get away with sub-standard work.
At the very least, it would also release Pincott and others from those interminable meetings about evaluation and allow them to get on with the real work of PR.
36% of agencies have had formal evaluation training 90% of clients believe evaluation should be used to determine agency pay 60% of agencies still do not use econometrics 20% of agencies consistently measure their contribution to client profits
'Around one fifth of our clients have some element of payment by results in their agreement with us,' says Hilary Meacham, MD of Focus PR. 'Eighty per cent of the fee will be a retainer and cover our costs. We will get paid the remaining 20 per cent if we hit our targets.'
But she still has mixed feelings about these arrangements: 'While they can be great for motivating us and making us want to prove we can hit the targets, I've found that they can cause problems. Often, our target is to get the client a feature in a publication it's not yet cracked.
We do it and send our bill with the feature attached, but the client still argues about paying for it. Basically, they never think we'll manage it and so budget based on 80 per cent of the fee, not 100 per cent.'
She is a firm believer in evaluating Focus's work, and does so by working out, for each pound spent on PR, how much of the target audience has heard the message or sampled the product. Meacham argues that brand tracking would be a better way to evaluate campaigns, but says no client has yet been willing to pay for it.
She does not yet believe that payment by results will replace the traditional method of payment by retainer. 'Basically, you need to hire good people, and you can't do that if you are only getting paid by results. You need to have the security that a retainer brings. Clients need to take a leap of faith when they hire an agency and work together so that both companies grow together.'
CASE STUDY: GCI/ELECTROLUX
Electrolux's Trilobite is the world's first robotic vacuum cleaner.
It uses bat-like acoustic radar to clean the floor all on its own. Its name was inspired by the robot's resemblance to a prehistoric creature, an extinct marine arthropod that crawled around on the seabed, sucking up small animals and plankton for food.
It costs £999 and is targeted at a small group of 'affluentials', households with incomes of more than £60,000, a group that is notoriously hard to influence. In Spring 2003, GCI was hired to generate media coverage that would convince this target market of the worth of the product and so drive sales. The budget was £30,000 and the only other marketing support was in-store demonstrations.
Evaluation played a major role in this campaign, not least because GCI had agreed to an element of payment by results. Eighty per cent of the project fee covered the work undertaken by GCI to support the launch.
This comprised preparation of materials, press targeting and organising, and attending a press event. The other 20 per cent was tied to the success of the campaign in increasing audience awareness and supporting sales of the Trilobite.
A consumer media relations campaign used teaser postcards and a media launch event to reinforce the brand proposition and message. To create a buzz in the media outside product review pages, GCI ran a product loan programme among radio and TV presenters. To reach affluentials directly, it set up product demonstrations at the upmarket gym chain Next Generation and at Jake, a gay and lesbian networking group in New York and London.
The evaluation, from a third-party research company, was comprehensive.
The PR campaign generated 116 million opportunities to see from more than 100 pieces of media coverage. Twenty-five national newspaper articles were secured and 93 per cent of coverage was in the target media. Ninety-two per cent of consumer coverage contained the key message that Electrolux offers intelligent solutions that make life easier.
The story also featured on Ananova and Sky News websites. Capital FM ran the piece as 'the story of the day' in the Drive Time show, while BBC Radio 2 covered the story in its breakfast show. BBC Breakfast presenters interviewed Electrolux Floorcare's UK managing director, and other coverage was secured on Talk Sport radio and ITV's This Morning.
Trade media articles highlighted Electrolux as a category leader in this new sector. The coverage from the electrical retailing titles conveyed all brand messages, and GCI reports that sales targets have been met.