Research & evaluation: The Toolkit Test - PROs put the IPR's media evaluation guide through its paces

As the column inches mount up for your client, you congratulate yourself for executing a brilliant PR campaign. Indeed, obtaining substantial media coverage remains the most visible initial output for demonstrating the immediate value of PR.

Nevertheless, merely obtaining coverage is not necessarily enough to fulfil client objectives. In addition to meeting sales, lobbying or brand awareness expectations, evaluating the kind of coverage received is equally essential. But to give clients a better understanding of PR's effectiveness, wouldn't it be more useful for the industry to have standard evaluation criteria for measuring campaigns? Otherwise, what one PRO claims is a worthwhile assessment of a campaign may not necessarily generate the same kind of results as another.

Following a survey of PR practitioners in the UK late last year, the IPR discovered that while the industry recognises the strength of proper media evaluation, there is still some confusion in the terminology used.

And the body's assistant director and head of marketing and PR Ann Mealor argues that the necessity to establish a standard evaluation process is growing, as clients want to see a clearer return on investment.

'From consultancies' perspectives, the need for a standard evaluation system is really hitting home, as in-house PR teams are becoming increasingly aware of the importance of media evaluation, and will be demanding more from their external agencies to justify expenditure,' she says.

To tackle this demand, the IPR launched the third edition of its media evaluation toolkit in July. 'We want to establish a guide for the industry that is standardised in terms of terminology and application,' adds Mealor.

'We want to establish this five-step process for practitioners, and encourage evaluation at all levels, to reinforce a standard across the industry.'

Choosing to focus on media evaluation as the first key step of PR measurement, the toolkit is designed as a practical guide to what media evaluation is, how it fits into the wider context of planning, research and evaluation (PRE), and how to do it. Along with members of the Association of Media Evaluation Companies (AMEC), media measurement companies and PR practitioners, the IPR has produced step-by-step sections. These include: why measure media publicity; media evaluation and the PRE process; a guide to media evaluation; and Q&As.

The PRE process covers five stages for PROs to follow: audit - where are we now?; setting objectives - where do we need to be?; strategy and plan - how do we get there?; ongoing measurement - are we getting there?; and results and evaluation - how did we do?

Yet, the question arises whether the toolkit actually tells experienced PROs anything new and innovative about how to handle media evaluation.

PRWeek therefore asked two consultancies and one in-house PR department to cherry-pick stages from the toolkit - applying the entire process to a campaign would take months - and apply them to a recent campaign they had conducted. If the team already had an evaluation method in place, it was asked to compare the IPR process to its in-house system. Would the toolkit reveal any methods of evaluation that the teams weren't already using?

Standardising evaluation methods remains a thorny issue across the industry, mainly due to disagreement on how it could be effectively implemented.

Haygarth PR director Kate Smith points out that a standardised evaluation document across the industry would make it easier to benchmark the success of different PR campaigns. But conversely, due to the varied nature of PR, she considers it unrealistic to think an evaluation tool that provides a 'one-size-fits-all' solution can be created.

In-house PR teams feel the same way. BT Major Business head of PR Ellen Ferrara thinks the idea is too simplistic. 'Yes, we need ways to measure what we do. But I don't see how we can come up with a single standard when you look at all of the different areas the term PR covers,' she says.

Mundipharma European director of PR Rob Cohen agrees. He argues that the complexity of the healthcare arena advocates against standardisation.

'But having said that, just because different sectors require different results, that doesn't diminish the need for a set formula to achieve ratings for specific elements of a campaign,' he adds.

Evaluating media coverage is not the first step the PR industry should be looking at, argues Kaizo director Crispin Manners. While he agrees the IPR's toolkits go some way towards establishing a standard across the indu-stry, he would like to see further clarity.

'Rather than focusing too much attention on calculating the value of the final media coverage, we should be looking at others elements of PR, such as ensuring campaigns achieve a change in attitudes and desires from the target audience,' he says. 'The IPR toolkit is a step forward, but I would prefer to see more towards helping PR professionals work harder in the planning stages of a campaign.'

If the trend of tighter budgets and growing client demand for a clearer ROI continues, so will the need for evaluation that goes beyond counting column inches. The question remains, however, that if an industry standard for media evaluation is formally established, will it be implemented by agencies with their own methods already in place? Nevertheless, as the road tests conducted for PRWeek confirm, a standard evaluation process should complement any existing method.

MENCAP

Mencap head of PR Mary Sweeting and senior PR officer Naomi Creeger decided to apply the IPR toolkit to help with the early planning stages of one of the charity's campaigns.

'One of our constant and most important challenges is how we can raise awareness and improve understanding of learning disabilities through the media, in order to generate support for our cause and change attitudes,' Sweeting says.

'Faced with such a vast task, and limited resources, we thought the IPR toolkit might enable us to focus on the essential aspects of a campaign that would make a real difference to our work.

'Having worked previously in PR agencies, I was familiar with the value of a strong planning and measurement cycle, and also the discipline of having a clear, defined brief.

'Naomi and I have worked in various positions in-house and agreed that because internally you can be pulled in so many different directions, it can occasionally be difficult to remain objective and see what the clear results from a campaign will be. Therefore, something that makes you really focus, such as this toolkit, has got to be beneficial.

'Stage one of the toolkit's PRE process - the 'where are we now?' section - would have had more relevance if we were giving a new brief to an external agency that did not know a great deal about our organisation.

'However, we still found it useful to work through each section of this stage and review our previous successes.

'One thing it did was make us realise we have more work to do to really define which of the many potential audiences we should be focusing our efforts on.

'It's always useful to do a SWOT analysis as suggested in the toolkit - that is, looking at the strengths, weaknesses, opportunities and threats of a campaign - to get you thinking around the problem and the potential up and downsides. And for a new recruit to the team, it will give a good idea of our starting point - the fundamental element of any good campaign.

'We haven't yet worked through stage two - setting our objectives - for the upcoming campaign, but again, having read the IPR's toolkit, it all made sense, and I've used similar systems based on audience/ message/desired responses on pre-vious campaigns.

'To make it more helpful to others, particularly if you're in-house and only have small budgets, the IPR could have given more direction on accessing Target Group Index (TGI) and National Readership Surveys, as mentioned in the 'setting measurable objectives' stage, and approximate costs or ways to reduce them. The toolkit did, however, give comprehensive media evaluation advice.

'The IPR's toolkit is not rocket science or even particularly revelatory, but it is a sound framework for running a PR programme. It can sometimes be easy to get carried away with a great idea or repeat things you believe are working well without really questioning why you're doing it, or whether things have changed.

'This toolkit should help make sure creativity is grounded in the right strategy. For that reason, we would recommend it to other in-house or consultancy practitioners, unless you already have similar disciplines in place. We will certainly carry on using most of it.'

BRANDS2LIFE

Brands2Life worked on a campaign for software company Kalido to promote the news of the firm's funding by Atlas Ventures and Benchmark Capital and the appointment of a new CEO, as well as the company's new corporate strategy.

Client services director Peter Jacob applied the IPR toolkit to the 28 pieces of media coverage achieved so far for the completed campaign, including articles in The Daily Telegraph, the Financial Times, The Times, the Wall Street Journal Online and Computing.

'Brands2Life had not previously used an IPR toolkit to evaluate PR campaigns. We have tended to use a range of evaluation techniques depending on budget, geographic focus and audience reach,' says Jacob.

'However, the toolkit enables you to make an objective assessment of a campaign's success or failure. It's certainly a useful reference point and embodies the core elements every campaign should include.

'Applying the principles to the campaign's results has helped in generating ideas for future PR campaigns for Kalido and other clients - for example, deciding on the appropriate type and level of research to use in developing the initial brief, researching target audience profiles and establishing media opinions and their specific requirements.

'We applied stages two and five to the recent campaign for Kalido - setting objectives, and results and evaluation. We did this by applying the principles outlined in each stage of the toolkit to both the initial campaign brief and our analysis of the results.

'It helped us establish and define which tactics worked most effectively across different types of media. It confirmed our initial plan was accurate and included key considerations, such as audiences, medium, messages and timing.

'The IPR toolkit can be used in a number of different ways depending on the PRO's experience. On a general level, it helps explain the process of evaluating a campaign, with some very useful case studies of what might work; the merits of internal versus external measurement techniques; definitions of commonly used PR terms; and what evaluation can do beyond a standard press cuttings analysis.

'The majority of points highlighted in the toolkit should be obvious to experienced PROs, but its value should be in acting as a reference to ensure every aspect of a campaign is considered properly. It is the perfect complement to measure against existing techniques and therefore is useful to even the most experienced PRO.'

COUNTRYWIDE PORTER NOVELLI

Countrywide Porter Novelli ran a summer campaign for Heinz Baked Beans to demonstrate the versatility of the product and other canned foods, giving consumers reasons to use them in different and more inventive ways.

The client had already decided that an on-pack offer for a recipe book should form part of the campaign. CPN was asked to propose a theme for the recipe book and develop a PR programme to run alongside it. The agency already had its own planning and evaluation system in place for this campaign. Assistant director of planning Adam Mack points out the differences and similarities between CPN's Compass process and the IPR toolkit.

'My first observation is that CPN's planning and evaluation system involves ten steps, as opposed to the IPR toolkit's five. Yet, one area common to both is the objective setting stage. Putting the PR brief in the context of the client's business and marketing objectives, so that we continually concentrate on what PR activities contribute to the business, is absolutely crucial to our approach. Although the toolkit incorporates this as part of the process, it does not highlight it as a key step, whereas I believe it should be.

'With specific regards to the Heinz campaign, we initially clarified the business objectives, as specified in the first step in the toolkit's second stage. It was clear that Heinz's commercial objective was fairly straightforward: to increase volume of sales for beans over the summer.

'As part of our Compass process, which is also highlighted as the third step of the PRE process in the tool-kit, we then audited the audience with a view to understanding how PR could best help generate more sales.

Our principal insight was that mums are looking for easy ways to use everyday products to bring variety and fun to mealtimes.

'We also looked at media channels, as specified in the fourth step of the PRE process, deciding that in order to shift behaviour sufficiently to increase sales we needed to penetrate the mainstream home news media.

We then backed this up with audience research to ascertain the key media needed to reach our target of BC1C2 mums aged 25 to 44.

'From following these three tool-kit-related steps, we were then able to set objectives around the campaign: percentage reach of BC1C2 mums aged 25 to 44, strength of voice in target media, percentage penetration of key PR collateral, such as messages, visuals and brand check.

'I would say the importance of the IPR's toolkit is more a result of its process than its individual parts. An experienced PRO would automatically carry out many of the different steps - audit phase, audience definition - but may miss out others.

'While I would say that, although the IPR toolkit doesn't teach us anything more about evaluation and planning than we already know, it will provide agencies that don't already have a system in place with the ability to structure their PR programmes more effectively and ensure it doesn't miss out any of the crucial steps.

'However, while it is a very comprehensive approach, it may benefit from a simpler, bite-size overview. A more succinct visual summary might make the kit more user-friendly.'

Have you registered with us yet?

Register now to enjoy more articles and free email bulletins

Register
Already registered?
Sign in