RESEARCH & EVALUATION: Roadtesting the IPR toolkit - Alastair Ray evaluates the industry's response to the IPR's updated toolkit

If the Institute of Public Relations has its way, discussions about

the role of PR will no longer be framed by warm words and woolly


Instead, the new language of the industry will be about graphs, facts

and figures. All of which will help prove the effectiveness of the

industry to clients.

The first update of the IPR's Planning Research and Evaluation Toolkit

promises to help the industry elbow aside competitors from other

marketing disciplines to take its place at the client's top table.

'What it really does is elevate the role of PR. For practitioners who

use the toolkit and carry out proper research and evaluation, it can

demonstrate the strategic importance of PR,' says The Fairchild

Consultancy MD and toolkit author, Mike Fairchild.

IPR chairman Ian Wright says the second toolkit aims to build on the

progress made by its predecessor: 'Practicioners were starting to

evaluate and recognise the importance of measurement, but they were all

doing it in different ways. The toolkit provided a standard approach

that could be used on any campaign.'

And the IPR says interest in the issue has been strong with nearly 1,000

copies already sold, on the back of wide recognition that evaluation is

an important issue for the industry.

Wright adds that despite occasional difficulties with evaluation, the

industry must continue to work hard to improve its position: 'All the

FTSE 100 have PR departments as they see the value of reputation

management, even though it is not straight measuring an intangible.

However, to progress and go further we must be able to measure and

evaluate what we do more thoroughly otherwise we will be seen as the

poor relation in the marketing mix.'

Fairchild says the new edition has been substantially updated from the

April 1999 version, with an increased number of case studies.

Among the changes are: details of how to combine new research techniques

to produce more accurate results; ideas for identifying the separate

effects of advertising and PR; techniques to help analyse post-campaign

research to assess the impact of PR on the people who have been exposed

to it; guidance on how to adapt planning, research and evaluation

methodologies to the internet; suggestions of ways to measure the value

of keeping a client out of the media; and ways of translating PR

effectiveness into a return on investment figure for the client.

Fairchild also argues that the industry needs to move away from using

Advertising Value Equivalents, which he says are an unreliable tool for

measurement. AVEs, he says, compare PR work to advertising when in

reality performance should be measured against the PR objectives that

have been set.

However, some questions are being asked about the new toolkit. Professor

Jon White, who teaches PR at City University, says that the very term

planning, research and evaluation is wrong.

'The PR process involves a number of steps - first of all research, then

planning, then an action phase, and then evaluation,' he says. 'For some

reason the toolkit is taking the whole established process and

suggesting that planning precedes research.'

White also questions who the guide is aimed at. 'The toolkit encourages

people to follow a process and my point would be that process has got to

be ingrained right from the outset,' he says. 'It's not something you

can come to after ten years of practice. It's got to be ingrained.

'An effective PR practitioner should be doing what the toolkit is

telling them already so it wouldn't really help such a practitioner. If

they need it they can't have been very effective.'

He adds that while the toolkit is a useful attempt to capture current

thinking, the industry needs to address the wider issue of training for

people coming into the business: 'The weakness across the board in

evaluation is to do with the way that practitioners are prepared for

work in practice. We are really looking at questions of education,

training, the early work in practice. These questions are not going to

be addressed by a thin document.'


RAC Foundation executive director Edmund King found the toolkit

reassuring as the independent motorists group already uses most of the

suggested techniques, although not necessarily all in one campaign.

'I agree with most of the theory. Where I think it has gaps is when it

comes to the reality on the streets. In any ongoing campaign using wit,

speed and opportunism is often as important as planning, research and

evaluation,' he says.

It's a similar case, he argues, when the toolkit suggests that planning,

research, evaluation 'does not have to be funded out of the PR budget.

As PR reinforces corporate reputation and provides benefits beyond the

remit of PR'. He says: 'In the ideal world it might be funded out of

other budgets but often it is not.'

King applied the toolkit to the RAC's recent 'Don't Dig There' campaign.

The campaign focused on the problem of utility companies digging up the

roads causing excessive congestion and costing the UK economy £2bn

per year. The aim was to get the Government to introduce legislation

that penalised companies if their work was not completed on time. The

RAC Foundation linked up with the Evening Standard to highlight daily

stories of bad practice, briefed leader writers of The Times to write an

editorial and convinced the BBC's Watchdog programme to cover the


'Despite the fact that the RAC Foundation uses ongoing media evaluation,

the use of media in this campaign was not evaluated specifically as the

objective of the campaign - to get legislation introduced - was won,'

says King.

Perhaps the most reassuring thing about using the toolkit was that

looking back on decisions made to short deadlines, the bulk of its

techniques had been followed as a matter of course: 'Sometimes in the

hectic life of a high-profile campaigner instinct has to take over. One

does not always have the time to check out the six steps before reacting

and embarking on a newspaper campaign. But when looking back, we have

normally adhered to the toolkit principles without consciously realising

it.' Overall, he says, the toolkit is a good start but the campaigners

should remember to be on the lookout to grasp 'opportunities with their

bare hands, to sometimes take risks and use creativity to get the story

across. However, as a tool to remind us all of best practice, this

toolkit is as good as they come'.


For Amanda Harrison the reality of life as a sole trader means that

applying the toolkit's suggested research and evaluation techniques to

her work gets short shrift.

'I felt it was unrealistic as a sole trader. If I followed the toolkit

to the letter, I would spend all my time proving my PR rather than

generating PR for my clients,' she says.

She concedes that the techniques will work better at a large agency

where account spends of upwards of £10,000 a month might justify

the time required.

'In a climate where you have an account team of three working on the

whole process, there's the time to apply post-campaign analysis,' she


In fact, she argues, the reason why clients opt for a sole trader is

that they much more personalised service and as a result they judge the

effectiveness of her work very quickly.

Harrison specialises in the marketing sector. Her clients includes

Quantum Media - an online and interactive media agency - London Taxi

Promotions, Continental Research and advertising agency WFCA


One of the challenges of applying evaluation in this kind of work is

that as agencies tend to be promoted as a brand on the basis of the work

that they do. Setting simple objectives when much of the activity

relates to the client's brand becomes 'way, way too complicated'.

'Clients' auditing of my work is down to the coverage I generate as well

as managing the process in a downturn of news,' she says.

Clients of Amanda Harrison Media also receive text messages when big

news stories break such as William Hague's resignation after the

election and the merger of Compaq and Hewlett-Packard.

'It's very difficult to audit that. Have you given me enough news?' she


A former journalist, Harrison has been in PR since 1997, and has worked

at MacLaurin and The Media Foundry. She says that the toolkit will be a

useful reference.

'Just as a way reassuring yourself that you are on the right track in

moments of darkness,' she says.

However, she does have other reservations.

The presence of a major grammatical error in the introduction was

disappointing. 'Full marks for wanting some kind of proof that the

industry is full of capable professionals but after that I took the

whole document less seriously,' she says.

And the reality is that even in larger PR organisations, evaluation is

often the last item on the 'to do' list. 'Auditing, from my experience,

seems to happen when you are trying to retain a client, by which time

you've probably lost them,' she says.


Age Concern campaigns manager Scott Davidson applied the toolkit to the

charity's recent health campaign.

The aim was to highlight the fact that ageism was a real issue in the

health service and that it was endemic rather than being a few isolated


His main impression was that while the toolkit has all the right

process, there is a real question mark for smaller organisations as to

how much time they can spend on research: 'Small organisations have to

think very hard about how they dedicate their research, sometimes you

have got to get out there.'

For the health campaign, Age Concern needed to look into the legislation

and guidelines covering the area, working out who was responsible for

the situation and how it might be changed. 'The key questions are: how

do you get there, does it need legislation or new guidance and who's got

the power to make the change that we want?' he says.

The crux of the issue was that while no-one disagreed with the aim of

providing a non-ageist health service, they needed to be convinced that

it was a real problem.

This was done by providing real people to talk to the media, cases where

medical notes proved that staff had unilaterally taken action based on a

patient's age. The level of coverage and the kind of message being

conveyed was monitored by media evaluation officer Margaret


Davidson says the campaign hit a nerve and rapidly produced the desired

result. Having launched the drive in summer 2000, the Government is now

actively involved in dealing with the issue.

In the spring it announced a national service framework and a full

programme to deal with the problem. Age Concern is now working with the

Government and testing local monitoring to see if the new measures have

made a difference.

He argues that with a primarily public affairs task, assessing the

effectiveness of the work is fairly straightforward: 'With public

affairs, we know what we want and either we have got it or we need to

convince someone else.'

One thing that might improve Age Concern's systems might be to

incorporate third parties such as health professionals to guage how they

have responded to the campaign. Davidson says the toolkit will be most

useful at the certain points of a campaign, such as the development

stage or in a mid-term assessment.

Two things that might have made it better were more detailed case

studies and some kind of guidance as to which of the research techniques

are essential: 'It needs some attempt to say here's the core essential,

where you are not going to have a clue where you are unless you do these

things and this is what you should do if you want to be really



Of PRWeek's four 'guinea pigs', QBO director Jo Carr was probably the

most familiar with the toolkit and some of the thinking from the first

edition is already part of the agency's everyday routine.

'I think that it's good in that it underlines that the whole industry is

taking evaluation really seriously,' she says of the new version. The

real challenge in implementing evaluation is to ensure that there are

targets in the first place. 'The big thing for us is getting clients to

set measurable objectives in the first place,' she says.

QBO, which has a client list that includes Royal Mail, Smile and British

Gas, road-tested the toolkit on a campaign for Norwich Union Healthcare

aimed at first-time dads.

The premise was that having children is often a trigger point to

consider health insurance. It looked to build a relationship with

first-time dads as most information is aimed at mums.

QBO tested the idea with focus groups of recent fathers to check that it

was right in thinking that dads could feel overlooked, there was an

information gap and that Norwich Union could be a credible


This pre-testing allowed QBO to tighten objectives and finalise

messages. At this stage it also sat down with the client to discuss how

the campaign might be measured.

As well as ongoing measurement, the media 'output' was also analysed at

the end of the campaign. A further measuring came from the response to a

hotline number. 'This is a good way of checking "out-take" - that the

audience has received and understood the message,' she says.

However, Carr argues that extending evaluation to the level where you

can demonstrate ROI in many cases depends on the client: 'Evidence is

really hard to get and it's sort of out of our control in many ways.

It's about the marketing team talking to the sales team talking to the

client services team.'

While it's something that happens with Smile - where Carr reports that

PR acquisition costs compare 'incredibly favourably' with other forms of

marketing - 'I think it requires the client to have given it thought

internally as do how you get that information.'

Another issue is the toolkit's attitude to Advertising Value Equivalent

figures. 'The toolkit is very negative about AVE and for some clients

that's what they are familiar with,' she says. 'You need to talk in some

terms that are recognised by clients.'

Have you registered with us yet?

Register now to enjoy more articles and free email bulletins

Already registered?
Sign in