Evaluation: Are in-house PROs made to measure?

In-house comms teams are increasingly turning to evaluation - but for different reasons. Robert Gray reports.

While the luddites who believe PR cannot be measured have dwindled to the minority, agency evaluation still inspires suspicion, while the appraisal of in-house departments is even less understood.

This month's CIPR research into the size of the industry shows that 77 per cent of PROs work in-house - a statistic that begs important questions around evaluation: are organisations as critical of their own PR performance as they are of the agencies they use? Should they work internally by the same rules they set external PROs? Does evaluation only happen when CEOs and FDs are pressuring PROs to justify their contribution, or does it occur as a matter of course?

According to a survey by Echo Research, which asked in-house communications directors how they maximised performance, 69 per cent said they had used evaluation tools to measure the impact of their work. 'We have seen many comms teams secure greater responsibilities and budgets because of the research and evaluation they have done,' says Echo Research CEO Sandra Macleod. 'It has opened up their management's eyes, and encouraged a far greater dialogue about strategy and direction.'

She adds that while in-house teams are tending to improve their status within companies, 'advertising's star has fallen. Getting a bigger slice of the cake was always likely to (happen for PR), but evidence shows that this has been turbo-charged in many cases.'

Hewlett-Packard is one company that takes in-house evaluation just as seriously as the appraisal of external agencies. It has even published a report, 'Measuring Media Relations', in which it identifies 148 ways in which PR can be analysed.

Equal Opportunities Commission media director Claire Hibbitt says: 'I think it's important to distinguish between tactical evaluation - such as researching how spokespeople are reported in relation to an issue - and strategic evaluation. The questions to ask are: How far is the media coverage helping you as an organisation?

How much has it generated response from stakeholders? How much has it helped your lobbyists?'

Company-wide performance

The question of whether internal evaluation is a result of pressure from bean-counting FDs, or the instigation of in-house PROs who appreciate the benefits of measurement, is difficult to answer.

Delahaye director of business development Paul Thomas says that in his experience, FDs increasingly want PR measurement to be tied in with department or company performance assessments. This includes everything from simple cost-versus-performance analysis, to measuring the effect of a campaign on sales. There is also demand for PR analysis to be linked with performance data from other marketing channels.

But Tom Vesey, managing director of global media analysis company Carma International, discerns little influence from finance directors in the push for in-house evaluation. He says FDs are often 'counter-productive as they generally flourish advertising value equivalents (AVEs) as their measuring tool'. Metrica MD Richard Bagnall agrees that FDs are behind the rise of AVEs as a means of justifying PR expenditure. He attributes this to finance departments finding it easier to understand measures with a pound sign in front of them.

Evaluation does not always have an impact on departmental budgets. At present, it is mainly used to assess the success of comms activity and inform future strategy. But some companies are linking it more tightly into budget-setting - and some on-the-ball PR departments are commissioning research as a means of securing additional resources.

Such was the case with The Walt Disney Company, where EMEA vice-president of corporate comms Joyce Lorigan gave the go-ahead for pan-European research into media satisfaction with her team. The findings helped Lorigan argue successfully for the creation of two PR posts in Paris and Munich.

In the UK, Disney generates more than 6,000 press cuttings a month and uses Precis to monitor and evaluate them. Lorigan believes in-house PROs are beginning to catch up with their marketing counterparts who have long condoned research and evaluation.

'It's been interesting to see which of our business units are driving branded coverage in the press,' she says. 'We look at the percentage share between the units to see which are generating the most coverage around specific messages. Some units are better at delivering messages about innovation, others about family-branded fun.'

Financial Services Authority head of media and web comms Rob McIvor says that at his organisation, evaluation is not used as a way of maintaining or increasing budget. He adds: 'Getting our communications right is certainly high on the agenda and senior management will have their own impression of how well we're doing. What formal evaluation provides is the hard data to support or challenge those impressions. It's all too easy for someone to see one piece of poor coverage of a subject and assume that all the other coverage is equally bad. Objective evaluation can kill misconceptions.'

In the US, a study sponsored by the Council of Public Relations Firms and conducted by the USC Annenberg Strategic Public Relations Center asked in-house PROs to describe the extent to which they used various measures. The top five measures were 'influence on corporate reputation'; 'content analysis of clips'; 'influence on employee attitudes'; 'influence on corporate culture'; and 'influence on stakeholder awareness'.

And there is evidence that in-house evaluation is on the rise internationally.

In September 2004, Benchpoint canvassed the views of PROs, directors and CEOs from 25 countries online. Sixty-nine per cent of PROs claimed to measure their work - with 77 per cent of those who did not saying they planned to do so in the future. Interestingly, while most PROs concurred that measurement was an important part of the PR process, the majority of board directors and CEOs agreed. This supports the view that the trend for greater in-house evaluation is being led from the top.

The international picture though is a patchy one. For example, a survey of New Zealand PR practitioners - conducted for the Public Relations Institute of New Zealand by Shattock Communications & Research - found the use of measurement varying from non-existent to continuous.

Methodology minefield

Of course, in-house practitioners will shy away from evaluation if they view the process as overly complicated - but technological advances and new methodologies are emerging all the time. These in themselves pose challenges, with in-house teams having to keep abreast of what measurement options are available on the market.

Earlier this year, consultant Biz360 added Nielsen NetRatings data to its corporate reputation measurement system, allowing clients - including AstraZeneca and Harley Davidson - to view the effects of their comms efforts online. But elsewhere there is evidence that companies are unsatisfied with media monitoring services.

The 2004 Worldwide Market Survey of Media Monitoring - conducted by the Slipstream Group for CyberAlert - found widespread dissatisfaction with news monitoring suppliers. Fewer than 20 per cent of in-house PROs were 'very satisfied', while 40 per cent expected to change their media monitoring supplier within a year. And while in-house PROs revealed that they spent the bulk of their media monitoring budgets on news evaluation, they were even less satisfied with suppliers than their agency counterparts.

Simply becoming more aware of what is out there could be the obvious solution, although busy PROs may not have time to assess these options.

Tools for change

To address this problem, software vendor Vocus, which moved into the UK market two years ago to help organisations measure the effectiveness of their comms, offers a 'desktop analytics module'. Managing director Andrew Muir claims the program removes a 'huge tier of manual reporting' - the practice PROs bemoaned the most in the Worldwide Marketing Survey.

TNS Media Intelligence deputy managing director Dean Wading argues that such evaluation tools can only help in-house PROs understand the direct effect of their work - and aid them in their quest for greater recognition from the FD and CEO.

'Most board members would agree that PR is hard to justify, therefore more effort is needed to adopt methods that track the effectiveness of a campaign in the market and the effect it had on brand sales, share price or sales leads,' he explains.

'One method historically owned by marketing but increasingly used by PR teams is market research,' Wading adds. 'It is the only way to track PR activity back to the target market to see if it has influenced customers.

'The CEO and FD may not understand AVE or "opportunities to see" figures, but showing how much money a campaign has made for shareholders is very easy to understand.'

The case for evaluation is compelling, but judging what to evaluate and how to do so, while securing the money for effective measurement tools, are major barriers - for in-house teams as well as agencies.


'The key is to alter public opinion - the hardest thing to measure,' says Toyota (GB) general manager of press and public affairs Scott Brownlee.

'We don't do monthly opinion polls, but we do try and influence journalists.'

The car manufacturer measures its press coverage using Precis, specifically looking at the slant journalists give to Toyota vehicles. It also uses PressWatch studies of business pages to gauge how the company is perceived more broadly. A third strand to its evaluation is an in-depth annual MORI poll based on hour-long interviews with 70 leading automotive journalists, which allows the manufacturer to compare its reputation and performance against those of its major rivals.

'We feed our monthly evaluation into broader management. It gives them a sense of where we are in the world, which makes PR results more real for them and gives them more confidence,' explains Brownlee.

Toyota has used analysis of competitor coverage to shape its own actions.

For instance, where early interview access to a rival model's designer has secured good coverage, Brownlee has pressed for similar assistance from Toyota's Japanese headquarters. 'Things have become more open,' he says. 'With each launch we compare coverage with that of our rivals.' Brownlee is part of a team of eight, and there is a separate PR team at Toyota's manufacturing facility. Brownlee says 80 to 90 per cent of the evaluation material he receives is useful for providing an insight into performance and shaping PR strategy.


Building society Nationwide does not retain external PR agencies, with the exception of public affairs firm AS Biss, and has a 28-strong in-house comms team, half of which is devoted to PR with the remainder working in areas such as community affairs and sustainability.

Evaluation firm Impact provides analysis of news cuttings and rates the team's performance against the delivery of messages and how the company is regarded. 'We've had ten years plus of using Impact,' says head of external affairs Alan Oliver. 'I'm able to track performance over time, which is useful. We also ask Impact to track our performance against several competitors - a mixture of building societies and ex-building societies that have become banks. We look at it closely as a team and tailor our practices accordingly.'

Performance findings are shown to directors and other senior managers.

Oliver says evaluation 'has a bearing' on the activities of the comms team and the way budget is allocated, but does not believe it 'materially affects' the overall size of the budget allocated for PR.

In a spirit of openness, Nationwide's in-house magazine Live carries features on the organisation's media performance.

Nationwide subscribes to a PressWatch financial survey as a benchmark against rivals and uses MORI's annual survey of personal finance journalists, which asks participants to rate brands and products and comment on companies' public profile and PR abilities.'That's very helpful,' says Oliver. 'If the picture is negative, you can take steps to address the problem. Evaluation is a vital part of managing and improving our performance over time.'

Meanwhile, tools such as the Business in the Community Index and services from the London Benchmarking Group are used to assess CSR and community relations performance.


Texas-based PR agency Idea Grove recently published a guide to evaluating the performance of external comms agencies. Addressing the questions (below) could also help in-house teams to develop best practice:

- Does the agency ask you for ideas more often than it provides you with ideas?

- Do you spot factual inaccuracies in agency-drafted news releases?

- Do agency-drafted releases miss the point - often burying important information?

- Does the agency seem to churn out releases, not offering other, more creative ways to gain attention?

- Has a journalist ever complained to you about your PR agency?

- Does the agency boast about delivering measurable results, but then only give you a stack of news clippings that mean nothing to your company's executives?

Visit www.ideagrove.com to see the full list


Severn Trent Water, whose PR is managed in-house, analyses its print and internet coverage via agency and in-house monitoring. It also analyses broadcast coverage to establish a return on investment figure around campaigns.

'We have a headline media-favourability index, which is the main element reviewed by my boss, the director of customer relations, and is reported to the executive director on a monthly basis,' says external communications manager Steve Hodgson. 'We do periodic breakdowns of that analysis dependent on issues and subjects. We might break down negative results to review policy issues, or positive results to assess value for money/ROI.'

Some analysis is included in the assessment of customer service problems, acting as an indicator of where problems lie. Reputation is a significant issue for the management team, and media coverage - especially as it informs the opinions of a range of stakeholders - is a significant component of that. With regards to this, analysis is also used as a method of target-setting for the media team.

'Evaluation is not strictly tied into the budget-setting process, but we do use it as a method of demonstrating the value of our communications, and can set that against other approaches, so in that sense it helps reinforce the budgeting process,' says Hodgson.

Evaluation is carried out by Echo Research, which has worked with Severn Trent Water for almost ten years. Hodgson says this longevity has generated a large amount of data for comparing current with projected performance and predicting the likely impact of recurring issues.

'We also carry out customer research and are looking at a more broadly based view of where our stakeholders are and what their expectations of us is,' says Hodgson. 'We have aligned some of the evaluation with our customer-tracking work, but it is difficult to establish direct correlations.'

He adds: 'We're working to use the evaluation to benchmark the success of our major activity against other water companies at a national level.

We look at their outputs and how our reputation stacks up against theirs.'


Children's charity NSPCC uses four kinds of evaluation: media evaluation through Metrica; public opinion tracking with HPI; comparative research from NFP Synergy; and internal communication research through in-house surveys.

'We do try very hard to pursue an integrated approach to communications and therefore try and follow an integrated approach to evaluation,' says comms director John Grounds. 'Often the one will fill in the gaps of the other.'

Getting the right messages to the right audiences is a vital part of the charity's fundraising activities. To ensure this is happening, Grounds favours an in-depth level of media evaluation. NSPCC looks at overall volume of coverage, favourability and audience reach.

It also breaks down coverage into five regions, in keeping with the organisation's structure, and assesses media by sector as part of its comms planning. 'We judge whether we need to hit audience X by media sector Y,' Grounds explains.

Bimonthly reports are produced for ongoing tracking, and specific campaign coverage is isolated to look at how a campaign per se is performing and its impact on NSPCC's long-term objectives. There is also research into which publications and journalists are covering NSPCC in a positive way.

The NFP research tracks the performance of a range of charities against several measures, allowing NSPCC to establish a benchmark for its performance in the highly competitive sector.

NSPCC's performance is reported to the leadership group, of which Grounds is a member, the charity's trustees and its volunteer bodies.

Grounds says evaluation has a bearing on the amount of money made available to his team, and is convinced it has had the benefit of making NSPCC much more audience focused.

Have you registered with us yet?

Register now to enjoy more articles and free email bulletins

Already registered?
Sign in

Recommended for you

Recommended for you

Explore further