FOCUS: EVALUATION; Measure of results valued like gold

STANDARDISATION: Demand has increased calls for a common standard to measure media effectiveness SYSTEMS: As well as cuttings services, companies are introducing software packages tailored to clients’ needs CASE STUDY: Media analysis helped BT to raise its profile and take on the competition in European markets

STANDARDISATION: Demand has increased calls for a common standard to measure media effectiveness

SYSTEMS: As well as cuttings services, companies are introducing software packages tailored to clients’ needs

CASE STUDY: Media analysis helped BT to raise its profile and take on the competition in European markets

As PR comes under growing pressure to get results, clients are also

demanding objective measurement of these achievements. Danny Rogers


A survey commissioned by The Reputation Managers late last year revealed

that 96 per cent of marketing professionals questioned saw demonstration

of results as important when selecting a consultancy, yet around two

thirds thought PR less accountable than other marketing disciplines.

There does indeed seem to be an industry-wide acceptance that those who

judge activity are demanding increasingly objective and professional

measurement of achievement - not only clients who are evaluating their

consultancies, but marketing personnel justifying PR spend to their own


While consultants and in-house practitioners feel this growing pressure,

they face an ever wider choice of approach. As basic evaluation systems

get cheaper due to vigorous demand, others become more advanced to

satisfy marketing management’s hunger for information.

And yet, as independent evaluation consultant David Hide points out,

‘The ten to 15 media evaluation, or media content analysis companies on

the market have one instantly recognisable thing in common - a

widespread dislike and mistrust of one another’s methodologies.’

PR is often criticised for this lack of standardisation in the way it

measures its efforts and an industry discussion group, chaired by Paul

Georgiou MD of Impacon, has been set up to identify some mutually

acceptable rules.

Quentin Bell is a member of this group and a long-standing supporter of

standardisation. Since 1989 he has been working on introducing a

standard of practice via the PRCA. He remains optimistic. ‘The

evaluation industry is still relatively young and as it matures I’m more

confident that it can agree a basic common standard to measure media

relations effectiveness. This is a first step towards other evaluation

standards,’ he says.

But David Philips, chairman of evaluation company Media Measurement, is

sceptical. ‘Those pushing for a single standard will be disappointed.

The advertising industry is much more advanced in terms of evaluation

and even it cannot agree on standardised values.’

However Hide believes that the difference is not as great as it may

seem: ‘Although each company has its own specialities, most are

basically similar in principle if not methodology.’

He identifies what may be a more illuminating divide: ‘The industry can

be split into two camps on evaluation. The in-house corporates, who

generally embrace media evaluation and take a realistic stance, and the

consultancies who, with some exceptions, tend towards the defensive.

There is a pervasive sense that objective professional evaluation may

show up any shortcomings in the work they undertake for their clients.’

Many leading evaluation specialists including CARMA and Media

Measurement have seen most growth for their services among the large

corporates. But some consultancies feel evaluation must be closer

aligned to clients’ specific needs. Tim Sutton, chief executive of

Charles Barker says: ‘What we should be measuring is our ability to

achieve what I term the basic ‘deliverables’ - essential performance

that the client expects. One can have a very impressive evaluation

formula, but is it focused on what the client actually wants for his

money? This can vary enormously.’

While this debate will run and run, there are those who believe that PR

evaluation is at a critical point in its evolution. In the introduction

to his recently published book Measuring Your Media Profile, Dermot

McKeone, deputy chairman of Infopress, says ‘Media analysis could...go

the way of the skateboard and the hula hoop and be consigned to oblivion

in a year or two, or it could lose a little of its gloss and evolve into

an essential working tool.’

Keone is highly critical of what he sees as the continuing ‘AVE

backwater’ - the measurement of editorial coverage in terms of

equivalent advertising costs. He says: ‘An article inspired by PR input

is no more a relevant comparison to an advertisement than a concert

grand piano is a pianola. It takes the board-level purse-string holder

about five seconds to see through the spuriousness of AVE data,’

Yet it could be a question of first educating the client. Hide points

out that some of the less expensive evaluation systems on the market are

based on AVEs but are often demanded by clients. ‘We can be self-

critical to a point but clients may have a lack of understanding about

how media evaluation works.’

Metrica has just launched its Cigma Direct service which is based on a

low monthly retainer fee and a fixed charge per cutting. This bureau

service analyses coverage according to branding achieved, favourability

gained and messages delivered. Metrica research analyst Paul Hender

says: ‘We try to avoid using AVEs but can provide this service if

clients want some form of financial evaluation.’

At the other end of the spectrum are sophisticated ‘content analysis’

systems like Media Measurement’s Strategic Media Monitor. These tend to

cost several thousand pounds per annum but give more qualitative

analysis of activity, providing an insight into media/opinion former

perceptions and even what factors influence these views.

Similar systems offered by CARMA, Mediatrack and Impacon highlight the

use of evaluation as an issues management tool. They can show who is

driving media issues, journalists’ standpoints, competitor’s profiles

and exactly where prevailing attitudes need to be addressed.

‘It is here that I have most faith in objective evaluation systems,’

says Sutton. ‘Regular tracking of campaigns and their tangible effect on

key audiences enables effective strategic planning.’

Larger agencies will often use content analysis on an ad-hoc basis. Big

campaigns mean big budgets justifying big analysis, and the consultancy

can bring in the specialists on a project basis - particularly where the

cost can be passed directly on to the client. Otherwise these agencies

will typically use in-house evaluation systems backed by occasional

market research for client reassurance.

Management consultancy KPMG’s in-house department combines internal

analysis with external perceptions measurement. Over the past year it

has carefully focused on three central messages that mould its PR

efforts - leadership in the industry, incorporation (from private

company to plc) and transparency through the first ever publication of

its annual results.

KPMG director of corporate affairs Tim Roberts has closely evaluated the

results: ‘Apart from internal analysis and the use of specialist

evaluation companies to enhance this, we also commissioned market

research agencies to gauge the effects on our audience of business


‘The data showed that we’re the only firm in the ‘big six’ to achieve

more positive than negative comment, that we’re identified with

incorporation by 64 per cent of those questioned - compared with ten per

cent or less among our competitors and that our senior partner is the

best known in the industry.’

PR evaluation continues to be characterised by diversity of approach but

this may be healthy in an industry which includes diverse activities

under its umbrella. It could be argued that PR has capabilities for

changing audience awareness that advertising and other marketing

disciplines do not. Therefore measurement will always suffer more


‘If PR evaluation were easy it would have been cracked long ago. An

activity like internal communications will require very different

measurement techniques to straightforward media relations’ says Bell.

‘But there’s still a need to agree some basic principles. If we can’t I

genuinely fear for the future of the industry. However as PR matures,

our measurement is rightly becoming more sophisticated and we’re moving


Low cost options: Software packages for cheap and cheerful evaluation

Those concerned about the complexity or cost of the ‘big name’

evaluation systems might consider some of the more modest solutions on

the market, which are proving particularly popular with smaller

consultancies and in-house departments in the public sector.

Companies like Metrica and Media Works offer cost-effective bureau

services where cuttings can be sent off for analysis, while The Cutting

Edge sells off-the-shelf evaluation software that can now be installed

on any PC for as little as pounds 400.

This system uses the sometimes contentious AVE measurement criteria, but

now has approaching 100 customers.. ‘We use AVEs because they are easy

to understand and they represent the influence of editorial in a

particular title. AVEs can also be weighted according to messages in

order to achieve a helpful gauge of coverage,’ says Cutting Edge MD Carl


The corporate communications department of the Civil Aviation Authority,

main provider of air traffic control services in the UK, uses The

Cutting Edge to improve internal reporting of the impact of media

coverage. The CAA uses the system to evaluate its coverage in six key

areas including National Air Traffic Services (eg. airmiss incidents)

and Air Traffic Operations (eg. delays).

The system shows which areas or issues have received significantly more

coverage than others and therefore provides a performance indicator.

Each cutting is also awarded a helpfulness rating based on the extent to

which it helps or hinders the CAA’s corporate objectives.

Over 1,500 cuttings have now been logged in and management can spot

check at any time to make sure it agrees with the analysis of a

particular cutting. Each quarter, a print out is pulled from the system,

backed by a report compiled by the in-house team. Press officer Jonathan

Nicholson says: ‘We wanted something that would give us quantified

reports which were easy to understand. We were keen to use advertising

values as we felt these were easier to understand than other measures.

‘The Cutting Edge has been able to provide use with the figures to back

up what we have been saying. It enables management to easily quantify

and understand the impact of our media coverage,’ he says.

More established players in the evaluation market can be dismissive of

such solutions, criticising them for their over simplified measurement

criteria. And while using a bought-in software package may be highly

convenient for an agency or department, it could be argued that the

total objectivity of outsourced reports is lost, reducing credibility.

Nevertheless these simpler systems continue to have a market. By

removing some of the financial obstacles they may also offer the user a

first step into the daunting world of evaluation techniques.

Case study: Getting BT recognised by European business

By the early nineties BT had firmly established Maureen Lipmann’s

‘Beattie’ in everyday British culture and was looking to expand into

lucrative European markets.

But, according to research by Harris in 1991, ‘spontaneous awareness’ of

BT stood at a level of a mere one in ten of BT’s target market within

the European business community. This was well behind foreign

competitors and would prove an obstacle to BT’s appearance on the

shortlist of major companies outsourcing their telecoms.

The research showed that although dominant, BT’s competitors were

vulnerable. Many had competitive weaknesses which were a legacy of

monopoly status. Shortcomings mentioned included ‘bureaucratic’, ‘dated’

and ‘offering poor service’.

BT’s relative anonymity in this market and the scale of the competition

presented it with an opportunity to position itself as the ‘challenger

brand’. This would enable BT to outflank some competitors on their

weaknesses and address customer expectations.

BT’s PR agency Countrywide studied the research, identified its

audiences and developed its communications messages accordingly.

Planning director Andrew Jones explained: ‘BT accepted that business and

communication objectives were tightly interlinked. An ambition to grow

European business couldn’t be achieved without the ability of

communications to help win access to new markets and build the company’s


From the final quarter of 1991 Countrywide ran a communications

programme across eight European markets. This included basic media

relations work like a contacts programme and facilities visits,

alongside more creative means of news generation such as market research

surveys and conferences for opinion formers.

To steer the programme in the right direction, Countrywide began using

CARMA in 1992 to analyse all coverage being generated. Through CARMA it

monitored quantity and quality of message delivery, media segmentation

and its ‘friends and foes’.

Countrywide could then target specific publications or journalists,

correct misinformation or devise creative means of changing perceptions.

While CARMAevaluated the messages sent and how successful the campaign

was in wooing the media, Countrywide also employed market research

agency MORI to gauge reception among target audiences.

‘From a combination of research we could continually tailor the

programme. We knew which messages were relevant and important in which

countries, and the transmission media that were effective in

communicating them,’ says account director Emma Brazier.

Through decoding pan-European signals Countrywide built a database of

communications knowledge. It learned how different company claims can

motivate audiences in different ways and how messages can be fine tuned

over time to achieve the desired effect.

Research conducted by MORI in late 1995 showed that in terms of

familiarity and favourability, BT had leapt ahead of many of its

competitors. It had also achieved parity with one major competitor which

enjoyed double the recognition and much stronger image values than when

the original benchmark was undertaken. This competitor had invested tens

of millions of dollars on pan-European advertising each year and BT had

not. And spontaneous awareness among European businessmen had moved from

a position four years ago where many thought BT stood for Bankers’

Trust, to quality awareness in a large majority of the audience.

Internet: The art of virtual evaluation

With thousands of users linking up to the Internet each month, the

information revolution may have a dramatic influence on the way PR

evaluation evolves. Not only does the Net provide a whole new medium for

information exchange between an organisation and its audiences, but also

a new means of communicating with journalists.

Worldwide content analysis firm CARMA International has until recently

focused on traditional media outlets such as newspapers, magazines and

the electronic media, but now recognises that the Internet is an

increasingly important medium for reaching mass consumers.

‘We’ve already run studies on particular issues for clients in the UK

pharmaceutical and environmental sectors and we expect this to be an

expanding area of our work,’ says European MD Sandra Macleod. ‘Usage of

the Internet in Europe is still relatively small compared to the US and

Canada however this will increase dramatically over the next year. In

these countries Internet analysis is growing fast.’

Media Measurement chairman David Philips although excited by the new

media, is quick to point out that it’s early days and he’s some way from

marketing an Internet evaluation service. ‘I like to think our

evaluation principles are internationally transferable and certainly

apply to the Internet. Internet forums are a growing means for

information exchange and the rules for analysis remain largely the same

as for traditional media.’

To show how organisations can assess product or issue debate on the

Internet, CARMA recently carried out a prototype study. From September

to November last year it monitored the launch of the Microsoft product

Windows 95.

CARMA looked at postings on three ‘newsgroups’ for comments on Windows

95, where it knew people would be discussing the new system. It found

that immediately after the launch there was a lot of activity and quite

a bit of criticism from users of competitive systems. By October this

had died down and the debate became more balanced as people started to

use the product. By November there was considerably more positive

comment. Overall Windows 95 received a slightly unfavourable rating.

CARMA compares such coverage to traditional ‘letters to the editor’ or

magazine opinion columns and assigns favourability ratings in a similar

way. It is also possible to identify important issues - in this case

‘functionality’ of the software and ‘ease of use’. However the problem

with Internet evaluation is the medium’s inherent lack of structure and

it will take time for any company to develop a methodology to tackle the


While some concentrate on the role of the Internet as opinion former,

others are beginning to exploit different potential. In March,

Benetton’s PR company Modus Publicity held one of world’s first Virtual

Press Launches, marking the launch of its own Web Site. PR agencies

worldwide spread the word and 9,138 journalists signed in, providing the

company with an invaluable record of press interest and contacts.

Have you registered with us yet?

Register now to enjoy more articles and free email bulletins

Already registered?
Sign in