STANDARDISATION: Demand has increased calls for a common standard to measure media effectiveness
SYSTEMS: As well as cuttings services, companies are introducing software packages tailored to clients’ needs
CASE STUDY: Media analysis helped BT to raise its profile and take on the competition in European markets
As PR comes under growing pressure to get results, clients are also
demanding objective measurement of these achievements. Danny Rogers
A survey commissioned by The Reputation Managers late last year revealed
that 96 per cent of marketing professionals questioned saw demonstration
of results as important when selecting a consultancy, yet around two
thirds thought PR less accountable than other marketing disciplines.
There does indeed seem to be an industry-wide acceptance that those who
judge activity are demanding increasingly objective and professional
measurement of achievement - not only clients who are evaluating their
consultancies, but marketing personnel justifying PR spend to their own
While consultants and in-house practitioners feel this growing pressure,
they face an ever wider choice of approach. As basic evaluation systems
get cheaper due to vigorous demand, others become more advanced to
satisfy marketing management’s hunger for information.
And yet, as independent evaluation consultant David Hide points out,
‘The ten to 15 media evaluation, or media content analysis companies on
the market have one instantly recognisable thing in common - a
widespread dislike and mistrust of one another’s methodologies.’
PR is often criticised for this lack of standardisation in the way it
measures its efforts and an industry discussion group, chaired by Paul
Georgiou MD of Impacon, has been set up to identify some mutually
Quentin Bell is a member of this group and a long-standing supporter of
standardisation. Since 1989 he has been working on introducing a
standard of practice via the PRCA. He remains optimistic. ‘The
evaluation industry is still relatively young and as it matures I’m more
confident that it can agree a basic common standard to measure media
relations effectiveness. This is a first step towards other evaluation
standards,’ he says.
But David Philips, chairman of evaluation company Media Measurement, is
sceptical. ‘Those pushing for a single standard will be disappointed.
The advertising industry is much more advanced in terms of evaluation
and even it cannot agree on standardised values.’
However Hide believes that the difference is not as great as it may
seem: ‘Although each company has its own specialities, most are
basically similar in principle if not methodology.’
He identifies what may be a more illuminating divide: ‘The industry can
be split into two camps on evaluation. The in-house corporates, who
generally embrace media evaluation and take a realistic stance, and the
consultancies who, with some exceptions, tend towards the defensive.
There is a pervasive sense that objective professional evaluation may
show up any shortcomings in the work they undertake for their clients.’
Many leading evaluation specialists including CARMA and Media
Measurement have seen most growth for their services among the large
corporates. But some consultancies feel evaluation must be closer
aligned to clients’ specific needs. Tim Sutton, chief executive of
Charles Barker says: ‘What we should be measuring is our ability to
achieve what I term the basic ‘deliverables’ - essential performance
that the client expects. One can have a very impressive evaluation
formula, but is it focused on what the client actually wants for his
money? This can vary enormously.’
While this debate will run and run, there are those who believe that PR
evaluation is at a critical point in its evolution. In the introduction
to his recently published book Measuring Your Media Profile, Dermot
McKeone, deputy chairman of Infopress, says ‘Media analysis could...go
the way of the skateboard and the hula hoop and be consigned to oblivion
in a year or two, or it could lose a little of its gloss and evolve into
an essential working tool.’
Keone is highly critical of what he sees as the continuing ‘AVE
backwater’ - the measurement of editorial coverage in terms of
equivalent advertising costs. He says: ‘An article inspired by PR input
is no more a relevant comparison to an advertisement than a concert
grand piano is a pianola. It takes the board-level purse-string holder
about five seconds to see through the spuriousness of AVE data,’
Yet it could be a question of first educating the client. Hide points
out that some of the less expensive evaluation systems on the market are
based on AVEs but are often demanded by clients. ‘We can be self-
critical to a point but clients may have a lack of understanding about
how media evaluation works.’
Metrica has just launched its Cigma Direct service which is based on a
low monthly retainer fee and a fixed charge per cutting. This bureau
service analyses coverage according to branding achieved, favourability
gained and messages delivered. Metrica research analyst Paul Hender
says: ‘We try to avoid using AVEs but can provide this service if
clients want some form of financial evaluation.’
At the other end of the spectrum are sophisticated ‘content analysis’
systems like Media Measurement’s Strategic Media Monitor. These tend to
cost several thousand pounds per annum but give more qualitative
analysis of activity, providing an insight into media/opinion former
perceptions and even what factors influence these views.
Similar systems offered by CARMA, Mediatrack and Impacon highlight the
use of evaluation as an issues management tool. They can show who is
driving media issues, journalists’ standpoints, competitor’s profiles
and exactly where prevailing attitudes need to be addressed.
‘It is here that I have most faith in objective evaluation systems,’
says Sutton. ‘Regular tracking of campaigns and their tangible effect on
key audiences enables effective strategic planning.’
Larger agencies will often use content analysis on an ad-hoc basis. Big
campaigns mean big budgets justifying big analysis, and the consultancy
can bring in the specialists on a project basis - particularly where the
cost can be passed directly on to the client. Otherwise these agencies
will typically use in-house evaluation systems backed by occasional
market research for client reassurance.
Management consultancy KPMG’s in-house department combines internal
analysis with external perceptions measurement. Over the past year it
has carefully focused on three central messages that mould its PR
efforts - leadership in the industry, incorporation (from private
company to plc) and transparency through the first ever publication of
its annual results.
KPMG director of corporate affairs Tim Roberts has closely evaluated the
results: ‘Apart from internal analysis and the use of specialist
evaluation companies to enhance this, we also commissioned market
research agencies to gauge the effects on our audience of business
‘The data showed that we’re the only firm in the ‘big six’ to achieve
more positive than negative comment, that we’re identified with
incorporation by 64 per cent of those questioned - compared with ten per
cent or less among our competitors and that our senior partner is the
best known in the industry.’
PR evaluation continues to be characterised by diversity of approach but
this may be healthy in an industry which includes diverse activities
under its umbrella. It could be argued that PR has capabilities for
changing audience awareness that advertising and other marketing
disciplines do not. Therefore measurement will always suffer more
‘If PR evaluation were easy it would have been cracked long ago. An
activity like internal communications will require very different
measurement techniques to straightforward media relations’ says Bell.
‘But there’s still a need to agree some basic principles. If we can’t I
genuinely fear for the future of the industry. However as PR matures,
our measurement is rightly becoming more sophisticated and we’re moving
Low cost options: Software packages for cheap and cheerful evaluation
Those concerned about the complexity or cost of the ‘big name’
evaluation systems might consider some of the more modest solutions on
the market, which are proving particularly popular with smaller
consultancies and in-house departments in the public sector.
Companies like Metrica and Media Works offer cost-effective bureau
services where cuttings can be sent off for analysis, while The Cutting
Edge sells off-the-shelf evaluation software that can now be installed
on any PC for as little as pounds 400.
This system uses the sometimes contentious AVE measurement criteria, but
now has approaching 100 customers.. ‘We use AVEs because they are easy
to understand and they represent the influence of editorial in a
particular title. AVEs can also be weighted according to messages in
order to achieve a helpful gauge of coverage,’ says Cutting Edge MD Carl
The corporate communications department of the Civil Aviation Authority,
main provider of air traffic control services in the UK, uses The
Cutting Edge to improve internal reporting of the impact of media
coverage. The CAA uses the system to evaluate its coverage in six key
areas including National Air Traffic Services (eg. airmiss incidents)
and Air Traffic Operations (eg. delays).
The system shows which areas or issues have received significantly more
coverage than others and therefore provides a performance indicator.
Each cutting is also awarded a helpfulness rating based on the extent to
which it helps or hinders the CAA’s corporate objectives.
Over 1,500 cuttings have now been logged in and management can spot
check at any time to make sure it agrees with the analysis of a
particular cutting. Each quarter, a print out is pulled from the system,
backed by a report compiled by the in-house team. Press officer Jonathan
Nicholson says: ‘We wanted something that would give us quantified
reports which were easy to understand. We were keen to use advertising
values as we felt these were easier to understand than other measures.
‘The Cutting Edge has been able to provide use with the figures to back
up what we have been saying. It enables management to easily quantify
and understand the impact of our media coverage,’ he says.
More established players in the evaluation market can be dismissive of
such solutions, criticising them for their over simplified measurement
criteria. And while using a bought-in software package may be highly
convenient for an agency or department, it could be argued that the
total objectivity of outsourced reports is lost, reducing credibility.
Nevertheless these simpler systems continue to have a market. By
removing some of the financial obstacles they may also offer the user a
first step into the daunting world of evaluation techniques.
Case study: Getting BT recognised by European business
By the early nineties BT had firmly established Maureen Lipmann’s
‘Beattie’ in everyday British culture and was looking to expand into
lucrative European markets.
But, according to research by Harris in 1991, ‘spontaneous awareness’ of
BT stood at a level of a mere one in ten of BT’s target market within
the European business community. This was well behind foreign
competitors and would prove an obstacle to BT’s appearance on the
shortlist of major companies outsourcing their telecoms.
The research showed that although dominant, BT’s competitors were
vulnerable. Many had competitive weaknesses which were a legacy of
monopoly status. Shortcomings mentioned included ‘bureaucratic’, ‘dated’
and ‘offering poor service’.
BT’s relative anonymity in this market and the scale of the competition
presented it with an opportunity to position itself as the ‘challenger
brand’. This would enable BT to outflank some competitors on their
weaknesses and address customer expectations.
BT’s PR agency Countrywide studied the research, identified its
audiences and developed its communications messages accordingly.
Planning director Andrew Jones explained: ‘BT accepted that business and
communication objectives were tightly interlinked. An ambition to grow
European business couldn’t be achieved without the ability of
communications to help win access to new markets and build the company’s
From the final quarter of 1991 Countrywide ran a communications
programme across eight European markets. This included basic media
relations work like a contacts programme and facilities visits,
alongside more creative means of news generation such as market research
surveys and conferences for opinion formers.
To steer the programme in the right direction, Countrywide began using
CARMA in 1992 to analyse all coverage being generated. Through CARMA it
monitored quantity and quality of message delivery, media segmentation
and its ‘friends and foes’.
Countrywide could then target specific publications or journalists,
correct misinformation or devise creative means of changing perceptions.
While CARMAevaluated the messages sent and how successful the campaign
was in wooing the media, Countrywide also employed market research
agency MORI to gauge reception among target audiences.
‘From a combination of research we could continually tailor the
programme. We knew which messages were relevant and important in which
countries, and the transmission media that were effective in
communicating them,’ says account director Emma Brazier.
Through decoding pan-European signals Countrywide built a database of
communications knowledge. It learned how different company claims can
motivate audiences in different ways and how messages can be fine tuned
over time to achieve the desired effect.
Research conducted by MORI in late 1995 showed that in terms of
familiarity and favourability, BT had leapt ahead of many of its
competitors. It had also achieved parity with one major competitor which
enjoyed double the recognition and much stronger image values than when
the original benchmark was undertaken. This competitor had invested tens
of millions of dollars on pan-European advertising each year and BT had
not. And spontaneous awareness among European businessmen had moved from
a position four years ago where many thought BT stood for Bankers’
Trust, to quality awareness in a large majority of the audience.
Internet: The art of virtual evaluation
With thousands of users linking up to the Internet each month, the
information revolution may have a dramatic influence on the way PR
evaluation evolves. Not only does the Net provide a whole new medium for
information exchange between an organisation and its audiences, but also
a new means of communicating with journalists.
Worldwide content analysis firm CARMA International has until recently
focused on traditional media outlets such as newspapers, magazines and
the electronic media, but now recognises that the Internet is an
increasingly important medium for reaching mass consumers.
‘We’ve already run studies on particular issues for clients in the UK
pharmaceutical and environmental sectors and we expect this to be an
expanding area of our work,’ says European MD Sandra Macleod. ‘Usage of
the Internet in Europe is still relatively small compared to the US and
Canada however this will increase dramatically over the next year. In
these countries Internet analysis is growing fast.’
Media Measurement chairman David Philips although excited by the new
media, is quick to point out that it’s early days and he’s some way from
marketing an Internet evaluation service. ‘I like to think our
evaluation principles are internationally transferable and certainly
apply to the Internet. Internet forums are a growing means for
information exchange and the rules for analysis remain largely the same
as for traditional media.’
To show how organisations can assess product or issue debate on the
Internet, CARMA recently carried out a prototype study. From September
to November last year it monitored the launch of the Microsoft product
CARMA looked at postings on three ‘newsgroups’ for comments on Windows
95, where it knew people would be discussing the new system. It found
that immediately after the launch there was a lot of activity and quite
a bit of criticism from users of competitive systems. By October this
had died down and the debate became more balanced as people started to
use the product. By November there was considerably more positive
comment. Overall Windows 95 received a slightly unfavourable rating.
CARMA compares such coverage to traditional ‘letters to the editor’ or
magazine opinion columns and assigns favourability ratings in a similar
way. It is also possible to identify important issues - in this case
‘functionality’ of the software and ‘ease of use’. However the problem
with Internet evaluation is the medium’s inherent lack of structure and
it will take time for any company to develop a methodology to tackle the
While some concentrate on the role of the Internet as opinion former,
others are beginning to exploit different potential. In March,
Benetton’s PR company Modus Publicity held one of world’s first Virtual
Press Launches, marking the launch of its own Web Site. PR agencies
worldwide spread the word and 9,138 journalists signed in, providing the
company with an invaluable record of press interest and contacts.