FOCUS: EVALUATION - Realising value in a global standard/An international task force is busy working towards common standards and language for the evaluation industry worldwide. Kate Nicholas reports on the latest developments

The last quarter of 1996 proved something of a watershed for the evaluation industry. After 30 years of discussion on both sides of the Atlantic about the potential, or even the advisability, of establishing some kind of minimum standard for PR evaluation, the countdown finally began towards an international consensus on industry standards.

The last quarter of 1996 proved something of a watershed for the

evaluation industry. After 30 years of discussion on both sides of the

Atlantic about the potential, or even the advisability, of establishing

some kind of minimum standard for PR evaluation, the countdown finally

began towards an international consensus on industry standards.

The stop watch was set at the ICO/GPRA-hosted international workshop on

evaluation on 22 to 24 November 1996 in Frankfurt, which brought

together for the first time evaluators and PR practitioners from the UK,

United States, France, Finland, Denmark, Italy and Germany. An

international task force is now working on the development of minimum

standards and a common language for the international evaluation

community, to be presented as a White Paper at an ICO/IPRA meeting at

the time of the PR World Congress in Helsinki in June.

The international evaluators are also now pooling their resources to

publish a definitive guide - for clients and practitioners - to media

evaluation and a client’s guide to setting objectives. This includes a

standard consultancy briefing form to help practitioners develop proper

briefs with their clients and to encourage them to ’set reasonable


While undoubtedly the most significant industry move to date, the

Frankfurt conference is by no means an isolated phenomenon. In the

United States, the Institute for Public Relations Research and

Education, Inside PR editor Paul Holmes and the Research Department of

Ketchum Public Relations Worldwide, recently came together with two

dozen leading US PR practitioners, counsellors, researchers and

academics at a ’PR Evaluation Summit Meeting’ in an attempt to resolve

the long running debate on the issue of measurement standards.

A task force, co-chaired by Walter K Lindenmann, senior vice

president/director of research Ketchum Public Relations Worldwide, and

Katherine Delahaye Paine, chief executive of The Delahaye Group, is

currently working on production of a reference guidebook to acceptable

standards and criteria for PR measurement encompassing PR output (media

analysis), outcomes (awareness and retention and behavioural changes). A

second task force is looking at the possibility of collecting and

electronically disseminating information on PR to be used as a field

tool for establishing fundamental criteria and standards. And a third

task force is examining the role of ’relationships’ in PR and assessing

the roles played by leading practitioners.

Within the UK the media evaluation and analysis industry has also

launched its own trade body called the Association of Media Evaluation

Companies (AMEC) which is currently working on a guidance paper on media

evaluation - including a general introduction to the concepts of media

analysis and a detailed glossary of terms - which will form the first

stage of the White Paper due to be presented in Helsinki.

The move to create an industry association follows the recent formation

of an informal cross-trade body made up of media evaluators, PR

consultants and clients. The group was set up to develop a common unit

of measurement to be known as the Public Relations Point (PRP) -

designed to dovetail with existing Gross Rating Points (GRP) and

Television Rating Points (TVRs).

The proposed PRP aims to provide a means of measuring a campaign’s reach

to a demographically defined target audience, and would be calculated

from a central database based on existing readership research.

However, the proposal of the PRP has prompted heated debate. According

to research undertaken last year by evaluation specialist Metrica, two

thirds of PRCA members supported the call for a media evaluation


But many evaluators and PR practitioners say that the proposal flies in

the face of other international developments and argue its lack of

content analysis does little to advance the cause of proper evaluation

of public relations activities.

’It is too simplistic and the issues are too complex,’ says Lindenmann

who, like many, believes that the international focus should be firmly

fixed on minimum standards rather than common measurement.

’When media measurement first started we were counting clips and column

inches but the business has changed and what is important now is the

clients’ key messages and themes. The attempt to narrow it down to a

magical score is rather naive.’

But why after 30 years of debate have international evaluators finally

been spurred into action? According to Delahaye Paine, much of the

impetus has come from clients.

’In the past it was research people pushing for (some kind of common

language), but the number of clients using measurement techniques has

risen dramatically,’ says Delahaye Paine, one of the major players

contributing to the initiative.

’In the 1990s recession everyone looked for measurement,


When the purchasing department gets involved they want to compare apples

to apples. Present them with impressions versus opportunities to see and

they say ’Can’t you make things simpler for us?’’

The other key motivation is the increasingly international nature of the

PR business, and the need for multinational clients to take the same

measurement programme and implement it globally.

The cultural and geographical hurdles this involves are likely to

present a significant challenge to the international task force. As

Mediatrack managing director Nick Grant points out media evaluation is

inevitably affected by the structure of the communications media in the

different markets and PR practices inevitably differ radically from

country to country.

’To set a minimum standard you are talking about different


In France, for example, public relations is mainly about media

relations, and in the US you have a very different level of focus to PR

activities,’ says Sandra Macleod managing director (Europe) Carma


Inevitably approaches to evaluation also vary. ’What concerns me is that

in Europe there is an equation of media analysis with evaluation, we

need to recognise that it is broader than that,’ says Lindenmann. And as

Dermot McKeone managing director of media analysis company Infopress

Communications, points out: ’A lot of people in continental Europe, in

particular Germany, are interested in input evaluation, analysing the

quality of what is going into the PR process at the beginning of the

cycle. But when you come to the UK and the States more emphasis is put

on outputs and outcomes.’

Such cultural differences combined with the plethora of methodologies

sold by competitive companies lead all too often to complete

incomprehension on the part of the client. Recent research undertaken by

Carma International into ’Worldwide Evaluation - Progress so Far’

flagged up lack of knowledge as a major factor in the reluctance to

commit budget to evaluation. The survey of clients’ attitudes towards

evaluation, ranging from multinationals to international NGO’s in US,

South Africa and Europe, showed that there is universally more talk than

action. While Europe may lead the field in terms of commitment to

carrying out evaluation, there is still an alarming disparity between

the 86 per cent of companies across Europe who believe that evaluation

is necessary and the 21 per cent who commit themselves in practice.

Another major problem is that evaluation tends to be seen as an expense

rather than an investment, a prejudice only likely to be overcome by a

greater concentration on planning in PR. ICO President Peter Hehir

believes that the focus on adequate briefings and objectives by the

international task force will help to sell the concept of evaluation to


’I have long believed that if we square the circle between what clients

really require from us by getting the objectives right in real

communications terms and being properly paid for what we do against

these objectives, the whole industry can grow up and take its place at

the board room table.

I believe that what has now come out of Frankfurt will lead us in that



’The UK is much further ahead than the US in terms of media evaluation,

but here we are talking more about strands of PR research as a much

broader topic,’ says Katherine Delahaye Paine, chief executive of The

Delahaye Group. Much of the emphasis of US evaluation and research

companies are on ’outcomes’ as well as ’outputs’. At the same time, many

of the leading PR practices have developed highly sophisticated in-house

systems reaching far beyond the confines of simple media analysis.

Ketchum for example, has employed for some years a broad ranging

proprietary strategic planning tool dubbed the ’Ketchum Effectiveness

Yardstick (KEY)’ The system was put in place by Walter Lindenmann, one

of the leading lights behind the recent movement towards minimum

standards of evaluation, and founder of Ketchum Worldwide’s PR Research

and Measurement Department. The department has a staff of six and

handles 80 to 95 full scale research projects per annum, 15 to 20 per

cent of which are undertaken for

non-Ketchum clients.

The department also handles multinational projects acting as a

counsellor on locally handled research.

In addition to basic measurement of PR outputs through content analysis,

publicity tracking studies, secondary analysis, segmentation analysis

and simple opinion polls, KEY also uses focus groups, depth interviews,

surveys, testing techniques and recall studies to measure PR outgrowth -

whether target audience groups actually received the messages directed

at them. At an advanced level KEY uses everything from experimental

research techniques to ethnographic studies and communication audits to

measure changes in attitude and or behaviour.

At the same time, US evaluation and research companies such as Carma and

Delahaye, lead the field in terms of ’Cyber Image Analysis’ - measuring

the impact of new media on corporate image. Delahaye/New Media tracks

conversations about client companies in key on-line forums and Internet

news groups, analysing the subject and tone of conversations,

identifying issues that need attention and recommending actionable

strategies. The company provides a tracking survey of Web site visitors,

identifying the number of people who visit the site, how they navigate

the site and how long they stay, as well as how the site impacted

visitors’ perceptions of companies and products.

The company has also developed one of the first services to help clients

measure the effectiveness of marketing on the Internet by providing a

demographic profile of site visitors. Measurement of Visitor Profile

(MVP) takes basic information gathered from Web site registration

fields, analyses it and provides a detailed profile of Web visits

including age, household composition, geographic location, lifestyle and

hobbies and interests of visitor base.


Another initiative is underway in the UK, fuelled by the extensive

research and evaluation work being undertaken by academic


On 9 December, a group of around 40 academics, PR practitioners and

media evaluators came together for the first LSE Forum on Communications

Research in London, to pool experiences and look at specific case

studies in a bid to promote a more rigorous stance on evaluation.

’Initially we aim to establish an agenda for joint research projects

including the vexed issue of evaluation,’ says George Gaskell director

of the London School of Economics and Political Science Methodology


’Such an agenda is likely to involve questioning some of the core

assumptions guiding current research in PR. Media evaluation is based on

some presumptions about relations between media coverage and public

opinion which researchers rejected years ago.’

Probably the greatest challenge for the forum will be to find a way to

bridge the gap between academic thinking and the more results-driven

approach of practitioners. ’The remit of the forum is not to have the

academics in one corner but to get them closer involved in industry and

vice versa,’ says Sandra Macleod managing director (Europe) Carma

International. ’We want to promote a broader debate and understanding of

how to use research, to push research further and make it work harder

for people.’

The evaluation industry has already taken a quasi-academic approach to

the issue .Scott Cutlip, Allan Center and Glenn Broom’s PII model

(Preparation, Implementation, Impact) featured in their book Effective

Public Relations has become something of a standard on US bookshelves.

The PII model looks at the adequacy of background information for

designing a programme, through to resulting social and cultural


An Australian practitioner Jim Macnamara has also proposed a model

called MACRO which looks at inputs, outputs and results against

activities and methodologies. But Tom Watson managing director of

Hallmark PR - who has developed two of his own models designed to take

into account the fluctuating nature of PR work - says: ’The problem with

existing models is that they are static. They don’t have the dynamic

element which reflects the real ongoing nature of PR practice - and PR

practitioners simply don’t use them.

’You can’t stop and say we will take a few weeks to review this, you

have to review as you go along. PR people aren’t natural scientists who

set up an experiment and then examine the results.

’The academic approach can draw concepts from other areas and well made

theory also makes things more predictable. A lot of people are half way

there, they just need to go a step further to make sure information is



Germany leads the field in terms of spending on evaluation and breadth

of practice, followed by the UK, Scandinavia and the South West

according to a recent survey of 107 European practitioners undertaken by

the German trade body Gesellschaft Public Relations Agenturen


The survey revealed a broader understanding of evaluation in Germany

with 86 per cent of respondents defining evaluation in its widest sense

(as opposed to 40 per cent in the UK) and only ten per cent of German

respondents confining their understanding of evaluation to

communications effect (as opposed to 48 per cent in the UK).

While in UK consultancies were more likely to use an outside evaluation

company, the German market has some highly developed in-house


Scheben Scheurer and Partner, for example, is just one of the agencies

to have developed a computer-based system for the analysis (as opposed

to simply measurement) of ’media resonance’. Its ’ComPlusControl.’

software evaluates each clipping according to reach, favourablity,

layout and position in publication as well as reflection of key messages

in the media and enables agencies to analyse the direct result of public

relations activity by measuring output of media activity against media

coverage during a certain period - independent of PR input.

The evaluation debate has been running for some time in Germany, the

GPRA being the prime mover behind the recent ICO/GPRA conference in


For over two years now the GPRA has had its guidelines on media

resonance and analysis on bookshelves of PR practitioners, and members

of the GPRA are all required to be accredited to ISO 9000, encompassing

a commitment to evaluation.

’Germany and the UK emerged as the most professional countries in Europe

with regard to evaluation,’ says GPRA vice-president Hannemie Stitz, who

is responsible for the trade body’s international affairs. ’We had a

very poor response from south west Europe - the Netherlands, Belgium,

France, Spain, Portugal and Switzerland. The further south you go the

less it has become a focus.’

Kirsten Berth managing director of Alpha Beta in Denmark, also recently

undertook a survey of evaluation in Denmark, Finland, Norway and


According to Berth, Sweden leads the field in terms of evaluation. The

Association of Public Relations Consultancies in Sweden (PRECIS)

recommends that all members commit to evaluation and has recently

published ’Return on Communications’ a booklet available in English

which attempts to define the financial value of research, taking case

studies from companies carried out over the last 18 months.

The survey showed that broadly in Scandinavia, while evaluation is

appreciated, not only by consultants but also by clients, it is seen as

situational, task specific, not methodological, unsystematic and

infrequent. Likewise clients appreciate the importance of evaluation but

do not want to pay for the service and feel it is the consultancy’s

responsibility. In general, clients tend to prefer easily understood

quantitative evaluation, and while agencies would prefer to use both

methods, qualitative methodologies are still underdeveloped.

According to Berth, evaluation is still undertaken mainly on an internal

basis using response cards, questionnaires, regular evaluation meetings

with clients and the occasional marketing tool.

Have you registered with us yet?

Register now to enjoy more articles and free email bulletins

Already registered?
Sign in