Steve - thanks for your comments. I thought some background might be useful. As you know, we launched this survey three years ago, but today's research bears only some resemblance to its first iteration. Over the years, we have sought the input of agency, in-house, and academic leaders.
In the beginning, we hoped to rank all agencies by reputation among their clients. So we conducted a survey asking clients which firms they worked for and how they rated their performance on a number of attributes. We did not have a set number of agencies - we just invited clients to rate any agency they worked for.
The problem that we ran into was in the sample sizes. The largest agencies had adequate (if not entirely convincing) sample sizes, while the smaller firms had samples so small they were basically meaningless.
We also had no idea how fragmented the market was until then. That first year some 600 clients rated more than 400 agencies. This was an interesting industry snapshot, but got us no closer to really understanding trends in client preference and needs. Agencies had no real substantive data that could be used to any meaningful degree.
Complicating our process was our firm belief - then and now - that in order to have a credible survey, we could not use client lists supplied by agencies, as other surveys have done. That was the case then and that is still the case now.
Last year we narrowed our research pool to agencies that we believed could achieve adequate sample sizes. But we still needed to crack the model, because the numbers were still inconsistent from firm to firm, especially among current and past clients.
During last year's survey process, Millward Brown, our research partner, identified a database called the ERI panel that we used this year in fielding the survey (and used partially last year). This helped us meet our goals of surveying 50 current and 50 past clients for all 14 agencies that participated. These agencies all purchased data packets with their full results.
Our success in reaching these sample sizes - which is truly a breakthrough for this survey - now gives us hope that we might be able to expand it next year and ongoing. But at the end of the day, we have to have an adequate sample size for each firm for it to be meaningful. By virtue of their size, large firms are at an advantage in this.
We are, however, not resting there and continue to work with Millward Brown in the hopes of expanding the survey, provided we can reach our sample goals for each participating firm. As we start work on this in the fall, I will give you an update on how that exploration is going.
Thanks again for the feedback. I'm happy to discuss further when I'm back next week.