Compass found itself in the eye of a media storm on the quality of school meals following the Channel 4 series Jamie's School Dinners.
It subsequently reported a pre-tax profit fall from £645m to £585m.
The results drew fire from analysts, who said the group was not generating sufficient returns on money invested.
The group will finish working with Brunswick at the end of this month after four years.
Finsbury founder Roland Rudd, who will lead the account, has a long-established relationship with incoming chairman Sir Roy Gardner, currently CEO of Finsbury client Centrica. Gardner replaces Sir Francis Mackay later this year.
Compass is currently trying to sell its retail food and drink arm Select Service Partner, which owns the Moto service station chain.
Brunswick, which was not invited to repitch for the business, told PRWeek: 'This is not a performance-related decision. We have been proud to represent Compass over a number of years and wish the new management team well.'
Compass group chief executive Mike Bailey said the appointment was part of a regular review of external advisers. 'We have enjoyed
a good relationship with Brunswick and are grateful for its support,' he added.