Bernard Ingham also warned Tory leader David Cameron not to rely too heavily on his comms director Andy Coulson to improve his standing.
Ingham was speaking to PRWeek following weeks of coverage focusing on Brown's new team of communications advisers, headed by ex-Brunswick CEO Stephen Carter.
Ingham said: 'One of the problems with the Government these days is that everything is looked at in PR terms. Brown is not failing because of his PR but because everything he touches turns to dust.
'All my sympathies are with Brown's £1.75m imported team, recruited at the rate of one a fortnight. They have bought a pup. They should have realised that they had nothing to work with before they started.'
Last week, PRWeek reported that the Prime Minister is becoming increasingly frustrated by Tory comms chief Coulson, who has been instrumental in reinforcing the perception that Brown is a 'ditherer'.
Ingham said: 'I don't begrudge Andy Coulson whatever glory comes from this, (but) David Cameron looks to be the best bet as the next prime minister not because of his PR, but because Brown is dithering, and because Cameron looks fresher and more human.'
Ingham warned the Tories that good PR would not disguise a poor performance by their leader. He said: 'Brown was thought to be the bees' knees until last autumn when he bottled out of an election. His is a failure of performance, not of PR. The eminently presentable Cameron will similarly perish if it is found he's all show.'
Ingham served as press secretary to a succession of ministers, including Barbara Castle and Tony Benn, before becoming Thatcher's chief press secretary for 11 years.