King has been chief executive and the public face of the retailer for the past 10 years and was rated as the most influential communicator in the FTSE 100 in a survey by Aberfield Communications in December.
His departure is not a complete surprise, after newspaper reports speculated he was close to moving on and King acknowledged that he would not be there for much longer.
King will leave Sainsbury’s after the annual general meeting on 9 July and will be replaced by group commercial director Mike Coupe, the company said in a statement.
It is unclear what King will do once he leaves the retailer.
Sainsbury’s chairman David Tyler said King was a "truly exceptional leader who has reshaped Sainsbury’s during his 10 years as CEO. He leaves a lasting legacy."
King said: "This was not an easy decision for me to make, and in truth it will never feel like the right time to leave a company like Sainsbury’s."