As he sat in the shepherd’s hut, surrounded by piles of his unwanted memoirs, David Cameron could be forgiven for wondering how it had come to this.