The Great British Bake Off judge says his family has faced ‘vindictiveness’ over Smarden pub sale.