Arch-Brexiter claims Sunak’s decision to leave D-Day commemorations early has gone down ‘very badly’ with voters