Labour party is the reason for the decolonialzation of British Empire IRL. They even pressured Liberal party to do so. Furthermore it was their failure at the Suez canal (and Labour) who madee the winds of change. So actually even with a Liberal party in goveernment the decolonialzation shouldn't have happened so rapidly.
Decolonisation is a moral duty, not a foreign policy failure. As someone who claims to be in favour of personal liberty and self-determination you should be highly in favour of it.