I had that same behavior with Cyprus. I got the province from rebels and they kept repeatedly sieging the province and turning it over to me. They did eventually vanish after many, many such cycles. But I think it has something to do with, islands where the rebels don't have movement options.