Thanks for the suggestions
In my custom species file, I tried using randomized=no, but that caused the custom races with the related portraits not to spawn for the AI at all, even though the races had spawn_enabled=always. So I had to set the portraits to randomized=yes. All the portraits are also listed under "non_randomized_portraits". I think this is working and I have not seen randomly generated empires with my custom species portraits
However the problem of duplicate portraits remains for the vanilla species I have used to make my own races. I cannot set them to randomized=no as that flag is CLASS wide and I only need to exempt some portraits. I also tried creating an extra file in my mod where I only overwrite the CLASS/non_randomized_portraits sections of the vanilla classes but that seems to work differently from what I expect also. The entire classes are overwritten, not just the non_randomized_portraits section and as a result all the portraits from all the CLASSES are lost.
I guess my one option is to make a complete copy of the vanilla 01_base_species_classes.txt file and overwrite the non_randomized_portraits section there, but it seems such a brute solution and maintenance hassle. Is there no way to be more precise with what I overwrite?