As far as I can tell, it's because all the rest of the confederacy with the exception of the Shardana and the Peleset are clearly identifiable as coming from Anatolia/The Aegean, and it therefore makes more sense for a unified confederacy to have been organized from a smaller area and subsequently settling out in the larger one than for a confederacy from a wider area to have organized itself over much greater distances.
Is there any meaningful evidence that they were actually a "confederacy" or "organized"? I thought all we knew is that they were all wreaking havoc.
If we had worse records in the 10th century AD, we might be wondering who the "heathens" really were that were attacking Christian Europe. It might have been tempting to assume these mysterious Saracens, Vikings and Magyars must have been somehow related but it would have been a mistake.