Coloring in all the inaccessible areas the same color as the surrounding territory also gives you a false sense of borders and ways of passage. The game is already bad about it's consistency at displaying inaccessible areas, sometimes it does, other times it doesn't. Best I can tell it is based on the map mode and whether all the territory being shown is within the same rank of title being shown or same criteria as what is being shown ( ie. faith, culture ).
For example: this is in de jure Duchies map mode. Notice how for Makuria the deserts surrounding it are colored in, but others are not.
View attachment 678313
Now, the same area in de jure Kingdoms map mode. Notice now how the deserts in Egypt and Nubia are all filled in with the same color as the titles, but the deserts between Nubia and Blemmyia are not.
View attachment 678314
Try viewing your game in religious map mode, and if all of your territory is the same faith I'm betting it will all be colored in.