This is only some 30% accurate and not related to my point, because the AI can be the strongest in galaxy and have good amount of space, and it still won't be able to declare war if he is boxed with friends, because he is a victim of "AI opinion".
Even if that AI is not very strong, sooner of later one of his neighbors might lose fleets in war against some other AI, and this is perfect chance for him to attack, but it wont do it, again because "opinion" won't let him.
The claims system can make that difficult. But that does seem like a hole in the AI. You can spend enough influence to claim a single system on the other end of friendly ally space, take it, and then use it to launch claims at reduced cost.
If you're referring to the ruthlessness of the AI, that's a potential hole in another direction. The AI never wants to trade systems, but it assigns a very low value to systems given to it. It's rather slanted, and if you're a smaller empire than a friendly neighbor, it should be possible to trade for some system you can at least develop with Orbitals and/or Ring Worlds.
This is again false, such AI (or you) should invite attackers to gain advantage, there really isn't any other option other than war when you're (or the AI is) boxed, because time will eat you a live, such player is a sitting duck waiting to be destroyed.
In this case opinion is again problem, because you won't be able to get attackers even if offering them resources or portion of victory.
That's not really an accurate description of the situation we're discussing. It's technically correct on a long enough time line, but there's a big difference between being hemmed into two or three systems at the start of the game and eventually hitting an expansion wall half or two thirds way through. At a certain point in the end-game, with the pop growth mallus, conquest doesn't even provide much benefit unless you can use the pops from the empire you're conquering.
Time will eat you alive if you're boxed in, but that's how the RNG rolls and why diplomacy matters. If you're the smaller nation you try to make friends and have enough of a fleet to make someone think twice about invading instead of being allies. Joining an aggressive Federation can help, but that's still a question of diplomacy. That presents its own challenges, trying to stay afloat and relevant despite not being a super power. Given how the AI tends to work, even with this disadvantage you can probably play catch up at some point. But realistically that's how the cookie crumbles, especially if you have a larger number of starting empires than is recommended for your galaxy type.
The Galactic Custodian offers a potential solution - if you can wheedle and deal your way into Custodianship, you potentially have access to a large fleet that's independent of your own military infrastructure to possibly throw at expansion targets.
Like you pointed out, inviting attackers doesn't really work if they know what they're doing. They either can stomp your defenses or they can risk losing due to Harmony and other defensive home territory advantages. If you're boxed in they can simply develop and expand until they beat your economy. You're contained, they're not.
I see you see no way out of such problematic situations, but there is always a way out, it just takes a bit of imagination, I gave just one example, but there are many more.
Peaceful coexistence is definitely not the answer because without expansion there is no victory.
That's a pretty limited perspective. Stellaris is a galactic civilization simulation. Join a large and successful Federation, or help found one. Become a galactic leader - the Custodian, or even the Emperor, if you can get the votes. Having a small, thriving empire standing tall among bigger, stronger empires, that survives and repels the mid-game and end-game crisis, that is a victory unless you're just looking at the arcade score that pops up at the victory date or how much of the map has your flag.