Specialized unit and they still put them in harms way throughout the campaign, and the losses are horrific in other units (good books though).
Well of course they put them in harm's way - they're
soldiers (well, airmen, but w/e). Their
job is to be in harm's way

. The point though is that they're not being thrown away as expendable mooks like you get with rather too many Imperial Guard regiments.
Regarding the Imperium they clearly have technology at their hands that would not endanger pilots in such a way but it's expensive and resource demanding
I'm not sure it's a question of resources so much as the Imperium's feudal setup and internal issues (and the fear of out of control or corrupted AIs). Even if (for sake of argument) Mars can manufacture the equipment necessary to automate the Imperial Navy's Lightning fighters or w/e, you still need to (a) persuade the Adeptus Mechanicus to do so, and (b) get them out there into a galaxy where the means of FTL travel is... not reliable, to put it mildly

. Frankly I suspect (a) is the real problem here - the AdMech
can build good AIs, but it tends to stick them into titans, where they merge with the titan pilot's mind. Robots in 40K tend to go bad far too regularly to be a safe investment.
In the end it's sci-fi and the fiction (story) is more important than the reasonability of the technological situation. Plausibility in sci-fi is largely dependent on the canon of the setting rather than modern day science
Yes and no. I mean... yes I like and enjoy Star Wars, but it would have been
better had George Lucas given us a reason for piloted fighters rather than "rule of cool". The best sci-fi IMHO is internally self-consistent, and considers fully (or as fully as possible) what impact the sci-fi aspects would have.
and in any case we don't even have the faintest idea on the feasibility of artificial gravity, AI, coherent beam-weapons, FTL or much of the other stuff making most sci-fi universes possible to begin with
Artificial gravity and FTL I'll grant you, but the physics of particle beams and lasers is pretty well understood, at least as far as
generating them goes. The real problem we have ATM is simply scaling them up. AI is in principle entirely possible, because we
know intelligence is possible (hi there

), and barring the existence of souls or something, what science can analyse, science can replicate.
The ultimate AI is of course better than a human in the cockpit (especially if it's a one-way trip) but it can be argued that it's not cost effective (depending on the cost of AI, sensors and flight-tech compared to a fighter) which is my point.
Most of those factors are irrelevant in the comparison: the sensors & flight-tech will need to be identical or nearly so for human pilots (using the mark 1 eyeball in space... eep), so the real cost difference is the cost of developing the AI (and I suppose periodically patching it), versus the cost of training a generation or two of pilots etc.
Another option for fighters (bombers really) over missiles: If the enemy point defense is good enough, they will pick off a large number of missiles, however, you could wrap a torpedo in black body material, and have it have no emissions other than a very, very tiny amount of power (if that) and they are instead contact weapons, a fighter(bomber) could carry these black body weapons close enough that they cant easily be evaded by larger ships and drop them at a fairly high speed and return to get more. These torpedoes being black body and radar/lidar absorbent, and having no emissions of their own due to a lack of engines would be hell on point defense to intercept.
Three points that spring to mind. First, if the point defence is that good, then why would a fighter be able to get close enough to launch its attack? It
must get close in your example because your stealth torpedo will be unguided (you need engines to change direction in space, and that means power, which means waste heat and/or other emissions, which means it's no longer a stealth weapon).
Second, those missiles must be very effective to justify the cost of a project to develop them and put them to use, as opposed to a project to strip out the missile launch tubes and replace them with more reactors and lasers or w/e

. If the defence against one weapon type is
that good, why stick with that weapon type?
Third, contact weapons are... iffy. They absolutely can be very effective, but the obviously problem is hitting a tiny target in the vastness of space (and yes, a multi-kilometre vessel is tiny in these circumstances

). Thus it depends on the specifics of the combat environment: it
may be better to use shaped nuclear weapons and bomb-pumped x-ray lasers, for example - these can have a stand-off range measured in thousands of kilometres (see the
Honor Harrington books). Omni-directional nuclear warheads will only ever work in very close range though, due to the inverse square rule.