At least in HOI3, each turn there was a chance to use either HA or SA, and the odds of one or the other were totally dependent upon the target's Softness. If the target had 90% Softness, then there was a 90% chance of using Soft Attack against it that turn, and a 10% chance of using HA. A 100% soft target (Infantry) will ALWAYS cause the firer to use its SA value, while no unit in the game had 0 Softness, meaning that a unit with NO Hard Attack always had at least some slim possibility of inflicting a small amount of damage against it. A 5% Softness division (all Heavy Armor) would still have a 5% chance each turn of being engaged by the opponent's Soft Attack value (typically halved because of the piercing mechanics), and a 95% chance of facing its HA value (again, probably halved).
As you point out, most often high Piercing and high Hard Attack values usually coincide, because the weapons that provide the one usually provide the other, although in HOI3, Infantry could already get fairly decent HA via the portable AT weapons techs, so you only needed a source of Piercing to make it useful against Armored targets. AT brigades provided high Piercing AND Hard Attack, but almost no SA, making them relatively useless against infantry and other soft targets, where their high HA value was totally wasted. Armor, on the other hand, generally provided both Armor values and Piercing values to the entire division, reduced Softness (so the opponent had to use its HA value more frequently, instead of SA value), and also contributed decent amounts of both HA and SA.