Hardness causes a random chance for the opposing unit to use its Hard Attack value instead of its Soft Attack value. The higher the Hardness, the greater the chance that the enemy will use its HA value against it for each attack, rather than SA. Since MOST units in the game have considerably higher SA than HA (other than AT and TD), it means that the hardened unit will typically suffer less "hits" each round. Even a relatively hard unit such as SHARM can be damaged by Soft Attack, but the odds are extremely low. For most Armor divisions, the softer components of the division will still lead to it taking fire from Soft Attack fairly often, and even infantry with NO Hard Attack can still damage it. Throw 5-6 INF divisions (or more) from multiple directions (to increase Frontage and to inflict an "outflanked" penalty on the target) against a typical ARM division, and you can crush it in spite of its higher Hardness, Armor, and Defensiveness values, inflicting more casualties than you suffer.
Armor and Piercing are a different mechanism, where units which fail to exceed the target's Armor with their own Piercing value will do significantly reduced damage AND suffer something like double the ORG loss from any return fire.
The combination of Toughness/Defensiveness, HA/Hardness, and Piercing/Armor together determine how many of the attacks manage to do damage in each round of combat. Giving up the Defensiveness of another Infantry brigade to add an AC or TD brigade for its higher Hardness is a tradeoff, and its effectiveness depends on what you're facing in that particular combat: Infantry and Artillery with a lot of Soft Attack but little HA or Piercing, Infantry with AT to provide greater Hard Attack and higher Piercing, or an ARM division with a combination of all of those factors plus greater hitting power and Hardness of its own. Then you have to think about how much supply and fuel the whole thing will draw. There's no single "right" answer, but plenty of options that will work well enough.