That's because a province can contain multiple terrain types. If you hover over a province in terrain map mode, you'll get a breakdown, e.g. 80% hills, 20% farmland. If the battle ends up taking place in the province's hills (80% of province means 80% chance), the attacker will suffer a big terrain penalty, if it takes place in the farmland, they won't.
It can be really annoying if you set up an army to defend a 95% mountains province, and they decide to do battle in the 5% flat terrain.