Barbarian Power does keep growing after you colonize a province - in fact, BP grows twice as fast in a colony compared to an uncolonized province. The Barbarian Revolt event can be reduced in frequency by having a high-Charisma governor and increasing the Civilization Value of the province.
To get rid of BP in colonies, you need to get the Barbarian Absorption event. This event reduces BP by 2 each time it fires. If BP is reduced to 0 by this event, you get the Successful Colony event. Successful Colony will set the province's culture and religion to your state ones, and remove the "colony=yes" flag - meaning BP will no longer increase for that province
The Barbarian Absorption event requires the province's governor to have at least 1 Finesse, and you can make it more likely to happen by doing the following:
Having a governor with 7 or more Finesse
Increasing the Civilization Value
Building Academies, Aqueducts, Arenas, Forums, Roads and Temples
Hope that helps.