Mods do not affect the game unless they're activated. If they did, that'd be a virus.
Activating a mod is basically telling the game "please load the mod's files, and replace any vanilla ones with the mod's files if they have the same name".
The only way a mod could affect the game without being activated, is if you manually put the mod's files in the games installation directory tree, overwriting vanilla files.
Deactivating a mod and continiung a save-game you were using them in is a different matter. Mod 'assets' are not merged into a save-game. They are referenced by pointers. So doing this will likely break an existing save-game, as it is now full of orphaned references. Common symptoms of this are the mod's assets vanishing.
Do not continue a game that had mods activated that you've deactivated. And do not expect a broken save-game to be magically fixed by reactivating the mods you deactivated. Too late, the save-game is usually toast.