Typically, the first nation (Italy or Japan) leaves the treaty early 1938, which means I can't start refitting to the increased levels until early 1939, leaving less than a year before the war starts (and the treaty ends completely). With almost any refit involving engines and/or armor taking 300+ days, these ships would still be in the que when the war starts. I do typically put my CV-2 (Yorktowns/Ark Royals) into the yards to add radar and better AA. I do tend to refit ALL of my Submarines (SS-1 get minelaying systems, but no changes to torpedos or engines, SS-2 get everything), and Destroyers. Sometimes I refit my CL so they are all the same standard, and if I'm really getting frisky I refit my CA-1 into CLs (only changing the guns)...particularly as the US because this allows me to build more Yorktowns without passing the UK.
I have more freedom as the US because I have an extra 2 years to play with, and sometimes refit the weaker BB-1 either into fully upgraded BC-1 or into Carriers.
This seems to be where our experience differs, I'm used to Italy or Japan leaving the naval treaty in 1936. Not 100% of the time, but most of the time. Every now and then the game goes a bit haywire for me when I'm playing the US or France and no one leaves, but typically Italy bails immediately. This might be based on default focuses versus using custom rulesets (I typically set Japan to fascist so they don't go communist and effectively kill themselves) though.