The Rise & Fall of the British Empire by Lawrence James starts two centuries in advance of Victorian era, with the British colonisation of the Americas, but the latter half is dedicated to the late 19th and early 20th centuries. It makes the interesting argument that Britain's imperial acquisitions were actually a sign of decline and weakness, since before, the British were able to control those territories through informal empire - exerting influence via proconsuls and missionary networks - but, as the other Great Powers began to catch up, the Empire was forced to annex swathes of Africa to maintain control of their markets and defend them against outside influence. It really explores the colonialist mindset and reveals how the pace of expansion was set and why it became necessary for the Great Powers.
Other recommended reading:
The Scramble for China by Robert Bickers, for an in-depth coverage of the decline of the Qing Empire and its subsequent exploitation at the hands of the Great Powers.
The Habsburg Monarchy, 1809-1918 by A.J.P. Taylor (again), which is more country-specific (being concerned only with the Austrian Empire), but which reveals why the Great Power of 1815 became the joke of 1918, and also incidentally shows how the European balance of power collapsed over the course of the 19th century, leading to the events of the early 20th century.