User blog comment:ShawnCognitionCP/The Empire vs Stormcloaks, the most likely answer/@comment-9062114-20140611211441/@comment-24728100-20140613145827

In responce to Shawn:

In the time of Skyrim, 4e201 the empire consists of Cyrodiil and half of Skyrim... that is all!

when the empire abandoned both Blackmarsh and Morrowind During the Oblivion Crisis it lost support of these nations, the empire refusal to aid the dunmer durring the red year sealed any chance of reunification with Morrowind.

Hammerfell was officially renounced as an imperial province by titus mede 2

Highrock has been uncooperative with the empire since the end of the war

Elsweyr has withdrawn from the empire and has after dividing into two rival nations become tributaries to the dominion

Valenwood was taken over by the thalmor with aid of their Bosmer allies and the 2nd Aldmeri Dominion was formed... then the purges started.

The empire is not as big anymore, and it is mostly due to it's greed and selfishness