Board Thread:Skyrim/@comment-120.28.157.86-20140406014559/@comment-26893431-20170215163342

1.32.72.184 wrote: In  4E 174 ,  Markarth  and  the Reach  was subjected to what is known today as the Forsworn Uprising. When the  Third Aldmeri Dominion  invaded the  Imperial City  during the  Great War, the  Imperial Legion  all but turned a blind eye to the other provinces. The native  Breton  people of the Reach, called the  Reachmen, took this opportunity to seize control of the city. This was only possible as the  Imperial Legionnaires  stationed in Markarth were recalled to fight, leaving the city very vulnerable. ( Empire did not bother about skyrim. )

During this time, Hrolfdir, his son  Igmund, and their Imperial colleagues sought help from  Ulfric Stormcloak  in ousting the Reachmen invaders. They promised Ulfric that if he re-took the Reach, then Hrolfdir as Jarl would allow free worship of  Talos  - an act that was recently outlawed at the end of the war. Ulfric agreed and marched his militia to the gates of Markarth and ruthlessly retook the city using the power of his  Thu'um. ( Ulfric cares about skyrim and talos worshipping. )

Eventually, the Empire rescinded upon the agreement due to pressure from the Aldmeri Dominion. Hrolfdir turned on Ulfric and his men and aided the Empire in arresting them. Ulfric and his militia were expelled from the city and imprisoned shortly after. This betrayal left Ulfric bitter towards the Empire and is considered to be the initial conception of the  Stormcloak  rebellion.

Source:  http://elderscrolls.wikia.com/wiki/Markarth_Incident

It wasn't an "Empire didn't care about Skyrim", it was more of a "Skyrim has no threats, so we use its forces to retake the capital". They did not know that the Reachmen would take Markarth.

During this time Ulfric was working with the Thalmor, aka Ulfric cares about appeasing his Thalmor masters and getting power, hence why he demanded free Talos worship, despite the Talos ban not being enforced.