The Great Collapse

The Great Collapse was a geological event that devastated Winterhold. On the 4E 122, the land in which the town stood crumbled away taking most of the town with it but miraculously left The College of Winterhold largely unharmed.

Volcanic eruption
The cause of the Great Collapse is assumed (by the The College of Winterhold) to be a far reaching consequence of the eruption of a dormant volcano, Red Mountain.

College mages
Many of the the locals, however, blame the College itself. Although the Jarl trusts Arch-Mage Aren's assurances that the college was not involved, the suspicions of the locals remain. Thus, the mages, and their associates, are generally disliked and avoided.

Effects
Since the cataclysmic event, Winterhold has never truly recovered. Immediate prejudices emerged surrounding the Dunmer refugees of the College, as detailed in a letter written by the then Arch-Mage Deneth, to the Jarl, Vladmir.

Despite the formal apology, the Dunmer and mages were never truly trusted again.