User blog comment:Zippertrain85/Harkon's backstory?/@comment-5450493-20130505155445

If it was hundreds of years ago like Harkon claims, Skyrim could have been different or Harkon could have come from a different province of Tamriel and moved to the castle later.

It might not make sense but it explains why he is the way he is.