Board Thread:Lore Discussion/@comment-77.172.125.107-20130901100104/@comment-3573184-20131002222817

Those Atmorans lived in Skyrim in small villages and were so few that they didn't bother the Falmer to any real degree. Ysgramor is the one who started bringing people over in mass, which caused the Falmer too get irritated.

And there most likely was some humans living in Skyrim before the Nords came, but like the humans of Morrowind, and the ones that ried to settle Blackmarsh, they either died or where enslaved.