Most times you see victims of the Falmer, they're farmers, villagers, and other defenseless/lightly defended folk, or inexperienced adventurers with poor equipment. They are inferior to the average hold guardsman, let alone to the average Legionnaire or Stormcloak, and those times that they are shown to overpower the latter are when those soldiers were heavily outnumbered in the middle of Falmer territory and separated from any source of reinforcement. It is important to remember that the Falmer do not have any organization to them by virtue of a severe lack of intellectual capacity, and thus cannot coordinate large-scale assaults on the surface world. Further, they have no way of knowing where any major towns or cities would be, and many of these cities are built upon stone foundations either way. They overwhelm the small groups of normally ill-equipped explorers as they live in those areas in large numbers, but only occasionally raid, and even then only light caravans or small villages close to the mouths of their caves. It is because their impact on the holds at large is so small that local militia focus on the Civil War rather than clearing them out: the very moment they begin to pose a threat rather than a nuisance is the moment the Jarls decide to send well armed, well armoured, and well trained militiamen, knights, or mercenaries to clear out their dens. And if they begin to pose a threat and their numbers are large enough to warrant it, that is when professional military organizations such as the Legion or Stormcloaks become involved.