User blog comment:ShawnCognitionCP/The Empire vs Stormcloaks, the most likely answer/@comment-9062114-20140611211441/@comment-12599067-20140613160015

The name Grey Quarter refers to the ash-grey color of the Dunmer people's skin, so it's basically saying that if you have grey skin, you have to live in the slums. They are both discriminated and oppressed, with people like Rolff, Agrenor, and Elda Early-Dawn constantly harrassing them and living in some pretty shabby conditions (cramped streets, small homes, fewer guard patrols, etc.). Even the wealthier merchants (the owner of Sadri's Used Wares, for example), have to scrape the bottom of the barrel to feed themselves.

I don't think the Empire would be happy about it, but I don't think they'd be able to do much if the Thalmor march for Skyrim. Given that the White-Gold Concordat makes the Empire basically sit by while the Aldmeri Dominion romps around Tamriel, they can't do all that much without risking another open war. The Empire and Aldmeri Dominion have basically entered a Cold War in which both opponents dislike each other, but one of which has to sit around and let the other do whatever they want or risk open war.

We don't see the Aldmeri Dominion armies in Skyrim, we only see the Thalmor. The Thalmor are High Elven supremacists, it makes sense that they would only have Altmer in their ranks. The Aldmeri Dominion, on the other hand, is the military of the Summerset Isles. If they have other provinces under their control, it makes sense that the Aldmeri Dominion (which is not a group of High Elvens supremacists) would have Bosmer and Khajiit in their ranks.