One of the ideas behind Tower lore is that they are supposedly responsible for the stability of Mundus, and prevent Oblivion and possibly beyond from just crashing into the mortal realm. More or less. By extension, it’s created the idea that if these Towers are destroyed/deactivated then that stability is gone.
However one being disagrees. The Daedric Prince Ithelia.
According to her, "Mortals are servants performing tasks to maintain reality. Every field plowed, child born, or war fought keeps the Aurbis stable by design.This fact is kept from them. The slave should not understand the master's plan, lest they seek to undo them."
According to her, it’s the people that keep reality stable. And I think this can make a lot of sense when we also consider the ideas that Sotha Sil has - that fate is a series of predetermined actions and consequences, with the intent behind it being to have mortals provide stability to creation. Taking the idea of the Prison to a whole new level.
But interestingly, it still leaves room for that stability to be weakened. Ithelia points out that mortals are made to not be aware of this, and if they were, it could lead to disaster. Ithelia also tells us that mortals have the ability to choose their own fate, “But mortals? You choose your paths. You don't need the powers of a Daedric Prince to alter your fate. I envy this.”
When I think on it, I think it suggests that instead of the whole “Towers Theory”, I think it’s much more complicated. Destabilising reality requires the complete degradation of predetermined fate, freeing mortals from the Prison, and a greater awareness about their reality to be able to accomplish this. This is backed up in ESO - where we witness the destruction of one reality due to fate being so broken that reality collapses on itself.