Your stolen items will be in the confiscated items chest in the jail. So, while those stolen items won't be returned to your inventory upon your release (or jail break), you can still attempt to steal them back from the chest when a guard is not looking.
No, you won't lose your items. This area is not a regular kind of prison with guards or a chest containing your gear. Near the end of the quest your items will be automatically returned. Dont want to spoil any story elements