I missed him on my first time doing the quest. I walked to the left and said "oh where does this lever lead?", then I picked up the black book and BAM transported away. Totally got distracted by the black book and forgot to investigate the suspicious chest in the previous room. Finished the quest without encountering the guy and then when I went back at least he still spawns when you approach the chest. I guess I'll just cheese my way thru with a follower.