I think I have it figured out. You have to be completely out of the area until midnight. Once it's 00:00(12A.M.) you can go there either manually or by fast travel and he'll be there. By out of the area I mean really far away from the Ramshackle Trading Post. Raven Rock is suitable.