Could it be related to going on the elevator? Maybe you could try and let him walk down the steps as many floors as possible and then take him back up the steps. If there is no apparent anxiety after that it would certainly narrow things down. You said he gets scared and cries when it's time to come in so maybe he is dreading the ride back up.
Just a thought.
Lukka Jan.20 2006 - Aug.19 2008