He is ordered to return to the Capital, so he does. Whether he looks forward to it or dreads it is not relevant. Whether he honestly prefers exile-in-all-but-name to the borders (it's dangerous, but at least its not a pit of snakes like the Capital) is not relevant. He has orders and a duty, and he will fulfill it.
(At least Xia Dong is leaving just as he arrives. That minimises the amount he has to talk to her, and that's... something.)
The first thing he notices amiss is that his horse trips. That's not completely surprising, even the most well trained and well coordinated horse falls some time. But it still takes him by surprise.
It takes him enough by surprise that the second amiss thing happens. He falls right off, sliding down his horse's neck before he even knows what's happening. Even that is unusual but not improbable. Even the best rider will fall off sometimes.
The third thing, and the one that retroactively makes everything else before it strange and concerning, is that he falls into a hole. A deep one. That keeps going and going and going until he lands-- somewhere. Not the Capital. (He shouldn't be relieved.)