The island of Escendit is also known as "the stepdown" for three reasons: it's the closest outpost to one of the outposts below; it kinda looks like a stair case if you are flying below and looking at it from the right angle; it's a transitory place for people that have seen better days and now have to resort to work at surface.
Esdras does not take a lot of comfort knowing that his half-brother came to this place willingly. Much less when he traces his steps around town (which takes forever because this is a place where you have to pay attention to your coin purse) and look, he actually did it! Not only he went to the surface but he smiling all the while and waving at people when the balloon takes off!
The man at the pier thinks the angry staring is directed at him, Esdras waves and signals at his eyes and then at his robes and the man apparently understands that Esdras was just using sensory magic. Typically the umbra shape and robes are enough, but a lot of people come here because they lost their shapes and need new ones. Esdras comes over and talks to the man to ask a few questions.
A short while later, Esdras is waiting for the balloon to depart.
Esdras focus on his exasperation because the alternative is imagining on the many horrible thinks that could've happened to his half-brother in the world below.