Virgil does not remember everything. His memory isn't eidetic; he doesn't have a huge advantage over anyone else if he sees something once. But he remembers the things that he does practice, rehearse, or study with startling clarity (according to his teachers), and he recalls his life with disturbing precision (according to his mother). Virgil isn't convinced this is all that unusual- most people just don't use their brains, because they don't like thinking. The Scholomance is going to be better, for that- everyone here has a vested interest in being the smartest person around- which Virgil knows he isn't, but he's working on changing that.

That's why, instead of socializing with his peers (that's why, and not that he hasn't had a single successful social interaction), Virgil is walking the stacks of the library, trying to memorize the layout. It's not using a completely familiar categorization system, but there is one, which is gratifying. He just needs to put the pieces together.