Lan Wangji is having a very small emotional crisis.
First of all, the math is telling him to break the rules of math and he doesn't understand how to do this. You're not supposed to just guess on math problems. There's a right answer.
Second, Wei Wuxian keeps smiling, which is normally good, except now he's smiling at Masozi and being interested in Masozi and Lan Wangji has never gotten him to smile that much. They're saying something in English and Lan Wangji doesn't speak English and they could be saying anything. Maybe Wei Wuxian is confessing his love to him. Maybe they're going to get married and Lan Wangji has accustomed himself to a life of celibate devotion but it is one thing to be accustomed to a life of celibate devotion and another to live it while your love is getting married to someone else and smiling at him. Lan Wangji wants those smiles to be reserved for himself alone.
Maybe he could stab Masozi with Bichen.