Another perspective on this is that it's kind of... infantilizing... to remove people's agency in what happens to them. Except, how much agency do they have in the first place? Except, that's the whole problem, isn't it, is that he doesn't really understand what's happening, or how people work, or the concept of "agency" when everyone else seems like they don't actually have any.
And of course, from the outside, this looks a hell of a lot like he is in fact trying to make decisions for other people regardless of what they say they want. Which isn't—okay, it is in fact less bad to do that to decide to refuse sex because he thinks other people "don't really mean it" than to do that to have sex that he thinks people might in fact not mean, but—it's still condescending as heck and, and forget thinking about the timeskip thing, actually. Imagine that the vague memories he has of his life prior to this week were somewhat less vague and when he woke up on Sunday he had in fact had the whole life he felt like he'd had, including the sex he wasn't there for. Would he have been feeling like he didn't meaningfully consent? Maybe, but only in this same abstract sense he's treating the whole thing right now; he wouldn't be feeling bad. The choices he didn't make still feel like they were his.
Plus, the way his timeskip self acts isn't not consistent with what a summarised version of what he'd be doing were he awake would be. His timeskip self isn't himself, but it's still playing The Peter Tarleton Aesthetic.