And they aren't sexualized. Having young girls or boys be the main characters of X or Y story is not 'lolicon'.
1) A drunk rant being told second-hand is far from reliable. Drunk people say all sorts of shit sometimes, and a second hand account of something from years ago is an unconfirmed anecdote
2) Bai-Niang was essentially a 'waifu' back when he was in highschool, which means he was around the same age, 2-4 years older than the main character. That's like saying a 60 year old man attracted/associating to a 56 year old woman is a gerontophile.