ANSWER:
What they say is absolutely baseless and fabricated. Such imbeciles should focus on praying themselves as those who do not pray at all are outside of the fold of Islam.
Start praying tahajjud but begin by little, like 2 rakahs every night so that you don’t get bored or tired after a few nights.
Be consistent in praying even if it’s just 2 rakahs because Allah loves a worship that is consistent regardless of how small it is.