ANSWER:
It can be as you didn’t stop him nor you left him!
Allah says in Quran: And it has already come down to you in the Book that when you hear the verses of Allah [recited], they are denied [by them] and ridiculed; so do not sit with them until they enter into another conversation. Indeed, you would then be like them. Indeed Allah will gather the hypocrites and disbelievers in Hell all together – 4:140