What I do is not evidence. However, what you were told is not right except if the trousers where too tight. Normal trousers are OK. As for the hadeeth he quoted, this is regarding females.

One of the Imams of our masjid told me that wearing trousers is like (aura) and he gave some hadith that said “Qasiat Aariat”, but I have seen many scholars including yourself putting on trousers so who do I follow here?