Yes, you should reply loudly as the prohibition is only while the Imam is giving his sermon.

My family and I watched numerous of your videos and browsed your website for numerous matters. May Allah give you even more knowledge so you can share. Sheikh, I need one clarification. If someone enters the masjid for Jummah, sit down quietly waiting for the khutbah to start and if a brother comes, say Assalamualaikum to this brother who is sitting down quietly, does the brother need to reply to the salam loudly? Please let me know.