ANSWER:
No problem in joining them and performing your prayers on time even if you have to disobey them in this regard. Your presence might be a deterrent for them from doing a lot of more sins and you might also be a reminder for them to pray on time.
There’s no obedience to the created at the cost of disobedience to the Creator.