ANSWER:

The imam does not move, it’s the follower who moves backwards and the other person joins him.

However your prayer is valid. 

I was the Imam in prayer with 1 follower. Then I see someone coming to join us so I moved to the front but then I realized that the third man joined late, was this rakah valid?

Prayer |