The correct view is that this does not invalidate your prayer, because doubts often come to a man without him wanting them to. The Prophet (peace and blessings of Allaah be upon him) explained the ruling concerning doubts in prayer, and that doubts fall into two categories:
1 – When the person doubts the number of rak’ahs he has prayed, but he thinks one of the two is more likely. In this case he should act upon what he is more certain of, and complete the prayer on that basis, then say the salaam and do the prostration of forgetfulness (sajdat al-sahw) after the salaam.
2 – When he doubts the number of rak’ahs, and neither number seems more likely to him. In this case he should act upon the lower number, because that is more certain and the higher number is doubtful, and he should complete the prayer based on the lower number, then do two prostrations for forgetfulness before the salaam. His prayer will not be invalidated in that case.