ANSWER:
This is from satan to distract you from remembering Allah. The more you engage in ibadah, the more he will distract you. If you back down and reduce it, he will reduce his whispering but if you increase it, he will increase it until you reach a level where it is not feasible any more for him. Only then, he will back down for few months and then try again! This is your jihad.