Well you must have some form of long term episodic memory before 3. When my older kid was 2 he would often talk about things he had done several weeks (or even months) before. He wouldn't be able to tell you what he had for lunch earlier in the day, but would suddenly start talking about something he had done months ago. I think the point is that at this age, the memories don't tend to be rehearsed so much, so they are more likely to be forgotten and not make it to the more permanent (ie that will survive to adulthood) long term memory. My son, at the age of 6 now doesn't seem to be able to remember anything before age 3 now.
But anyway, though I doubt whether there is a fixed lower age limit of 3 years, I still think the idea of memories of babyhood and prebirth are nonsense.