Any reincarnated successor "self" of mine... doesn't have my current memories of my current life. Doesn't have my learned knowledge or beliefs (those being types of memories). Doesn't have my specific genetic physical characteristics. Doesn't have my life history of interactions with the world. Doesn't share the follow-ons of that history (such as, people who know me, things I've written, property I used to own... all of which, if I had merely lost all my memories but were still living, would still be associated with me even if I didn't remember them myself).
So, in what way is that successor myself? The only thing we share is eventual future access to some hypothetical locked-away memory store that will be restored to "me" at some future time along my supposedly educational spiritual journey.
Aside from that, the reincarnation narrative tells me that I have far more in common with you than with any of my predecessor or reincarnated successor "selves." You and I share large amounts of culture, similar environments we've lived in and experiences we've lived through, and perhaps a common language in which we can actually converse. That's not true of my reincarnated selves. I can't talk to them, they can't talk to me; they live in my past and future, and most likely in far-away places with different ways of life. They're complete strangers.
Why would I be tempted to single out a few specific past and future lives, rather than the people I know around me today (or, perhaps, every conscious being), as additional "selves" of mine? What qualifies them for that distinction?