Two things come to mind:
1) Jets are LOUD, really LOUD. Folks may get used to the noise of aircraft on climbs or landing approaches because in the case of the former there are often noise abatement procedures in place, and in the latter due to the fact the jet is moving relatively slowly and at low power settings.
But a jet, flying along at low altitude at a very high rate of speed, that's going to make a noise folks are going to notice.
2) A large, fast-moving metallic object is sure to catch people's eyes. Human eyes are relatively good at detecting movement within their visual field, at least in my experience.
Put these two together and any jet overflying the Pentagon would have been witnessed by a heck of a lot of people, not the paltry few from whom you've supposedly been able to get statements.