This is confusing the binary of sex gametes (egg vs sperm) with the physiological expression of the underlying genetic makeup, and also combining it with the judgements humans as a society have made on how each sex looks and behaves---what makes a male a man or a female a woman. Phenotype is the physical outcome of the genotype an individual starts with, where the genetic interaction and womb environment and even early childhood take the original two flavors and spread them into a spectrum. Genital size and shape, limb dimensions, body shape, all start from the same source and take infinite paths from there. It's a fool's errand to try to narrow down exactly what marks an individual as belonging to one category or another, because while the source is simple (ish), it's only the beginning of development and all we see is the end.
That's also entirely leaving out the societal influences, what non-physical traits are assigned to each sex and the relative value associated with them.