Plus, while the impact of age on a couple’s fertility has historically focused on the woman, “there has been a lot of data gathered over the last 10 years that indicates that, as men age, their fertility potential does decline over time,” said Daniel H. Williams, a urologist who specializes in male infertility at the University of Wisconsin School of Medicine and Public Health.