There are limited data on physical activity in relation to trajectories in cognitive function. The aim was to examine the association of physical activity with trajectories in cognitive function, measured from repeated assessments over 10 years. We conducted a 10-year follow-up of 10 652 (aged 65±10.1 years) men and women from the English Longitudinal Study of Ageing, a cohort of community dwelling older adults. Self-reported physical activity was assessed at baseline and neuropsychological tests of memory and executive function were administered at regular 2-year intervals. Data from six repeated measurements of memory over 10 years and five repeated measurements of executive function over 8 years were used. The multivariable models revealed relatively small baseline differences in cognitive function by physical activity status in both men and women. Over the 10-year follow-up, physically inactive women experienced a greater decline in their memory (-0.20 recalled words, 95% CI -0.29 to -0.11, per study wave) and in executive function ability (-0.33 named animals; -0.54 to -0.13, per study wave) in comparison with the vigorously active reference group. In men, there were no differences in memory (-0.08 recalled words, 95% CI -0.18 to 0.01, per study wave), but small differences in executive function (-0.23 named animals; -0.46 to -0.01, per study wave) between inactive and vigorously active. Physical activity was associated with preservation of memory and executive function over 10 years follow-up. The results were, however, more pronounced in women.