In Angiosperms, there exists a strong association between mating system and lifespan. Most self-fertilizing species are short-lived, and most predominant or obligate outcrossers are long-lived. This association is generally explained by the influence of lifespan on the evolution of the mating system, considering lifespan as fixed. Yet, lifespan can itself evolve, and the mating system may as well influence the evolution of lifespan, as is suggested by joint evolutionary shifts of lifespan and mating system between sister species. In this paper, we build modifier models to study the joint evolution of self-fertilization and lifespan, including both juvenile and adult inbreeding depression. We show that provided that inbreeding depression affects adult survival, self-fertilization is expected to promote evolution towards shorter lifespan, and that the range of conditions under which selfing can evolve rapidly shrinks as lifespan increases. We study the effects of inbreeding depression affecting various steps in the life cycle and discuss how extrinsic mortality conditions are expected to affect evolutionary associations. In particular, we show that selfers may sometimes remain short-lived even in a very stable habitat, as a strategy to avoid the deleterious effects of inbreeding.