The goal of this work is to study how brain volume loss at old age is affected by factors such as age, APOE gene, sex, and school level. The study of brain volume loss at old age relative to young age requires at least in principle two MRI scans performed at both young and old age. There is, however, a way to address the problem by having only one MRI scan at old age. We compute the total brain loss of elderly subjects as the ratio between the estimated brain volume and the estimated total intracranial volume. Magnetic resonance imaging (MRI) scans of 890 healthy subjects aged 69 to 85 were assessed. The causal analysis of factors affecting brain atrophy was performed using Probabilistic Bayesian Modeling and the mathematics of Causal Inference. We find that healthy subjects get into their seventies with an average brain volume loss of 30% from their maximum brain volume at a young age. Both age and sex are causally related to brain atrophy, with women getting to elderly age with 1% larger brain volume relative to intracranial volume than men. How the brain ages and what are the reasons for sex differences in adult lifespan are causal questions that need to be addressed with causal inference and empirical data. The graphical causal modeling presented here can be instrumental in understanding a puzzling scientific inquiry-the biological age of the brain.