The long-term effects of calcium and vitamin D supplementation on bone material and structural properties in older men are not known. The aim of this study was to examine the effects of high calcium (1000 mg/day)- and vitamin-D3 (800 IU/day)-fortified milk on cortical and trabecular volumetric BMD (vBMD) and bone geometry at the axial and appendicular skeleton in men aged over 50 years. One hundred and eleven men who were part of a larger 2-year randomized controlled trial had QCT scans of the mid-femur and lumbar spine (L1–L3) to assess vBMD, bone geometry and indices of bone strength [polar moment of inertia (Ipolar)]. After 2 years, there were no significant differences between the milk supplementation and control group for the change in any mid-femur or L1–L3 bone parameters for all men aged over 50 years. However, the mid-femur skeletal responses to the fortified milk varied according to age, with a split of ≤62 versus >62 years being the most significant for discriminating the changes between the two groups. Subsequent analysis revealed that, in the older men (>62 years), the expansion in mid-femur medullary area was 2.8% (P < 0.01) less in the milk supplementation compared to control group, which helped to preserve cortical area in the milk supplementation group (between group difference 1.1%, P < 0.01). Similarly, for mid-femur cortical vBMD and Ipolar, the net loss was 2.3 and 2.8% less in the milk supplementation compared to control group (P < 0.01 and <0.001, respectively). In conclusion, calcium–vitamin-D3-fortified milk may represent an effective strategy to maintain bone strength by preventing endocortical bone loss and slowing the loss in cortical vBMD in elderly men.