Aromatase inhibitors (AIs), the standard therapy for estrogen receptor- or progesterone receptor-positive breast cancer in postmenopausal women, lead to increased hip fractures in breast cancer patients. To investigate the mechanism of increased incidence of hip fractures in breast cancer patients treated with AIs, we evaluated bone mineral density (BMD) in the cortical and trabecular compartments and assessed femoral geometry using quantitative computed tomography (QCT) in breast cancer patients. In total, 249 early breast cancer patients who underwent QCT in their fifties (mean age 54.3 years) were retrospectively analyzed. Proximal femoral BMD and geometrical parameters were compared. In all regions of the proximal femur, cortical areal BMDs were lower in the AI group than in the non-AI group (p < 0.05). Cortical thickness of the femoral neck, trochanter, and total hip was significantly lower in the AI group compared with the non-AI group (p < 0.05). Analysis of the narrowest section of the femoral neck showed significantly thinner cortical bone and smaller cortical area in the AI group than in the non-AI group (p < 0.05), especially in the superoposterior quadrant. Bone strength parameters in the femoral neck, such as the section modulus and cross-sectional moment of inertia, were significantly lower in the AI group than in the non-AI group (p < 0.05). In conclusion, AI treatment in breast cancer patients is associated with deterioration of femoral cortical BMD and geometry, which could contribute in site-specific weakened bone strength and increased incidence of hip fractures.