The aim of this study was to perform a comparative investigation of white matter integrity in patients with prelingual and postlingual deafness; we carried out a tract-based statistical analysis of diffusion tensor anisotropy in eight and ten adults with prelingual and postlingual deafness, respectively. Patients with deafness showed significant decreases in diffusion anisotropy at the right internal capsule, the right thalamus, and the splenium of the corpus callosum as well as within the bilateral superior temporal gyrus (including Heschl gyrus) and right temporal white matter. Furthermore, relative to patients with postlingual deafness, those with prelingual deafness showed lower anisotropy in the right superior temporal gyrus, bilateral temporal white matter, and the genu and anterior body of the corpus callosum. We believe that, in patients with deafness, reception of early auditory stimuli before language acquisition might be more critical to white matter maturation and brain reorganization than the nature of auditory stimuli itself or the duration of disuse. These findings provide the theoretical background for early auditory rehabilitation.