In this Paper, a new DCT HLMS DFE(Discrete Cosine Transform Hierarchical Least Mean Square Decision Feedback Equalizer) algorithm is proposed to improve the convergence speed and MSE(Mean Square Error) performance of a receive channel equalizer in ATSC(Advanced Television System Committee) 8VSB(Vestigial Side Band) digital terrestrial TV system. The proposed algorithm reduces the eigenvalue range of input data autocorrelation by transforming LMS (Least Mean Square) DFE into the subfilter of hierarchical structure. Moreover, the use of DCT and power estimation algorithm makes it possible to reduce the eigenvalue deviation of input data which results from distortion and delay of the receive signal in the miulti-path environment. Simulation results show that proposed DCT HLMS DFE has SNR improvement of approximately 3.8dB, 5dB and 2dB as compared to LMS DFE when the equalized symbol error rate is 0.2 in ATTC defined digital terrestrial TV broadcasting channels A, B and F, respectively.