A process for up-converting an existing video source signal having a low frequency (frames/second) to a high frequency signal for use with High Definition Television (HDTV). The process samples the existing frames in the existing video signal and calculates integer displacements of pels within the existing frames. A polynomial curve fit is then performed on the displacements to obtain estimates of horizontal and vertical displacements of each block in each existing frame. Based on the alignment of the blocks within a sampling grid on each frame, the blocks are segregated into groups. The block groups are then used to interpolate missing or required frames of the high frequency signal in a piecemeal manner by utilizing blocks of a particular block group to estimate a corresponding block in a frame of the high frequency signal.