A video compression system and method for compressing video data for transmission or storage by reducing the temporal redundancy in the video data is described. A frame of video data is divided into a variable number of blocks of pixel data of varying size, and each block of data is compared to a window of pixel data in a reference frame of pixel data, typically the previous frame. A best matched block of pixel data is selected from the window of pixel data in the reference frame, and a displacement vector is assigned to describe the selected block location in the reference frame relative to the current block of pixel data. The number and size of the blocks of pixel data are permitted to vary, in order to adapt to motion discontinuities in the sequential frames of pixel data. This is to allow prediction blocks of pixel data in the current frame to be smaller in areas of high activity, while maintaining high levels of compression, achieved by using larger prediction blocks, in areas of the frame with low levels of activity. A frame of predicted pixel data is assembled from variable size blocks of prediction data and subtracted from the current frame of pixel data. Only the residual difference, the displacement vectors and an indication of the block sizes used in the prediction are needed for transmission or storage.