A method for tracking motion from field to field in a sequence of related video broadcast images. The method uses template correlation to follow a set of predetermined landmarks within a scene in order to provide position information of objects in the current image. The current image object position information is compared to position information of the same objects within a reference array data table. The comparison is accomplished through the use of warp equations that map points in the current image to points in the reference array. Motion is tracked according to a velocity prediction scheme utilizing a weighted formula that emphasizes the weight of landmarks that are closer to their predicted position.