Techniques and tools for using motion vector block patterns in video encoding and decoding are described. In general, a motion vector block pattern signals the presence or absence of motion vector data for a macroblock with multiple motion vectors. For example, a video decoder decodes variable length codes that represent motion vector block patterns. Each motion vector block pattern has one bit per corresponding luminance motion vector of a macroblock with multiple luminance motion vectors, where the one bit indicates whether or not motion vector data for the corresponding luminance motion vector is signaled. A video encoder performs corresponding encoding.