An analog video tagging and encoding system is disclosed. The invention inserts an analog tag frame into the analog video stream such that the luminance (Y) values are made to approach zero, i.e., black, allowing a potential tag frame to be quickly identified. Tag information is then encoded as a structured color pattern into the chrominance (Cb and Cr) portions of the frame, making the tag itself invisible to the casual viewer and contains program start/stop information, program segment identification, or data downloads. The color pattern is chosen such that when it is converted into digital form according to the Moving Picture Experts Group (MPEG) standard. In a simple embodiment of the invention, the structured analog color pattern may appear as a pattern of colored blocks aligned with the macroblock segmentation performed on the analog signal by the MPEG encoding process. The simplest color pattern uses two colors with extremely different Cr or Cb values, and derives a binary encoding sequence where a binary zero (0) might be represented by values near one color, and a binary one (1) represented by values near the other color. This is easily extended to create a number of different chrominance levels within each encoded macroblock, allowing multiple bits of tag information to be encoded by each macroblock, where the number of chrominance levels available for data encoding are limited by the transmission accuracy and color distortion caused by the transmission process.