A multi-display computer system comprises a host computer system that processes windowed desktop environments for multiple remote displays, multiple users or a combination of the two. For each display and for each frame, the multi-display processor responsively manages each necessary portion of a windowed desktop environment. The necessary portions of the windowed desktop environment are further processed, encoded, and where necessary, transmitted over the network to the remote display for each user. Embodiments integrate the multi-display processor with the graphics processing unit, network controller, main memory controller or a combination of the three. The encoding process is optimized for network traffic and special attention is made to assure that all users have low latency interactive capabilities.