A stereo matching method simultaneously recovers disparities, colors and opacities from input images to reconstruct 3-dimensional surfaces depicted in the input images. The method includes formulating a general disparity space by selecting a projective sampling of a 3D working volume, and then mapping the input images into cells in the general disparity space. After computing statistics on the color samples at each cell, the method computes initial estimates of color and opacity from the statistics (e.g., the mean and variance of the color samples). The method uses opacity to compute visibility and then uses this visibility information to make more accurate estimates by giving samples not visible from a given camera less or no weight. The method uses the initial estimates as input to refining process which tries to match re-projected image layers to the input images.