CSx495 Computer Vision

Project 2 – Window-based Stereo Matching

In class and in Forsyth and Ponce, chapter 7 we discussed window-based approaches to estimating dense

stereo correspondence. In this project you will implement such approaches and evaluate it on some

standard stereo pairs.

As a reminder as to what you hand in: A Zip file that has

(1) Images (either as JPG or PNG or some other easy to recognize format) clearly labeled using the

convention PS<number>-<question number>-<question sub>-counter.jpg

(2) Code you used for each question. It should be clear how to run your code to generate the

results. Code should be in different folders for each main part with names like PS1-1-code. For

some parts – especially if using Matlab – the entire code might be just a few lines.

(3) Finally a PDF file that shows all the results you need for the problem set. This will include the

images appropriately labeled so it is clear which section they are for and the small number of

written responses necessary to answer some of the questions. Also, for each main section, if it

is not obvious how to run your code please provide brief but clear instructions.

1 There are a few images located in the project directory at:

http://www.cc.gatech.edu/~afb/classes/CS4495-Fall2013/ProblemSets/PS2/Data. For this first

problem we provided a test pair leftTest.png and rightTest.png that are just a central square moved 2

pixels to the right.

Implement the basic stereo algorithm of taking a window around every pixel in one image and

search for the best match along the same scan line in the other image. You will do this both left to

right and right to left. Remember: because of depth changes (discontinuities) some pixels visible in

the left image are not in the right and vice a versa. So you will match in both directions.

For this part implement the simplest thing imaginable: look for the smallest difference between the

template window (source) and the proposed location window. Use the sum of squared differences

measure (SSD). I am going to take the definitions from:

http://software.intel.com/sites/products/documentation/hpc/ipp/ippi/ippi_ch11/ch11_image_proxi

mity_measures.html

SSD is defined by:

Basically you just sum up the squares. A “good” match, then, is when this value is at a minimum.

That is, you are looking for the same image patch in both images. YOU MAY NOT USE BUILT-IN SSD

FUNCTIONS THAT BASICALLY DO THIS ENTIRE STEP. But it is easy in Matlab to sum the squares of a

matrix; check out the operator: .^2 (Yes you need the ‘dot’.)

a) Implement the SSD match algorithm, and create a disparity image D(x,y) such that L(x,y) =

R(x+D(x,y),y) when matching from left to right. Also match from right to left.

Output: Show DL(x,y) [matching from left to right] and DR(x,y)[matching from right to left] as

images. These disparity images may need to be scaled and shifted to display correctly. They

should show a central square moved 2 pixels to the left or right.

2 Now we’re going to try this on a real image pair: proj2-pair1-L .png and proj2-pair1-R .png. Since

these are color images create gray scale versions. You can use rgb2gray or your own function.

a) Again apply your SSD match algorithm, and again create a disparity image D(x,y) such that

L(x,y) = R(x+DL(x,y),y) when matching from left to right. Also match from right to left.

Output: Show DL(x,y) [matching from left to right] and DR(x,y)[matching from right to left] as

images. These disparity images may need to be scaled and shifted to display correctly.

b) Also in this directory are the ground truth disparity images proj2-pair1-Disp-L .png and

proj2-pair1-Disp-R .png. Compare your results.

Output: description of the differences between your results and the ground truth

3 SSD is not very robust to certain perturbations. We’re going to try to see the effect of perturbations:

a. Add some Gaussian noise to the image; either one or both. Make the noise sigma big

enough that you can tell some noise has been added. Run SSD again.

Output: Disparity images and analysis of result compared to part 1.

b. Instead of the Gaussian noise, increase the contrast (multiplication) of one of the images by

just 10%. Run SSD again.

Output: Disparity images and analysis of result compared to part 1.

4 Now you’re going to use (not implement yourself unless you want) an improved method, called

normalized correlation – this is discussed in the book. The basic idea is that we think of two image

patches as vectors and to compute the angle between them – much like normalized dot products.

This can be written as:

where the above is just the explicit dot product which then has to be normalized by the magnitude

of each:

a) Implement a window matching stereo algorithm using some form of normalized correlation.

Matlab has its own function normxcorr2(template, A) which implements:

OpenCV has a variety of relevant functions as well such as CV_TM_CCOEFF_NORMED. You

MAY use these built in normalized correlation functions.

Test it on the original images both left to right and right to left.

Output: disparity images and description of how it compares to the SSD version and to the

ground truth

b) Now test it on both the Gaussian noise and contrast boosted versions from 2a and 2b.

Output: Again disparity images and analysis of results.

5 Finally, there is a second pair of images: proj2-pair2-L .png and proj2-pair2-R .png

a. Try your algorithms on this pair. Play with the images – smooth, sharpen, etc. Keep

comparing to the ground truth.

Output: disparity images and analysis of what it takes to make stereo work using a window

based approach.

Sale!