Jonathan Beall - Robin Schriebman - Whitney Buchanan
As a practice project to work on vision processing, we have adapted the OpenCV vision code to play the game Set.
In the game of set, twelve cards are layed out. Each card contains one, two or three objects of one of three colors (red, green, or blue/purple), one of three shapes (oval, diamond, and squiggle), and one of three patterns (empty, filled, or striped). A set is three cards that are either all different or all the same in each of the attributes. Identifying sets (once the cards have been identified) is a much easier task for a machine than a human, but machines find it much more difficult to perform the initial identification of the cards, making it an interesting problem. difficult to identify the individual cards.
We began using the OpenCV Set example provided by Professor Dodds. We then cut out much of the unnecesary code, including extraneous functions, any access to the camera, and the continual refreshing and re-processing of the images (as opposed to processing and displaying them once when they are loaded). We then integrated the improvements we had made earlier while writing our ball tracking code. This allowed us to correctly identify the color of the pixels in the images (using rectangular regions in HSV-space) and group them into blobs using cvBlobsLib.
This was a good start, identifying the color (indicated by the thin circle at the upper left) and number of the shapes. To identify which shapes the blobs were we looked at the top slice of the images. We then found the blobs composed of pixels identified as colored as well as any pixels below them. This eliminated differences in blob shape due to the fill of the shapes. The shapes were then identified by comparing the center of mass of the blob to the center of it's bounding box (the pink and pale blue diamonds in the above image). A large vertical separation indicated a diamond, horizontal separation, a squiggle, and neither, an oval (indicated by a red, green, or blue thick circle over the individual shape). We then examined a vertical stripe below the center of the blob and in the vertical middle of the image (the magenta box in the picture) and compared the minimum and maximum intensities of the pixels. High, low, and varying intensities corresponded to plain, filled, and striped fills (indicated by the thickness of the magenta box).
With a little tweaking of parameters, this algorithm was able to correctly identify all of the Set cards. After this, it was a simple matter to check for the existance of sets and indicate the first set found by circling all of the cards in red. All sets found are also printed to standard out.