CS 152: Neural Networks - Final Project: Cellular Neural Networks

	Cellular Neural Networks	Max Pflueger CS 152: Neural Networks Final Project Page
Presentations: First Second Referenced Papers: Ocean Modeling High Speed Image Processing CNN Tech in Action Face and Eye Detection Analog Implementation of Discrete Time CNNs Gradient Computation	Project Proposal I will be doing a survey of techniques and applications of cellular neural networks (CNN) and the CNN Universal Machine (CNN-UM). This survey will include both discrete and continuous time CNNs used in a variety of applications. CNN Background and Theory Take a look at the powerpoint slides from my first presentation: here Cellular neural networks are a variation on Hopfield networks. Hopfield nets are a recurrent model where the input to each neuron consists of the input data to the net as well as the output from every other neuron. CNNs modify that by giving the cells (a nomenclature change from neurons) a spatial arrangement, and making the input to each cell be the input to the net combined with the output of the cells in spatial proximity to the cell in question. This arrangement makes it possible to implement CNNs containing thousands of cells in hardware on a single chip. The behavior of an individual cell can be described by the differential equation below: A detailed definition of all the variables is in my powerpoint presentation, but the relevant details are that A and B are the feedback and input synaptic operators respectively, and z is the threshold. Those three variables together can be interpreted at the template for a CNN, as they define how it will behave for any input and initial state. The CNN becomes much more powerful when you allow it to change templates during execution. This allows templates to behave much like instructions to a traditional microprocessor. This device is called a CNN Universal Machine, and with the right program it can perform very complex image processing tasks. It is notable that theoretical results have shown the CNN-UM to be Turing complete (in the weak sense of assuming it has infinite memory). Techniques and Applications Ocean Modeling CNNs hold a lot of promise for working on fluid mechanics problems. Exact solutions to fluid mechanics problems require solutions to systems of partial differential equations. In the vast majority of cases, analytical solutions do not exist. When a precise solution is necessary for some application, usually the only option available is a numerical solution, unfortunately numerical solutions are very computationally expensive. CNNs may offer a fast new way to solve fluid dynamics problems. Nagy and Szolgay designed a simulation of a CNN-UM with modified cell architecture to model ocean currents. They ran their simulation on an Athlon XP 1800+ and an XC2V1000 FPGA (this is a mid-size FPGA). Running time to complete the simulation on the Athlon was 56 minutes, this compares with 41 seconds to complete the simulation on the FPGA. The authors speculated that if one were to do this simulation with the largest currently (2004) available FPGA the simulation could be completed in just 1 second, a 3300 time speed up over the Athlon. Based on this data the potential benefits of CNNs to computational fluid mechanics seem substantial. High Speed Image Processing Most CNN-UM chips on the market take input projected visually onto the chip itself. This suggests the possibility of doing parallel image processing on the chip, and with CNN-UM architecture, this processing can be very fast. Zarandy, Csapodi and Roska conducted a demonstration of the capabilities of a CNN-UM chip have having it discriminate between one of 6 different images projected onto the chip. The input to the chip was one bit per pixel, and the monochrome base images were mounted on a wheel that could be spun to present the images quite quickly. Using this setup they were able to achieve identification of the image at 10,000 frames per second, but were limited by their presentation hardware. The authors speculate that an optimized test setup could go as high as 50,000 frames per second, though this speed would depend on the complexity of the recognition program. The image sensor in this paper had a resolution of 22x24, which is quite low. Chips exist now with much higher resolutions (128x128 and larger), and it is important to note that processing time (and hence frame rate) would not change by going to a higher resolution chip. The authors speculated on some possible industrial applications including robot visual arm control and part positioning in SMD mounting. Face and Eye Detection Detecting faces is a classic and important problem in computer science that may be well suited for the capabilities of a CNN. In "Face and Eye Detection by CNN Algorithms" Balya and Roska designed a CNN algorithm for recognizing and normalizing faces from color images. Their system used a series of programs for a CNN-UM. The first step identifies the area of the image that may be a face by exploiting the unique color and texture of skin tones in an image. The second step finds the eyes by using their unique structure and specifying that they must be within the face and horizontal from each other. Then, by using the eyes as a reference the face can be normalized. Their algorithm was shown to be quite reliable on a standard database, and the response time of the algorithm was only 3 ms, which would make a hardware implementation suitable for real-time applications. Analog Implementation of Discrete Time CNNs All of the CNN models I have discussed so far are based on continuous time CNNs, but there is also a model for doing CNNs in discrete time. Analog electronics are well suited to implement continuous time CNNs, but Harrer and Nossek showed that it is also possible to do discrete time CNNs in analog hardware. They fabricated and tested their design for a small 4x4 hexagonal network and showed that it could be easily extended to larger sizes or different cell geometries. Gradient Computation for Training As with other neural computation techniques, it is possible and desirable to be able to do learning with a CNN. However, most popular training techniques require one to be able to compute the gradient of the error function for the network. This is a non-trivial calculation in a CNN. Brendel, Roska, and Bártfai published a paper titled "Gradient Computation of Continuous-Time Cellular Neural/Nonlinear Networks with Linear Templates via the CNN Universal Machine" that described the mathematics of how to compute this gradient. In addition to describing the math necessary to compute the gradient, they also show that for a given network, the equations describing the gradient had the same neighborhood and connectivity as the original network. Therefore, they show how one could use a CNN to compute a gradient quickly. They suggest that it would be possible to do on-line training very quickly for a CNN by allowing it to compute its own gradient. References Balya, D. and Roska, T. 1999. "Face and Eye Detection by CNN Algorithms." J. VLSI Signal Process. Syst. 23, 2-3 (Nov. 1999), 497-511. Brendel, M., Roska, T., and Bártfai, G. 2002. "Gradient Computation of Continuous-Time Cellular Neural/Nonlinear Networks with Linear Templates via the CNN Universal Machine." Neural Process. Lett. 16, 2 (Oct. 2002), 111-120. Chua, Leon O. and Tamás Roska. Cellular Neural Networks and Visual Computing. Cambridge: Cambridge University Press, 2002. Harrer, H.; Nossek, J.A.; Stelzl, R., "An analog implementation of discrete-time cellular neural networks ," Neural Networks, IEEE Transactions on , vol.3, no.3pp.466-476, May 1992. Nagy, Z.; Szolgay, P., "Emulated digital CNN-UM implementation of a barotropic ocean model," Neural Networks, 2004. Proceedings. 2004 IEEE International Joint Conference on, vol.4, no.pp. 3137- 3142 vol.4, 25-29 July 2004. Zarandy, A.; Csapodi, M.; Roska, T., "20 ?sec focal plane image processing," Cellular Neural Networks and Their Applications, 2000. (CNNA 2000). Proceedings of the 2000 6th IEEE International Workshop on , vol., no.pp.267-271, 2000. Zarandy, A.; Espejo, S.; Foldesy, P.; Kek, L.; Linan, G.; Rekeczky, C.; Rodriguez-Vazquez, A.; Roska, T.; Szatmari, I.; Sziranyi, T.; Szolgay, P., "CNN technology in action," Cellular Neural Networks and Their Applications, 2000. (CNNA 2000). Proceedings of the 2000 6th IEEE International Workshop on , vol., no.pp.79-81, 2000.