Intelligent image sensor for heterogeneous smart camera architectures.

Abstract

In recent years, real-time image processing has gained an essential space in mobile devices, such as IoT endpoints, smartphones, and laptops. This interest is mainly fueled by the need to improve biometrics, such as recognizing different features from humans, security, such as detecting intruders in restricted areas, and safety, such as detecting pedestrians in assisted driving, among others. Mobile devices typically have two essential characteristics, a small form factor, to allow portability, and low energy consumption, to extend battery life. Designing and creating solutions to achieve real-time processing with those two characteristics is not straightforward; moreover if we consider that most image processing methods are not thought to be mobile-friendly. As a consequence, several researchers in the field of hardware design have focused on designing specific purpose circuits. Among these, we can find external digital coprocessors, next to the imager, and smart image sensors (SIS), that add extra circuitry to the imager (i.e., at pixel level) to capture and process the image in the same die. This thesis report presents the architecture of two smart imaging sensor for face recognition and motion-based object detection, two commonly used image-processing methods. The SISs are based on custom smart pixels capable of computing part of computer vision algorithms in the analog domain, and a respective digital coprocessor that performs the rest of the algorithm in the same die. On the one hand, the SIS for face recognition can compute local spatial gradi ents in the analog domain, on the smart pixel, and perform image classification on the digital coprocessor. The SIS uses spatial gradients to compute a lightweight version of local-binary patterns (LBP), which was named Ringed LBP (RLBP). The face-recognition method, which is based on Ahonen’s algorithm, operates in three stages: (1) it extracts local image features using RLBP, (2) it computes a feature vector using RLBP histograms, and (3) it projects the vector onto a subspace that maximizes class separation and classifies the image using a nearest neighbor criterion. On the other hand, the SIS for motion-based object detection can compute frame differences in the analog domain, on the smart pixel, and perform morphological opera tions and connected components to determine the bounding boxes of the detected objects on the digital coprocessor. The smart-pixel array implements on-pixel temporal difference computa tion using analog memories to detect motion between consecutive frames. The object detection SIS can operate in two modes: (1) as a conventional image sensor and (2) as a smart sensor which delivers a binary image that highlights the pixels in which movement is detected between consecutive frames and the object bounding boxes. The smart pixels were designed the smart ii pixel using a 0.18 µm and 0.35 µm mixed-signal CMOS processes. The evaluation of the perfor mance were performed using post-layout parasitic extraction. With a pixel-pitch of 32 µm × 32 µm, and considering the 0.18 µm and 0.35 µm processes, the fill factor of the face recognition smart pixel is 34% and 76%, respectively, and of 28% and 74% for the object detection smart pixel, respectively. The pixel array for face recognition operates at up to 556 frames per second. Implemented, validated and tested on a Xilinx XC7Z020 field-programmable gate array, the digital coprocessor achieves 96.5% classification accuracy on a database of infrared face images, can classify a 150 × 80-pixel image in 94 µs, and consumes 71 mW of power. On an array of 320 × 240 smart pixels, the object detection SIS operates at 60 frames per second. The digital coprocessor was implemented and validated on a Xilinx Artix-7 XC7A35T field-programmable gate array that can run at 125 MHz, detects objects in a frame in 0.614 µs, and has a power consumption of 58 mW. Thanks to the results obtained in this thesis, we can enumerate the following contributions. First, it is possible to design heterogeneous smart cameras that combine smart pixels based on intelligent readout circuits and a digital coprocessor in the same die. Using an array of smart pixels can contribute to exploiting the parallelism present in the image processing algorithms. Designing smart pixels based on intelligent readout circuits allows computing during capture time, thus, reducing processing time and memory resources. Finally, modifying and adapting the image processing algorithms while considering the SIS architecture can lead to simpler methods, mathematically, without significantly impacting results such as precision.