WO2012117583A1

WO2012117583A1 - Color imaging device

Info

Publication number: WO2012117583A1
Application number: PCT/JP2011/067419
Authority: WO
Inventors: 田中　誠二
Original assignee: 富士フイルム株式会社
Priority date: 2011-02-28
Filing date: 2011-07-29
Publication date: 2012-09-07
Also published as: EP2683166A4; JP5054857B1; US8531563B2; CN103974044B; RU2551649C2; EP2683166B1; JP5872407B2; US20130308022A1; CN102870417B; EP2683166A1; RU2013138394A; US8704922B2; JPWO2012117583A1; US20120293695A1; CN103974044A; BR112012027306A2; JP2013048409A; CN102870417A

Abstract

A color imaging device uses a single-plate color imaging element obtained through the positioning of a color filter having a color filter arrangement in which filters of all RGB colors are positioned periodically within each line in the horizontal and vertical directions. A weighted-average filter is used that has filter factors set in a manner such that the proportions of the total filter factor for each color within each line in the horizontal and vertical directions are equal, and the weighted average of the pixel values of the pixels in a mosaic image outputted from the color imaging element is calculated for each color. Also, when calculating the pixel value for another color at the pixel location of the pixel to be subjected to simultaneous processing in the center section of the weighted-average filter, the pixel value of the other color is estimated by interpolating the pixel value of the pixel to be subjected to simultaneous processing, by using the color proportions or color difference of the calculated weighted-average value.

Description

Color imaging device

The present invention relates to a color imaging apparatus, and more particularly to a color imaging apparatus capable of suppressing the occurrence of color moire.

In a color imaging device having a single-plate color imaging device, since the output image from the color imaging device is a RAW image (mosaic image), processing for interpolating missing color pixels from surrounding pixels (simultaneous) Multi-channel images are obtained by processing). In this case, the problem is the reproduction characteristics of high-frequency image signals.

In the primary color Bayer array, which is the most widely used color array in a single-plate color image sensor, green (G) pixels are arranged in a checkered pattern, and red (R) and blue (B) are line-sequentially arranged. Therefore, there is a problem that low-frequency coloring (color moire) occurs due to the folding of the high-frequency signal exceeding the reproduction band of each color and the phase shift of each color.

For example, when a black and white vertical stripe pattern (high-frequency image) as shown in FIG. 10A is incident on an image sensor having the Bayer arrangement shown in FIG. 10B, the color is assigned to the Bayer color arrangement. Compared to each other, as shown in FIGS. 10C to 10E, R is a light flat, B is a dark flat, and G is a light and shaded mosaic image, which is originally a monochrome image, Although there is no density difference (level difference) between RGB, depending on the color arrangement and the input frequency, a color is added.

Generally, in a color image pickup apparatus using a single-plate color image pickup device, an optical low-pass filter made of a birefringent material such as crystal is disposed in front of the color image pickup device, and this is avoided by optically dropping a high frequency. However, with this method, coloring due to folding of the high-frequency signal can be reduced, but there is a problem that the resolution is lowered due to its adverse effect.

In order to solve such a problem, the color filter arrangement of the color image sensor is determined based on an arrangement restriction condition in which any target pixel is adjacent in any one of three colors including the color of the target pixel and four sides of the target pixel. A color image sensor having a three-color random array that satisfies the above has been proposed (Patent Document 1).

In addition, a plurality of filters having different spectral sensitivities are provided, and the first filter and the second filter are alternately arranged at a first predetermined period in one diagonal direction of the pixel grid of the image sensor. At the same time, an image sensor having a color filter array alternately arranged at the second predetermined period in the other diagonal direction has been proposed (Patent Document 2).

Furthermore, an image pickup apparatus is proposed that includes a color image pickup device in which R and B of the three primary colors of RGB are arranged every three pixels in the horizontal and vertical directions, and G is arranged between these R and B. (Patent Document 3). The color image pickup device described in Patent Document 3 is arranged so that the G pixel that contributes most to obtain the luminance signal is much more than the RB pixel because the resolution of the color difference signal may be lower than the luminance signal. Thus, the horizontal and vertical resolutions can be increased.

JP 2000-308080 A JP 2005-136766 A JP-A-8-23543

The three-color random arrangement described in Patent Document 1 is effective for low-frequency color moire, but is not effective for false colors in the high-frequency part.

On the other hand, in the color filter array of the image sensor described in Patent Document 2, R, G, and B filters are periodically arranged in the horizontal and vertical lines of the color filter array. In the described invention, when the mosaic image output from the image sensor having the color filter array is synchronized, a local region having a predetermined image size is extracted around the target pixel, and the target pixel in the local region is extracted. Calculates statistics on the color distribution shape of the color and the color distribution shape of the other color to be estimated, and linearly regresses the color distribution shape based on the color intensity at the target pixel position and the color distribution shape statistics Thus, the estimated value of the other color of the target pixel position is calculated. The invention described in Patent Document 2 requires a calculation of a statistic (covariance value) relating to a color distribution shape and a regression calculation process, and has a problem that image processing becomes complicated.

Also, the color image sensor described in Patent Document 3 is not effective for false colors in the high-frequency part in the horizontal or vertical direction because there are lines of only G pixels in the horizontal or vertical direction.

The present invention has been made in view of such circumstances, and an object of the present invention is to provide a color imaging apparatus capable of suppressing the generation of false colors in a high-frequency part by simple image processing.

In order to achieve the above object, an invention according to one aspect of the present invention is directed to a method in which filters of all colors are arranged in a horizontal direction and a vertical direction on a plurality of pixels including photoelectric conversion elements arranged in a horizontal direction and a vertical direction. A single-plate color image pickup element in which color filters of a color filter array periodically arranged in each line are arranged, an image acquisition unit for acquiring a mosaic image from the color image pickup element, and a predetermined filter coefficient A weighted average filter having a relationship between the color of each pixel in the local area extracted from the mosaic image corresponding to the weighted average filter and the filter coefficient in a color in each line in the horizontal and vertical directions A weighted average filter set so that the ratio of the sum of the filter coefficients for each is equal, the filter coefficient of the weighted average filter, and the weighted average filter corresponding to the weighted average filter A weighted average calculating unit that calculates a weighted average value for each color based on a pixel value of each pixel in a local region extracted from a zyke image, and a pixel of a target pixel for synchronization processing at the center of the weighted average filter A synchronization processing unit for calculating a pixel value of another color at a position, the pixel of the target pixel based on a color ratio or color difference of the calculated weighted average value of the color of the target pixel and the other color A synchronization processing unit that interpolates values and calculates pixel values of other colors, and a local region that is extracted from the mosaic image corresponding to the weighted average filter is moved for each target pixel unit of the synchronization processing And a control unit that repeatedly operates the weighted average calculation unit and the synchronization processing unit.

According to the color imaging device of one aspect of the present invention, a single color filter array in which all color filters are periodically arranged in the horizontal and vertical lines is provided. Since a plate-type color image sensor is used, for each color in each line in the horizontal and vertical directions as the filter coefficient for the color of each pixel in the local area extracted from the mosaic image corresponding to the weighted average filter The sum of the filter coefficients can be set to be equal.

The filter coefficient of the weighted average filter is equal to the ratio of the sum of each color in any line in the horizontal and vertical directions. Therefore, the filter coefficient is multiplied regardless of the frequency input in the horizontal and vertical directions. The resulting color relationship does not shift, and coloring due to high-frequency folding does not occur. That is, the weighted average value for each color calculated based on the filter coefficient of the weighted average filter and the pixel value of each pixel in the local region extracted from the mosaic image corresponding to the weighted average filter is Whatever frequency is input to the local area in the horizontal and vertical directions, the correct color in the local area is indicated. Therefore, when calculating the pixel value of the other color at the pixel position of the target pixel of the target pixel in the central processing of the weighted average filter, the pixel value of the target pixel is calculated based on the color ratio or color difference of the calculated weighted average value. By interpolating, the pixel values of other colors can be estimated with high accuracy.

In the color imaging device according to another aspect of the present invention, the color filter array of the color imaging element includes a first filter corresponding to the first color that contributes most to obtain a luminance signal and other than the first color. And a second filter corresponding to the second color corresponding to the second color or more, and the basic array pattern is repeatedly arranged in the horizontal direction and the vertical direction, and corresponds to the first filter. The ratio between the number of pixels of the first color and the number of pixels of each color of the second color corresponding to the second filter is made different. That is, even if the ratio between the number of pixels of the first color corresponding to the first filter and the number of pixels of each color of the second color corresponding to the second filter is different, the weighted average filter The filter coefficients have the same ratio of the sum of the filter coefficients for each color in both the horizontal and vertical lines, so that the color relationship as a result of applying the filter coefficients does not shift, and coloring due to high-frequency folding occurs. There is nothing.

In the color imaging device according to still another aspect of the present invention, the ratio of the number of pixels of the first color corresponding to the first filter is a pixel of each color of the second color corresponding to the second filter. It is preferably larger than the ratio of the numbers. That is, since the ratio of the number of pixels of the first color that contributes most to obtain the luminance signal is made larger than the ratio of the number of pixels of each color of the second color corresponding to the second filter, Singing can be suppressed and high frequency reproducibility is also good.

In the color imaging device according to still another aspect of the present invention, it is preferable that the weighted average filter is a filter that is weighted so that a filter coefficient at the center is increased. Thereby, the color at the pixel position of the target pixel of the synchronization process can be obtained with high accuracy.

In the color imaging device according to still another aspect of the present invention, it is preferable that the weighted average filter has filter coefficients that are left-right symmetric, up-down symmetric, and point symmetric. Thereby, when extracting a local area | region from the said mosaic image and calculating the weighted average for every color, even if it moves the local area | region to extract, the same weighted average filter can be used.

In the color imaging device according to still another aspect of the present invention, the color filter includes an R filter, a G filter, and a B filter corresponding to red (R), green (G), and blue (B) colors. A weighted average value for each color of pixel values of R, G, and B pixels corresponding to the R filter, G filter, and B filter calculated by the weighted average calculation unit, arranged in a filter array, is Rf, Assuming Gf and Bf, the synchronization processing unit, when the target pixel of the synchronization processing is a G pixel and the pixel value is G, the pixel values R and B of the R and B pixels at the position of the target pixel With the following formula:
R = G × (Rf / Gf), B = G × (Bf / Gf)
When the target pixel of the synchronization processing is an R pixel and the pixel value is R, the pixel values G and B of the G and B pixels at the target pixel position are expressed by the following equations:
G = R × (Gf / Rf), B = R × (Bf / Rf)
When the target pixel of the synchronization processing is a B pixel and the pixel value is B, the pixel values G and R of the G and R pixels at the position of the target pixel are expressed by the following equations:
G = B × (Gf / Bf), R = B × (Rf / Bf)
It is made to calculate by.

The ratio of the weighted average value (Rf, Gf, Bf) for each color in the local region indicates the RGB ratio (color ratio) of the original color at the pixel position of the target pixel in the local region. Thus, by interpolating the pixel value at the position of the target pixel, the pixel values of other colors can be accurately estimated.

In the color imaging device according to still another aspect of the present invention, the color filter includes an R filter, a G filter, and a B filter corresponding to red (R), green (G), and blue (B) colors. A weighted average value for each color of pixel values of R, G, and B pixels corresponding to the R filter, G filter, and B filter calculated by the weighted average calculation unit, arranged in a filter array, is Rf, Assuming Gf and Bf, the synchronization processing unit, when the target pixel of the synchronization processing is a G pixel and the pixel value is G, the pixel values R and B of the R and B pixels at the position of the target pixel With the following formula:
R = G + (Rf−Gf), B = G + (Bf−Gf)
When the target pixel of the synchronization processing is an R pixel and the pixel value is R, the pixel values G and B of the G and B pixels at the target pixel position are expressed by the following equations:
G = R + (Gf−Rf), B = R + (Bf−Rf)
When the target pixel of the synchronization processing is a B pixel and the pixel value is B, the pixel values G and R of the G and R pixels at the position of the target pixel are expressed by the following equations:
G = B + (Gf−Bf), R = B + (Rf−Bf)
It is made to calculate by.

The difference in the weighted average value (Rf, Gf, Bf) for each color in the local area indicates the RGB difference (color difference) of the original color at the pixel position of the target pixel in the local area. By interpolating the pixel values at the positions, the pixel values of other colors can be estimated with high accuracy.

In the color imaging device according to still another aspect of the present invention, the color filter includes an R filter, a G filter, and a B filter corresponding to red (R), green (G), and blue (B) colors. The filter array is a first array corresponding to 3 × 3 pixels, and a G filter is arranged at the center and four corners, and a B filter is arranged above and below the center G filter. Are arranged, and a second array corresponding to 3 × 3 pixels, with G filters arranged at the center and four corners, with the center G filter interposed therebetween. And a second array in which R filters are arranged on the top and bottom and B filters on the left and right are alternately arranged in the horizontal direction and the vertical direction, and the weighted average filter has a kernel size of 9 × 9 The control unit has The weighted average the filter first sequence or the second sequence so that the order weighted average repeating the calculation unit and the synchronization unit is operated while sequentially moved so that the center.

The first array and the second array both have color filters that are symmetrical vertically and horizontally, and the first array and the second array are simply replaced by an R filter and a B filter. It is. Therefore, when processing while moving the weighted average filter every 3 × 3 pixels, the total sum of the filter coefficients for each color in each line in the horizontal and vertical directions can be obtained without changing the filter coefficient of the weighted average filter. The ratio can be made equal.

According to the present invention, the filter coefficient for the color of each pixel in the local region extracted from the mosaic image corresponding to the weighted average filter is the sum of the filter coefficients for each color in each line in the horizontal and vertical directions. The ratio is set to be equal, and the weighted average value for each color based on the filter coefficient of the weighted average filter and the pixel value of each pixel in the local area extracted from the mosaic image corresponding to the weighted average filter Therefore, the color ratio or color difference of the weighted average value for each color in the local area indicates the color ratio or color difference of the original color at the pixel position of the target pixel in the local area. Thereby, the pixel value of another color can be accurately estimated by interpolating the pixel value of the target pixel based on the color ratio or color difference of the calculated weighted average value.

Note that no matter what frequency is input to the local area in the horizontal and vertical directions, the color ratio or the color difference of the weighted average value for each color in the local area does not change, which may cause erroneous color determination. Therefore, the generation of false colors in the high-frequency part can be suppressed, and the pixel values of other colors can be estimated by a simple calculation such as weighted average or interpolation.

FIG. 1 is a block diagram showing an embodiment of a color imaging device according to the present invention; FIG. 2 is a diagram showing a color filter array of color filters provided in the color image sensor of the first embodiment; FIG. 3 is a diagram showing a basic array pattern included in the color filter array of the color image sensor of the first embodiment; FIG. 4 is a diagram showing a state in which the basic array pattern of 6 × 6 pixels included in the color filter array of the color image sensor of the first embodiment is divided into an A array and a B array of 3 × 3 pixels, and these are arranged. Is; FIG. 5 is a diagram showing a weighted average filter applied to the color image sensor of the first embodiment; FIG. 6A is a diagram showing an image when there is a high frequency input of vertical stripes; FIG. 6B is a diagram used to explain that color shift does not occur in the weighted average color that has been subjected to the weighted average filter when high-frequency vertical stripes are input; FIG. 7 is a diagram showing a second embodiment of a color imaging device and a weighted average filter applied to the present invention; FIG. 8 is a diagram showing a third embodiment of a color image sensor applied to the present invention; FIG. 9A is a diagram showing a third embodiment of a weighted average filter applied to the color image sensor of the third embodiment; FIG. 9B is a diagram showing a weighted average filter applied when the 6 × 6 pixel local region shown in FIG. 9A is moved by two pixels in the horizontal direction; FIG. 10 is a diagram used for explaining a problem of a color image sensor having a color filter with a conventional Bayer array.

Hereinafter, preferred embodiments of the color imaging apparatus according to the present invention will be described in detail with reference to the accompanying drawings.

[Overall configuration of color imaging device]
FIG. 1 is a block diagram showing an embodiment of a color imaging apparatus according to the present invention.

The subject is imaged by the photographing optical system 10, and a light image indicating the subject image is formed on the light receiving surface of the color image sensor 12 (the color image sensor of the first embodiment).

The color image sensor 12 includes a plurality of pixels (not shown) made of photoelectric conversion elements arranged in a horizontal direction and a vertical direction (two-dimensional arrangement), and a predetermined color filter arranged on the light receiving surface of each pixel. This is a single-plate type color image pickup device composed of an array of color filters. Note that the color filter array of the color image pickup device 12 is such that filters of all colors of red (R), green (G), and blue (B) are periodically arranged in the horizontal and vertical lines. The details will be described later.

The subject image formed on the color image sensor 12 is converted into a signal charge corresponding to the amount of incident light by the photoelectric conversion element. The signal charge accumulated in each photoelectric conversion element is sequentially read out from the color imaging element 12 as a voltage signal (image signal) corresponding to the signal charge based on a drive pulse given from the drive unit 18 according to a command from the control unit 20. The image signals read from the color image sensor 12 are R, G, and B signals that indicate R, G, and B mosaic images corresponding to the color filter array of the color image sensor 12. The color image sensor 12 is not limited to a CCD (Charge-Coupled Device) color image sensor, but may be another type of image sensor such as a CMOS (Complementary Metal Oxide Semiconductor) image sensor.

The image signal read from the color imaging device 12 is input to the imaging processing unit 14. The imaging processing unit 14 includes a correlated double sampling circuit (CDS) for removing reset noise included in the image signal, an AGC circuit for amplifying the image signal and controlling it to a certain level, and an A / D A conversion unit is included, and the input image signal is subjected to correlated double sampling processing and amplified, and thereafter, the RAW data converted into a digital image signal is output to the image processing unit 16.

The image processing unit 16 includes a white balance correction circuit, a gamma correction circuit, and a synchronization processing circuit according to the present invention (all RGB colors for each pixel from an RGB mosaic image associated with the color filter array of the single-plate color image sensor 12). Processing circuit for calculating information (converting into simultaneous equations), luminance / color difference signal generation circuit, contour correction circuit, color correction circuit, and the like, and in accordance with instructions from the control unit 20, the mosaic image input from the imaging processing unit 14 The RAW data is subjected to required signal processing to generate image data (YUV data) composed of luminance data (Y data) and color difference data (Cr, Cb data).

The image data generated by the image processing unit 16 is subjected to compression processing conforming to the JPEG standard for still images by a compression / decompression processing circuit, and compression processing conforming to the MPEG2 standard for moving images. After that, it is recorded on a recording medium (memory card), and is output and displayed on a display means (not shown) such as a liquid crystal monitor.

The details of the processing contents by the synchronization processing circuit according to the present invention in the image processing unit 16 will be described later.

<Characteristics of color filter array>
The color filter array of the color image sensor 12 has the following features (1), (2), and (3).

(Feature (1))
FIG. 2 is a diagram showing a color filter array of color filters provided in the color image sensor 12. As shown in FIG. 2, the color filter array of the color image sensor 12 includes a basic array pattern P (pattern indicated by a thick frame) composed of a square array pattern corresponding to 6 × 6 pixels. It is repeatedly arranged in the horizontal and vertical directions. That is, in this color filter array, R, G, and B color filters (R filter, G filter, and B filter) are arrayed with a predetermined periodicity.

Since the R filter, G filter, and B filter are arranged with a predetermined periodicity in this way, when performing R, G, B signal readout processing from the color image sensor 12, processing is performed according to a repetitive pattern. It can be performed.

(Feature (2))
In the color filter array shown in FIG. 2, filters of all colors of R, G, and B are arranged in each line in the horizontal direction and the vertical direction.

FIG. 3 shows a state in which the basic array pattern P shown in FIG. 2 is divided into 4 × 3 × 3 pixels.

As shown in FIG. 4, the basic array pattern P includes a 3 × 3 pixel A array surrounded by a solid frame and a 3 × 3 pixel B array surrounded by a broken frame as shown in FIG. It can also be understood that the array is arranged alternately in the vertical direction.

In the A array and the B array, G filters, which are luminance system pixels, are arranged at the four corners and the center, and are arranged on both diagonal lines. In the A arrangement, the R filter is arranged in the horizontal direction with the central G filter interposed therebetween, and the B filter is arranged in the vertical direction. On the other hand, in the B arrangement, the B filter is arranged in the horizontal direction with the central G filter interposed therebetween. The R filters are arranged in the vertical direction. That is, in the A array and the B array, the positional relationship between the R filter and the B filter is reversed, but the other arrangements are the same.

(Feature (3))
The basic array pattern of the color filter array shown in FIG. 2 is point-symmetric with respect to the center of the basic array pattern (the centers of the four G filters). As shown in FIG. 3, the A array and the B array in the basic array pattern are also point-symmetric with respect to the central G filter.

(Feature (4))
The basic arrangement pattern of the color filter array shown in FIG. 2 is that the number of R, G, and B pixels corresponding to the R, G, and B filters in the basic arrangement pattern is 8 pixels, 20 pixels, and 8 pixels, respectively. It has become. That is, the ratio of the number of pixels of RGB pixels is 2: 5: 2, and the ratio of the number of G pixels that contributes most to obtain a luminance signal is the ratio of R pixels and B pixels of other colors. It is larger than the ratio of the number of pixels.

[Weighted average filter used in the synchronization processing circuit of the image processing unit 16]
FIG. 5 is a diagram showing an embodiment of a weighted average filter used in the synchronization processing circuit of the image processing unit 16, and particularly shows filter coefficients of the weighted average filter.

As shown in FIG. 5, this weighted average filter (the weighted average filter of the first embodiment) has a 9 × 9 kernel size, and the filter coefficients shown in FIG. 5 are set.

That is, the filter coefficient of the weighted average filter is obtained by extracting a 9 × 9 pixel local area from the mosaic image obtained from the color image sensor 12 so that the A array is in the center, and changing the color of each pixel in the local area. Correspondingly, when the filter coefficients are extracted for each color and the sum of the filter coefficients for each color is obtained, the ratio of the sum of the filter coefficients for each RGB color in each line in the horizontal and vertical directions is equal ( The filter coefficient is set so as to be 1: 1: 1.

For example, in FIG. 5, the filter coefficient in the top row is 0,2,1,1,4,1,1,2,0, but when the sum is calculated for each color, R = 4, G = 0 + 1 + 1 + 1 + 1 + 0 = 4, and B = 2 + 2 = 4, showing that the relationship is 4: 4: 4 = 1: 1: 1. All rows and columns (horizontal and vertical lines) have filter coefficients that satisfy this relationship.

In addition, this weighted average filter has a large 3 × 3 filter coefficient in the center when comparing the filter coefficients for each region divided into 3 × 3 sizes, and then 3 × 3 on the top, bottom, left and right sides of the center. The filter coefficients are weighted so that the filter coefficients are large and the 3 × 3 filter coefficients at the four corners are the smallest.

Furthermore, the filter coefficient is set so that this weighted average filter is left-right symmetric, up-down symmetric, and point symmetric.

When the weighted average value for each RGB is calculated based on the weighted average filter having the above configuration and the pixel value of each pixel in the 9 × 9 pixel local region extracted from the mosaic image, the color based on the weighted average value for each RGB is Regardless of the frequency input in the horizontal and vertical directions, no color shift occurs and coloring due to high-frequency folding does not occur.

For example, when there is a high frequency input of vertical stripes shown in FIG. 6A, the sum of the filter coefficients of each color is 32:32:32 (see FIG. 6B), and it can be seen from the color ratio that it is black and white.

[Synchronization processing by the synchronization processing circuit of the image processing unit 16]
Next, a description will be given of a method of performing an RGB mosaic image synchronization process using the synchronization processing circuit of the image processing unit 16.

As shown in FIG. 5, a local area of 9 × 9 pixels is extracted from the mosaic image obtained from the color image sensor 12 so that the A array is in the center, and the pixel value of each pixel in the local area and the weight A weighted average value for each color of RGB is calculated based on the filter coefficient of the average filter. That is, the pixel value of each pixel in the local area is multiplied by the filter coefficient of the weighted average filter at each pixel position, the multiplication result is added for each color to obtain the sum for each color, and the sum for each color is further calculated. The weighted average value is calculated by dividing by 64. Note that 64 is the total sum of the filter coefficients for each RGB of the weighted average filter.

Next, a ratio (color ratio) of the RGB weighted average values is calculated from the RGB weighted average values calculated as described above. Then, the central 3 × 3 pixel (the pixel in the thick frame shown in FIG. 5) in the local region of 9 × 9 pixels is the target pixel of the synchronization process, and the pixel value at the pixel position of the target pixel is Interpolation is performed based on the calculated color ratio, and pixel values of other colors at the pixel position are calculated.

Specifically, when the calculated weighted average value for each RGB color is Rf, Gf, and Bf, the target pixel of the synchronization process is a G pixel, and when the pixel value is G, the position of the target pixel The pixel values R and B at
R = G × (Rf / Gf), B = G × (Bf / Gf) (1)
Calculated by

Similarly, when the target pixel of the synchronization process is an R pixel and the pixel value is R, the pixel values G and B of the G and B pixels at the target pixel position are expressed by the following equations:
G = R × (Gf / Rf), B = R × (Bf / Rf) (2)
Calculated by

Further, when the target pixel of the synchronization process is a B pixel and the pixel value is B, the pixel values G and R of the G and R pixels at the target pixel position are expressed by the following equations:
G = B × (Gf / Bf), R = B × (Rf / Bf) (3)
Calculated by

When the synchronization process for obtaining the RGB pixel values for all the 3 × 3 pixels in the central portion of the 9 × 9 pixel local region is completed, the local region extracted from the mosaic image is determined for each 3 × 3 pixel. The same processing as above is performed while moving.

When the local area of 9 × 9 pixels is moved 3 pixels in the horizontal direction or the vertical direction from the state shown in FIG. 5, a B array of 3 × 3 pixels is arranged at the center of the local area of 9 × 9 pixels after the movement. Will be located (see FIG. 4). As described above, the A array and the B array have the same G filter array, and only the positions of the R filter and the B filter are different. On the other hand, since the same value is assigned as the filter coefficient at the position corresponding to the R filter and the position corresponding to the B filter, the weighted average filter has the same weight when performing the synchronization process while moving the local region. An average filter can be used.

When the weighted average filter is applied to the RGB mosaic image obtained from the color image sensor 12 as described above, the ratio of the sum of the filter coefficients for each RGB color in each line in the horizontal and vertical directions is equal. Since the filter coefficient is set so that the weighted average value for each RGB is calculated based on this weighted average filter, the frequency of the color based on the weighted average value for each RGB can be input in the horizontal and vertical directions. Even if there is, it is possible to accurately represent the color of the local region. Since the synchronization processing is performed based on the color based on the weighted average value for each RGB, it is possible to suppress the occurrence of false colors, thereby providing an optical low-pass filter for suppressing the occurrence of false colors. It is possible not to arrange it in the optical path from the incident surface to the imaging surface, or even when applying an optical low-pass filter, it is possible to apply one that has a weak function of cutting high frequency components to prevent false color generation , Resolution can be maintained.

In the above embodiment, the pixel value at the pixel position of the target pixel is interpolated based on the color ratio of the weighted average value of RGB, and the pixel values of other colors at the pixel position are calculated. The pixel values of other colors may be calculated by interpolating the pixel values of the target pixel based on the color difference of the weighted average values of RGB.

Specifically, when the weighted average value for each RGB color is Rf, Gf, and Bf, the target pixel of the synchronization process is a G pixel, and when the pixel value is G, the pixel value R at the position of the target pixel , B is the following formula:
R = G + (Rf−Gf), B = G + (Bf−Gf) (4)
Calculated by

Similarly, when the target pixel of the synchronization process is an R pixel and the pixel value is R, the pixel values G and B of the G and B pixels at the target pixel position are expressed by the following equations:
G = R + (Gf−Rf), B = R + (Bf−Rf) (5)
Calculated by

Further, when the target pixel of the synchronization process is a B pixel and the pixel value is B, the pixel values G and R of the G and R pixels at the target pixel position are expressed by the following equations:
G = B + (Gf−Bf), R = B + (Rf−Bf) (6)
Calculated by

[Second Embodiment of Color Image Sensor and Weighted Average Filter]
FIG. 7 is a diagram showing a second embodiment of a color image sensor and a weighted average filter applied to the present invention, and in particular, a color filter array of color filters provided in the color image sensor and the color filter array. The filter coefficient of the weighted average filter is shown.

As shown in FIG. 7, in the color filter array of the color imaging device of the second embodiment, a certain horizontal line is RGBRGBRGB, the next horizontal line is GBRGBRGBR, and the next horizontal line is BRGBRGBRG. This is repeated.

That is, in this color filter array, R, G, and B color filters (R filter, G filter, and B filter) are arrayed with a predetermined periodicity.

Further, as shown in FIG. 7, in the color filter array, filters of all colors R, G, and B are arranged in each line in the horizontal direction and the vertical direction.

Thus, the color filter array of the color image sensor of the second embodiment has the same characteristics as the features (1) and (2) of the color filter array of the color image sensor 12 of the first embodiment.

On the other hand, the weighted average filter of the second embodiment applied to the color image sensor has a kernel size of 6 × 6 as shown by a thick frame in FIG. 7, and the filter coefficient shown in FIG. Yes.

That is, the filter coefficient of the weighted average filter of the second embodiment is obtained by extracting a 6 × 6 pixel local area from the mosaic image obtained from the color image sensor of the second embodiment, and each pixel in the local area. When the filter coefficients are extracted for each color corresponding to the colors and the sum of the filter coefficients for each color is obtained, the ratio of the sum of the filter coefficients for each RGB color in each line in the horizontal and vertical directions is The filter coefficients are set to be equal (1: 1: 1).

For example, in FIG. 7, the filter coefficients in the top row are 0,1,2,2,1,0, but when the sum is calculated for each color, R = 0 + 2 = 2 G = 1 + 1 = 2 and B = 2 + 0 = 2, and it can be seen that the relationship is 2: 2: 2 = 1: 1: 1. All rows and columns (horizontal and vertical lines) have filter coefficients that satisfy this relationship.

In addition, this weighted average filter has a large 2 × 2 filter coefficient in the center when comparing the filter coefficients for each region divided into 2 × 2 sizes, and then 2 × 2 on the top, bottom, left and right sides of the center. The filter coefficients are weighted so that the filter coefficients are large and the 2 × 2 filter coefficients at the four corners are the smallest.

When the weighted average value for each RGB is calculated based on the weighted average filter having the above configuration and the pixel value of each pixel in the 6 × 6 pixel local area extracted from the mosaic image, the color based on the weighted average value for each RGB is Regardless of the frequency input in the horizontal and vertical directions, no color shift occurs and coloring due to high-frequency folding does not occur.

Then, as in the first embodiment, the weighted average value for each RGB is calculated based on the pixel value of each pixel in the 6 × 6 pixel local region extracted from the mosaic image and the filter coefficient of the weighted average filter. By interpolating the pixel value of the target pixel of 2 × 2 pixels in the central part in the local area of 6 × 6 pixels by the calculated color ratio or color difference of the weighted average values for each of RGB, Pixel values can be calculated.

When the synchronization process for obtaining the RGB pixel values for all the 2 × 2 pixels in the central portion of the 6 × 6 pixel local region is completed, the local region extracted from the mosaic image is set in the horizontal direction or the vertical direction. The same processing as above is performed while moving two pixels in the direction, but in this case, the same weighted average filter can be used.

[Third Embodiment of Color Image Sensor and Weighted Average Filter]
FIG. 8 is a diagram showing a third embodiment of a color image sensor applied to the present invention, and particularly shows a color filter array of color filters provided in the color image sensor.

FIG. 9 shows filter coefficients of a weighted average filter applied to this color image sensor.

As shown in FIG. 8, the color filter array of the color imaging device of the third embodiment includes a basic array pattern (pattern indicated by a thick frame) including a square array pattern corresponding to 4 × 4 pixels. The array pattern is repeatedly arranged in the horizontal direction and the vertical direction. That is, in this color filter array, R, G, and B color filters (R filter, G filter, and B filter) are arrayed with a predetermined periodicity.

In this color filter array, filters of all the colors R, G, and B are arranged in each line in the horizontal direction and the vertical direction.

Furthermore, the basic array pattern of the color filter array is point-symmetric with respect to the center of the basic array pattern.

Furthermore, in the basic arrangement pattern of the color filter arrangement shown in FIG. 8, the numbers of R pixels, G pixels, and B pixels corresponding to the R, G, and B filters in the basic arrangement pattern are 4 pixels and 8 pixels, respectively. There are 4 pixels. That is, the ratio of the number of RGB pixels is 1: 2: 1, and the ratio of the number of G pixels that contributes most to obtain a luminance signal is the ratio of the R and B pixels of the other colors. It is larger than the ratio of the number of pixels.

As described above, the color filter array of the color image sensor of the third embodiment includes the features (1), (2), (3), and (4) of the color filter array of the color image sensor 12 of the first embodiment. Has the same characteristics.

On the other hand, the weighted average filter of the third embodiment applied to the color imaging device has a kernel size of 6 × 6 as shown by the thick frames in FIGS. 9A and 9B, and the filter coefficient shown in FIG. Is set.

That is, the filter coefficient of the weighted average filter of the third embodiment is obtained by extracting a 6 × 6 pixel local area from the mosaic image obtained from the color image sensor of the third embodiment, and each pixel in the local area. When the filter coefficients are extracted for each color corresponding to the colors, and the sum of the filter coefficients for each color is obtained, the ratio of the sum of the filter coefficients for each RGB color in each line in the horizontal and vertical directions is The filter coefficients are set to be equal (1: 1: 1).

For example, in FIG. 9A, the filter coefficient in the top row is 2,1,2,4,2,1, but when the sum is calculated for each color, R = 4 and G = 1. It can be seen that + 2 + 1 = 4 and B = 2 + 2 = 4, and the relationship is 4: 4: 4 = 1: 1: 1. All rows and columns (horizontal and vertical lines) have filter coefficients that satisfy this relationship.

When the synchronization process for obtaining the RGB pixel values for all the 2 × 2 pixels in the central portion of the 6 × 6 pixel local area is completed, the local area extracted from the mosaic image is set in the horizontal direction or the vertical direction. The same process as above is performed while moving two pixels in the direction, but in this case, a different weighted average filter is used.

That is, as shown in FIG. 9A, when G pixels come in the upper left and lower right of 2 × 2 pixels in the center of the 6 × 6 pixel local area extracted from the mosaic image, the filter coefficients shown in FIG. 9B, the local area extracted from the mosaic image is moved to two pixels in the horizontal direction, and G pixels are located at the upper right and lower left of the 2 × 2 pixels in the center of the local area. Is used, a weighted average filter having a filter coefficient shown in FIG. 9B is used.

[Others]
In the above-described embodiment, a color imaging device including a color imaging device having color filters of RGB three primary colors has been described. However, the present invention is not limited to this, and the RGB primary colors + other colors (for example, emerald ( The present invention can also be applied to a color image pickup apparatus including a color image pickup element having four color filters of E)).

Further, the present invention provides a color image pickup including a color image pickup element having a color filter of four complementary colors, in which G is added to C (cyan), M (magenta), and Y (yellow) which are complementary colors of the primary colors RGB. It can also be applied to devices.

Furthermore, the present invention is not limited to the above-described embodiments, and it goes without saying that various modifications can be made without departing from the spirit of the present invention.

DESCRIPTION OF SYMBOLS 10 ... Shooting optical system, 12 ... Color image sensor, 14 ... Imaging processing part, 16 ... Image processing part, 18 ... Drive part, 20 ... Control part

Claims

On a plurality of pixels composed of photoelectric conversion elements arranged in the horizontal direction and the vertical direction, color filters of a color filter arrangement in which all color filters are periodically arranged in each line in the horizontal direction and the vertical direction are arranged. A single-plate color image pickup device,
An image acquisition unit for acquiring a mosaic image from the color imaging device;
A weighted average filter having a predetermined filter coefficient, wherein the relationship between the color of each pixel in the local region extracted from the mosaic image corresponding to the weighted average filter and the filter coefficient is in the horizontal and vertical directions. A weighted average filter set so that the ratio of the sum of the filter coefficients for each color in each line is equal;
A weighted average calculating unit for calculating a weighted average value for each color based on a filter coefficient of the weighted average filter and a pixel value of each pixel in the local region extracted from the mosaic image corresponding to the weighted average filter; ,
A synchronization processing unit that calculates a pixel value of another color at a pixel position of a target pixel of a synchronization process in a central part of the weighted average filter, the calculated weight of the color of the target pixel and the other color A synchronization processing unit that interpolates the pixel value of the target pixel and calculates the pixel value of the other color based on the color ratio or color difference of the average value;
A control unit that repeatedly operates the weighted average calculation unit and the synchronization processing unit while moving a local region extracted from the mosaic image corresponding to the weighted average filter for each target pixel unit of the synchronization processing;
A color imaging apparatus comprising:
The color filter array of the color imaging device has a first filter corresponding to the first color that contributes most to obtain a luminance signal and a second color corresponding to two or more second colors other than the first color. A basic arrangement pattern in which two filters are arranged, and the basic arrangement pattern is repeatedly arranged in a horizontal direction and a vertical direction,
2. The color imaging device according to claim 1, wherein a ratio between the number of pixels of the first color corresponding to the first filter and the number of pixels of each color of the second color corresponding to the second filter is different.
The color imaging device according to claim 2, wherein a ratio of the number of pixels of the first color corresponding to the first filter is larger than a ratio of the number of pixels of each color of the second color corresponding to the second filter. .
The color imaging device according to any one of claims 1 to 3, wherein the weighted average filter is a filter weighted so that a filter coefficient at a central portion is increased.
The color imaging device according to any one of claims 1 to 4, wherein the weighted average filter has filter coefficients that are bilaterally symmetric, vertically symmetric, and point symmetric.
The color filter includes R filters, G filters, and B filters corresponding to red (R), green (G), and blue (B) colors arranged in a predetermined color filter array,
When the weighted average value for each color of the R, G, and B pixel values corresponding to the R filter, G filter, and B filter calculated by the weighted average calculation unit is Rf, Gf, and Bf,
The synchronization processing unit includes:
When the target pixel of the synchronization processing is a G pixel and the pixel value is G, the pixel values R and B of the R and B pixels at the target pixel position are expressed by the following equations:
R = G × (Rf / Gf), B = G × (Bf / Gf)
When the target pixel of the synchronization processing is an R pixel and the pixel value is R, the pixel values G and B of the G and B pixels at the target pixel position are expressed by the following equations:
G = R × (Gf / Rf), B = R × (Bf / Rf)
When the target pixel of the synchronization processing is a B pixel and the pixel value is B, the pixel values G and R of the G and R pixels at the position of the target pixel are expressed by the following equations:
G = B × (Gf / Bf), R = B × (Rf / Bf)
The color imaging device according to claim 1, which is calculated by:
The color filter includes R filters, G filters, and B filters corresponding to red (R), green (G), and blue (B) colors arranged in a predetermined color filter array,
When the weighted average value for each color of the R, G, and B pixel values corresponding to the R filter, G filter, and B filter calculated by the weighted average calculation unit is Rf, Gf, and Bf,
The synchronization processing unit includes:
When the target pixel of the synchronization processing is a G pixel and the pixel value is G, the pixel values R and B of the R and B pixels at the target pixel position are expressed by the following equations:
R = G + (Rf−Gf), B = G + (Bf−Gf)
When the target pixel of the synchronization processing is an R pixel and the pixel value is R, the pixel values G and B of the G and B pixels at the target pixel position are expressed by the following equations:
G = R + (Gf−Rf), B = R + (Bf−Rf)
When the target pixel of the synchronization processing is a B pixel and the pixel value is B, the pixel values G and R of the G and R pixels at the position of the target pixel are expressed by the following equations:
G = B + (Gf−Bf), R = B + (Rf−Bf)
The color imaging device according to claim 1, which is calculated by:
The color filter includes R filters, G filters, and B filters corresponding to red (R), green (G), and blue (B) colors arranged in a predetermined color filter array,
The filter array is a first array corresponding to 3 × 3 pixels, in which G filters are arranged at the center and four corners, B filters are arranged above and below the center G filter, and R filters are arranged on the left and right Are arranged in the first array and the second array corresponding to 3 × 3 pixels, and G filters are arranged at the center and four corners, and R filters are arranged above and below the center G filter. The second array in which B filters are arranged on the left and right are alternately arranged in the horizontal direction and the vertical direction,
The weighted average filter has a kernel size of 9 × 9;
8. The control unit according to claim 1, wherein the control unit repeatedly operates the weighted average calculation unit and the synchronization processing unit while sequentially moving the weighted average filter so that the first array or the second array is centered. The color imaging device according to any one of the above.